feat: provenance attributes on the embedding dump
VR-010 — a dump made with one detector/embedder pair was byte-indistinguishable from one made with another, except for the two attributes GR-004 added. Replayed against a gallery from a different model, cosine similarities are meaningless but look entirely plausible. The register states the principle directly: a fixture whose provenance is unknown is worse than no fixture, because it will be trusted. Sixteen attributes now record everything that determines the dump's content: detector model and thresholds, min_face_px, max_faces, cut_threshold, dense_scale, bbox_upscale, start/end, track_assoc_min_prob, and scene_detect. scene_detect is the one that matters most. is_scene_boundary is all-zero both when the detector found nothing and when it never ran, and those mean completely different things to a consumer — without the flag they are indistinguishable. No schema_version bump: new root attributes are additive and replay.py already reads attributes with a default, so older dumps stay readable and the committed fixtures — which predate this — still load. Also corrects SCHEMA.md, which claimed bbox was already mapped to original resolution at dump time. It is not; the upscale is applied downstream in the matcher, after the dump tap. Harmless while dense_scale is 1 and silently wrong otherwise, so bbox_upscale is now recorded and the doc says what the code does. Verified end to end: all sixteen attributes present and correct on a freshly generated dump. Suite: 92 cases, 6136 assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: VR-010, VR-001 | PR-002
This commit is contained in:
@@ -19,11 +19,32 @@ variable-length HDF5 types and reads straight into numpy.
|
||||
/ (root)
|
||||
attrs:
|
||||
schema_version : int = 1
|
||||
movie : str (source video path)
|
||||
sample_fps : float
|
||||
embed_dim : int = 512
|
||||
embedder_model : str basename of the embedding model (GR-004)
|
||||
embedder_sha256: str SHA-256 of that model file (GR-004)
|
||||
|
||||
# ── what produced the vectors (GR-004) ──────────────────────────────────
|
||||
embedder_model : str basename of the embedding model
|
||||
embedder_sha256: str SHA-256 of that model file
|
||||
|
||||
# ── what produced the faces (VR-010) ────────────────────────────────────
|
||||
detector_model : str basename of the detector .onnx
|
||||
detector_conf : float score floor a detection had to clear to be dumped
|
||||
detector_nms : float NMS IoU threshold
|
||||
min_face_px : float minimum box side, ORIGINAL-resolution px (AR-002)
|
||||
max_faces : int per-frame cap; 0 = uncapped, the default (AR-003)
|
||||
|
||||
# ── what produced the frames (VR-010) ───────────────────────────────────
|
||||
movie : str source video path
|
||||
sample_fps : float frames analysed per second of movie
|
||||
start_sec : float seek point
|
||||
end_sec : float stop point; -1 = end of file
|
||||
cut_threshold : float histogram correlation below which is_cut fires
|
||||
dense_scale : float decoded-frame downscale in dense mode; 1 = off
|
||||
bbox_upscale : float multiply faces/bbox and faces/landmarks by this to
|
||||
reach original video pixels; 1 when dense_scale is 1
|
||||
scene_detect : uint8 0/1 — was TransNetV2 running at all (see below)
|
||||
|
||||
# ── downstream setting recorded for comparability (VR-010) ──────────────
|
||||
track_assoc_min_prob : float the run's tracker admission probability
|
||||
|
||||
frames/ group — one row per sampled frame
|
||||
timestamp_sec : float64 [F]
|
||||
@@ -35,14 +56,47 @@ variable-length HDF5 types and reads straight into numpy.
|
||||
|
||||
faces/ group — one row per detected face, concatenated
|
||||
embedding : float32 [N, 512] L2-normalised ArcFace embedding
|
||||
bbox : float32 [N, 4] x, y, w, h in original video pixels
|
||||
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order
|
||||
bbox : float32 [N, 4] x, y, w, h in DECODED-frame pixels
|
||||
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order,
|
||||
same space as bbox
|
||||
confidence : float32 [N] detector confidence
|
||||
```
|
||||
|
||||
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
|
||||
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.
|
||||
|
||||
## Provenance (VR-010)
|
||||
|
||||
The attributes above are not documentation; they are the only thing that makes a
|
||||
dump interpretable. Two dumps of the same film at `detector_conf` 0.5 and 0.7, or
|
||||
at `dense_scale` 1.0 and 0.5, or with scene detection on and off, are different
|
||||
measurements of different things — and they are byte-shaped identically. Without
|
||||
provenance a consumer that mixes them gets a plausible number from an incoherent
|
||||
input, and nothing anywhere reports a problem.
|
||||
|
||||
**`scene_detect` is the one that cannot be inferred.** `is_scene_boundary` is
|
||||
all-zero both when TransNetV2 found no boundaries in the clip and when it was
|
||||
never enabled, and those mean opposite things: the first says *this footage has
|
||||
no shot changes*, the second says *nobody looked*. A consumer that reads the
|
||||
array alone must guess. The flag is what removes the guess. (`dump_embeddings`
|
||||
has no `--scene-detect`, so every dump it writes records `false` — which is
|
||||
exactly the fact the committed fixtures needed to state.)
|
||||
|
||||
**`bbox_upscale` is recorded, not applied.** See the coordinate-space note below.
|
||||
|
||||
Reading is by name with a default or an existence check on **both** sides —
|
||||
`replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in
|
||||
`src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are
|
||||
additive and `schema_version` stays 1: a pre-VR-010 dump still loads, and a
|
||||
post-VR-010 dump still reads on old code.
|
||||
|
||||
A missing attribute means **unknown**, never a default value. Substituting
|
||||
`detector_conf = 0.5` for a dump that does not say so manufactures the provenance
|
||||
the requirement exists to prevent — per `docs/requirements.md`, *"a fixture whose
|
||||
provenance is unknown is worse than no fixture, because it will be trusted."*
|
||||
The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these
|
||||
attributes; re-dump to bind them, as with GR-004.
|
||||
|
||||
## Model binding (GR-004)
|
||||
|
||||
`embedder_model` / `embedder_sha256` record which embedder produced every vector
|
||||
@@ -57,10 +111,36 @@ warning, or a hard error under `SAE_REQUIRE_GALLERY_STAMP=1`) rather than as a
|
||||
pass. Re-dump to bind an old dump; there is no in-place migration, because unlike
|
||||
a gallery nobody can assert after the fact which model produced a vector.
|
||||
|
||||
## Coordinate space — `bbox`, `landmarks`, `bbox_upscale`
|
||||
|
||||
`bbox` and `landmarks` are in **decoded-frame pixels**: exactly the numbers SCRFD
|
||||
produced, untransformed. To reach original video pixels, multiply by
|
||||
`bbox_upscale`. With `dense_scale == 1` (the default, and every committed
|
||||
fixture) `bbox_upscale == 1` and the two spaces coincide.
|
||||
|
||||
> Earlier revisions of this document claimed the upscale was applied at dump time.
|
||||
> It never was. `embedding_dump_node.hpp` writes `f.bbox` raw; the upscale lives
|
||||
> in `identity_matcher_node.hpp`, which is *downstream* of the dump tap. The
|
||||
> claim was harmless only because `dense_scale` was 1 in practice.
|
||||
|
||||
The fix is to record the factor rather than to apply it, because the dump's whole
|
||||
contract is to be a **faithful tap** at the `EmbeddedSceneFrame` channel — VR-002
|
||||
requires replay to drive the real nodes, and a replay is only equivalent to the
|
||||
live run if the tracker is fed the geometry the live tracker saw. Rescaling at
|
||||
the tap would break that: the replayed tracker would associate on boxes the live
|
||||
one never received. Two further reasons:
|
||||
|
||||
- The matcher's upscale is applied to `bbox` **only**, not to `landmarks`.
|
||||
Pre-multiplying at the tap would leave the two arrays in different coordinate
|
||||
spaces inside one file — a worse trap than the one being fixed.
|
||||
- Pre-multiplying is lossy in the sense that matters: a dump that had been
|
||||
upscaled would be indistinguishable from one taken at `dense_scale == 1`, so
|
||||
you would have to record `bbox_upscale` anyway to know which you were holding.
|
||||
|
||||
## Invariants
|
||||
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
|
||||
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
|
||||
- `bbox` is already mapped to original resolution (bbox_upscale applied at dump time),
|
||||
matching what the identity matcher would emit.
|
||||
- `bbox` and `landmarks` share one coordinate space; `bbox_upscale` maps both to
|
||||
original resolution (see above).
|
||||
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
|
||||
- EOF sentinel frames are NOT written.
|
||||
|
||||
Reference in New Issue
Block a user