feat(quality): score every face on sharpness and alignment before it is evidence
Every embedding now carries the quality of the input it came from. Both axes fall out of the AR-005 warp for free: crop_sharpness() is the normalised Laplacian variance over the aligned 112x112, so contrast and size cannot leak into it, and the alignment residual is the part of the landmark deformation a similarity transform cannot explain, so in-plane roll reads as zero and foreshortening does not. Carried, not consumed. Nothing discounts or thresholds on either number yet -- that is AR-030 and VR-012, and the knee has to be located against recorded data before a gate is chosen. What this change buys is that the data exists to locate it with. No face is admitted unscored: the -1 sentinel is preserved rather than clamped, and a degenerate landmark fit is counted rather than silently dropped. Takes the VR-001 dump to schema_version 2. The bump is not for readers, which check for the datasets by name and replay a v1 dump unchanged; it is so a consumer can tell "never scored" from "scored zero", which is not recoverable from the arrays afterwards. TRACES: AR-028, AR-029, AR-030 | VR-001 | SR-002
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
# Embedding-dump HDF5 schema (v1)
|
||||
# Embedding-dump HDF5 schema (v2)
|
||||
|
||||
One file per analysed title. Captures the pipeline state at the `EmbeddedSceneFrame`
|
||||
channel — i.e. after decode → detect → align → embed, but **before** tracking and
|
||||
@@ -18,7 +18,7 @@ variable-length HDF5 types and reads straight into numpy.
|
||||
```
|
||||
/ (root)
|
||||
attrs:
|
||||
schema_version : int = 1
|
||||
schema_version : int = 2
|
||||
embed_dim : int = 512
|
||||
|
||||
# ── what produced the vectors (GR-004) ──────────────────────────────────
|
||||
@@ -60,6 +60,12 @@ variable-length HDF5 types and reads straight into numpy.
|
||||
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order,
|
||||
same space as bbox
|
||||
confidence : float32 [N] detector confidence
|
||||
|
||||
# ── embedding input quality (AR-028), v2 onward ─────────────────────────
|
||||
sharpness : float32 [N] normalised Laplacian variance on the
|
||||
112x112 aligned crop (AR-029)
|
||||
alignment_residual : float32 [N] RMS landmark misfit in canonical px,
|
||||
after the AR-005 similarity fit (AR-030)
|
||||
```
|
||||
|
||||
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
|
||||
@@ -87,8 +93,9 @@ exactly the fact the committed fixtures needed to state.)
|
||||
Reading is by name with a default or an existence check on **both** sides —
|
||||
`replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in
|
||||
`src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are
|
||||
additive and `schema_version` stays 1: a pre-VR-010 dump still loads, and a
|
||||
post-VR-010 dump still reads on old code.
|
||||
additive and did not themselves move `schema_version` off 1: a pre-VR-010 dump
|
||||
still loads, and a post-VR-010 dump still reads on old code. (AR-028 later took
|
||||
it to 2 by adding *datasets* — see below.)
|
||||
|
||||
A missing attribute means **unknown**, never a default value. Substituting
|
||||
`detector_conf = 0.5` for a dump that does not say so manufactures the provenance
|
||||
@@ -97,6 +104,40 @@ provenance is unknown is worse than no fixture, because it will be trusted."*
|
||||
The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these
|
||||
attributes; re-dump to bind them, as with GR-004.
|
||||
|
||||
## Embedding input quality (AR-028) — and why this one bumps the version
|
||||
|
||||
`sharpness` and `alignment_residual` are two of the three AR-028 quality axes,
|
||||
written beside the embedding they describe. **The third axis, size, is already
|
||||
here**: it is `bbox`, scaled by `bbox_upscale` to reach the original resolution
|
||||
AR-002 thresholds in. It is not duplicated into a third column, because that
|
||||
would put the same quantity in two coordinate spaces inside one file — the trap
|
||||
the `bbox_upscale` note below records — and the copy is the one that drifts.
|
||||
|
||||
The vector is **carried, not consumed**. Nothing in the pipeline thresholds or
|
||||
discounts on it yet; VR-012 locates the knees from these columns, which is only
|
||||
possible if they were recorded at inference. A study cannot recover how sharp a
|
||||
face was from an embedding, any more than it can recover which model produced it.
|
||||
|
||||
**This is the change that bumps `schema_version` to 2**, where VR-010's
|
||||
attributes did not. The rule is unchanged — a bump is for the *datasets* — and
|
||||
so is the reason behind it. Readers are fine either way: `replay.py` and
|
||||
`test_replay_fixtures.cpp` take these datasets by name with an existence check,
|
||||
so a v1 dump still replays and loses only what it never had. The version exists
|
||||
for a *consumer of the quality vector*, which otherwise cannot tell **"this
|
||||
film's faces were never scored"** from **"this film's faces scored zero"** —
|
||||
sharpness 0 is a real reading, meaning a featureless crop. That is the same
|
||||
distinction `scene_detect` exists to make, and it is equally unrecoverable from
|
||||
the arrays.
|
||||
|
||||
A v1 dump reports the vector as **unknown, never as a default** — `load_frames`
|
||||
omits the keys rather than filling zeros, and the C++ side leaves the
|
||||
`DetectedFace` fields at their -1 "unscored" sentinel. Re-dump to acquire it;
|
||||
there is no migration, for the same reason GR-004 has none.
|
||||
|
||||
> The committed `tests/fixtures/dumps/*.h5` are v1 and carry no quality vector.
|
||||
> Re-dumping needs a GPU host (`scripts/make_fixtures.sh`), so until that runs,
|
||||
> anything driven from the fixtures sees the sentinel.
|
||||
|
||||
## Model binding (GR-004)
|
||||
|
||||
`embedder_model` / `embedder_sha256` record which embedder produced every vector
|
||||
@@ -144,3 +185,7 @@ one never received. Two further reasons:
|
||||
original resolution (see above).
|
||||
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
|
||||
- EOF sentinel frames are NOT written.
|
||||
- v2 onward: `sharpness` and `alignment_residual` are `[N]`, parallel to
|
||||
`confidence`, so face *i*'s quality indexes with the same slice as its
|
||||
embedding. Both are `>= 0` for any face the aligner admitted; a negative value
|
||||
means unscored and must never be read as a quality.
|
||||
|
||||
@@ -63,6 +63,14 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
|
||||
bbox = f["faces/bbox"][:]
|
||||
lmk = f["faces/landmarks"][:]
|
||||
conf = f["faces/confidence"][:]
|
||||
# TRACES: AR-028 | SR-002
|
||||
# The quality vector, present from schema v2. A v1 dump predates AR-028
|
||||
# and simply has no such dataset — read as absent, never as a default,
|
||||
# so a face from an old dump stays at the C++ -1 "unscored" sentinel
|
||||
# rather than acquiring a fabricated sharpness of 0 (which is a real
|
||||
# value on this axis, meaning a featureless crop).
|
||||
qual = {k: f[f"faces/{k}"][:] for k in ("sharpness", "alignment_residual")
|
||||
if f"faces/{k}" in f}
|
||||
movie = f.attrs.get("movie", "")
|
||||
fps = float(f.attrs.get("sample_fps", 1.0))
|
||||
|
||||
@@ -81,6 +89,8 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
|
||||
"landmarks": np.ascontiguousarray(lmk[keep][sel], dtype=np.float32),
|
||||
"confidence": np.ascontiguousarray(c[sel], dtype=np.float32),
|
||||
"embeddings": np.ascontiguousarray(emb[keep][sel], dtype=np.float32),
|
||||
**{k: np.ascontiguousarray(v[keep][sel], dtype=np.float32)
|
||||
for k, v in qual.items()},
|
||||
})
|
||||
else:
|
||||
frames.append({
|
||||
@@ -90,6 +100,8 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
|
||||
"landmarks": np.ascontiguousarray(lmk[keep], dtype=np.float32),
|
||||
"confidence": c,
|
||||
"embeddings": np.ascontiguousarray(emb[keep], dtype=np.float32),
|
||||
**{k: np.ascontiguousarray(v[keep], dtype=np.float32)
|
||||
for k, v in qual.items()},
|
||||
})
|
||||
last_ts = float(ts[-1]) if len(ts) else 0.0
|
||||
frames.append({"timestamp_sec": last_ts, "eof": True})
|
||||
|
||||
Reference in New Issue
Block a user