feat(quality): score every face on sharpness and alignment before it is evidence

Every embedding now carries the quality of the input it came from. Both
axes fall out of the AR-005 warp for free: crop_sharpness() is the
normalised Laplacian variance over the aligned 112x112, so contrast and
size cannot leak into it, and the alignment residual is the part of the
landmark deformation a similarity transform cannot explain, so in-plane
roll reads as zero and foreshortening does not.

Carried, not consumed. Nothing discounts or thresholds on either number
yet -- that is AR-030 and VR-012, and the knee has to be located against
recorded data before a gate is chosen. What this change buys is that the
data exists to locate it with.

No face is admitted unscored: the -1 sentinel is preserved rather than
clamped, and a degenerate landmark fit is counted rather than silently
dropped.

Takes the VR-001 dump to schema_version 2. The bump is not for readers,
which check for the datasets by name and replay a v1 dump unchanged; it
is so a consumer can tell "never scored" from "scored zero", which is
not recoverable from the arrays afterwards.

TRACES: AR-028, AR-029, AR-030 | VR-001 | SR-002
This commit is contained in:
2026-08-05 14:37:30 +02:00
parent c1155cb607
commit 777c98cb33
10 changed files with 726 additions and 17 deletions
+49 -4
View File
@@ -1,4 +1,4 @@
# Embedding-dump HDF5 schema (v1) # Embedding-dump HDF5 schema (v2)
One file per analysed title. Captures the pipeline state at the `EmbeddedSceneFrame` One file per analysed title. Captures the pipeline state at the `EmbeddedSceneFrame`
channel — i.e. after decode → detect → align → embed, but **before** tracking and channel — i.e. after decode → detect → align → embed, but **before** tracking and
@@ -18,7 +18,7 @@ variable-length HDF5 types and reads straight into numpy.
``` ```
/ (root) / (root)
attrs: attrs:
schema_version : int = 1 schema_version : int = 2
embed_dim : int = 512 embed_dim : int = 512
# ── what produced the vectors (GR-004) ────────────────────────────────── # ── what produced the vectors (GR-004) ──────────────────────────────────
@@ -60,6 +60,12 @@ variable-length HDF5 types and reads straight into numpy.
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order, landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order,
same space as bbox same space as bbox
confidence : float32 [N] detector confidence confidence : float32 [N] detector confidence
# ── embedding input quality (AR-028), v2 onward ─────────────────────────
sharpness : float32 [N] normalised Laplacian variance on the
112x112 aligned crop (AR-029)
alignment_residual : float32 [N] RMS landmark misfit in canonical px,
after the AR-005 similarity fit (AR-030)
``` ```
`F` = number of sampled frames, `N` = total faces (= sum of face_count). `F` = number of sampled frames, `N` = total faces (= sum of face_count).
@@ -87,8 +93,9 @@ exactly the fact the committed fixtures needed to state.)
Reading is by name with a default or an existence check on **both** sides — Reading is by name with a default or an existence check on **both** sides —
`replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in `replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in
`src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are `src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are
additive and `schema_version` stays 1: a pre-VR-010 dump still loads, and a additive and did not themselves move `schema_version` off 1: a pre-VR-010 dump
post-VR-010 dump still reads on old code. still loads, and a post-VR-010 dump still reads on old code. (AR-028 later took
it to 2 by adding *datasets* — see below.)
A missing attribute means **unknown**, never a default value. Substituting A missing attribute means **unknown**, never a default value. Substituting
`detector_conf = 0.5` for a dump that does not say so manufactures the provenance `detector_conf = 0.5` for a dump that does not say so manufactures the provenance
@@ -97,6 +104,40 @@ provenance is unknown is worse than no fixture, because it will be trusted."*
The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these
attributes; re-dump to bind them, as with GR-004. attributes; re-dump to bind them, as with GR-004.
## Embedding input quality (AR-028) — and why this one bumps the version
`sharpness` and `alignment_residual` are two of the three AR-028 quality axes,
written beside the embedding they describe. **The third axis, size, is already
here**: it is `bbox`, scaled by `bbox_upscale` to reach the original resolution
AR-002 thresholds in. It is not duplicated into a third column, because that
would put the same quantity in two coordinate spaces inside one file — the trap
the `bbox_upscale` note below records — and the copy is the one that drifts.
The vector is **carried, not consumed**. Nothing in the pipeline thresholds or
discounts on it yet; VR-012 locates the knees from these columns, which is only
possible if they were recorded at inference. A study cannot recover how sharp a
face was from an embedding, any more than it can recover which model produced it.
**This is the change that bumps `schema_version` to 2**, where VR-010's
attributes did not. The rule is unchanged — a bump is for the *datasets* — and
so is the reason behind it. Readers are fine either way: `replay.py` and
`test_replay_fixtures.cpp` take these datasets by name with an existence check,
so a v1 dump still replays and loses only what it never had. The version exists
for a *consumer of the quality vector*, which otherwise cannot tell **"this
film's faces were never scored"** from **"this film's faces scored zero"** —
sharpness 0 is a real reading, meaning a featureless crop. That is the same
distinction `scene_detect` exists to make, and it is equally unrecoverable from
the arrays.
A v1 dump reports the vector as **unknown, never as a default**`load_frames`
omits the keys rather than filling zeros, and the C++ side leaves the
`DetectedFace` fields at their -1 "unscored" sentinel. Re-dump to acquire it;
there is no migration, for the same reason GR-004 has none.
> The committed `tests/fixtures/dumps/*.h5` are v1 and carry no quality vector.
> Re-dumping needs a GPU host (`scripts/make_fixtures.sh`), so until that runs,
> anything driven from the fixtures sees the sentinel.
## Model binding (GR-004) ## Model binding (GR-004)
`embedder_model` / `embedder_sha256` record which embedder produced every vector `embedder_model` / `embedder_sha256` record which embedder produced every vector
@@ -144,3 +185,7 @@ one never received. Two further reasons:
original resolution (see above). original resolution (see above).
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense). - A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
- EOF sentinel frames are NOT written. - EOF sentinel frames are NOT written.
- v2 onward: `sharpness` and `alignment_residual` are `[N]`, parallel to
`confidence`, so face *i*'s quality indexes with the same slice as its
embedding. Both are `>= 0` for any face the aligner admitted; a negative value
means unscored and must never be read as a quality.
+12
View File
@@ -63,6 +63,14 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
bbox = f["faces/bbox"][:] bbox = f["faces/bbox"][:]
lmk = f["faces/landmarks"][:] lmk = f["faces/landmarks"][:]
conf = f["faces/confidence"][:] conf = f["faces/confidence"][:]
# TRACES: AR-028 | SR-002
# The quality vector, present from schema v2. A v1 dump predates AR-028
# and simply has no such dataset — read as absent, never as a default,
# so a face from an old dump stays at the C++ -1 "unscored" sentinel
# rather than acquiring a fabricated sharpness of 0 (which is a real
# value on this axis, meaning a featureless crop).
qual = {k: f[f"faces/{k}"][:] for k in ("sharpness", "alignment_residual")
if f"faces/{k}" in f}
movie = f.attrs.get("movie", "") movie = f.attrs.get("movie", "")
fps = float(f.attrs.get("sample_fps", 1.0)) fps = float(f.attrs.get("sample_fps", 1.0))
@@ -81,6 +89,8 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
"landmarks": np.ascontiguousarray(lmk[keep][sel], dtype=np.float32), "landmarks": np.ascontiguousarray(lmk[keep][sel], dtype=np.float32),
"confidence": np.ascontiguousarray(c[sel], dtype=np.float32), "confidence": np.ascontiguousarray(c[sel], dtype=np.float32),
"embeddings": np.ascontiguousarray(emb[keep][sel], dtype=np.float32), "embeddings": np.ascontiguousarray(emb[keep][sel], dtype=np.float32),
**{k: np.ascontiguousarray(v[keep][sel], dtype=np.float32)
for k, v in qual.items()},
}) })
else: else:
frames.append({ frames.append({
@@ -90,6 +100,8 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
"landmarks": np.ascontiguousarray(lmk[keep], dtype=np.float32), "landmarks": np.ascontiguousarray(lmk[keep], dtype=np.float32),
"confidence": c, "confidence": c,
"embeddings": np.ascontiguousarray(emb[keep], dtype=np.float32), "embeddings": np.ascontiguousarray(emb[keep], dtype=np.float32),
**{k: np.ascontiguousarray(v[keep], dtype=np.float32)
for k, v in qual.items()},
}) })
last_ts = float(ts[-1]) if len(ts) else 0.0 last_ts = float(ts[-1]) if len(ts) else 0.0
frames.append({"timestamp_sec": last_ts, "eof": True}) frames.append({"timestamp_sec": last_ts, "eof": True})
+91 -1
View File
@@ -1,5 +1,5 @@
#pragma once #pragma once
/// TRACES: AR-005, AR-030 | SR-002 /// TRACES: AR-005, AR-029, AR-030 | SR-002
#include "types.hpp" #include "types.hpp"
#include <opencv2/core.hpp> #include <opencv2/core.hpp>
@@ -143,6 +143,96 @@ inline cv::Mat align_face(const cv::Mat& img,
return crop; return crop;
} }
// ── crop_sharpness ────────────────────────────────────────────────────────────
/// TRACES: AR-029 | SR-002
//
// Normalised variance of the Laplacian over the aligned 112×112 crop: the AR-029
// sharpness axis. Returns -1 for an empty crop (unscored), matching the
// DetectedFace sentinel.
//
// sharpness = Var(∇²I) / Var(I)
//
// Two normalisations, each removing a quantity that would otherwise be read as
// blur:
//
// - **Divided by the image variance, so contrast cannot leak in.** Scaling
// intensity by α scales the Laplacian by α too, so both variances scale by α²
// and the ratio is unchanged. A raw Var(∇²I) — the textbook measure — instead
// falls with exposure, so a dim scene reads as soft and a graded-up one as
// sharp. VR-012 has to locate one knee across films whose grading differs by
// more than their focus does; an uncalibrated measure would put the knee in a
// different place per film, which is the AR-024 failure in another metric.
// - **Measured on the aligned crop, so size cannot leak in.** The destination
// frame is fixed at 112×112 (AR-002 owns size, and double-counting it here
// would make every small face read as blurred). What the ratio reports is the
// detail actually present in the embedder's input — so a small sharp face can
// and does outscore a large soft one. That is the claim; it is *not* a claim
// of invariance to source resolution, because a 40 px face warped up to 112
// genuinely carries less detail, and hiding that would defeat the point.
//
// Frequency-domain reading of why the blur ladder is monotone: with
// Var(∇²I) = ∫|ω|⁴|F(ω)|² and Var(I) = ∫|F(ω)|², the ratio is E[|ω|⁴] under the
// image's own spectral measure. Gaussian blur multiplies that measure by
// e^{-σ²|ω|²}, concentrating it at low |ω|, so the expectation falls strictly
// with σ. It is a property of the construction, not a fitted behaviour.
//
// **Three known hazards, for VR-012 to check rather than for a threshold to
// absorb.** All are recorded here because they are properties of the measure,
// visible in the dumped distribution, and neither should be papered over by a
// correction chosen before that distribution has been looked at.
//
// 1. **Border fill.** `align_face` warps with BORDER_CONSTANT, so a face
// crossing the frame edge brings a hard black step into the crop, and a
// step edge is high-frequency. The normalisation blunts it — the fill
// inflates Var(I) as well as Var(∇²I) — but does not remove it, so
// heavily-cropped faces may read sharper than they are. The fix is either a
// validity mask or a different border mode, and the second changes what the
// embedder is fed (AR-011).
//
// 2. **The contrast invariance is exact in the algebra and approximate in
// 8 bits.** Scaling I by α cancels exactly; what does not cancel is the
// quantisation floor of a stored crop, which is broadband and so lands in
// the numerator. It matters only where there is little signal left to
// compete with it: on the AR-029 test texture a half-contrast copy reads
// 0.9% high when sharp, 24% high at sigma 1.2 and 148% high at sigma 2.5.
// A crop that is both **dim and soft therefore reads sharper than it is** —
// the low corner of the axis, and the corner VR-012 must put a knee in.
//
// 3. **It reports where the energy sits, not how much there is.** A crop whose
// energy is *already* concentrated at high frequency — dense film grain,
// a face against foliage — loses numerator and denominator together under
// blur, so the ratio moves less than the damage does. Measured on a
// flat-spectrum synthetic, an anisotropic (motion) smear even makes it rise,
// because the surviving perpendicular detail really is as fine as before.
// Natural crops have the low-frequency mass that keeps the denominator
// steady, and on those both ladders fall (see the AR-029 tests, which use a
// 1/f texture for exactly this reason). The same property means the axis
// conflates focus with intrinsic texture — a bearded face outscores a smooth
// one at equal focus — which is true of every no-reference sharpness measure
// and is why AR-028 carries the number instead of thresholding on it.
inline float crop_sharpness(const cv::Mat& crop) {
if (crop.empty()) return -1.f;
cv::Mat gray;
if (crop.channels() == 3) cv::cvtColor(crop, gray, cv::COLOR_BGR2GRAY);
else gray = crop;
cv::Mat lap;
cv::Laplacian(gray, lap, CV_32F, 3);
cv::Scalar mean_i, sd_i, mean_l, sd_l;
cv::meanStdDev(gray, mean_i, sd_i);
cv::meanStdDev(lap, mean_l, sd_l);
const double var_i = sd_i[0] * sd_i[0];
// A flat crop has no detail to be sharp or soft about, and the ratio is 0/0.
// Zero is the honest answer and keeps the axis finite; -1 would claim the
// face was never scored, which is a different fact.
if (var_i < 1e-6) return 0.f;
return static_cast<float>((sd_l[0] * sd_l[0]) / var_i);
}
// ── enhance_for_retry ──────────────────────────────────────────────────────── // ── enhance_for_retry ────────────────────────────────────────────────────────
// Used when initial face detection finds nothing. Pads the image by 50% // Used when initial face detection finds nothing. Pads the image by 50%
// (border-replicated, so the detector doesn't see a hard edge) and applies // (border-replicated, so the detector doesn't see a hard edge) and applies
+19
View File
@@ -28,6 +28,7 @@
#include <nanobind/stl/map.h> #include <nanobind/stl/map.h>
#include <memory> #include <memory>
#include <optional>
#include <variant> #include <variant>
namespace nb = nanobind; namespace nb = nanobind;
@@ -69,6 +70,22 @@ template<> struct PythonConverter<EmbeddedSceneFrame> {
auto conf = nb::cast<nb::ndarray<float, nb::shape<-1>, nb::c_contig>>(d["confidence"]); auto conf = nb::cast<nb::ndarray<float, nb::shape<-1>, nb::c_contig>>(d["confidence"]);
auto emb = nb::cast<nb::ndarray<float, nb::shape<-1, 512>, nb::c_contig>>(d["embeddings"]); auto emb = nb::cast<nb::ndarray<float, nb::shape<-1, 512>, nb::c_contig>>(d["embeddings"]);
// AR-028 quality vector. Optional because a v1 dump predates it — absent
// leaves the DetectedFace sentinels at -1, which reads as *unscored*, not
// as a bad face. There is no live aligner on this path to recompute it:
// the replay starts at the embedded-frame channel, so what the dump does
// not carry is genuinely gone.
//
// Held in named locals, like the four above, because the ndarray owns the
// reference that keeps the buffer alive — reading .data() off a temporary
// would leave the pointer dangling at the end of the statement.
using FloatCol = nb::ndarray<float, nb::shape<-1>, nb::c_contig>;
std::optional<FloatCol> sharp_col, resid_col;
if (d.contains("sharpness")) sharp_col = nb::cast<FloatCol>(d["sharpness"]);
if (d.contains("alignment_residual")) resid_col = nb::cast<FloatCol>(d["alignment_residual"]);
const float* sp = sharp_col ? sharp_col->data() : nullptr;
const float* rp = resid_col ? resid_col->data() : nullptr;
const size_t n = bbox.shape(0); const size_t n = bbox.shape(0);
ef.faces.reserve(n); ef.faces.reserve(n);
ef.embeddings.reserve(n); ef.embeddings.reserve(n);
@@ -82,6 +99,8 @@ template<> struct PythonConverter<EmbeddedSceneFrame> {
for (int k = 0; k < 5; ++k) for (int k = 0; k < 5; ++k)
f.landmarks[k] = cv::Point2f(lp[i*10 + k*2], lp[i*10 + k*2 + 1]); f.landmarks[k] = cv::Point2f(lp[i*10 + k*2], lp[i*10 + k*2 + 1]);
f.confidence = cp[i]; f.confidence = cp[i];
if (sp) f.sharpness = sp[i];
if (rp) f.alignment_residual = rp[i];
ef.faces.push_back(f); ef.faces.push_back(f);
Embedding e; Embedding e;
+26 -3
View File
@@ -1,5 +1,5 @@
#pragma once #pragma once
/// TRACES: VR-001, VR-010 | PR-002 /// TRACES: AR-028 | VR-001, VR-010 | PR-002
#include "types.hpp" #include "types.hpp"
#include "config.hpp" #include "config.hpp"
#include "gallery/embedder_stamp.hpp" #include "gallery/embedder_stamp.hpp"
@@ -178,6 +178,16 @@ struct EmbeddingDumpFunc {
lmk_.push_back(f.landmarks[k].y); lmk_.push_back(f.landmarks[k].y);
} }
conf_.push_back(f.confidence); conf_.push_back(f.confidence);
/// TRACES: AR-028 | SR-002
// The quality vector, carried rather than consumed: written beside
// the embedding it describes so VR-012 can locate its knees against
// recorded data instead of by re-running video. Size is the third
// axis and is already here as bbox + the bbox_upscale attribute.
// Both are -1 only if a face reached the dump unscored, which the
// aligner does not allow — the sentinel is preserved rather than
// clamped so that a future path which did would be visible.
sharp_.push_back(f.sharpness);
resid_.push_back(f.alignment_residual);
const auto& e = ef.embeddings[i]; const auto& e = ef.embeddings[i];
emb_.insert(emb_.end(), e.begin(), e.end()); emb_.insert(emb_.end(), e.begin(), e.end());
} }
@@ -194,11 +204,20 @@ struct EmbeddingDumpFunc {
} }
private: private:
// Root attributes are additive: schema_version stays 1 across VR-010, because // Root attributes are additive: schema_version stayed 1 across VR-010, because
// every reader takes attributes by name with a default (replay.py) or an // every reader takes attributes by name with a default (replay.py) or an
// existence check (read_dump_provenance), so an old dump loses nothing and a // existence check (read_dump_provenance), so an old dump loses nothing and a
// new dump breaks nothing. A bump is for a change to the *datasets*. // new dump breaks nothing. A bump is for a change to the *datasets*.
static constexpr int kSchemaVersion = 1; //
// v2 is that change: AR-028 adds faces/sharpness and faces/alignment_residual.
// The bump is not about readers — those check for the datasets by name, and a
// v1 dump still replays. It is so a *consumer of the quality vector* can tell
// "this film's faces were never scored" from "this film's faces scored zero",
// which is the same distinction scene_detect exists to make and is likewise
// not recoverable from the arrays. A v1 dump reports the vector as unknown;
// re-dump to acquire it, since nobody can assert after the fact how sharp a
// face was.
static constexpr int kSchemaVersion = 2;
static constexpr int kEmbedDim = 512; static constexpr int kEmbedDim = 512;
static std::string basename_of(const std::string& path) { static std::string basename_of(const std::string& path) {
@@ -274,6 +293,9 @@ private:
write_vec(faces, "bbox", bbox_, H5::PredType::NATIVE_FLOAT, 4); write_vec(faces, "bbox", bbox_, H5::PredType::NATIVE_FLOAT, 4);
write_vec(faces, "landmarks", lmk_, H5::PredType::NATIVE_FLOAT, 10); write_vec(faces, "landmarks", lmk_, H5::PredType::NATIVE_FLOAT, 10);
write_vec(faces, "confidence", conf_, H5::PredType::NATIVE_FLOAT); write_vec(faces, "confidence", conf_, H5::PredType::NATIVE_FLOAT);
/// TRACES: AR-028 | SR-002
write_vec(faces, "sharpness", sharp_, H5::PredType::NATIVE_FLOAT);
write_vec(faces, "alignment_residual", resid_, H5::PredType::NATIVE_FLOAT);
std::cerr << "[embedding_dump] wrote " << ts_.size() << " frames, " std::cerr << "[embedding_dump] wrote " << ts_.size() << " frames, "
<< conf_.size() << " faces → " << path_ << "\n"; << conf_.size() << " faces → " << path_ << "\n";
@@ -292,4 +314,5 @@ private:
std::vector<int64_t> face_off_; std::vector<int64_t> face_off_;
std::vector<int32_t> face_cnt_; std::vector<int32_t> face_cnt_;
std::vector<float> emb_, bbox_, lmk_, conf_; std::vector<float> emb_, bbox_, lmk_, conf_;
std::vector<float> sharp_, resid_; // AR-028 quality vector, parallel to conf_
}; };
+59 -6
View File
@@ -1,25 +1,55 @@
#pragma once #pragma once
#include "face_utils.hpp" #include "face_utils.hpp"
#include <cstdint>
#include <iostream> #include <iostream>
// ── FaceAlignerFunc ─────────────────────────────────────────────────────────── // ── FaceAlignerFunc ───────────────────────────────────────────────────────────
/// TRACES: AR-005, AR-030 | SR-002 /// TRACES: AR-005, AR-028, AR-029, AR-030 | SR-002
/// ///
// KPN node: applies a 5-point similarity transform to each detected face, // KPN node: applies a 5-point similarity transform to each detected face,
// producing a 112×112 BGR crop suitable for ArcFace inference. // producing a 112×112 BGR crop suitable for ArcFace inference.
// //
// Alignment is an Umeyama least-squares fit over all five landmarks (AR-005), // Alignment is an Umeyama least-squares fit over all five landmarks (AR-005),
// not a robust one: a RANSAC fit discards the very landmarks AR-030 reads. // not a robust one: a RANSAC fit discards the very landmarks AR-030 reads.
// Degenerate detections (where the fit fails) are dropped from the output //
// vectors. The fit's residual is the AR-030 visibility measure and comes free, // This is also where the AR-028 quality vector is filled in, because this is
// since the warp needs the transform anyway. // where the inputs to it already exist:
//
// - **Visibility** (AR-030) is the fit's residual, and is genuinely free — the
// transform is computed for the warp regardless, and the residual is what
// that fit could not explain.
// - **Sharpness** (AR-029) is measured on the crop this node just produced,
// which is the only place it *can* be measured: the aligned canvas is what
// makes the number scale-normalised, and downstream of the embedder the crop
// is only forwarded for debug rendering. It is not free — 33 us per face
// single-threaded (cvtColor, one Laplacian, two meanStdDev over 112x112) —
// but it is two orders below the embedder inference it qualifies, and it
// runs per face rather than per frame, so a landscape shot costs nothing.
//
// Size, the third axis, is `bbox` and needs no work here.
//
// No face is admitted unscored: every face in the output carries both numbers,
// so a negative value downstream is a bug rather than a poor-quality face.
// Nothing is dropped or discounted on quality — that is AR-030's discount and
// VR-012's knee, both still open.
//
// Degenerate detections (where the fit fails) cannot be scored, since there is
// no crop and no residual to score, and are therefore dropped — but they are
// **counted**, not silently discarded. A nonzero tally means the detector is
// emitting landmark sets the aligner cannot use, which is a fact about the
// detector; losing it leaves a hole in the dump that looks like footage with
// no faces in it.
struct FaceAlignerFunc { struct FaceAlignerFunc {
static constexpr std::string_view label() { return "face_aligner"; } static constexpr std::string_view label() { return "face_aligner"; }
AlignedSceneFrame operator()(SceneFrame sf) { AlignedSceneFrame operator()(SceneFrame sf) {
if (sf.source.eof || sf.faces.empty()) if (sf.source.eof) {
report();
return {std::move(sf.source), {}, {}};
}
if (sf.faces.empty())
return {std::move(sf.source), {}, {}}; return {std::move(sf.source), {}, {}};
std::vector<DetectedFace> good_faces; std::vector<DetectedFace> good_faces;
@@ -33,15 +63,38 @@ struct FaceAlignerFunc {
float residual = -1.f; float residual = -1.f;
cv::Mat crop = align_face(sf.source.image, face.landmarks, &residual); cv::Mat crop = align_face(sf.source.image, face.landmarks, &residual);
if (crop.empty()) { if (crop.empty()) {
std::cerr << "[face_aligner] degenerate detection skipped\n"; ++degenerate_;
continue; continue;
} }
face.alignment_residual = residual; face.alignment_residual = residual;
face.sharpness = crop_sharpness(crop);
good_faces.push_back(face); good_faces.push_back(face);
crops.push_back(std::move(crop)); crops.push_back(std::move(crop));
++scored_;
} }
return {std::move(sf.source), std::move(good_faces), std::move(crops)}; return {std::move(sf.source), std::move(good_faces), std::move(crops)};
} }
/// Faces that carry a full quality vector, and faces the fit could not use.
uint64_t scored() const { return scored_; }
uint64_t degenerate() const { return degenerate_; }
private:
// Reported once at EOF rather than per occurrence: a run with a systematic
// landmark problem would otherwise emit one line per face for the length of
// a film, which is how the count came to be ignored.
void report() {
if (reported_) return;
reported_ = true;
if (degenerate_)
std::cerr << "[face_aligner] " << degenerate_ << " of "
<< (degenerate_ + scored_)
<< " detections had a degenerate landmark fit and were dropped"
" (no crop, so no embedding and no quality vector)\n";
}
uint64_t scored_{0};
uint64_t degenerate_{0};
bool reported_{false};
}; };
+25
View File
@@ -59,11 +59,36 @@ inline constexpr float kArcFaceRef[5][2] = {
// Landmark order matches ArcFace convention (same as SCRFD output order): // Landmark order matches ArcFace convention (same as SCRFD output order):
// [0] right-eye-centre [1] left-eye-centre [2] nose // [0] right-eye-centre [1] left-eye-centre [2] nose
// [3] right-mouth [4] left-mouth // [3] right-mouth [4] left-mouth
/// TRACES: AR-028 | SR-002
struct DetectedFace { struct DetectedFace {
cv::Rect2f bbox; cv::Rect2f bbox;
std::array<cv::Point2f, 5> landmarks; std::array<cv::Point2f, 5> landmarks;
float confidence{0.f}; float confidence{0.f};
// ── AR-028 quality vector ────────────────────────────────────────────────
// Three axes, kept separate and never collapsed into one scalar: they fail
// for different reasons, have different remedies, and do not earn the same
// response. Carried, not consumed — the vector travels with the face into
// the VR-001 dump so a threshold can be re-litigated against recorded data
// rather than by re-running video.
//
// **Size is the third axis and is deliberately not a field here.** It is
// `bbox`, which every consumer already has, scaled by the frame's
// `bbox_upscale` to reach the original resolution AR-002 thresholds in.
// Copying it into a second field would put the same quantity in two
// coordinate spaces inside one struct — the trap SCHEMA.md records for
// `bbox_upscale` — and the copy would be the one that drifts.
//
// Both fields below are -1 until the aligner runs, so *unscored* is
// distinguishable from *scored badly*. Nothing downstream may read a
// negative value as a quality.
// AR-029 sharpness: normalised Laplacian variance over the aligned crop,
// dimensionless. Falls with motion blur and soft focus; invariant to
// contrast, and taken on the fixed 112×112 canvas so it cannot re-measure
// face size. See crop_sharpness() for the construction and its one hazard.
float sharpness{-1.f};
// AR-030 visibility: RMS landmark misfit, in canonical 112×112 pixels, left // AR-030 visibility: RMS landmark misfit, in canonical 112×112 pixels, left
// over after the best similarity fit to the ArcFace template. Rises with // over after the best similarity fit to the ArcFace template. Rises with
// out-of-plane pose and with occlusion; blind to in-plane roll and to face // out-of-plane pose and with occlusion; blind to in-plane roll and to face
+1
View File
@@ -25,6 +25,7 @@ add_executable(sae_tests
test_face_detector_node.cpp test_face_detector_node.cpp
test_scene_detector_node.cpp test_scene_detector_node.cpp
test_replay_fixtures.cpp test_replay_fixtures.cpp
test_embedding_dump.cpp
test_audio_signature.cpp test_audio_signature.cpp
${CMAKE_SOURCE_DIR}/src/backends/gemm_backend.cpp ${CMAKE_SOURCE_DIR}/src/backends/gemm_backend.cpp
${CMAKE_SOURCE_DIR}/src/gallery/gallery_store.cpp ${CMAKE_SOURCE_DIR}/src/gallery/gallery_store.cpp
+169
View File
@@ -0,0 +1,169 @@
// TRACES: AR-028 | VR-001 | UT-139, UT-140, UT-141 | SR-002
//
// The other half of AR-028: the quality vector has to *survive into the dump*.
// Measuring it at inference and then leaving it in a struct that dies at the
// EmbeddedSceneFrame channel would satisfy the letter of "assessed" and none of
// the point — VR-012 sets its knees from recorded data, and what the dump does
// not carry cannot be re-litigated without re-running video on a GPU.
//
// Tier T2, but cheap: EmbeddingDumpFunc is a sink, so it can be driven directly
// with hand-built frames. No model, no video, no gallery — the embedder stamp
// tolerates an unset model path (GR-004 records it as unverifiable).
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "config.hpp"
#include "nodes/embedding_dump_node.hpp"
#include "types.hpp"
#include <H5Cpp.h>
#include <atomic>
#include <cstdio>
#include <filesystem>
#include <string>
#include <vector>
using Catch::Matchers::WithinAbs;
namespace {
namespace fs = std::filesystem;
// Removes the file on scope exit so a failing assertion cannot leave the next
// run reading a stale dump.
struct TempDump {
fs::path path;
explicit TempDump(const char* stem)
: path(fs::temp_directory_path() / (std::string("sae_") + stem + ".h5")) {
std::remove(path.c_str());
}
~TempDump() { std::error_code ec; fs::remove(path, ec); }
};
EmbeddedSceneFrame frame_with(double ts, const std::vector<std::pair<float, float>>& quality) {
EmbeddedSceneFrame ef;
ef.source.timestamp_sec = ts;
ef.source.frame_idx = static_cast<int64_t>(ts * 5.0);
for (const auto& [sharpness, residual] : quality) {
DetectedFace f;
f.bbox = cv::Rect2f(10.f, 20.f, 60.f, 60.f);
f.confidence = 0.8f;
f.sharpness = sharpness;
f.alignment_residual = residual;
ef.faces.push_back(f);
Embedding e{};
e[0] = 1.f;
ef.embeddings.push_back(e);
}
return ef;
}
EmbeddedSceneFrame eof_frame() {
EmbeddedSceneFrame ef;
ef.source.eof = true;
return ef;
}
std::vector<float> read_face_col(const H5::H5File& f, const char* name) {
H5::DataSet ds = f.openDataSet(std::string("faces/") + name);
hsize_t n = 0;
ds.getSpace().getSimpleExtentDims(&n, nullptr);
std::vector<float> out(n);
if (n) ds.read(out.data(), H5::PredType::NATIVE_FLOAT);
return out;
}
int read_schema_version(const H5::H5File& f) {
int v = 0;
f.openAttribute("schema_version").read(H5::PredType::NATIVE_INT, &v);
return v;
}
} // namespace
TEST_CASE("the quality vector survives into the dump", "[dump][AR-028][UT-139]") {
TempDump tmp("quality_roundtrip");
Config cfg;
cfg.dump_embeddings_path = tmp.path.string();
cfg.movie_path = "synthetic";
cfg.sample_fps = 5.f;
std::atomic<bool> done{false};
{
EmbeddingDumpFunc dump(cfg, done);
dump(frame_with(0.0, {{3.25f, 0.75f}, {0.5f, 4.5f}}));
dump(frame_with(0.2, {})); // a frame with no faces
dump(frame_with(0.4, {{12.0f, 0.0f}}));
dump(eof_frame());
}
REQUIRE(done.load());
REQUIRE(fs::exists(tmp.path));
H5::H5File f(tmp.path.string(), H5F_ACC_RDONLY);
const std::vector<float> sharp = read_face_col(f, "sharpness");
const std::vector<float> resid = read_face_col(f, "alignment_residual");
const std::vector<float> conf = read_face_col(f, "confidence");
// Parallel to every other per-face array, so a consumer can index the
// quality of face i with the same slice it uses for the embedding.
REQUIRE(sharp.size() == conf.size());
REQUIRE(resid.size() == conf.size());
REQUIRE(sharp.size() == 3);
CHECK_THAT(sharp[0], WithinAbs(3.25f, 1e-6f));
CHECK_THAT(sharp[1], WithinAbs(0.50f, 1e-6f));
CHECK_THAT(sharp[2], WithinAbs(12.0f, 1e-6f));
CHECK_THAT(resid[0], WithinAbs(0.75f, 1e-6f));
CHECK_THAT(resid[1], WithinAbs(4.50f, 1e-6f));
CHECK_THAT(resid[2], WithinAbs(0.00f, 1e-6f));
}
TEST_CASE("a dump carrying the quality vector announces itself as v2", "[dump][AR-028][UT-140]") {
// The bump is not for readers — they check for the datasets by name, and a
// v1 dump still replays. It is so a consumer of the vector can tell "these
// faces were never scored" from "these faces scored zero", which is not
// recoverable from the arrays. Same reason scene_detect is an attribute.
TempDump tmp("quality_version");
Config cfg;
cfg.dump_embeddings_path = tmp.path.string();
cfg.movie_path = "synthetic";
std::atomic<bool> done{false};
{
EmbeddingDumpFunc dump(cfg, done);
dump(frame_with(0.0, {{1.f, 1.f}}));
dump(eof_frame());
}
H5::H5File f(tmp.path.string(), H5F_ACC_RDONLY);
CHECK(read_schema_version(f) == 2);
}
TEST_CASE("an unscored face keeps its sentinel through the dump", "[dump][AR-028][UT-141]") {
// The aligner admits no unscored face, so this state should be unreachable.
// The dump still must not clamp it: -1 is how a future path that skipped
// scoring would be caught, and rewriting it to 0 would hide that path behind
// a legitimate-looking "featureless crop" reading.
TempDump tmp("quality_sentinel");
Config cfg;
cfg.dump_embeddings_path = tmp.path.string();
cfg.movie_path = "synthetic";
std::atomic<bool> done{false};
{
EmbeddingDumpFunc dump(cfg, done);
dump(frame_with(0.0, {{-1.f, -1.f}}));
dump(eof_frame());
}
H5::H5File f(tmp.path.string(), H5F_ACC_RDONLY);
CHECK(read_face_col(f, "sharpness")[0] < 0.f);
CHECK(read_face_col(f, "alignment_residual")[0] < 0.f);
}
+275 -3
View File
@@ -1,18 +1,26 @@
// TRACES: AR-005, AR-030 | SR-002 // TRACES: AR-005, AR-028, AR-029, AR-030 | UT-130, UT-131, UT-132, UT-133, UT-134, UT-135, UT-136, UT-137, UT-138 | SR-002
// //
// Unit tests for the geometric/numeric helpers in types.hpp and face_utils.hpp: // Unit tests for the geometric/numeric helpers in types.hpp and face_utils.hpp:
// cosine_similarity, the ArcFace 5-point alignment transform, and the alignment // cosine_similarity, the ArcFace 5-point alignment transform, and the two
// residual that AR-030 reads as its visibility measure. GPU-free, model-free. // measured axes of the AR-028 quality vector — the alignment residual AR-030
// reads as visibility, and the normalised Laplacian variance AR-029 reads as
// sharpness. The aligner node is exercised here too, since it is the unit that
// fills the vector in. GPU-free, model-free.
#include <catch2/catch_test_macros.hpp> #include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp> #include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "face_utils.hpp" #include "face_utils.hpp"
#include "nodes/face_aligner_node.hpp"
#include "types.hpp" #include "types.hpp"
#include <algorithm>
#include <array> #include <array>
#include <cmath> #include <cmath>
#include <cstdint>
#include <limits>
using Catch::Matchers::WithinAbs; using Catch::Matchers::WithinAbs;
using Catch::Matchers::WithinRel;
TEST_CASE("cosine_similarity of a unit vector with itself is 1", "[types]") { TEST_CASE("cosine_similarity of a unit vector with itself is 1", "[types]") {
std::array<float, 512> raw{}; std::array<float, 512> raw{};
@@ -179,3 +187,267 @@ TEST_CASE("degenerate landmarks report not-ok rather than a residual", "[face_ut
CHECK_FALSE(a.ok); CHECK_FALSE(a.ok);
CHECK(a.M.empty()); CHECK(a.M.empty());
} }
// ── AR-029: normalised Laplacian variance as a sharpness measure ──────────────
// As with AR-030 above, these assert the *properties* the measure is relied on
// for rather than magic values: no threshold is set here or anywhere else, so a
// number that drifted with the OpenCV version would still be usable — a number
// that stopped falling with blur, or started tracking exposure, would not.
namespace {
// Pink noise: broadband, but with a 1/f spectrum, so most of the energy sits at
// low frequency the way it does in a photograph. An LCG rather than cv::randu so
// the ladder is identical on every machine and every OpenCV build.
//
// **The 1/f part is load-bearing, not decoration.** On a flat-spectrum texture
// (raw white noise) the Gaussian ladder still falls, but the motion-blur ladder
// *rises* — 80.1 → 90.0 across the same kernel lengths used below. That is not a
// bug in the measure, it is what a normalised measure must do on such an input:
// a horizontal smear takes energy out of the numerator and the denominator
// together, and what survives is vertical detail that really is just as fine.
// Real crops have the low-frequency mass that keeps the denominator steady while
// the numerator falls. See the second hazard note on crop_sharpness().
cv::Mat pink(int size, uint32_t seed = 12345u) {
cv::Mat white(size, size, CV_32F);
uint32_t s = seed;
for (int y = 0; y < size; ++y)
for (int x = 0; x < size; ++x) {
s = s * 1664525u + 1013904223u;
white.at<float>(y, x) = float((s >> 16) & 0xFFFF) / 65535.f - 0.5f;
}
// Octaves weighted 1/f. The sigma=0 band keeps genuine per-pixel detail in,
// so the top of the blur ladder is a sharp image rather than an already-soft
// one.
cv::Mat acc = cv::Mat::zeros(size, size, CV_32F);
const double sigma[] = {0.0, 1.0, 2.0, 4.0, 8.0};
const double weight[] = {1.0, 2.0, 4.0, 8.0, 16.0};
for (int k = 0; k < 5; ++k) {
cv::Mat band;
if (sigma[k] <= 0.0) band = white.clone();
else cv::GaussianBlur(white, band, {0, 0}, sigma[k], sigma[k], cv::BORDER_REPLICATE);
acc += band * weight[k];
}
// Into [40, 215]: 8-bit like a real crop, with headroom at both ends so the
// contrast test can halve it without clipping.
double lo = 0, hi = 0;
cv::minMaxLoc(acc, &lo, &hi);
const double scale = 175.0 / (hi - lo);
cv::Mat out;
acc.convertTo(out, CV_8U, scale, 40.0 - lo * scale);
return out;
}
// One master pattern, resampled. So "the same face at 40 px and at 400 px" is
// literally the same image at two resolutions, and a test about source
// resolution is not accidentally a test about two different textures.
// INTER_AREA because area-averaging is what a sensor does when it images the
// same subject onto fewer pixels.
cv::Mat texture(int size) {
static const cv::Mat master = pink(448);
if (size == master.cols) return master;
cv::Mat out;
cv::resize(master, out, {size, size}, 0, 0, cv::INTER_AREA);
return out;
}
cv::Mat gaussian(const cv::Mat& in, double sigma) {
if (sigma <= 0.0) return in.clone();
cv::Mat out;
cv::GaussianBlur(in, out, {0, 0}, sigma, sigma, cv::BORDER_REPLICATE);
return out;
}
// Horizontal box smear — motion blur, which is anisotropic and so attenuates
// only one axis of the spectrum. A measure tuned to the isotropic case can
// miss it.
cv::Mat motion(const cv::Mat& in, int len) {
if (len <= 1) return in.clone();
const cv::Mat k(1, len, CV_32F, cv::Scalar(1.0 / len));
cv::Mat out;
cv::filter2D(in, out, -1, k, {-1, -1}, 0, cv::BORDER_REPLICATE);
return out;
}
// The pipeline reaches 112×112 through warpAffine's INTER_LINEAR; resize with
// the same interpolation so a test about source resolution is not really a test
// about which resampler was used.
cv::Mat to_crop(const cv::Mat& in) {
cv::Mat out;
cv::resize(in, out, {112, 112}, 0, 0, cv::INTER_LINEAR);
return out;
}
} // namespace
TEST_CASE("sharpness falls monotonically along a Gaussian blur ladder", "[face_utils][AR-029][UT-130]") {
const cv::Mat src = texture(112);
float prev = std::numeric_limits<float>::infinity();
for (double sigma : {0.0, 0.6, 1.0, 1.6, 2.5, 4.0}) {
const float s = crop_sharpness(gaussian(src, sigma));
CHECK(s < prev);
CHECK(s > 0.f);
prev = s;
}
}
TEST_CASE("sharpness falls monotonically under motion blur too", "[face_utils][AR-029][UT-131]") {
// Motion blur is the failure mode that leaves the bounding box looking
// perfectly healthy, so it is the one the measure exists for.
const cv::Mat src = texture(112);
float prev = std::numeric_limits<float>::infinity();
for (int len : {1, 3, 5, 9, 15}) {
const float s = crop_sharpness(motion(src, len));
CHECK(s < prev);
CHECK(s > 0.f);
prev = s;
}
}
TEST_CASE("contrast does not leak into sharpness", "[face_utils][AR-029][UT-132]") {
// The normalisation that makes the axis mean the same thing in a dim scene
// and a bright one. Without it VR-012 would locate a different knee per
// film — a magic number wearing a measurement's clothes (AR-024).
const cv::Mat src = texture(112);
cv::Mat dim;
src.convertTo(dim, CV_8U, 0.5, 64.0); // half contrast, re-centred, no clipping
const float a = crop_sharpness(src);
const float b = crop_sharpness(dim);
REQUIRE(a > 0.f);
CHECK_THAT(b, WithinRel(a, 0.03f));
}
TEST_CASE("the contrast invariance is exact, and 8-bit sampling is what bends it",
"[face_utils][AR-029]") {
// Worth separating because the two have different consequences. The
// algebra is exact — scaling I by α scales the Laplacian by α, so both
// variances scale by α² and cancel — which is why halving a float crop
// changes nothing at all.
//
// What deviates is the 8-bit *round trip*: halving the contrast of a stored
// crop throws away a bit of dynamic range, and the quantisation floor it
// leaves behind is broadband, so it lands almost entirely in the numerator.
// The effect scales with how little signal is left to compete with it —
// measured on this texture, a half-contrast copy reads 0.9% high when sharp,
// 24% high at sigma 1.2 and 148% high at sigma 2.5.
//
// So: a dim *and* soft crop reads sharper than it is, and that is the corner
// of the axis VR-012 has to put a knee in. Asserted here rather than left as
// a comment, because "the measure is contrast-invariant" is the kind of claim
// that gets repeated without its precondition.
cv::Mat src;
gaussian(texture(112), 1.2).convertTo(src, CV_32F);
const cv::Mat half = src * 0.5 + 64.0;
const float a = crop_sharpness(src);
const float b = crop_sharpness(half);
REQUIRE(a > 0.f);
CHECK_THAT(b, WithinRel(a, 1e-5f));
}
TEST_CASE("a small sharp face outscores a large soft one", "[face_utils][AR-029][UT-134]") {
// The register's named edge case: "size must not leak into this axis". What
// that means operationally is that the measure is not a monotone function of
// source face size — it reports the detail present in the embedder's input,
// so the ordering can and must invert when the large face is the blurred one.
//
// Small sharp: 40 px of real detail, upsampled 2.8x → finest scale ~2.8 crop px.
// Large soft: 400 px blurred at sigma 20, downsampled 3.57x → ~5.6 crop px.
const float small_sharp = crop_sharpness(to_crop(texture(40)));
const float large_soft = crop_sharpness(to_crop(gaussian(texture(400), 20.0)));
CHECK(small_sharp > large_soft);
}
TEST_CASE("a flat crop scores zero rather than dividing by zero", "[face_utils][AR-029][UT-135]") {
const cv::Mat flat(112, 112, CV_8UC3, cv::Scalar(90, 90, 90));
const float s = crop_sharpness(flat);
CHECK(std::isfinite(s));
CHECK_THAT(s, WithinAbs(0.0f, 1e-6f));
}
TEST_CASE("an empty crop is unscored, not zero", "[face_utils][AR-029][UT-136]") {
// -1 says "nothing measured this"; 0 says "measured, and there was no
// detail". Collapsing them would put unscored faces at the bottom of the
// quality axis, where VR-012 would read them as the blurriest in the film.
CHECK(crop_sharpness(cv::Mat()) < 0.f);
}
// ── AR-028: the aligner fills the vector, and loses nothing quietly ───────────
namespace {
// A face at `centre` in an image with enough texture for sharpness to be a real
// number rather than the flat-crop zero.
std::array<cv::Point2f, 5> face_at(cv::Point2f centre, float scale) {
std::array<cv::Point2f, 5> lm;
for (int i = 0; i < 5; ++i)
lm[i] = {centre.x + (kArcFaceRef[i][0] - 56.f) * scale,
centre.y + (kArcFaceRef[i][1] - 56.f) * scale};
return lm;
}
cv::Mat textured_frame(int w, int h) {
cv::Mat gray = texture(std::max(w, h));
cv::Mat bgr;
cv::cvtColor(gray(cv::Rect(0, 0, w, h)), bgr, cv::COLOR_GRAY2BGR);
return bgr;
}
} // namespace
TEST_CASE("every face the aligner admits carries a full quality vector", "[face_utils][AR-028][UT-137]") {
SceneFrame sf;
sf.source.image = textured_frame(400, 300);
for (auto c : {cv::Point2f{120.f, 100.f}, cv::Point2f{280.f, 190.f}}) {
DetectedFace f;
f.landmarks = face_at(c, 1.2f);
f.bbox = cv::Rect2f(c.x - 60.f, c.y - 60.f, 120.f, 120.f);
f.confidence = 0.9f;
sf.faces.push_back(f);
}
FaceAlignerFunc aligner;
const AlignedSceneFrame out = aligner(std::move(sf));
REQUIRE(out.faces.size() == 2);
for (const auto& f : out.faces) {
// Not "is it good quality" — that is VR-012's to decide. Only that the
// sentinel is gone, so no embedding reaches the matcher unscored.
CHECK(f.sharpness >= 0.f);
CHECK(f.alignment_residual >= 0.f);
}
CHECK(aligner.scored() == 2);
CHECK(aligner.degenerate() == 0);
}
TEST_CASE("a degenerate detection is counted, not silently vanished", "[face_utils][AR-028][UT-138]") {
// It cannot be scored — there is no crop and no fit to score — so it is
// dropped. The requirement is that the drop leaves a trace: without the
// tally, a detector emitting unusable landmark sets produces a dump that
// looks exactly like footage with fewer faces in it.
SceneFrame sf;
sf.source.image = textured_frame(400, 300);
DetectedFace good;
good.landmarks = face_at({150.f, 140.f}, 1.2f);
good.confidence = 0.9f;
sf.faces.push_back(good);
DetectedFace degenerate;
for (auto& p : degenerate.landmarks) p = {200.f, 200.f};
degenerate.confidence = 0.9f;
sf.faces.push_back(degenerate);
FaceAlignerFunc aligner;
const AlignedSceneFrame out = aligner(std::move(sf));
CHECK(out.faces.size() == 1);
CHECK(out.crops.size() == 1);
CHECK(aligner.scored() == 1);
CHECK(aligner.degenerate() == 1);
}