Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 dtourolle
|
889018aa34 | ||
|
dtourolle
|
26de01b2e3 | ||
|
dtourolle
|
41d30395da | ||
|
dtourolle
|
1ae88376e1 |
+77
-14
@@ -248,11 +248,29 @@ the same response.
|
||||
- **Size** — already AR-002, floor at 40×40 px in original resolution, measured
|
||||
end to end by VR-013. It is the precedent for the other two: the
|
||||
threshold was *located*, not chosen.
|
||||
- **Sharpness** — motion blur and soft focus destroy the high-frequency detail
|
||||
the embedder keys on, and unlike size they leave the bounding box looking
|
||||
perfectly healthy. Measured on the **112×112 aligned crop**, not the raw box:
|
||||
the crop is already scale-normalised, so a measure taken there cannot silently
|
||||
re-measure face size and double-count it against AR-002.
|
||||
- **Sharpness** — motion blur and optical defocus destroy the high-frequency
|
||||
detail the embedder keys on, and unlike size they leave the bounding box
|
||||
looking perfectly healthy. Measured on the **112×112 aligned crop**, not the
|
||||
raw box.
|
||||
|
||||
An earlier version of this clause argued the crop is scale-normalised and so a
|
||||
measure taken there "cannot re-measure face size and double-count it against
|
||||
AR-002". **That reasoning is wrong and VR-012 measured it wrong.** The
|
||||
normalisation is geometric, not informational: a 40 px face upscaled into the
|
||||
canonical frame genuinely carries less high-frequency content than a 400 px
|
||||
one downscaled into it, so every candidate measure *does* respond to source
|
||||
size. What the crop yields is **effective resolution in canonical space** —
|
||||
the union of "was small" and "was blurred", not blur alone.
|
||||
|
||||
The conclusion survives, for a better reason. VR-012 sorted its grid by
|
||||
measured sharpness and found the six cells at effectively identical sharpness
|
||||
(0.0003–0.0005) spanning **15.3% to 91.0% TPI**, ordered entirely by source
|
||||
size. Sharpness is therefore not a sufficient statistic for identity loss: a
|
||||
scalar keyed on high-frequency energy cannot separate *attenuated* high
|
||||
frequencies from *destroyed* spatial sampling, because blur preserves
|
||||
mid-frequency facial geometry exactly while downsampling destroys it. The two
|
||||
axes are not redundant and neither substitutes for the other — which is what
|
||||
"not collapsed into one scalar" above now rests on.
|
||||
- **Visibility** — extreme pose or occlusion means the face presents fewer of the
|
||||
features the embedding assumes are present. The measure is the **residual of
|
||||
the AR-005 alignment fit**: the RMS landmark error, in canonical 112×112
|
||||
@@ -333,18 +351,63 @@ hand-chosen cutoff on an uncalibrated measure is the same unfalsifiable magic
|
||||
number AR-024 retired for similarity, and it would fail the same way: meaning
|
||||
something different for every detector, every embedder and every film.
|
||||
|
||||
**A discount curve on sharpness must be flat, then steep.** VR-012 measured the
|
||||
response as a cliff rather than a gradient: Gaussian sigma up to 1.5 costs under
|
||||
1.5 points of TPI in every cell — at 16 px it is very slightly *positive*,
|
||||
smoothing upscale artifacts — sigma 2 costs 1–3, and the 2→3 step costs 7–19. A
|
||||
linear or sigmoid discount over the measure would penalise the whole flat region
|
||||
where blur demonstrably costs nothing.
|
||||
|
||||
**Which blur is modelled is a first-order decision, not a detail.** VR-012 swept
|
||||
three families at matched per-axis PSF spread, and at σ=3 px on a 112 px face
|
||||
they cost 9%, 18% and **53%** error for Gaussian, motion and optical defocus
|
||||
respectively. Defocus is the destructive one because its disc PSF has a jinc
|
||||
transfer function with **exact zeros** — bands annihilated rather than
|
||||
attenuated — where a Gaussian merely rolls off. It is also the case AR-002
|
||||
cannot catch, since a defocused face is large and confidently detected. Any
|
||||
future study that sweeps blur states its family and its justification; a
|
||||
Gaussian-only sweep understated the effect by a factor of five and would have
|
||||
retired this axis as not worth its cost.
|
||||
|
||||
**The cost of blur is proportional to proximity to the decision boundary, not to
|
||||
blur itself.** Sigma 3 costs −22.5 points at 24 px, but only −7.9 at 112 px
|
||||
(margin to spare) and −8.3 at 16 px (already below threshold). This is why the
|
||||
axes must combine multiplicatively in `EvidenceDiscounter` rather than each
|
||||
gating independently.
|
||||
|
||||
**Sharpness discounts; it must never gate.** VR-012 tried the gate directly, as
|
||||
a compute saving: skipping the embed below a sharpness threshold costs 15.1% of
|
||||
true identifications to save 20% of the work, against the size filter's 4.7% at
|
||||
16.7% — three times the damage, from a measure that needs the warped crop plus a
|
||||
DFT where size is a bbox dimension available for free. The reason is a ceiling
|
||||
no measure can beat: **at 112 px with defocus radius 6 — visually destroyed —
|
||||
46.9% of faces still identify correctly, and rank-1 is still 94.8%.** Apparent
|
||||
blur does not determine the outcome. The size filter wins only because smallness
|
||||
destroys identity more completely than blur does (16 px succeeds 23.5% of the
|
||||
time), and that asymmetry is the measured justification for the rule above:
|
||||
**failing sharpness discounts the observation, failing size may drop it.**
|
||||
|
||||
**Current:** visibility is measured and carried — `estimate_alignment()` in
|
||||
`src/face_utils.hpp` returns the residual alongside the transform, and
|
||||
`FaceAlignerFunc` writes it to `DetectedFace::alignment_residual`. Size is
|
||||
`min_face_px` (40, decoded-frame space — AR-002 still open). Sharpness is
|
||||
unmeasured. Nothing yet *consumes* any of it: no discount is applied, and
|
||||
`align_face()` still drops the degenerate-fit case without counting it.
|
||||
`FaceAlignerFunc` writes it to `DetectedFace::alignment_residual`. Sharpness is
|
||||
measured: `assess_sharpness()` in `src/quality.hpp` returns five AR-029
|
||||
candidates over a fixed 64×64 window on the face interior, and VR-012 has ranked
|
||||
them — `var_laplacian` and `tenengrad` are disqualified as discounts (see
|
||||
AR-029), leaving `hf_energy_ratio` as the only correctly-signed survivor. Size
|
||||
is `min_face_px` (40, decoded-frame space — AR-002 still open). All three are
|
||||
exposed to studies through `sae_embed`. Nothing yet *consumes* any of it: no
|
||||
discount is applied, and `align_face()` still drops the degenerate-fit case
|
||||
without counting it.
|
||||
|
||||
**Gap:** AR-029 entirely. For AR-030, the measure exists but the discount does
|
||||
not — it must reach `EvidenceDiscounter` as the reliability term. For AR-028, the
|
||||
residual does not yet reach the VR-001 dump, which is what VR-012 needs to run
|
||||
from fixtures; that is the next step, since it unblocks the study that sets
|
||||
every remaining behaviour.
|
||||
**Gap:** the discount itself, on every axis. Neither sharpness nor the residual
|
||||
reaches `EvidenceDiscounter`, whose weight remains pure novelty — so a profile
|
||||
or defocused view still moves a track's belief hardest when it deserves the
|
||||
least trust. Neither reaches the VR-001 dump either, so VR-012 must still re-run
|
||||
video rather than replay fixtures. VR-012's **pose half is not started**: the
|
||||
AR-030 residual has no arm in the grid, so whether the 5-point proxy suffices or
|
||||
a dedicated landmark model is needed remains open. And the sharpness result is
|
||||
weak enough (best within-cell AUC 0.530) that whether AR-029 earns a discount at
|
||||
all is still a judgement, not a measurement.
|
||||
|
||||
## AR-007, AR-008 — Tracking
|
||||
|
||||
|
||||
@@ -53,6 +53,14 @@ every finding below.
|
||||
A training-set effect that did not reproduce on 5 held-out films once
|
||||
two methodology bugs in the comparison harness were found and fixed.
|
||||
|
||||
- :material-blur:{ .lg .middle } **[What does blur cost?](quality-knee.md)**
|
||||
|
||||
---
|
||||
|
||||
Sharpness is not a sufficient statistic for identity loss, blur breaks
|
||||
confidence rather than ranking, and variance-of-Laplacian is
|
||||
anti-predictive at fixed resolution.
|
||||
|
||||
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
# Quality knee: what does a blurred or small face cost, and can a measure predict it?
|
||||
|
||||
VR-012. Companion to the minimum-face-size studies VR-005 and VR-013 (see the
|
||||
[requirement register](requirements.md)), which located the size floor at 40 px;
|
||||
this asks the same question for **sharpness**, and asks whether any cheap
|
||||
measure taken on the aligned crop can be acted on at inference.
|
||||
|
||||
Run by
|
||||
[`scripts/validation/quality_knee.py`](https://REPOLINK/scripts/validation/quality_knee.py)
|
||||
through the `sae_embed` bindings — detection, the ArcFace warp, the embedder,
|
||||
the five candidate measures and the Platt calibration are all the shipped C++.
|
||||
|
||||
## Protocol
|
||||
|
||||
1670 gallery actors with 3 or more mugshots (of 2456 total), one image held out
|
||||
per actor as a probe, the remaining 10326 embeddings staying in the gallery at
|
||||
native resolution. Only the probe degrades — reference mugshots are clean and
|
||||
the face coming out of the video is not.
|
||||
|
||||
Each probe passes through a **joint grid**: downscale to *S*×*S* and back to
|
||||
112 (the sampling loss), then blur at level *L* in canonical pixels. Three blur
|
||||
families, 36 cells each, 60120 probe-cell records per family:
|
||||
|
||||
| family | models | parameter |
|
||||
|---|---|---|
|
||||
| Gaussian | soft focus, a generic stand-in | sigma 0 … 3 |
|
||||
| **Disc** | **real optical defocus** — the circle of confusion | radius 0 … 6 |
|
||||
| Motion | camera pan or moving subject | length 0 … 21 px |
|
||||
|
||||
The three are not interchangeable, and sweeping only the first was the original
|
||||
design error — one that would have produced a wrong answer, not merely an
|
||||
incomplete one (Result 3). A defocused lens spreads a point into a **uniform
|
||||
disc**, whose transfer function is a jinc — `2·J1(x)/x` — that crosses zero and
|
||||
goes negative, annihilating whole frequency bands and returning the ones beyond
|
||||
each zero phase-reversed. A Gaussian MTF is strictly positive and monotone and
|
||||
does neither. More practically: defocus and motion are how a face ends up
|
||||
**large and useless**, while Gaussian blur as swept here mostly co-occurs with
|
||||
small faces. That difference decides whether sharpness carries anything the size
|
||||
filter does not.
|
||||
|
||||
Families are compared at matched **per-axis PSF standard deviation** (σ for a
|
||||
Gaussian, R/2 for a disc, L/√12 for a linear smear), never at equal raw
|
||||
parameter, which would compare different amounts of damage.
|
||||
|
||||
Identification is the pipeline's own decision: per-actor best-of-N cosine →
|
||||
Platt sigmoid → accept above `prob_threshold` 0.754. Never a raw cosine
|
||||
(AR-024).
|
||||
|
||||
## Result 1 — sharpness is not a sufficient statistic
|
||||
|
||||
Sorting the 36 Gaussian cells by `hf_energy_ratio`, the six sigma-3 cells land
|
||||
at effectively identical measured sharpness:
|
||||
|
||||
| size | sigma | hf_energy_ratio | TPI |
|
||||
|---|---|---|---|
|
||||
| 16 | 3 | 0.0003 | **15.3%** |
|
||||
| 24 | 3 | 0.0003 | 63.2% |
|
||||
| 32 | 3 | 0.0003 | 79.4% |
|
||||
| 48 | 3 | 0.0003 | 86.6% |
|
||||
| 64 | 3 | 0.0004 | 88.4% |
|
||||
| 112 | 3 | 0.0005 | **91.0%** |
|
||||
|
||||
Same measured sharpness, a **76-point spread in identification**. It inverts
|
||||
too: 16 px unblurred measures 0.0033 and scores 23.5%, while 48 px at sigma 2
|
||||
measures *lower* at 0.0021 and scores 96.6%.
|
||||
|
||||
A canonical-frame sharpness scalar cannot separate *attenuated* high
|
||||
frequencies from *destroyed* spatial sampling. Blur suppresses the high band
|
||||
while preserving mid-frequency facial geometry exactly; downsampling to 16 px
|
||||
destroys that geometry outright. Both look alike to any measure keyed on
|
||||
high-frequency energy.
|
||||
|
||||
This is the measured basis for AR-028's rule that the axes are **kept separate
|
||||
and not collapsed into one scalar**, and it settles the double-counting
|
||||
question: size and sharpness are not redundant, and neither substitutes for the
|
||||
other.
|
||||
|
||||
## Result 2 — blur is a cliff, and it breaks confidence, not identity
|
||||
|
||||
TPI % by size (rows) against Gaussian sigma (columns):
|
||||
|
||||
| size | 0 | 0.5 | 1 | 1.5 | 2 | 3 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 16 | 23.5 | 24.0 | 25.0 | 24.6 | 23.9 | 15.3 |
|
||||
| 24 | 85.7 | 85.1 | 85.6 | 85.9 | 82.6 | 63.2 |
|
||||
| 32 | 95.9 | 95.9 | 96.0 | 95.5 | 93.7 | 79.4 |
|
||||
| 48 | 98.7 | 98.7 | 98.4 | 98.1 | 96.6 | 86.6 |
|
||||
| 64 | 98.6 | 98.6 | 98.8 | 98.4 | 97.5 | 88.4 |
|
||||
| 112 | 98.9 | 98.9 | 98.8 | 98.6 | 98.0 | 91.0 |
|
||||
|
||||
Three regimes: **sigma ≤ 1.5 is free** (every cell moves under 1.5 points, sign
|
||||
flipping at random — at 16 px it slightly *improves*, smoothing upscale
|
||||
artifacts); sigma 2 costs 1–3 points; the 2→3 step costs 7–19. A smooth
|
||||
discount curve is therefore the wrong shape — the response is flat, then falls
|
||||
off a cliff.
|
||||
|
||||
**The cost peaks at the size knee, not at full resolution.** Sigma 3 costs
|
||||
−22.5 points at 24 px but only −7.9 at 112 px and −8.3 at 16 px. Blur has no
|
||||
intrinsic cost; it costs in proportion to how close the observation already sits
|
||||
to the decision boundary. At 112 px there is margin to spare, at 16 px the probe
|
||||
is already below threshold, and at 24 px it sits exactly on the knee.
|
||||
|
||||
**What blur destroys is confidence, not ranking.** Rank-1 barely moves: 99.3% →
|
||||
99.2% at 112 px across the whole sigma range. The extreme case is 16 px at sigma
|
||||
3, where rank-1 is **80.2%** while TPI is **15.3%** — 65 points of probes have
|
||||
the correct actor ranked first and are rejected anyway for falling under the
|
||||
probability threshold.
|
||||
|
||||
That is why **FPI never left 0.1% in any of the 108 cells across all three
|
||||
families**. Degradation produces TBI, never a wrong name. The calibration
|
||||
degrades gracefully, which is what SR-002 needs.
|
||||
|
||||
## Result 3 — the blur *family* matters more than the blur *amount*
|
||||
|
||||
Comparing families by their raw parameter is meaningless — sigma, radius and
|
||||
length are different units. They are matched here by the **per-axis standard
|
||||
deviation of the PSF**, which puts them on one scale:
|
||||
|
||||
| family | per-axis σ | level giving σ = 3 px |
|
||||
|---|---|---|
|
||||
| Gaussian σ | σ | 3 |
|
||||
| Disc radius R | R/2 | 6 |
|
||||
| Motion length L | L/√12 | 10.4 |
|
||||
|
||||
For reference the ArcFace template places the eyes 35.2 canonical px apart, so
|
||||
σ = 3 px is 9% of the inter-ocular distance.
|
||||
|
||||
TPI at matched severity, interpolated within each family:
|
||||
|
||||
| size | σ=3 Gaussian | σ=3 Motion | σ=3 **Defocus** | defocus penalty |
|
||||
|---|---|---|---|---|
|
||||
| 16 | 15.3 | 15.1 | 11.0 | +4.3 |
|
||||
| 24 | 63.2 | 61.1 | 41.4 | +21.9 |
|
||||
| 32 | 79.4 | 76.1 | 50.4 | +29.0 |
|
||||
| 48 | 86.6 | 80.9 | 52.6 | +34.0 |
|
||||
| 64 | 88.4 | 81.7 | 51.0 | +37.4 |
|
||||
| 112 | 91.0 | 82.1 | **46.9** | **+44.0** |
|
||||
|
||||
**Optical defocus is up to 44 points more destructive than a Gaussian of
|
||||
identical spread**, and the ordering is defocus ≫ motion > Gaussian throughout.
|
||||
At σ=1 the three families are indistinguishable, and at σ=2 they differ by under
|
||||
5 points; the divergence appears only when both the blur is severe *and* the face
|
||||
is large.
|
||||
|
||||
That pattern is physically consistent. At 16 px the resampling has already
|
||||
removed the high frequencies, so the PSF's shape has nothing left to act on and
|
||||
all three agree. At 112 px the full spectrum is present and shape decides: a
|
||||
Gaussian MTF rolls off gently and always leaves *some* energy at every
|
||||
frequency, so the embedder receives a merely attenuated signal, while a disc MTF
|
||||
is a jinc that **hits exact zeros** — whole frequency bands annihilated rather
|
||||
than attenuated, with the bands beyond each zero returning phase-reversed.
|
||||
Motion sits between them because it ruins one axis and leaves the perpendicular
|
||||
one untouched.
|
||||
|
||||
**The methodological consequence is the important one.** This study originally
|
||||
swept Gaussian blur alone and concluded blur was a minor effect. On the family
|
||||
that actually occurs in film, the same nominal severity costs **53% error
|
||||
instead of 9%** at full resolution. A threshold set from the Gaussian arm would
|
||||
have been wrong by a factor of five in error rate, and the axis would probably
|
||||
have been dropped as not worth its cost.
|
||||
|
||||
**Defocus is also the case a size gate cannot catch.** Every one of those 112 px
|
||||
faces is large and confidently detected, and sails through AR-002 untouched.
|
||||
That, not the Gaussian result, is what justifies a sharpness axis existing at
|
||||
all.
|
||||
|
||||
## Result 4 — variance of Laplacian is anti-predictive at fixed degradation
|
||||
|
||||
Pooled across all cells, every candidate scores AUC 0.76–0.80 for predicting
|
||||
correct identification, with textbook `var_laplacian` top. That number is close
|
||||
to worthless: it rewards a measure for detecting *how degraded the crop is*,
|
||||
which all five do. The question a per-observation discount needs is whether, at
|
||||
a **fixed** degradation, the measure predicts which faces fail:
|
||||
|
||||
| measure | Gaussian | Defocus | Motion |
|
||||
|---|---|---|---|
|
||||
| `hf_energy_ratio` | **0.530** | **0.521** | **0.557** |
|
||||
| `norm_var_laplacian` | 0.520 | 0.507 | 0.539 |
|
||||
| `dir_min_tenengrad` | 0.524 | 0.512 | 0.506 |
|
||||
| `tenengrad` | 0.433 | 0.437 | 0.457 |
|
||||
| `var_laplacian` | 0.423 | 0.422 | 0.473 |
|
||||
|
||||
Best is 0.557 — barely above chance, and `hf_energy_ratio` wins on all three
|
||||
families. `var_laplacian` is anti-predictive on all three too, so that finding
|
||||
does not depend on the blur model.
|
||||
|
||||
**The two metrics measure different jobs, and the candidates split along that
|
||||
line.** On the motion arm `dir_min_tenengrad` has the best *pooled* AUC by a
|
||||
wide margin — **0.854** against 0.792 for the next — exactly as its synthetic
|
||||
directional-blur ladder predicted, yet its within-cell AUC there is 0.506. It is
|
||||
an excellent detector of *how badly smeared a crop is* and no guide at all to
|
||||
*which face will be recognised*. Pooled AUC is the right metric for a
|
||||
gross-degradation flag; within-cell AUC is the right one for a per-observation
|
||||
discount; a measure can be strong at one and useless at the other.
|
||||
|
||||
Deciles within the 16 px Gaussian cell, where 1277 failures give the test real
|
||||
power:
|
||||
|
||||
| `var_laplacian` decile | TPI |
|
||||
|---|---|
|
||||
| 0.00071–0.00192 (blurriest) | **37.1%** |
|
||||
| 0.00242–0.00278 | 22.8% |
|
||||
| 0.00397–0.00447 | 25.7% |
|
||||
| 0.00625–0.01445 (sharpest) | **14.4%** |
|
||||
|
||||
The faces the measure calls sharpest are **2.6x less identifiable** than those
|
||||
it calls blurriest, monotone across ten bins of 167. Within a cell every crop
|
||||
received identical degradation, so the residual variance is *native contrast*,
|
||||
not native detail — and hard shadows, high-contrast lighting, sharpening halos
|
||||
and JPEG ringing all raise Laplacian variance while making a face harder to
|
||||
match. The measure reads photographic style and encoding artifacts and calls
|
||||
them sharpness.
|
||||
|
||||
`hf_energy_ratio` is the only candidate with a correctly-signed within-cell
|
||||
trend (16.2% → 35.3% across the same deciles), being a pure ratio in which the
|
||||
contrast factor cancels.
|
||||
|
||||
**Consequence:** a per-face quality *discount* keyed on variance of Laplacian —
|
||||
the most widely used blur metric in production vision pipelines — would
|
||||
systematically down-weight the *more* identifiable faces. It is worse than no
|
||||
discount.
|
||||
|
||||
## Result 5 — as a compute gate, sharpness loses to the size filter
|
||||
|
||||
Skipping the embed for crops below a threshold, measured as compute saved
|
||||
against true identifications lost:
|
||||
|
||||
| gate | skipped | true IDs lost | of skipped, doomed anyway |
|
||||
|---|---|---|---|
|
||||
| `hf_energy_ratio` < 0.00023 | 10.0% | 7.6% | 37.9% |
|
||||
| `hf_energy_ratio` < 0.00051 | 20.0% | 15.1% | 38.7% |
|
||||
| **source size < 24 px** | **16.7%** | **4.7%** | **77.3%** |
|
||||
|
||||
At a comparable skip rate the size filter loses **4.7% against sharpness's
|
||||
15.1%** — three times less damage — and it is free, being a bbox dimension
|
||||
available before alignment or embedding, where sharpness needs the warped crop
|
||||
plus a colour convert, three convolutions and a 64×64 DFT.
|
||||
|
||||
Restricting to large faces (≥64 px) on the **defocus** arm, where the size
|
||||
filter is blind, improves the gate's precision 3.5x (37% of skipped crops doomed
|
||||
versus 10.7% on the Gaussian arm) but not its trade: skip 10%, lose 7.0%.
|
||||
|
||||
A hard ceiling explains why. **At 112 px with defocus radius 6 — visually
|
||||
destroyed — 46.9% of faces still identify correctly and rank-1 is still 94.8%.**
|
||||
Blur does not determine the outcome, so any gate keyed on apparent blur is
|
||||
predicting a coin flip. The size filter wins not because size is better
|
||||
measured, but because *smallness destroys identity more completely than blur
|
||||
does*: 16 px faces succeed only 23.5% of the time, so discarding them is cheap.
|
||||
|
||||
## What this means for the requirements
|
||||
|
||||
**Do not gate on sharpness; discount on it.** Heavily defocused faces remain
|
||||
~47% identifiable, so a gate destroys recoverable evidence. This is the first
|
||||
hard evidence that AR-028's "**discounts the observation, never deletes the
|
||||
detection**" is right on the merits rather than merely cautious. Since ranking
|
||||
survives where confidence does not, the per-track accumulation (AR-025) should
|
||||
recover much of what a single-frame threshold rejects — which is also the
|
||||
argument for the discount living in `EvidenceDiscounter` rather than in a filter.
|
||||
|
||||
**`var_laplacian` and `tenengrad` are disqualified as discounts** by Result 4,
|
||||
on all three blur families. They remain usable as coarse *gross-degradation*
|
||||
detectors, the role in which their pooled AUC is real — the same role the size
|
||||
filter plays — but they must never weight a per-observation belief.
|
||||
|
||||
**`hf_energy_ratio` is the only surviving discount candidate**, best on all
|
||||
three families, and its within-cell signal (0.52–0.56) is weak enough that
|
||||
shipping a discount on it needs justification beyond this study.
|
||||
|
||||
**`dir_min_tenengrad` earns a different job.** Its pooled 0.854 on the motion arm
|
||||
makes it the best available detector of gross directional smear — useful as a
|
||||
per-frame "this shot is unusable" flag, which is a decision about a *frame*, not
|
||||
a weighting of an *observation*. If AR-029 ships two measures for two roles, this
|
||||
is the second one, and it must not be confused with the first.
|
||||
|
||||
**Model the blur family, not just its amount.** Result 3 makes the choice of
|
||||
degradation model a first-order design decision rather than a detail: the same
|
||||
matched severity costs 9% or 53% error depending on the PSF. Any future study
|
||||
that sweeps blur must state which family it used and why.
|
||||
|
||||
**Any discount curve must be flat then steep**, not linear or sigmoid over the
|
||||
measure. Blur costs nothing until it costs a great deal.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Cooperative population.** Gallery mugshots are frontal and well-lit;
|
||||
within-cell failures are likely dominated by cross-view mismatch, which no
|
||||
sharpness measure can predict. Read the ~chance within-cell AUCs as "sharpness
|
||||
does not predict the dominant failure mode *here*", not as "sharpness is
|
||||
meaningless".
|
||||
- **Uniform grid, not a natural distribution.** Sizes and blur levels are
|
||||
sampled evenly, so "skip 16.7%" is exactly the 16 px row. The gate comparisons
|
||||
are like-for-like on identical records, but the absolute savings are not what
|
||||
a film would show.
|
||||
- **TensorRT fp16.** A different realisation of the embedder from the fp32 ONNX
|
||||
reference — VR-005 measured ~0.85 cosine agreement with separation intact.
|
||||
Gallery and probes share one session so the study is internally consistent,
|
||||
but the absolute knee belongs to the fp16 space.
|
||||
- **Blur is applied in the canonical frame**, after resampling, so its width is
|
||||
independent of the cell's size. Real optics blur before sampling.
|
||||
- **The top motion rung is an anchor, not an operating point.** Length 21 is a
|
||||
per-axis σ of 6.1 — 17% of the inter-ocular distance, a streak rather than a
|
||||
face — and it is swept to bound the curve, not because a frame like that is
|
||||
worth reasoning about. Its 3.4% TPI at 112 px should not be quoted as a
|
||||
headline. The same caution applies less severely to defocus radius 6 (σ = 3).
|
||||
- **Per-axis σ equates spread, not perceptual damage.** It is the fairest single
|
||||
scalar for comparing PSFs, but Result 3 is precisely the finding that equal
|
||||
spread does *not* mean equal harm, so the matched-severity tables compare
|
||||
like-for-like inputs, not like-for-like severity as a face would experience it.
|
||||
@@ -56,7 +56,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
|
||||
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
|
||||
| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | Planned |
|
||||
| AR-029 | Sharpness measure on the **aligned crop** (scale-normalised, so it cannot re-measure size) | SR-002 | Medium | Planned |
|
||||
| AR-029 | Sharpness measure on the **aligned crop**, consumed as a discount and **never as a gate** | SR-002 | Medium | **In Progress** — five candidates implemented (`src/quality.hpp`) and ranked by VR-012 over three blur families. `var_laplacian` and `tenengrad` are **disqualified as discounts**: within a fixed degradation they are anti-predictive on *all three* families (AUC 0.42–0.47; the decile the measure calls sharpest is 2.6× *less* identifiable), since their residual variance is native contrast, not detail. `hf_energy_ratio` is the only correctly-signed survivor, best on all three, and weak (0.52–0.56). `dir_min_tenengrad` is the best *gross-smear detector* (pooled AUC 0.854 on motion) but ~chance within-cell, so it serves a per-frame flag, not a per-observation weight. The parenthetical this row used to carry — "scale-normalised, so it cannot re-measure size" — was wrong: every candidate responds to source size, and the axes are separable for a different reason (see AR-028) |
|
||||
| AR-030 | Visibility measure from the AR-001 5-point landmarks — extreme pose or occlusion **discounts the observation, never deletes the detection** | SR-002 | Medium | **In Progress** — measure is the AR-005 alignment residual (`estimate_alignment()`), carried on `DetectedFace`; roll/scale invariance and monotonicity under foreshortening asserted. Nothing consumes it as a discount yet |
|
||||
|
||||
## Deployment (DP)
|
||||
@@ -114,7 +114,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
|
||||
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned |
|
||||
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned |
|
||||
| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | Planned |
|
||||
| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | **In Progress** — sharpness half done ([`docs/quality-knee.md`](quality-knee.md)): 1670 actors, joint size×blur grid over three blur families (Gaussian, disc defocus, linear motion), 60120 probe-cell records each. Sharpness is **not a sufficient statistic** (equal measured sharpness spans 15.3–91.0% TPI, ordered by source size); **the blur family matters more than its amount** — at matched per-axis σ=3 on a 112 px face, Gaussian/motion/defocus cost 9/18/**53**% error, so a Gaussian-only sweep understates real lens blur fivefold; blur breaks **confidence, not ranking** (rank-1 80.2% where TPI is 15.3%), so FPI never left 0.1% in any of the 108 cells; a sharpness **gate** loses 3× more true presence than the free size filter at equal saving, because even destroyed faces stay 46.9% identifiable. **Pose half not started** — the AR-030 residual is exposed via `sae_embed.alignment_residual` but no pose arm has been run, so the dedicated-landmark-model question is still open |
|
||||
| VR-014 | Audio-signature **offset recovery on real content** — a known trim recovered from film audio, not from the synthetic golden tone | PR-002 | Medium | **Done** — 40 random in-cap offsets, every one recovered to the nearest frame: **worst error 46 ms against a 500 ms budget**, and 46 ms is the floor rather than a result, since the offset is quantised to whole 92.88 ms frames. The `runtime/2` anchor confirmed through real head-trimmed files (a `delta` trim moves the window by `delta/2`). The one soft spot is **tier labelling, not accuracy**: the score falls with sub-frame misalignment (0.94–0.99 near a frame boundary, 0.69–0.73 at half a frame), so 27/40 correct alignments were demoted to `loose`. ±1 frame of slack in the *score* fixes it — measured, all 40 back to `audio` (min 0.906), false matches unmoved at 0.12–0.16, costing 81 ms of the budget |
|
||||
| VR-013 | Cross-source identification probe — gallery from one recording, probes from another, swept over input resolution end to end | PR-002 | Medium | **In Progress** — holding 90% of the plateau needs ~50 px end to end against VR-005's ~22 px, the gap being detection and landmark error; **`min_face_px` 40, since 32 admits faces in the falling region** (AR-002). FPI 0.0% at every scale. Ceiling is cross-view, not resolution |
|
||||
|
||||
|
||||
@@ -39,6 +39,7 @@ nav:
|
||||
- Best Model: best-model.md
|
||||
- Gallery Scope (Full vs. Limited): gallery-scope.md
|
||||
- Pose Expansion: pose-expansion.md
|
||||
- Quality Knee (Blur and Size): quality-knee.md
|
||||
- LVFace Deep Dive: lvface-deep-dive.md
|
||||
- Full Experiment Log: model-bakeoff.md
|
||||
- Service Conversion (proposal): service-conversion.md
|
||||
|
||||
@@ -183,10 +183,16 @@ DEDUP_SIM = 1.0 - 1e-7
|
||||
class Stages:
|
||||
"""Thin holder so the rest of the script has one object to call."""
|
||||
|
||||
def __init__(self, detector: str, arcface: str, conf: float, nms: float):
|
||||
def __init__(self, detector: str, arcface: str, conf: float, nms: float,
|
||||
detector_engine: str = "", arcface_engine: str = ""):
|
||||
# The engine paths are only consulted by a TRT-backend build, where they
|
||||
# are mandatory — that backend loads a pre-built .engine and will not
|
||||
# fall back to reading the .onnx. An ORT build ignores them, so passing
|
||||
# them unconditionally is safe and keeps one constructor for both.
|
||||
self.engine = sae_embed.FaceEmbedder(
|
||||
detector_model=detector, arcface_model=arcface,
|
||||
conf=conf, nms=nms, max_side=0)
|
||||
conf=conf, nms=nms, max_side=0,
|
||||
detector_engine=detector_engine, arcface_engine=arcface_engine)
|
||||
|
||||
def detect(self, img):
|
||||
return self.engine.detect(img)
|
||||
|
||||
@@ -0,0 +1,667 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
quality_knee.py — VR-012: what does a blurred or small face cost in identification,
|
||||
and which sharpness measure predicts it?
|
||||
|
||||
TRACES: VR-012, AR-028, AR-029
|
||||
|
||||
VR-005 located the size floor by degrading held-out gallery mugshots and watching
|
||||
TPI/FPI fall. This does the same over a **joint size x blur grid**, and adds the
|
||||
part that makes the result usable at inference.
|
||||
|
||||
Why a joint grid and not two sweeps
|
||||
-----------------------------------
|
||||
A 16 px face upscaled to 112 has already lost its high frequencies, so additional
|
||||
blur costs it far less than it costs a 112 px one. Sweeping the axes separately
|
||||
measures each in the presence of an implicit "other axis at its best" and misses
|
||||
that interaction entirely — and the interaction is the whole question, because
|
||||
AR-002 already gates on size and AR-029 proposes to discount on sharpness. If
|
||||
identity loss turns out to be a function of the sharpness measure alone, then one
|
||||
axis carries the information and discounting on both double-counts. If a
|
||||
small-but-sharp and a large-but-blurred probe at equal measure lose different
|
||||
amounts, the axes are genuinely separate and both belong.
|
||||
|
||||
Why sigma is not the answer
|
||||
---------------------------
|
||||
Sigma is a lab variable. At inference nothing knows how blurred a face is, so a
|
||||
knee expressed in sigma cannot be acted on. What AR-028/AR-030 can consume is
|
||||
measure value -> expected identity reliability
|
||||
so the controlled degradation exists to *select and calibrate the measure*, and
|
||||
the measure is what ships. Every candidate is therefore scored on every degraded
|
||||
crop, and the candidates are ranked by how well each predicts the identification
|
||||
outcome (AUC over probe-cell records), not by how smooth its ladder looks.
|
||||
|
||||
Protocol (VR-005's, extended)
|
||||
-----------------------------
|
||||
1. Every gallery actor with at least `--min-images` mugshots. At the default 3,
|
||||
holding one out still leaves two references per actor.
|
||||
2. Hold out ONE image per actor as the probe; the rest stay in the gallery at
|
||||
native resolution. Only the probe degrades — reference mugshots are clean and
|
||||
the face coming out of the video is not, which is the production case.
|
||||
3. For each (size, sigma) cell: downscale the probe crop to size x size and back
|
||||
to 112 (the sampling loss), then Gaussian blur at sigma canonical px (the
|
||||
optical/motion loss). Resolution first, then blur, so sigma always means the
|
||||
same thing in the frame AR-029 measures in, whatever the cell's size.
|
||||
4. Score all five AR-029 candidates on the degraded crop, through the C++
|
||||
binding.
|
||||
5. Embed, match against the whole gallery, record TPI/FPI/unidentified.
|
||||
|
||||
Decision rule is the pipeline's: per-actor best-of-N cosine -> Platt sigmoid ->
|
||||
accept if P > prob_threshold. Never a raw cosine (CLAUDE.md invariant, AR-024).
|
||||
|
||||
Everything runs through `sae_embed` — detection, the ArcFace warp, the embedder,
|
||||
the sharpness measures and the calibration are all the shipped C++. Nothing here
|
||||
re-implements a pipeline stage in numpy; the analysis on top of the recorded
|
||||
numbers (AUC, knee location) is analysis and is numpy's job.
|
||||
|
||||
CAVEAT — FPI IS RELATIVE, NOT ABSOLUTE
|
||||
--------------------------------------
|
||||
False positives grow with the number of actors competing. Read FPI as a curve
|
||||
across cells, not as a production rate. This runs the whole eligible gallery
|
||||
rather than VR-005's 100-actor sample, so the understatement is much smaller,
|
||||
but a production library is larger still.
|
||||
|
||||
Usage
|
||||
-----
|
||||
python scripts/validation/quality_knee.py \
|
||||
--images images --gallery gallery_lvface.h5 \
|
||||
--arcface models/LVFace-B_Glint360K.onnx \
|
||||
--min-images 3 --out experiments/results/vr012_quality_knee
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(REPO / "scripts"))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
# min_face_size owns the shared scaffolding — actor discovery, the sae_embed
|
||||
# locator, the Stages wrapper, the calibration-through-the-binding and the house
|
||||
# plot palette. Importing it keeps one copy of each; a second copy of the
|
||||
# calibration path in particular is what AR-024 exists to prevent.
|
||||
import min_face_size as vr005 # noqa: E402
|
||||
from min_face_size import ( # noqa: E402
|
||||
DEDUP_SIM, INTERP, Stages, calibrate_gallery, discover_actors,
|
||||
gallery_keys, normalise_name, probability, err,
|
||||
INK, MUTED, GRID, SURFACE, BLUE, GREEN, RED, AMBER,
|
||||
)
|
||||
|
||||
import cv2 # noqa: E402
|
||||
import numpy as np # noqa: E402
|
||||
import sae_embed # noqa: E402
|
||||
|
||||
# The five AR-029 candidates, in quality.hpp's order. Names match the binding's
|
||||
# attributes so the CSV columns and the C++ fields cannot drift apart.
|
||||
MEASURES = ["var_laplacian", "norm_var_laplacian", "tenengrad",
|
||||
"hf_energy_ratio", "dir_min_tenengrad"]
|
||||
|
||||
|
||||
# ── Degradation ───────────────────────────────────────────────────────────────
|
||||
|
||||
def disc_kernel(radius: float) -> np.ndarray:
|
||||
"""The circle-of-confusion PSF of a defocused lens.
|
||||
|
||||
Optical defocus is **not** Gaussian, and the difference is not cosmetic. A
|
||||
lens out of focus spreads a point into a uniform disc, whose transfer
|
||||
function is a jinc — `2·J1(x)/x` — which crosses zero and goes negative.
|
||||
Defocus therefore reverses contrast at particular spatial frequencies and
|
||||
can leave *more* energy in some high bands than a Gaussian of the same
|
||||
nominal width. A Gaussian MTF is strictly positive and monotonically
|
||||
decreasing and does neither.
|
||||
|
||||
That matters here beyond realism: defocus is how a face ends up **large and
|
||||
useless**. A focus pull, a shallow depth of field, an actor stepping off the
|
||||
focal plane — all leave a big, confidently-detected face carrying no usable
|
||||
detail, and all sail straight through a size gate. Gaussian blur was the one
|
||||
family that mostly co-occurs with small faces, which is precisely why
|
||||
sharpness looked redundant against AR-002 on the first grid.
|
||||
|
||||
The disc is supersampled 8x before downsampling so its edge is
|
||||
anti-aliased; a hard-edged binary disc at small radii is a poor circle and
|
||||
its spectrum carries the staircase, not the optics.
|
||||
"""
|
||||
ss = 8
|
||||
n = int(np.ceil(radius)) * 2 + 1
|
||||
hi = np.zeros((n * ss, n * ss), np.float32)
|
||||
c = (n * ss - 1) / 2.0
|
||||
y, x = np.ogrid[:n * ss, :n * ss]
|
||||
hi[((x - c) ** 2 + (y - c) ** 2) <= (radius * ss) ** 2] = 1.0
|
||||
k = hi.reshape(n, ss, n, ss).mean(axis=(1, 3))
|
||||
s = k.sum()
|
||||
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
|
||||
|
||||
|
||||
def motion_kernel(length: int, angle_deg: float) -> np.ndarray:
|
||||
"""Linear motion blur — a camera pan or a moving subject.
|
||||
|
||||
Directional by construction: it destroys detail along one axis and leaves
|
||||
the perpendicular axis untouched. That is the property that separates the
|
||||
AR-029 candidates, since a measure normalising by total energy divides out
|
||||
the loss and reads a heavy smear as mild (see tests/test_quality.cpp).
|
||||
"""
|
||||
k = np.zeros((length, length), np.float32)
|
||||
k[length // 2, :] = 1.0
|
||||
m = cv2.getRotationMatrix2D(((length - 1) / 2.0, (length - 1) / 2.0),
|
||||
angle_deg, 1.0)
|
||||
k = cv2.warpAffine(k, m, (length, length))
|
||||
s = k.sum()
|
||||
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
|
||||
|
||||
|
||||
def degrade(crop: np.ndarray, size: int, level: float, kind: str,
|
||||
down: int, up: int, angle: float = 0.0) -> np.ndarray:
|
||||
"""Resolution loss, then blur of the requested family.
|
||||
|
||||
Order matters and this one is deliberate. Sampling happens in the source
|
||||
frame, so the downscale/upscale pair models a face that was `size` px when
|
||||
detected. The blur is then applied in the canonical frame, so `level` means
|
||||
the same number of canonical pixels in every cell of the grid — which is what
|
||||
lets the two axes be read independently. Blurring first would make the
|
||||
effective width depend on the cell's size, and the grid would no longer be
|
||||
factorial.
|
||||
|
||||
`level` is the family's natural parameter: Gaussian sigma, disc radius, or
|
||||
motion length in canonical px. They are NOT equivalent at equal numbers —
|
||||
matching families by parameter would compare different amounts of damage, so
|
||||
the analysis matches them on measured effect instead.
|
||||
"""
|
||||
out = crop
|
||||
if size != 112:
|
||||
small = cv2.resize(out, (size, size), interpolation=down)
|
||||
out = cv2.resize(small, (112, 112), interpolation=up)
|
||||
if level > 0:
|
||||
if kind == "gaussian":
|
||||
out = cv2.GaussianBlur(out, (0, 0), level, level)
|
||||
elif kind == "disc":
|
||||
out = cv2.filter2D(out, -1, disc_kernel(level))
|
||||
elif kind == "motion":
|
||||
out = cv2.filter2D(out, -1, motion_kernel(int(round(level)), angle))
|
||||
else:
|
||||
raise ValueError(f"unknown blur kind: {kind}")
|
||||
return out
|
||||
|
||||
|
||||
def score_sharpness(crop: np.ndarray) -> dict:
|
||||
"""All five candidates, from the shipped C++ (quality.hpp)."""
|
||||
s = sae_embed.assess_sharpness(np.ascontiguousarray(crop))
|
||||
d = {m: float(getattr(s, m)) for m in MEASURES}
|
||||
d["ok"] = bool(s.ok)
|
||||
return d
|
||||
|
||||
|
||||
# ── Analysis ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def auc(scores: np.ndarray, positive: np.ndarray) -> float:
|
||||
"""Area under the ROC for `scores` predicting `positive`, by the rank
|
||||
(Mann-Whitney U) identity. 0.5 is chance; 1.0 is a measure that orders every
|
||||
correctly-identified probe above every failure.
|
||||
|
||||
This is the ranking criterion for AR-029. A measure earns the job by
|
||||
predicting *the decision the pipeline makes*, not by having a tidy response
|
||||
to synthetic blur — a candidate can be beautifully monotone in sigma and
|
||||
still be a poor guide to whether this particular face will be recognised.
|
||||
"""
|
||||
pos = scores[positive]
|
||||
neg = scores[~positive]
|
||||
if pos.size == 0 or neg.size == 0:
|
||||
return float("nan")
|
||||
order = np.argsort(np.concatenate([pos, neg]), kind="mergesort")
|
||||
ranks = np.empty(order.size, dtype=np.float64)
|
||||
ranks[order] = np.arange(1, order.size + 1)
|
||||
# Average ranks over ties, or a measure with many equal values is scored
|
||||
# arbitrarily by input order.
|
||||
vals = np.concatenate([pos, neg])
|
||||
sv = vals[order]
|
||||
i = 0
|
||||
while i < sv.size:
|
||||
j = i
|
||||
while j + 1 < sv.size and sv[j + 1] == sv[i]:
|
||||
j += 1
|
||||
if j > i:
|
||||
ranks[order[i:j + 1]] = ranks[order[i:j + 1]].mean()
|
||||
i = j + 1
|
||||
r_pos = ranks[:pos.size].sum()
|
||||
return float((r_pos - pos.size * (pos.size + 1) / 2) / (pos.size * neg.size))
|
||||
|
||||
|
||||
def knee_from_measure(records: list[dict], measure: str, retention: float,
|
||||
n_bins: int = 20) -> dict:
|
||||
"""Where on `measure`'s own scale does identification start to fall apart?
|
||||
|
||||
Bins the probe-cell records by measure value and reports the TPI rate in
|
||||
each. The threshold is the lowest bin edge whose bin and every bin above it
|
||||
retain `retention` of the undegraded control's TPI rate — a stated rule, so
|
||||
changing the answer means changing the rule rather than picking a number.
|
||||
"""
|
||||
vals = np.array([r[measure] for r in records], dtype=np.float64)
|
||||
tpi = np.array([r["outcome"] == "TPI" for r in records])
|
||||
control = np.array([r["size_px"] == 112 and r["sigma"] == 0.0 for r in records])
|
||||
if control.sum() == 0:
|
||||
return {}
|
||||
floor = retention * float(tpi[control].mean())
|
||||
|
||||
# Quantile edges: the measures have wildly different scales and heavy tails,
|
||||
# so equal-width bins would put almost everything in one bucket.
|
||||
edges = np.unique(np.quantile(vals, np.linspace(0, 1, n_bins + 1)))
|
||||
if edges.size < 3:
|
||||
return {}
|
||||
idx = np.clip(np.digitize(vals, edges[1:-1]), 0, edges.size - 2)
|
||||
|
||||
bins = []
|
||||
for b in range(edges.size - 1):
|
||||
m = idx == b
|
||||
if m.sum() == 0:
|
||||
continue
|
||||
bins.append({"lo": float(edges[b]), "hi": float(edges[b + 1]),
|
||||
"n": int(m.sum()), "tpi_rate": float(tpi[m].mean()),
|
||||
"fpi_rate": float(np.mean([r["outcome"] == "FPI"
|
||||
for r, k in zip(records, m) if k]))})
|
||||
# Walk down from the top; the threshold is where retention first breaks.
|
||||
thr = None
|
||||
for b in reversed(bins):
|
||||
if b["tpi_rate"] < floor:
|
||||
thr = b["hi"]
|
||||
break
|
||||
return {"measure": measure, "control_tpi": float(tpi[control].mean()),
|
||||
"tpi_floor": floor, "threshold": thr, "bins": bins}
|
||||
|
||||
|
||||
# ── Plot ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
def write_plots(cells: list[dict], records: list[dict], ranking: list[dict],
|
||||
out_png: Path, meta: dict) -> None:
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
plt.rcParams.update({
|
||||
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE,
|
||||
"savefig.facecolor": SURFACE, "text.color": INK,
|
||||
"axes.edgecolor": MUTED, "axes.labelcolor": INK,
|
||||
"xtick.color": MUTED, "ytick.color": MUTED,
|
||||
"axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.8,
|
||||
"axes.spines.top": False, "axes.spines.right": False,
|
||||
})
|
||||
|
||||
sizes = sorted({c["size_px"] for c in cells})
|
||||
sigmas = sorted({c["sigma"] for c in cells})
|
||||
fig, axes = plt.subplots(1, 3, figsize=(17, 5.4))
|
||||
|
||||
# (a) the joint grid as TPI heat map
|
||||
grid = np.full((len(sigmas), len(sizes)), np.nan)
|
||||
for c in cells:
|
||||
grid[sigmas.index(c["sigma"]), sizes.index(c["size_px"])] = 100 * c["tpi_rate"]
|
||||
im = axes[0].imshow(grid, origin="lower", aspect="auto", cmap="viridis",
|
||||
vmin=0, vmax=100)
|
||||
axes[0].set_xticks(range(len(sizes)), [str(s) for s in sizes])
|
||||
axes[0].set_yticks(range(len(sigmas)), [f"{s:g}" for s in sigmas])
|
||||
axes[0].set_xlabel("probe size before upscaling (px)")
|
||||
axes[0].set_ylabel("Gaussian sigma (canonical px)")
|
||||
axes[0].set_title("TPI % over the joint grid", fontsize=11, loc="left")
|
||||
axes[0].grid(False)
|
||||
fig.colorbar(im, ax=axes[0], fraction=0.046)
|
||||
|
||||
# (b) TPI against the winning measure — the curve a discount is built from
|
||||
best = ranking[0]["measure"]
|
||||
vals = np.array([r[best] for r in records])
|
||||
tpi = np.array([r["outcome"] == "TPI" for r in records])
|
||||
edges = np.unique(np.quantile(vals, np.linspace(0, 1, 21)))
|
||||
centres, rates = [], []
|
||||
for i in range(edges.size - 1):
|
||||
m = (vals >= edges[i]) & (vals <= edges[i + 1])
|
||||
if m.sum() > 20:
|
||||
centres.append(0.5 * (edges[i] + edges[i + 1]))
|
||||
rates.append(100 * tpi[m].mean())
|
||||
axes[1].plot(centres, rates, "-o", color=GREEN, lw=2)
|
||||
axes[1].set_xscale("log")
|
||||
axes[1].set_xlabel(f"{best} (log scale)")
|
||||
axes[1].set_ylabel("TPI %")
|
||||
axes[1].set_title(f"identification vs the measure\nbest predictor: {best} "
|
||||
f"(AUC {ranking[0]['auc']:.3f})", fontsize=11, loc="left")
|
||||
|
||||
# (c) how well each candidate predicts the decision
|
||||
names = [r["measure"] for r in ranking]
|
||||
aucs = [r["auc"] for r in ranking]
|
||||
axes[2].barh(range(len(names)), aucs, color=BLUE)
|
||||
axes[2].axvline(0.5, color=RED, lw=1.4, ls="--")
|
||||
axes[2].set_yticks(range(len(names)), names, fontsize=9)
|
||||
axes[2].set_xlim(0.4, 1.0)
|
||||
axes[2].set_xlabel("AUC — predicts correct identification")
|
||||
axes[2].set_title("AR-029 candidate ranking", fontsize=11, loc="left")
|
||||
axes[2].invert_yaxis()
|
||||
|
||||
fig.suptitle(f"VR-012 — quality knee, {meta['model']}, {meta['n_actors']} actors, "
|
||||
f"{meta['n_probes']} probes x {len(cells)} cells",
|
||||
fontsize=12, x=0.01, ha="left")
|
||||
fig.tight_layout(rect=(0, 0.02, 1, 0.97))
|
||||
out_png.parent.mkdir(parents=True, exist_ok=True)
|
||||
fig.savefig(out_png, dpi=150)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
# ── Main ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(
|
||||
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--images", default=str(REPO / "images"))
|
||||
p.add_argument("--gallery", default=str(REPO / "gallery_lvface.h5"))
|
||||
p.add_argument("--out", default=str(REPO / "experiments/results/vr012_quality_knee"))
|
||||
p.add_argument("--actors", type=int, default=0,
|
||||
help="cap the actor pool (0 = every eligible actor, the default: "
|
||||
"FPI is gallery-size dependent and the whole gallery is the "
|
||||
"least understated estimate available)")
|
||||
p.add_argument("--min-images", type=int, default=3,
|
||||
help="minimum mugshots to be eligible (default 3, so holding one "
|
||||
"out still leaves two references)")
|
||||
p.add_argument("--seed", type=int, default=0)
|
||||
p.add_argument("--sizes", default="16,24,32,48,64,112",
|
||||
help="probe sizes before upscaling; 112 is undegraded")
|
||||
p.add_argument("--sigmas", default="0,0.5,1,1.5,2,3",
|
||||
help="blur level in canonical px; 0 is unblurred. Meaning "
|
||||
"depends on --blur-kind: Gaussian sigma, disc radius, "
|
||||
"or motion length")
|
||||
p.add_argument("--blur-kind", default="gaussian",
|
||||
choices=["gaussian", "disc", "motion"],
|
||||
help="blur family. gaussian is a soft-focus stand-in; disc "
|
||||
"is the circle-of-confusion PSF of real optical "
|
||||
"defocus (non-Gaussian, jinc MTF with zero crossings); "
|
||||
"motion is a linear smear. The last two are how a face "
|
||||
"ends up large and useless, which a size gate cannot "
|
||||
"catch")
|
||||
p.add_argument("--motion-angle", type=float, default=0.0,
|
||||
help="motion blur direction in degrees (--blur-kind motion)")
|
||||
p.add_argument("--keep-duplicates", action="store_true")
|
||||
|
||||
p.add_argument("--models-dir", default=str(REPO / "models"))
|
||||
p.add_argument("--arcface", default=None)
|
||||
p.add_argument("--detector", default=None)
|
||||
p.add_argument("--conf", type=float, default=0.5)
|
||||
p.add_argument("--nms", type=float, default=0.4)
|
||||
p.add_argument("--max-side", type=int, default=500)
|
||||
# Required by a TRT-backend build, ignored by an ORT one. A TensorRT fp16
|
||||
# run is a different realisation of the embedder — VR-005 measured ~0.85
|
||||
# cosine agreement with the fp32 ONNX path on LVFace-B, with separation
|
||||
# essentially intact — so a knee located here belongs to the fp16 space.
|
||||
# The study stays internally consistent because gallery and probes are both
|
||||
# embedded in this one session.
|
||||
p.add_argument("--detector-engine", default="",
|
||||
help="pre-built SCRFD .engine (TRT builds only)")
|
||||
p.add_argument("--arcface-engine", default="",
|
||||
help="pre-built ArcFace .engine (TRT builds only)")
|
||||
|
||||
p.add_argument("--prob-threshold", type=float, default=0.754)
|
||||
p.add_argument("--match-prior", type=float, default=0.5)
|
||||
p.add_argument("--tpi-retention", type=float, default=0.95)
|
||||
p.add_argument("--down-interp", default="area", choices=sorted(INTERP))
|
||||
p.add_argument("--up-interp", default="linear", choices=sorted(INTERP))
|
||||
args = p.parse_args()
|
||||
|
||||
models_dir = Path(args.models_dir)
|
||||
arcface = Path(args.arcface) if args.arcface else models_dir / "LVFace-B_Glint360K.onnx"
|
||||
detector = Path(args.detector) if args.detector else models_dir / "scrfd_500m_bnkps.onnx"
|
||||
for path, what in ((arcface, "embedder"), (detector, "detector")):
|
||||
if not path.is_file():
|
||||
return err(f"{what} model not found: {path}")
|
||||
|
||||
images_root = Path(args.images)
|
||||
if not images_root.is_dir():
|
||||
return err(f"image cache not found: {images_root}")
|
||||
|
||||
sizes = sorted({int(s) for s in args.sizes.split(",") if s.strip()})
|
||||
sigmas = sorted({float(s) for s in args.sigmas.split(",") if s.strip()})
|
||||
cv2.setRNGSeed(args.seed)
|
||||
|
||||
# ── actor pool ────────────────────────────────────────────────────────────
|
||||
pool = discover_actors(images_root)
|
||||
print(f"[select] {len(pool)} actor dirs under {images_root}", file=sys.stderr)
|
||||
if args.gallery and Path(args.gallery).is_file():
|
||||
ids, names = gallery_keys(Path(args.gallery))
|
||||
pool = [a for a in pool
|
||||
if (a["jellyfin_id"] and a["jellyfin_id"] in ids)
|
||||
or normalise_name(a["name"]) in names]
|
||||
print(f"[select] {len(pool)} are in {args.gallery}", file=sys.stderr)
|
||||
|
||||
eligible = [a for a in pool if len(a["images"]) >= args.min_images]
|
||||
print(f"[select] {len(eligible)} have >= {args.min_images} mugshots",
|
||||
file=sys.stderr)
|
||||
if len(eligible) < 2:
|
||||
return err(f"need at least 2 eligible actors; found {len(eligible)}")
|
||||
|
||||
rng = random.Random(args.seed)
|
||||
selected = (sorted(rng.sample(eligible, min(args.actors, len(eligible))),
|
||||
key=lambda a: a["dir"].name)
|
||||
if args.actors else eligible)
|
||||
|
||||
# ── detect + align every mugshot once ─────────────────────────────────────
|
||||
stages = Stages(str(detector), str(arcface), args.conf, args.nms,
|
||||
args.detector_engine, args.arcface_engine)
|
||||
print(f"[models] detector={detector.name} embedder={arcface.name} "
|
||||
f"batch={stages.engine.max_batch}", file=sys.stderr)
|
||||
|
||||
t0 = time.time()
|
||||
crops, rows, actors = [], [], []
|
||||
n_nodetect = 0
|
||||
for a in selected:
|
||||
actor_crops, actor_paths = [], []
|
||||
for img_path in a["images"]:
|
||||
img = cv2.imread(str(img_path))
|
||||
if img is None:
|
||||
n_nodetect += 1
|
||||
continue
|
||||
if args.max_side > 0 and max(img.shape[:2]) > args.max_side:
|
||||
s = args.max_side / max(img.shape[:2])
|
||||
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
|
||||
faces = stages.detect(img)
|
||||
if not faces:
|
||||
enhanced = stages.enhance(img)
|
||||
faces = stages.detect(enhanced)
|
||||
if faces:
|
||||
img = enhanced
|
||||
if not faces:
|
||||
n_nodetect += 1
|
||||
continue
|
||||
best = max(faces, key=lambda f: f.confidence)
|
||||
crop = stages.align(img, best.landmarks)
|
||||
if crop is None:
|
||||
n_nodetect += 1
|
||||
continue
|
||||
actor_crops.append(crop)
|
||||
actor_paths.append(img_path)
|
||||
if len(actor_crops) < 2:
|
||||
continue
|
||||
ai = len(actors)
|
||||
actors.append({"name": a["name"], "jellyfin_id": a["jellyfin_id"],
|
||||
"dir": a["dir"].name, "n_images": len(actor_crops)})
|
||||
for crop, img_path in zip(actor_crops, actor_paths):
|
||||
rows.append({"actor_idx": ai, "image": str(img_path)})
|
||||
crops.append(crop)
|
||||
if len(actors) % 200 == 0:
|
||||
print(f" [align] {len(actors)}/{len(selected)} actors, "
|
||||
f"{len(crops)} crops", file=sys.stderr)
|
||||
|
||||
if len(actors) < 2:
|
||||
return err(f"only {len(actors)} actors survived detection/alignment")
|
||||
print(f"[align] {len(actors)} actors, {len(crops)} crops, {n_nodetect} skipped "
|
||||
f"in {time.time() - t0:.1f}s", file=sys.stderr)
|
||||
|
||||
actor_of = np.array([r["actor_idx"] for r in rows], dtype=int)
|
||||
|
||||
t0 = time.time()
|
||||
native = stages.embed(crops)
|
||||
print(f"[embed] {len(crops)} native crops in {time.time() - t0:.1f}s",
|
||||
file=sys.stderr)
|
||||
|
||||
# ── drop duplicate mugshots ───────────────────────────────────────────────
|
||||
if not args.keep_duplicates:
|
||||
keep = np.ones(len(rows), bool)
|
||||
for ai in range(len(actors)):
|
||||
kept: list[int] = []
|
||||
for i in np.nonzero(actor_of == ai)[0]:
|
||||
if any(float(native[i] @ native[k]) > DEDUP_SIM for k in kept):
|
||||
keep[i] = False
|
||||
else:
|
||||
kept.append(int(i))
|
||||
n_dup = int((~keep).sum())
|
||||
counts = np.bincount(actor_of[keep], minlength=len(actors))
|
||||
drop_actor = counts < 2
|
||||
keep &= ~drop_actor[actor_of]
|
||||
remap = np.full(len(actors), -1, dtype=int)
|
||||
remap[~drop_actor] = np.arange(int((~drop_actor).sum()))
|
||||
actors = [a for a, d in zip(actors, drop_actor) if not d]
|
||||
rows = [r for r, k in zip(rows, keep) if k]
|
||||
crops = [c for c, k in zip(crops, keep) if k]
|
||||
native = native[keep]
|
||||
actor_of = remap[actor_of[keep]]
|
||||
print(f"[dedup] dropped {n_dup} duplicates and {int(drop_actor.sum())} "
|
||||
f"actors; {len(actors)} actors, {len(rows)} images remain",
|
||||
file=sys.stderr)
|
||||
|
||||
# ── hold out one probe per actor ──────────────────────────────────────────
|
||||
is_probe = np.zeros(len(rows), bool)
|
||||
for ai in range(len(actors)):
|
||||
idx = np.nonzero(actor_of == ai)[0]
|
||||
r = random.Random(f"{args.seed}:{actors[ai]['dir']}")
|
||||
is_probe[r.choice(list(idx))] = True
|
||||
probe_rows = np.nonzero(is_probe)[0]
|
||||
gal_rows = np.nonzero(~is_probe)[0]
|
||||
print(f"[holdout] {len(probe_rows)} probes, {len(gal_rows)} gallery embeddings",
|
||||
file=sys.stderr)
|
||||
|
||||
gal_emb = native[gal_rows]
|
||||
gal_actor = actor_of[gal_rows]
|
||||
probe_actor = actor_of[probe_rows]
|
||||
actor_cols = [np.nonzero(gal_actor == ai)[0] for ai in range(len(actors))]
|
||||
if not all(len(c) for c in actor_cols):
|
||||
return err("an actor has no gallery references left; raise --min-images")
|
||||
|
||||
cal = calibrate_gallery(gal_emb, gal_actor)
|
||||
if not cal["valid"]:
|
||||
return err("calibration could not be fitted; this study will not fall back "
|
||||
"to a raw cosine threshold (CLAUDE.md invariant)")
|
||||
log_prior_odds = float(np.log(args.match_prior / (1.0 - args.match_prior)))
|
||||
|
||||
# ── the grid ──────────────────────────────────────────────────────────────
|
||||
down, up = INTERP[args.down_interp], INTERP[args.up_interp]
|
||||
probe_crops = [crops[i] for i in probe_rows]
|
||||
cells, records = [], []
|
||||
n = len(probe_rows)
|
||||
|
||||
for size in sizes:
|
||||
for sigma in sigmas:
|
||||
t0 = time.time()
|
||||
degraded = [degrade(c, size, sigma, args.blur_kind, down, up,
|
||||
args.motion_angle) for c in probe_crops]
|
||||
sharp = [score_sharpness(d) for d in degraded]
|
||||
q = stages.embed(degraded)
|
||||
|
||||
sims = q @ gal_emb.T
|
||||
best_per_actor = np.stack([sims[:, c].max(axis=1) for c in actor_cols],
|
||||
axis=1)
|
||||
best_actor = best_per_actor.argmax(axis=1)
|
||||
best_sim = best_per_actor.max(axis=1)
|
||||
p_match = np.asarray(probability(best_sim, cal["a"], cal["b"],
|
||||
log_prior_odds))
|
||||
accept = p_match > args.prob_threshold
|
||||
correct = best_actor == probe_actor
|
||||
tpi = int(np.sum(accept & correct))
|
||||
fpi = int(np.sum(accept & ~correct))
|
||||
unid = int(np.sum(~accept))
|
||||
|
||||
cell = {"size_px": size, "sigma": sigma, "blur_kind": args.blur_kind,
|
||||
"n_probes": n,
|
||||
"tpi": tpi, "fpi": fpi, "unidentified": unid,
|
||||
"tpi_rate": tpi / n, "fpi_rate": fpi / n,
|
||||
"unidentified_rate": unid / n,
|
||||
"rank1_rate": float(np.mean(correct)),
|
||||
"mean_p_match": float(np.mean(p_match))}
|
||||
for m in MEASURES:
|
||||
cell[f"mean_{m}"] = float(np.mean([s[m] for s in sharp]))
|
||||
cells.append(cell)
|
||||
|
||||
for j in range(n):
|
||||
rec = {"size_px": size, "sigma": sigma,
|
||||
"blur_kind": args.blur_kind,
|
||||
"probe_image": rows[probe_rows[j]]["image"],
|
||||
"p_match": float(p_match[j]),
|
||||
"outcome": ("TPI" if accept[j] and correct[j]
|
||||
else "FPI" if accept[j] else "unidentified")}
|
||||
rec.update({m: sharp[j][m] for m in MEASURES})
|
||||
records.append(rec)
|
||||
|
||||
print(f"[grid] {size:3d}px {args.blur_kind[:4]} {sigma:<4g} TPI {100*tpi/n:5.1f}% "
|
||||
f"FPI {100*fpi/n:5.1f}% unid {100*unid/n:5.1f}% "
|
||||
f"rank1 {100*np.mean(correct):5.1f}% [{time.time()-t0:.1f}s]",
|
||||
file=sys.stderr)
|
||||
|
||||
# ── rank the candidates, then locate the knee on the winner ───────────────
|
||||
is_tpi = np.array([r["outcome"] == "TPI" for r in records])
|
||||
ranking = sorted(
|
||||
({"measure": m,
|
||||
"auc": auc(np.array([r[m] for r in records], dtype=np.float64), is_tpi)}
|
||||
for m in MEASURES),
|
||||
key=lambda d: -d["auc"])
|
||||
knees = [knee_from_measure(records, r["measure"], args.tpi_retention)
|
||||
for r in ranking]
|
||||
|
||||
# ── outputs ───────────────────────────────────────────────────────────────
|
||||
out = Path(args.out)
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
csv_path = out.with_name(out.name + ".csv")
|
||||
with open(csv_path, "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=list(cells[0].keys()))
|
||||
w.writeheader()
|
||||
w.writerows(cells)
|
||||
rec_path = out.with_name(out.name + ".records.csv")
|
||||
with open(rec_path, "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=list(records[0].keys()))
|
||||
w.writeheader()
|
||||
w.writerows(records)
|
||||
|
||||
meta = {
|
||||
"requirement": "VR-012",
|
||||
"model": arcface.stem, "detector": detector.stem,
|
||||
"n_actors": len(actors), "n_probes": len(probe_rows),
|
||||
"n_gallery_embeddings": len(gal_rows),
|
||||
"min_images": args.min_images, "seed": args.seed,
|
||||
"sizes": sizes, "sigmas": sigmas, "blur_kind": args.blur_kind,
|
||||
"motion_angle": args.motion_angle,
|
||||
"prob_threshold": args.prob_threshold, "match_prior": args.match_prior,
|
||||
"calibration": cal,
|
||||
"measure_ranking": ranking,
|
||||
"knees": knees,
|
||||
"sharpness_window": list(sae_embed.sharpness_window()),
|
||||
"caveat": (f"FPI grows with gallery size; this ran against {len(actors)} "
|
||||
f"actors and still understates a production library."),
|
||||
"grid": cells,
|
||||
}
|
||||
json_path = out.with_name(out.name + ".json")
|
||||
json_path.write_text(json.dumps(meta, indent=2) + "\n")
|
||||
png_path = out.with_name(out.name + ".png")
|
||||
write_plots(cells, records, ranking, png_path, meta)
|
||||
|
||||
# ── stdout report ─────────────────────────────────────────────────────────
|
||||
print(f"\nVR-012 — quality knee, {arcface.stem}")
|
||||
print(f"{len(actors)} actors, {len(probe_rows)} probes x {len(cells)} cells\n")
|
||||
print(f"{'size':>5} {'sigma':>6} {'TPI':>8} {'FPI':>8} {'unid':>8} {'rank1':>8}")
|
||||
for c in cells:
|
||||
print(f"{c['size_px']:>5} {c['sigma']:>6g} {100*c['tpi_rate']:>7.1f}% "
|
||||
f"{100*c['fpi_rate']:>7.1f}% {100*c['unidentified_rate']:>7.1f}% "
|
||||
f"{100*c['rank1_rate']:>7.1f}%")
|
||||
print("\nAR-029 candidate ranking — AUC for predicting correct identification:")
|
||||
for r in ranking:
|
||||
print(f" {r['measure']:>20} {r['auc']:.4f}")
|
||||
print(f"\n[out] {csv_path}\n[out] {rec_path}\n[out] {json_path}\n[out] {png_path}")
|
||||
print(f"\n{meta['caveat']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -23,12 +23,23 @@
|
||||
enum class OrtProvider { CPU, CUDA, ROCm, TensorRT };
|
||||
|
||||
inline OrtProvider detect_ort_provider() {
|
||||
// ORT returns these in its own preference order (TensorRT, CUDA, ..., CPU
|
||||
// last), so the first recognised entry is the best available and the loop
|
||||
// returns on it.
|
||||
auto available = Ort::GetAvailableProviders();
|
||||
for (const auto& p : available) {
|
||||
// Only the TensorRT *EP* is a build-time opt-in — it needs the headers
|
||||
// and the profile plumbing below. CUDA is not: it is a plain ORT
|
||||
// provider, and gating its detection on the TRT flag (as this did) made
|
||||
// the CUDA branch unreachable in every build that did not also ask for
|
||||
// TensorRT. The symptom is silent rather than loud — inference simply
|
||||
// runs on the CPU and everything still returns correct answers — which
|
||||
// is why it survived: a 300-actor VR-012 grid cell took 76 s on the CPU
|
||||
// with the GPU idle at 212 MiB.
|
||||
#ifdef SAE_ORT_WITH_TRT_EP
|
||||
if (p == "TensorrtExecutionProvider") return OrtProvider::TensorRT;
|
||||
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
|
||||
#endif
|
||||
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
|
||||
if (p == "ROCMExecutionProvider") return OrtProvider::ROCm;
|
||||
}
|
||||
return OrtProvider::CPU;
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
#include "face_embedder_engine.hpp"
|
||||
#include "gallery/gallery_calibration.hpp"
|
||||
#include "gallery/gallery_store.hpp"
|
||||
#include "quality.hpp"
|
||||
|
||||
#include <nanobind/nanobind.h>
|
||||
#include <nanobind/ndarray.h>
|
||||
@@ -176,6 +177,55 @@ NB_MODULE(sae_embed, m) {
|
||||
}, "image"_a,
|
||||
"Border-replicate pad by 50% and CLAHE, for a detector second try.");
|
||||
|
||||
// ── Quality (AR-028 … AR-030) ────────────────────────────────────────────
|
||||
// Exposed for the same reason the calibration is: VR-012 has to select
|
||||
// among the AR-029 candidates, and the measure it selects must be the one
|
||||
// that ships. A numpy copy scored during the study would leave the shipped
|
||||
// measure unmeasured, which is precisely the failure the whole quality axis
|
||||
// exists to prevent.
|
||||
nb::class_<SharpnessScores>(m, "SharpnessScores")
|
||||
.def_ro("var_laplacian", &SharpnessScores::var_laplacian)
|
||||
.def_ro("norm_var_laplacian", &SharpnessScores::norm_var_laplacian)
|
||||
.def_ro("tenengrad", &SharpnessScores::tenengrad)
|
||||
.def_ro("hf_energy_ratio", &SharpnessScores::hf_energy_ratio)
|
||||
.def_ro("dir_min_tenengrad", &SharpnessScores::dir_min_tenengrad)
|
||||
.def_ro("ok", &SharpnessScores::ok)
|
||||
.def("__repr__", [](const SharpnessScores& s) {
|
||||
return "<SharpnessScores varlap=" + std::to_string(s.var_laplacian) +
|
||||
" normvarlap=" + std::to_string(s.norm_var_laplacian) +
|
||||
" tenengrad=" + std::to_string(s.tenengrad) +
|
||||
" hf=" + std::to_string(s.hf_energy_ratio) +
|
||||
(s.ok ? ">" : " NOT-OK>");
|
||||
});
|
||||
|
||||
m.def("assess_sharpness", [](ImageArray crop) {
|
||||
return ::assess_sharpness(as_mat(crop));
|
||||
}, "crop"_a,
|
||||
"All four AR-029 sharpness candidates for a 112x112 aligned crop "
|
||||
"(quality.hpp). Higher is sharper for every measure; scales are not "
|
||||
"comparable between measures. Scored over a fixed 64x64 window on the "
|
||||
"face interior, so background bokeh and hairstyle do not enter.");
|
||||
|
||||
m.def("sharpness_window", [] {
|
||||
const cv::Rect w = ::sharpness_window();
|
||||
return std::vector<int>{w.x, w.y, w.width, w.height};
|
||||
},
|
||||
"The (x, y, w, h) canonical-pixel window every sharpness measure is "
|
||||
"taken over, so a study can show the pixels a score came from.");
|
||||
|
||||
m.def("alignment_residual", [](nb::ndarray<const float, nb::shape<5, 2>,
|
||||
nb::c_contig, nb::device::cpu> landmarks)
|
||||
-> std::optional<float> {
|
||||
const Alignment a = ::estimate_alignment(as_landmarks(landmarks));
|
||||
if (!a.ok) return std::nullopt; // degenerate landmarks
|
||||
return a.residual;
|
||||
}, "landmarks"_a,
|
||||
"The AR-030 visibility measure: RMS landmark error in canonical "
|
||||
"112x112 px left over after the best similarity fit onto the ArcFace "
|
||||
"template (face_utils.hpp). None when the landmarks are degenerate. "
|
||||
"Exposed so VR-012 can check whether blur leaks into the pose axis — "
|
||||
"if it does, discounting on both would double-count one cause.");
|
||||
|
||||
// ── Calibration ──────────────────────────────────────────────────────────
|
||||
// AR-024: the pipeline reasons in one probability space. Exposed so Python
|
||||
// scores through the same sigmoid the C++ matcher uses, rather than a numpy
|
||||
|
||||
+256
@@ -0,0 +1,256 @@
|
||||
#pragma once
|
||||
/// TRACES: AR-028, AR-029 | SR-002
|
||||
///
|
||||
/// Sharpness of the aligned crop — candidate measures for AR-029.
|
||||
///
|
||||
/// Motion blur and soft focus destroy the high-frequency detail the embedder
|
||||
/// keys on, and unlike face size they leave the bounding box looking perfectly
|
||||
/// healthy. An embedder handed such a face does not fail: it returns a
|
||||
/// confident, plausible, wrong vector that then competes on equal terms with
|
||||
/// every good one in the gallery.
|
||||
///
|
||||
/// **Why four measures and not one.** AR-029's threshold has to be *located*,
|
||||
/// the way VR-005 located the size floor, not chosen. Locating it means letting
|
||||
/// a study rank candidates by how well each predicts real identity loss, so all
|
||||
/// four ship and VR-012 picks the winner. Until that study reports, none of
|
||||
/// these is "the" sharpness measure.
|
||||
///
|
||||
/// **They are computed on the 112×112 aligned crop**, never the raw box. The
|
||||
/// crop is geometrically scale-normalised, so a measure taken there cannot
|
||||
/// re-express face size the way a raw-pixel one would.
|
||||
///
|
||||
/// That normalisation is geometric, not informational, and the distinction
|
||||
/// matters: a 40 px face upscaled into the canonical frame genuinely carries
|
||||
/// less high-frequency detail than a 400 px one downscaled into it, so every
|
||||
/// measure here *does* respond to source face size. It reads **effective
|
||||
/// resolution in canonical space**, which is the union of "was small" and "was
|
||||
/// blurred", not blur alone. Whether that makes a sharpness discount a
|
||||
/// double-count against AR-002's size gate is VR-012's joint size×sigma grid to
|
||||
/// settle: if identity loss is a function of the measure alone, one axis
|
||||
/// suffices; if a small-but-sharp and a large-but-blurred face at equal measure
|
||||
/// lose different amounts, the axes are genuinely separate. The unit test
|
||||
/// `sharpness falls under downscale-upscale as well as under blur` pins this as
|
||||
/// known behaviour rather than leaving it to be discovered as a surprise.
|
||||
|
||||
#include <opencv2/core.hpp>
|
||||
#include <opencv2/imgproc.hpp>
|
||||
|
||||
#include <cmath>
|
||||
|
||||
// ── The measurement window ────────────────────────────────────────────────────
|
||||
// All four measures see the same pixels, so a comparison between them is about
|
||||
// the operator and not about the window each happened to pick.
|
||||
//
|
||||
// A 64×64 region centred on the face interior, not the whole crop. Under the
|
||||
// ArcFace template the landmarks span x ∈ [38.3, 73.5], y ∈ [51.5, 92.4]; this
|
||||
// window covers that plus the surrounding cheeks, brow and chin while excluding
|
||||
// the corners.
|
||||
//
|
||||
// The corners are excluded because they are where the background lives, and
|
||||
// studio headshots — the gallery's entire population — are very often shot at a
|
||||
// wide aperture with a deliberately blurred background. Measured over the full
|
||||
// crop, that bokeh drags the score down on exactly the sharpest, most
|
||||
// cooperative images in the set, which would put the measure's response
|
||||
// backwards on the population used to calibrate it. Hair is excluded for the
|
||||
// weaker version of the same reason: its high-frequency content varies with
|
||||
// hairstyle rather than with capture quality.
|
||||
//
|
||||
// 64 is also a power of two, so the DFT below gets its natural size.
|
||||
inline constexpr int kSharpWindow = 64;
|
||||
inline constexpr int kSharpWindowX = 24; // (24,36) … (88,100) in canonical px
|
||||
inline constexpr int kSharpWindowY = 36;
|
||||
|
||||
/// Every candidate, computed in one pass over the window.
|
||||
///
|
||||
/// Higher is sharper for all four, so a discount curve has the same orientation
|
||||
/// whichever one VR-012 selects. Scales are *not* comparable between measures —
|
||||
/// only within one.
|
||||
struct SharpnessScores {
|
||||
/// Variance of the Laplacian. The textbook measure, included as the
|
||||
/// baseline every other candidate has to beat. Second derivatives amplify
|
||||
/// sensor noise, and the value scales with image contrast, so a
|
||||
/// low-contrast sharp face reads as blurred. Expected to lose; it should
|
||||
/// lose on the record rather than by assertion.
|
||||
float var_laplacian{0.f};
|
||||
|
||||
/// Variance of the Laplacian over the variance of the intensity. Divides
|
||||
/// out the first-order contrast dependence that var_laplacian carries,
|
||||
/// which is the single confound most likely to matter on a gallery drawn
|
||||
/// from thousands of different cameras, lighting setups and JPEG pipelines.
|
||||
float norm_var_laplacian{0.f};
|
||||
|
||||
/// Tenengrad: mean squared Sobel gradient magnitude. A first derivative, so
|
||||
/// markedly less noise-amplifying than the Laplacian, at the cost of
|
||||
/// responding to coarser structure. Still contrast-dependent.
|
||||
float tenengrad{0.f};
|
||||
|
||||
/// Fraction of spectral energy above a quarter of Nyquist, DC excluded.
|
||||
/// A ratio, so contrast divides out by construction rather than by an
|
||||
/// explicit correction, and it is the most direct statement of "how much
|
||||
/// fine detail is actually present". Bounded in [0,1], which makes it the
|
||||
/// easiest of the four to turn into a discount.
|
||||
float hf_energy_ratio{0.f};
|
||||
|
||||
/// The worse of the two Sobel axes, normalised by a low-frequency contrast
|
||||
/// estimate. The only candidate here that satisfies both requirements at
|
||||
/// once, and it exists because the other four do not.
|
||||
///
|
||||
/// Two independent fixes, each answering a measured failure of the four
|
||||
/// above (numbers from the T1 ladders in tests/test_quality.cpp):
|
||||
///
|
||||
/// - **Normalise by low frequencies, not by total energy.** Dividing by
|
||||
/// the whole intensity variance puts the detail being measured into the
|
||||
/// denominator as well as the numerator, so a blur shrinks both and the
|
||||
/// quotient barely moves. A Gaussian at sigma 4 canonical px keeps
|
||||
/// illumination and coarse facial structure and discards detail, giving
|
||||
/// a contrast estimate that blur leaves alone.
|
||||
/// - **Take the minimum over direction, not the sum.** Motion blur is
|
||||
/// directional: a horizontal smear destroys horizontal detail and
|
||||
/// leaves vertical detail untouched. Summing the two axes (as
|
||||
/// Tenengrad does) lets the surviving axis mask the destroyed one — the
|
||||
/// reason both ratio measures are U-shaped in blur length, scoring a
|
||||
/// 21 px smear about as sharp as a 3 px one. The minimum tracks the
|
||||
/// axis that was ruined, which is the one the embedder suffers from.
|
||||
///
|
||||
/// Falls 510 → 12 monotonically across that same motion-blur ladder, stays
|
||||
/// monotone under Gaussian blur and resampling, and moves 0.4% when
|
||||
/// contrast is halved.
|
||||
float dir_min_tenengrad{0.f};
|
||||
|
||||
/// False when the crop was the wrong size or degenerate (flat). Scored,
|
||||
/// never silently dropped: a face whose sharpness cannot be computed is a
|
||||
/// fact the dump should record, not an absence.
|
||||
bool ok{false};
|
||||
};
|
||||
|
||||
/// The window every measure is taken over. Exposed so a study can show the
|
||||
/// pixels a score was computed from rather than trusting the constants.
|
||||
inline cv::Rect sharpness_window() {
|
||||
return {kSharpWindowX, kSharpWindowY, kSharpWindow, kSharpWindow};
|
||||
}
|
||||
|
||||
namespace detail {
|
||||
|
||||
/// Fraction of spectral energy above `cutoff` × Nyquist, DC bin excluded.
|
||||
///
|
||||
/// A Hann window is applied first. Without it the DFT sees the region's edges
|
||||
/// as a step discontinuity, and that step is broadband: it deposits energy at
|
||||
/// every frequency including the high band being measured, so a uniformly
|
||||
/// blurry crop still scores a substantial high-frequency fraction and the
|
||||
/// measure's dynamic range collapses.
|
||||
///
|
||||
/// **The mean is removed before the window, not after.** Windowing a signal
|
||||
/// that still carries its DC offset multiplies that constant by the Hann taper,
|
||||
/// and the taper's own spectrum is not a single bin — the offset smears across
|
||||
/// the low-frequency neighbourhood, where dropping bin (0,0) no longer removes
|
||||
/// it. The leaked energy lands in the denominator without scaling with image
|
||||
/// contrast, so the "ratio" silently becomes a function of absolute brightness:
|
||||
/// on the synthetic crop, a 20/255 brightening moved it 23% and halving the
|
||||
/// contrast moved it by a factor of 3.6. Subtracting the mean first restores
|
||||
/// the invariance the ratio form is supposed to provide for free.
|
||||
inline float hf_ratio(const cv::Mat& gray32, float cutoff = 0.25f) {
|
||||
static const cv::Mat hann = [] {
|
||||
cv::Mat w(kSharpWindow, kSharpWindow, CV_32F);
|
||||
for (int y = 0; y < kSharpWindow; ++y) {
|
||||
const float wy = 0.5f * (1.f - std::cos(2.f * float(CV_PI) * y / (kSharpWindow - 1)));
|
||||
for (int x = 0; x < kSharpWindow; ++x) {
|
||||
const float wx = 0.5f * (1.f - std::cos(2.f * float(CV_PI) * x / (kSharpWindow - 1)));
|
||||
w.at<float>(y, x) = wx * wy;
|
||||
}
|
||||
}
|
||||
return w;
|
||||
}();
|
||||
|
||||
cv::Mat centred;
|
||||
cv::subtract(gray32, cv::mean(gray32), centred);
|
||||
|
||||
cv::Mat windowed;
|
||||
cv::multiply(centred, hann, windowed);
|
||||
|
||||
cv::Mat spectrum;
|
||||
cv::dft(windowed, spectrum, cv::DFT_COMPLEX_OUTPUT);
|
||||
|
||||
// Quadrants are wrapped: frequency index n maps to the signed frequency
|
||||
// n - N for n > N/2, so the radius has to be computed on the wrapped index.
|
||||
const int N = kSharpWindow;
|
||||
const float nyquist = N / 2.f;
|
||||
const float r_cut = cutoff * nyquist;
|
||||
|
||||
double total = 0.0, high = 0.0;
|
||||
for (int y = 0; y < N; ++y) {
|
||||
const float fy = (y <= N / 2) ? float(y) : float(y - N);
|
||||
for (int x = 0; x < N; ++x) {
|
||||
if (x == 0 && y == 0) continue; // DC carries no detail
|
||||
const float fx = (x <= N / 2) ? float(x) : float(x - N);
|
||||
const auto& c = spectrum.at<cv::Vec2f>(y, x);
|
||||
const double e = double(c[0]) * c[0] + double(c[1]) * c[1];
|
||||
total += e;
|
||||
if (std::sqrt(fx * fx + fy * fy) > r_cut) high += e;
|
||||
}
|
||||
}
|
||||
if (total < 1e-12) return 0.f; // flat region
|
||||
return static_cast<float>(high / total);
|
||||
}
|
||||
|
||||
} // namespace detail
|
||||
|
||||
/// Score a 112×112 aligned BGR (or single-channel) crop on all four candidates.
|
||||
///
|
||||
/// Costs one colour conversion and three small convolutions over a 64×64 window
|
||||
/// — negligible beside the embedder inference it guards.
|
||||
inline SharpnessScores assess_sharpness(const cv::Mat& crop) {
|
||||
SharpnessScores s;
|
||||
const cv::Rect win = sharpness_window();
|
||||
if (crop.empty() ||
|
||||
win.x + win.width > crop.cols || win.y + win.height > crop.rows)
|
||||
return s;
|
||||
|
||||
cv::Mat gray;
|
||||
if (crop.channels() == 3) cv::cvtColor(crop(win), gray, cv::COLOR_BGR2GRAY);
|
||||
else gray = crop(win).clone();
|
||||
|
||||
// Scale into [0,1] so a score does not depend on the 8-bit convention, and
|
||||
// so the two contrast-normalised measures are comparable across builds.
|
||||
cv::Mat g32;
|
||||
gray.convertTo(g32, CV_32F, 1.0 / 255.0);
|
||||
|
||||
cv::Scalar mu, sigma;
|
||||
cv::meanStdDev(g32, mu, sigma);
|
||||
const double var_img = sigma[0] * sigma[0];
|
||||
|
||||
cv::Mat lap;
|
||||
cv::Laplacian(g32, lap, CV_32F, 3);
|
||||
cv::Scalar lmu, lsigma;
|
||||
cv::meanStdDev(lap, lmu, lsigma);
|
||||
const double var_lap = lsigma[0] * lsigma[0];
|
||||
|
||||
cv::Mat gx, gy;
|
||||
cv::Sobel(g32, gx, CV_32F, 1, 0, 3);
|
||||
cv::Sobel(g32, gy, CV_32F, 0, 1, 3);
|
||||
cv::Mat gx2, gy2;
|
||||
cv::multiply(gx, gx, gx2);
|
||||
cv::multiply(gy, gy, gy2);
|
||||
cv::Mat mag2 = gx2 + gy2;
|
||||
|
||||
// Contrast from low frequencies only — see dir_min_tenengrad. Blur leaves
|
||||
// this denominator alone, which is exactly what the other two normalised
|
||||
// measures lack.
|
||||
cv::Mat lf;
|
||||
cv::GaussianBlur(g32, lf, cv::Size(0, 0), 4.0);
|
||||
cv::Scalar lfmu, lfsigma;
|
||||
cv::meanStdDev(lf, lfmu, lfsigma);
|
||||
const double var_lf = lfsigma[0] * lfsigma[0];
|
||||
|
||||
s.var_laplacian = static_cast<float>(var_lap);
|
||||
// A flat window has no contrast to normalise by. Reporting 0 (rather than a
|
||||
// huge quotient) keeps "less sharp" pointing the same way for a degenerate
|
||||
// input as for a blurred one.
|
||||
s.norm_var_laplacian = var_img > 1e-9 ? static_cast<float>(var_lap / var_img) : 0.f;
|
||||
s.tenengrad = static_cast<float>(cv::mean(mag2)[0]);
|
||||
s.hf_energy_ratio = detail::hf_ratio(g32);
|
||||
s.dir_min_tenengrad = var_lf > 1e-9
|
||||
? static_cast<float>(std::min(cv::mean(gx2)[0], cv::mean(gy2)[0]) / var_lf)
|
||||
: 0.f;
|
||||
s.ok = var_img > 1e-9;
|
||||
return s;
|
||||
}
|
||||
@@ -19,6 +19,7 @@ add_executable(sae_tests
|
||||
test_calibration.cpp
|
||||
test_gallery_store.cpp
|
||||
test_face_utils.cpp
|
||||
test_quality.cpp
|
||||
test_track_gallery.cpp
|
||||
test_face_tracker.cpp
|
||||
test_track_registry.cpp
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
// TRACES: AR-029 | SR-002
|
||||
//
|
||||
// T1 for the AR-029 sharpness candidates: the properties that have to hold
|
||||
// before a study is allowed to pick between them. GPU-free, model-free.
|
||||
//
|
||||
// The register's acceptance criterion is "synthetic blur ladder ->
|
||||
// monotonically falling sharpness; Gaussian vs motion blur; small sharp face vs
|
||||
// large soft one — size must not leak into this axis". The last clause needs
|
||||
// care, and the tests below split it in two:
|
||||
//
|
||||
// - What must NOT leak is *geometric* scale. The measure is taken in the
|
||||
// canonical frame, so changing how big the face was in the source while
|
||||
// preserving its detail must not move the score. That is structural: the
|
||||
// window is fixed at 64x64 canonical px.
|
||||
// - What DOES legitimately move the score is lost *detail*. A face that was
|
||||
// 40 px before being warped up to 112 really does carry less
|
||||
// high-frequency content than one that was 400 px, and a measure blind to
|
||||
// that would be blind to the thing it exists to catch.
|
||||
//
|
||||
// So "size must not leak" cannot mean "invariant to the source face size", and
|
||||
// the ladder test below asserts the opposite on purpose. What it buys is that
|
||||
// the overlap with AR-002 is a recorded property with a test naming it, rather
|
||||
// than a surprise VR-012 discovers when the two axes turn out to be correlated.
|
||||
#include <catch2/catch_test_macros.hpp>
|
||||
#include <catch2/matchers/catch_matchers_floating_point.hpp>
|
||||
|
||||
#include "quality.hpp"
|
||||
#include "types.hpp" // kArcFaceRef, for the window-placement test
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
using Catch::Matchers::WithinAbs;
|
||||
using Catch::Matchers::WithinRel;
|
||||
|
||||
namespace {
|
||||
|
||||
// A deterministic 112x112 stand-in for a face crop.
|
||||
//
|
||||
// **Broadband, not a sum of a few sinusoids.** An earlier version of this
|
||||
// fixture used three discrete spatial frequencies, and the resampling ladder
|
||||
// below was non-monotone for hf_energy_ratio because of it: a period-7
|
||||
// component downsampled to 32 px lands exactly at Nyquist and aliases, so the
|
||||
// ratio rose at one rung instead of falling. That is a property of a
|
||||
// three-tone test pattern meeting a resampler, not of the measure or of any
|
||||
// face — a real crop has energy spread across the band, where such a
|
||||
// resonance averages out. Deterministic value noise, smoothed to give the
|
||||
// roughly 1/f falloff of a photograph, exercises the whole band at once.
|
||||
//
|
||||
// Mid-grey base with bounded amplitude, so scaling the contrast in the tests
|
||||
// below does not clip.
|
||||
cv::Mat synthetic_crop() {
|
||||
// Fixed LCG rather than cv::randu: the suite must not depend on OpenCV's
|
||||
// RNG state, which other tests share.
|
||||
uint32_t seed = 0x5eed1234u;
|
||||
auto next = [&seed] {
|
||||
seed = seed * 1664525u + 1013904223u;
|
||||
return (seed >> 16) & 0xffffu;
|
||||
};
|
||||
|
||||
cv::Mat noise(112, 112, CV_32F);
|
||||
for (int y = 0; y < 112; ++y)
|
||||
for (int x = 0; x < 112; ++x)
|
||||
noise.at<float>(y, x) = float(next()) / 65535.f - 0.5f;
|
||||
|
||||
// Mild smoothing: white noise is flat to Nyquist, which no lens produces
|
||||
// and which would make the sharpest rung of every ladder unrealistic.
|
||||
cv::Mat smooth;
|
||||
cv::GaussianBlur(noise, smooth, cv::Size(0, 0), 0.8);
|
||||
cv::normalize(smooth, smooth, -1.0, 1.0, cv::NORM_MINMAX);
|
||||
|
||||
cv::Mat img(112, 112, CV_8UC3);
|
||||
for (int y = 0; y < 112; ++y) {
|
||||
for (int x = 0; x < 112; ++x) {
|
||||
double v = 128.0 + 70.0 * smooth.at<float>(y, x);
|
||||
const auto b = static_cast<uchar>(std::clamp(v, 0.0, 255.0));
|
||||
img.at<cv::Vec3b>(y, x) = {b, b, b};
|
||||
}
|
||||
}
|
||||
return img;
|
||||
}
|
||||
|
||||
cv::Mat gaussian(const cv::Mat& src, double sigma) {
|
||||
cv::Mat out;
|
||||
cv::GaussianBlur(src, out, cv::Size(0, 0), sigma, sigma);
|
||||
return out;
|
||||
}
|
||||
|
||||
// Horizontal box blur — the camera-pan case, and the one an isotropic measure
|
||||
// could in principle miss.
|
||||
cv::Mat motion(const cv::Mat& src, int len) {
|
||||
cv::Mat kernel = cv::Mat::zeros(1, len, CV_32F);
|
||||
kernel.setTo(1.0f / len);
|
||||
cv::Mat out;
|
||||
cv::filter2D(src, out, -1, kernel);
|
||||
return out;
|
||||
}
|
||||
|
||||
// Throw away detail a face detected at size x size never had, then warp back up
|
||||
// to the 112x112 the embedder is fed — the VR-005 degradation.
|
||||
cv::Mat rescale(const cv::Mat& src, int size) {
|
||||
if (size == 112) return src.clone();
|
||||
cv::Mat small, out;
|
||||
cv::resize(src, small, {size, size}, 0, 0, cv::INTER_AREA);
|
||||
cv::resize(small, out, {112, 112}, 0, 0, cv::INTER_LINEAR);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<float> field(const std::vector<SharpnessScores>& s,
|
||||
float SharpnessScores::* m) {
|
||||
std::vector<float> v;
|
||||
v.reserve(s.size());
|
||||
for (const auto& x : s) v.push_back(x.*m);
|
||||
return v;
|
||||
}
|
||||
|
||||
void check_strictly_falling(const std::vector<float>& v, const char* what) {
|
||||
INFO(what);
|
||||
for (size_t i = 1; i < v.size(); ++i) {
|
||||
INFO("step " << i << ": " << v[i - 1] << " -> " << v[i]);
|
||||
CHECK(v[i] < v[i - 1]);
|
||||
}
|
||||
}
|
||||
|
||||
const std::vector<std::pair<const char*, float SharpnessScores::*>> kMeasures{
|
||||
{"var_laplacian", &SharpnessScores::var_laplacian},
|
||||
{"norm_var_laplacian", &SharpnessScores::norm_var_laplacian},
|
||||
{"tenengrad", &SharpnessScores::tenengrad},
|
||||
{"hf_energy_ratio", &SharpnessScores::hf_energy_ratio},
|
||||
{"dir_min_tenengrad", &SharpnessScores::dir_min_tenengrad},
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
TEST_CASE("every candidate falls monotonically along a Gaussian blur ladder",
|
||||
"[quality][AR-029]") {
|
||||
const cv::Mat base = synthetic_crop();
|
||||
std::vector<SharpnessScores> ladder;
|
||||
for (double sigma : {0.0, 0.5, 1.0, 1.5, 2.0, 3.0})
|
||||
ladder.push_back(assess_sharpness(sigma == 0.0 ? base : gaussian(base, sigma)));
|
||||
|
||||
for (const auto& [name, m] : kMeasures) {
|
||||
REQUIRE(ladder.front().ok);
|
||||
check_strictly_falling(field(ladder, m), name);
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("only the absolute and directional measures survive motion blur",
|
||||
"[quality][AR-029]") {
|
||||
// Motion blur is the commonest way a film frame is unusable, and it is
|
||||
// where the candidates separate. A horizontal smear destroys horizontal
|
||||
// detail and leaves vertical detail untouched, so what a measure does here
|
||||
// depends on whether it can be fooled by the surviving axis.
|
||||
const cv::Mat base = synthetic_crop();
|
||||
std::vector<SharpnessScores> ladder{assess_sharpness(base)};
|
||||
for (int len : {3, 5, 9, 15, 21})
|
||||
ladder.push_back(assess_sharpness(motion(base, len)));
|
||||
|
||||
// Total gradient/Laplacian energy keeps falling: nothing replaces what the
|
||||
// smear removed.
|
||||
check_strictly_falling(field(ladder, &SharpnessScores::var_laplacian),
|
||||
"var_laplacian");
|
||||
check_strictly_falling(field(ladder, &SharpnessScores::tenengrad),
|
||||
"tenengrad");
|
||||
// The fix for the two below: low-frequency denominator, and the worse of
|
||||
// the two axes rather than their sum.
|
||||
check_strictly_falling(field(ladder, &SharpnessScores::dir_min_tenengrad),
|
||||
"dir_min_tenengrad");
|
||||
|
||||
// The disqualifying behaviour, pinned rather than hidden. Both measures
|
||||
// normalise by a quantity that contains the detail they are measuring, so
|
||||
// once the horizontal band is gone the quotient climbs back toward its
|
||||
// unblurred value: each is U-shaped in blur length, and a single score
|
||||
// maps to two very different amounts of blur. A 21 px smear scores about
|
||||
// as sharp as a 3 px one.
|
||||
for (const auto& [name, m] : {
|
||||
std::pair{"norm_var_laplacian", &SharpnessScores::norm_var_laplacian},
|
||||
std::pair{"hf_energy_ratio", &SharpnessScores::hf_energy_ratio}}) {
|
||||
const std::vector<float> v = field(ladder, m);
|
||||
INFO(name);
|
||||
const auto trough = std::min_element(v.begin(), v.end());
|
||||
CHECK(trough != v.begin()); // it does fall at first …
|
||||
CHECK(trough != v.end() - 1); // … then turns back up
|
||||
CHECK(v.back() > 0.8f * v[1]); // recovering most of one rung
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("sharpness falls under downscale-upscale as well as under blur",
|
||||
"[quality][AR-029]") {
|
||||
// The overlap with AR-002, asserted rather than assumed. Losing resolution
|
||||
// and losing focus are the same loss of high-frequency content, so every
|
||||
// candidate reads a small upscaled face as less sharp. VR-012's joint
|
||||
// size x sigma grid decides whether that makes a sharpness discount a
|
||||
// double-count against the size gate, or whether the two axes carry
|
||||
// separable information.
|
||||
const cv::Mat base = synthetic_crop();
|
||||
std::vector<SharpnessScores> ladder;
|
||||
for (int size : {112, 64, 48, 32, 24, 16})
|
||||
ladder.push_back(assess_sharpness(rescale(base, size)));
|
||||
|
||||
for (const auto& [name, m] : kMeasures)
|
||||
check_strictly_falling(field(ladder, m), name);
|
||||
}
|
||||
|
||||
TEST_CASE("the ratio measures are contrast-free and the raw ones are not",
|
||||
"[quality][AR-029]") {
|
||||
// The confound that decides the bake-off. A gallery drawn from thousands of
|
||||
// cameras, lighting setups and JPEG pipelines varies enormously in
|
||||
// contrast, and a measure that reads a low-contrast sharp face as blurred
|
||||
// would discount it for the photographer's choices rather than for anything
|
||||
// the embedder cares about.
|
||||
const cv::Mat base = synthetic_crop();
|
||||
|
||||
// Halve the contrast about mid-grey, leaving spatial structure untouched.
|
||||
cv::Mat low;
|
||||
base.convertTo(low, CV_8UC3, 0.5, 64.0);
|
||||
|
||||
const auto s_hi = assess_sharpness(base);
|
||||
const auto s_lo = assess_sharpness(low);
|
||||
REQUIRE(s_hi.ok);
|
||||
REQUIRE(s_lo.ok);
|
||||
|
||||
// Invariant by construction: both are ratios in which the contrast factor
|
||||
// cancels.
|
||||
CHECK_THAT(s_lo.norm_var_laplacian,
|
||||
WithinRel(s_hi.norm_var_laplacian, 0.02f));
|
||||
CHECK_THAT(s_lo.hf_energy_ratio, WithinRel(s_hi.hf_energy_ratio, 0.02f));
|
||||
|
||||
// Not invariant: both scale with the square of the contrast factor, so
|
||||
// halving the contrast quarters them. This is the disqualifying behaviour,
|
||||
// pinned so that a change making them contrast-free is a deliberate one.
|
||||
CHECK_THAT(s_lo.var_laplacian, WithinRel(0.25f * s_hi.var_laplacian, 0.05f));
|
||||
CHECK_THAT(s_lo.tenengrad, WithinRel(0.25f * s_hi.tenengrad, 0.05f));
|
||||
}
|
||||
|
||||
TEST_CASE("brightness alone moves nothing", "[quality][AR-029]") {
|
||||
const cv::Mat base = synthetic_crop();
|
||||
cv::Mat bright;
|
||||
base.convertTo(bright, CV_8UC3, 1.0, 20.0);
|
||||
|
||||
const auto a = assess_sharpness(base);
|
||||
const auto b = assess_sharpness(bright);
|
||||
for (const auto& [name, m] : kMeasures) {
|
||||
INFO(name);
|
||||
CHECK_THAT(b.*m, WithinRel(a.*m, 0.02f));
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("a flat crop is scored not-ok rather than given a number",
|
||||
"[quality][AR-029]") {
|
||||
// A face whose sharpness cannot be computed is a fact to record, not an
|
||||
// absence — the same rule AR-030 follows for degenerate landmarks.
|
||||
const cv::Mat flat(112, 112, CV_8UC3, cv::Scalar(128, 128, 128));
|
||||
const auto s = assess_sharpness(flat);
|
||||
CHECK_FALSE(s.ok);
|
||||
for (const auto& [name, m] : kMeasures) {
|
||||
INFO(name);
|
||||
CHECK_THAT(s.*m, WithinAbs(0.0f, 1e-6f));
|
||||
CHECK_FALSE(std::isnan(s.*m));
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("a crop smaller than the measurement window is scored not-ok",
|
||||
"[quality][AR-029]") {
|
||||
const cv::Mat small(64, 64, CV_8UC3, cv::Scalar(40, 90, 160));
|
||||
CHECK_FALSE(assess_sharpness(small).ok);
|
||||
CHECK_FALSE(assess_sharpness(cv::Mat()).ok);
|
||||
}
|
||||
|
||||
TEST_CASE("the measurement window covers the face interior of the crop",
|
||||
"[quality][AR-029]") {
|
||||
// The landmarks the ArcFace template pins must all fall inside the window,
|
||||
// or the measure is scoring background and hair rather than the face.
|
||||
const cv::Rect w = sharpness_window();
|
||||
CHECK(w.x >= 0);
|
||||
CHECK(w.y >= 0);
|
||||
CHECK(w.x + w.width <= 112);
|
||||
CHECK(w.y + w.height <= 112);
|
||||
for (int i = 0; i < 5; ++i) {
|
||||
INFO("landmark " << i);
|
||||
CHECK(w.contains(cv::Point(static_cast<int>(kArcFaceRef[i][0]),
|
||||
static_cast<int>(kArcFaceRef[i][1]))));
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("a single-channel crop scores the same as its BGR equivalent",
|
||||
"[quality][AR-029]") {
|
||||
// The dump replays crops; nothing should depend on whether they arrived as
|
||||
// three identical channels or one.
|
||||
const cv::Mat base = synthetic_crop();
|
||||
cv::Mat gray;
|
||||
cv::cvtColor(base, gray, cv::COLOR_BGR2GRAY);
|
||||
|
||||
const auto a = assess_sharpness(base);
|
||||
const auto b = assess_sharpness(gray);
|
||||
for (const auto& [name, m] : kMeasures) {
|
||||
INFO(name);
|
||||
CHECK_THAT(b.*m, WithinRel(a.*m, 1e-3f));
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user