scene-actor-extraction/docs/quality-knee.md

Quality knee: what does a blurred or small face cost, and can a measure predict it?

VR-012. Companion to the minimum-face-size studies VR-005 and VR-013 (see the requirement register), which located the size floor at 40 px; this asks the same question for sharpness, and asks whether any cheap measure taken on the aligned crop can be acted on at inference.

Run by scripts/validation/quality_knee.py through the sae_embed bindings — detection, the ArcFace warp, the embedder, the five candidate measures and the Platt calibration are all the shipped C++.

Protocol

1670 gallery actors with 3 or more mugshots (of 2456 total), one image held out per actor as a probe, the remaining 10326 embeddings staying in the gallery at native resolution. Only the probe degrades — reference mugshots are clean and the face coming out of the video is not.

Each probe passes through a joint grid: downscale to S×S and back to 112 (the sampling loss), then blur at level L in canonical pixels. Three blur families, 36 cells each, 60120 probe-cell records per family:

family	models	parameter
Gaussian	soft focus, a generic stand-in	sigma 0 … 3
Disc	real optical defocus — the circle of confusion	radius 0 … 6
Motion	camera pan or moving subject	length 0 … 21 px

The three are not interchangeable, and sweeping only the first was the original design error — one that would have produced a wrong answer, not merely an incomplete one (Result 3). A defocused lens spreads a point into a uniform disc, whose transfer function is a jinc — 2·J1(x)/x — that crosses zero and goes negative, annihilating whole frequency bands and returning the ones beyond each zero phase-reversed. A Gaussian MTF is strictly positive and monotone and does neither. More practically: defocus and motion are how a face ends up large and useless, while Gaussian blur as swept here mostly co-occurs with small faces. That difference decides whether sharpness carries anything the size filter does not.

Families are compared at matched per-axis PSF standard deviation (σ for a Gaussian, R/2 for a disc, L/√12 for a linear smear), never at equal raw parameter, which would compare different amounts of damage.

Identification is the pipeline's own decision: per-actor best-of-N cosine → Platt sigmoid → accept above prob_threshold 0.754. Never a raw cosine (AR-024).

Result 1 — sharpness is not a sufficient statistic

Sorting the 36 Gaussian cells by hf_energy_ratio, the six sigma-3 cells land at effectively identical measured sharpness:

size	sigma	hf_energy_ratio	TPI
16	3	0.0003	15.3%
24	3	0.0003	63.2%
32	3	0.0003	79.4%
48	3	0.0003	86.6%
64	3	0.0004	88.4%
112	3	0.0005	91.0%

Same measured sharpness, a 76-point spread in identification. It inverts too: 16 px unblurred measures 0.0033 and scores 23.5%, while 48 px at sigma 2 measures lower at 0.0021 and scores 96.6%.

A canonical-frame sharpness scalar cannot separate attenuated high frequencies from destroyed spatial sampling. Blur suppresses the high band while preserving mid-frequency facial geometry exactly; downsampling to 16 px destroys that geometry outright. Both look alike to any measure keyed on high-frequency energy.

This is the measured basis for AR-028's rule that the axes are kept separate and not collapsed into one scalar, and it settles the double-counting question: size and sharpness are not redundant, and neither substitutes for the other.

Result 2 — blur is a cliff, and it breaks confidence, not identity

TPI % by size (rows) against Gaussian sigma (columns):

size	0	0.5	1	1.5	2	3
16	23.5	24.0	25.0	24.6	23.9	15.3
24	85.7	85.1	85.6	85.9	82.6	63.2
32	95.9	95.9	96.0	95.5	93.7	79.4
48	98.7	98.7	98.4	98.1	96.6	86.6
64	98.6	98.6	98.8	98.4	97.5	88.4
112	98.9	98.9	98.8	98.6	98.0	91.0

Three regimes: sigma ≤ 1.5 is free (every cell moves under 1.5 points, sign flipping at random — at 16 px it slightly improves, smoothing upscale artifacts); sigma 2 costs 1–3 points; the 2→3 step costs 7–19. A smooth discount curve is therefore the wrong shape — the response is flat, then falls off a cliff.

The cost peaks at the size knee, not at full resolution. Sigma 3 costs −22.5 points at 24 px but only −7.9 at 112 px and −8.3 at 16 px. Blur has no intrinsic cost; it costs in proportion to how close the observation already sits to the decision boundary. At 112 px there is margin to spare, at 16 px the probe is already below threshold, and at 24 px it sits exactly on the knee.

What blur destroys is confidence, not ranking. Rank-1 barely moves: 99.3% → 99.2% at 112 px across the whole sigma range. The extreme case is 16 px at sigma 3, where rank-1 is 80.2% while TPI is 15.3% — 65 points of probes have the correct actor ranked first and are rejected anyway for falling under the probability threshold.

That is why FPI never left 0.1% in any of the 108 cells across all three families. Degradation produces TBI, never a wrong name. The calibration degrades gracefully, which is what SR-002 needs.

Result 3 — the blur family matters more than the blur amount

Comparing families by their raw parameter is meaningless — sigma, radius and length are different units. They are matched here by the per-axis standard deviation of the PSF, which puts them on one scale:

family	per-axis σ	level giving σ = 3 px
Gaussian σ	σ	3
Disc radius R	R/2	6
Motion length L	L/√12	10.4

For reference the ArcFace template places the eyes 35.2 canonical px apart, so σ = 3 px is 9% of the inter-ocular distance.

TPI at matched severity, interpolated within each family:

size	σ=3 Gaussian	σ=3 Motion	σ=3 Defocus	defocus penalty
16	15.3	15.1	11.0	+4.3
24	63.2	61.1	41.4	+21.9
32	79.4	76.1	50.4	+29.0
48	86.6	80.9	52.6	+34.0
64	88.4	81.7	51.0	+37.4
112	91.0	82.1	46.9	+44.0

Optical defocus is up to 44 points more destructive than a Gaussian of identical spread, and the ordering is defocus ≫ motion > Gaussian throughout. At σ=1 the three families are indistinguishable, and at σ=2 they differ by under 5 points; the divergence appears only when both the blur is severe and the face is large.

That pattern is physically consistent. At 16 px the resampling has already removed the high frequencies, so the PSF's shape has nothing left to act on and all three agree. At 112 px the full spectrum is present and shape decides: a Gaussian MTF rolls off gently and always leaves some energy at every frequency, so the embedder receives a merely attenuated signal, while a disc MTF is a jinc that hits exact zeros — whole frequency bands annihilated rather than attenuated, with the bands beyond each zero returning phase-reversed. Motion sits between them because it ruins one axis and leaves the perpendicular one untouched.

The methodological consequence is the important one. This study originally swept Gaussian blur alone and concluded blur was a minor effect. On the family that actually occurs in film, the same nominal severity costs 53% error instead of 9% at full resolution. A threshold set from the Gaussian arm would have been wrong by a factor of five in error rate, and the axis would probably have been dropped as not worth its cost.

Defocus is also the case a size gate cannot catch. Every one of those 112 px faces is large and confidently detected, and sails through AR-002 untouched. That, not the Gaussian result, is what justifies a sharpness axis existing at all.

Result 4 — variance of Laplacian is anti-predictive at fixed degradation

Pooled across all cells, every candidate scores AUC 0.76–0.80 for predicting correct identification, with textbook var_laplacian top. That number is close to worthless: it rewards a measure for detecting how degraded the crop is, which all five do. The question a per-observation discount needs is whether, at a fixed degradation, the measure predicts which faces fail:

measure	Gaussian	Defocus	Motion
hf_energy_ratio	0.530	0.521	0.557
norm_var_laplacian	0.520	0.507	0.539
dir_min_tenengrad	0.524	0.512	0.506
tenengrad	0.433	0.437	0.457
var_laplacian	0.423	0.422	0.473

Best is 0.557 — barely above chance, and hf_energy_ratio wins on all three families. var_laplacian is anti-predictive on all three too, so that finding does not depend on the blur model.

The two metrics measure different jobs, and the candidates split along that line. On the motion arm dir_min_tenengrad has the best pooled AUC by a wide margin — 0.854 against 0.792 for the next — exactly as its synthetic directional-blur ladder predicted, yet its within-cell AUC there is 0.506. It is an excellent detector of how badly smeared a crop is and no guide at all to which face will be recognised. Pooled AUC is the right metric for a gross-degradation flag; within-cell AUC is the right one for a per-observation discount; a measure can be strong at one and useless at the other.

Deciles within the 16 px Gaussian cell, where 1277 failures give the test real power:

var_laplacian decile	TPI
0.00071–0.00192 (blurriest)	37.1%
0.00242–0.00278	22.8%
0.00397–0.00447	25.7%
0.00625–0.01445 (sharpest)	14.4%

The faces the measure calls sharpest are 2.6x less identifiable than those it calls blurriest, monotone across ten bins of 167. Within a cell every crop received identical degradation, so the residual variance is native contrast, not native detail — and hard shadows, high-contrast lighting, sharpening halos and JPEG ringing all raise Laplacian variance while making a face harder to match. The measure reads photographic style and encoding artifacts and calls them sharpness.

hf_energy_ratio is the only candidate with a correctly-signed within-cell trend (16.2% → 35.3% across the same deciles), being a pure ratio in which the contrast factor cancels.

Consequence: a per-face quality discount keyed on variance of Laplacian — the most widely used blur metric in production vision pipelines — would systematically down-weight the more identifiable faces. It is worse than no discount.

Result 5 — as a compute gate, sharpness loses to the size filter

Skipping the embed for crops below a threshold, measured as compute saved against true identifications lost:

gate	skipped	true IDs lost	of skipped, doomed anyway
hf_energy_ratio < 0.00023	10.0%	7.6%	37.9%
hf_energy_ratio < 0.00051	20.0%	15.1%	38.7%
source size < 24 px	16.7%	4.7%	77.3%

At a comparable skip rate the size filter loses 4.7% against sharpness's 15.1% — three times less damage — and it is free, being a bbox dimension available before alignment or embedding, where sharpness needs the warped crop plus a colour convert, three convolutions and a 64×64 DFT.

Restricting to large faces (≥64 px) on the defocus arm, where the size filter is blind, improves the gate's precision 3.5x (37% of skipped crops doomed versus 10.7% on the Gaussian arm) but not its trade: skip 10%, lose 7.0%.

A hard ceiling explains why. At 112 px with defocus radius 6 — visually destroyed — 46.9% of faces still identify correctly and rank-1 is still 94.8%. Blur does not determine the outcome, so any gate keyed on apparent blur is predicting a coin flip. The size filter wins not because size is better measured, but because smallness destroys identity more completely than blur does: 16 px faces succeed only 23.5% of the time, so discarding them is cheap.

What this means for the requirements

Do not gate on sharpness; discount on it. Heavily defocused faces remain ~47% identifiable, so a gate destroys recoverable evidence. This is the first hard evidence that AR-028's "discounts the observation, never deletes the detection" is right on the merits rather than merely cautious. Since ranking survives where confidence does not, the per-track accumulation (AR-025) should recover much of what a single-frame threshold rejects — which is also the argument for the discount living in EvidenceDiscounter rather than in a filter.

var_laplacian and tenengrad are disqualified as discounts by Result 4, on all three blur families. They remain usable as coarse gross-degradation detectors, the role in which their pooled AUC is real — the same role the size filter plays — but they must never weight a per-observation belief.

hf_energy_ratio is the only surviving discount candidate, best on all three families, and its within-cell signal (0.52–0.56) is weak enough that shipping a discount on it needs justification beyond this study.

dir_min_tenengrad earns a different job. Its pooled 0.854 on the motion arm makes it the best available detector of gross directional smear — useful as a per-frame "this shot is unusable" flag, which is a decision about a frame, not a weighting of an observation. If AR-029 ships two measures for two roles, this is the second one, and it must not be confused with the first.

Model the blur family, not just its amount. Result 3 makes the choice of degradation model a first-order design decision rather than a detail: the same matched severity costs 9% or 53% error depending on the PSF. Any future study that sweeps blur must state which family it used and why.

Any discount curve must be flat then steep, not linear or sigmoid over the measure. Blur costs nothing until it costs a great deal.

Limitations

• Cooperative population. Gallery mugshots are frontal and well-lit; within-cell failures are likely dominated by cross-view mismatch, which no sharpness measure can predict. Read the ~chance within-cell AUCs as "sharpness does not predict the dominant failure mode here", not as "sharpness is meaningless".
• Uniform grid, not a natural distribution. Sizes and blur levels are sampled evenly, so "skip 16.7%" is exactly the 16 px row. The gate comparisons are like-for-like on identical records, but the absolute savings are not what a film would show.
• TensorRT fp16. A different realisation of the embedder from the fp32 ONNX reference — VR-005 measured ~0.85 cosine agreement with separation intact. Gallery and probes share one session so the study is internally consistent, but the absolute knee belongs to the fp16 space.
• Blur is applied in the canonical frame, after resampling, so its width is independent of the cell's size. Real optics blur before sampling.
• The top motion rung is an anchor, not an operating point. Length 21 is a per-axis σ of 6.1 — 17% of the inter-ocular distance, a streak rather than a face — and it is swept to bound the curve, not because a frame like that is worth reasoning about. Its 3.4% TPI at 112 px should not be quoted as a headline. The same caution applies less severely to defocus radius 6 (σ = 3).
• Per-axis σ equates spread, not perceptual damage. It is the fairest single scalar for comparing PSFs, but Result 3 is precisely the finding that equal spread does not mean equal harm, so the matched-severity tables compare like-for-like inputs, not like-for-like severity as a face would experience it.