Cut the eye box from a landmark contour, and refuse eyes that cannot be read
SCRFD's eye point places a face, not an eye: on turned and smiling heads the classifier's window had the eye in a corner, and two model-free ways of re-centring it — the darkest blob, the most contrasty window — both lost open eyes (19 → 15 and 19 → 9 of 25). Three landmark models were then run over the same faces; Face Mesh V2 and InsightFace's 2d106det tied at 22 of 25 and 2d106det ships, being the cheapest by far and under the grant the detector and embedder already carry. The eye box is the tight bounding box of its ten lid points, cut upright from the native render, which is what the classifier was trained on. The larger change is that the reading now carries, per eye, the source pixels across the box and the sharpness of the patch — because the commonest wrong answer on the reference library was a soft eye read as closed, and a classifier shown a smear will always say something. An eye under either floor, or narrower than six tenths of its partner (the far eye of a turned head, whose contour collapses), is not asked; a face with no readable eye is a fourth state, Unreadable, that no filter drops. On twenty native renders the one real blink is caught, the laughing faces are closed, the profiles are judged on the near eye, and the one thing left beyond any floor is a face with a pot held over it.
This commit is contained in:
@@ -0,0 +1,185 @@
|
||||
//! TRACES: FR-CULL-13
|
||||
//! Dense facial landmarks — InsightFace's `2d106det` (docs/faces.md §17.2).
|
||||
//!
|
||||
//! SCRFD's five points place a face; they do not place an eye. Its eye
|
||||
//! point is loose enough that a window centred on it left the eye in a
|
||||
//! corner on turned and smiling heads, and two model-free ways of
|
||||
//! re-centring it made things worse. So a second model draws the eye's lid
|
||||
//! contour, and the eye box is cut from that.
|
||||
//!
|
||||
//! **Why this one.** Three were measured on the same faces — MediaPipe Face
|
||||
//! Mesh V2, PIPNet and this — and tied on what the eye classifier made of
|
||||
//! their boxes (22 of 25 open eyes read open, against 19 from the SCRFD
|
||||
//! point). This is the cheapest of the three by a wide margin (5 MB, 106
|
||||
//! points, ~24 ms in tract), and it is under the grant the detector and
|
||||
//! embedder already carry rather than a new one to read.
|
||||
//!
|
||||
//! # Pre-processing
|
||||
//!
|
||||
//! Ported from InsightFace's `landmark.py`: a square crop centred on the
|
||||
//! detector box, 1.5× its longer edge, resized to 192; **RGB in 0..255**
|
||||
//! (the graph carries its own `bn_data` normalisation, so `input_mean` is
|
||||
//! 0 and `input_std` 1); 106 `(x, y)` in −1..1 mapped back through
|
||||
//! `(p + 1) · 96`. The graph's batch dimension is the literal `None` and
|
||||
//! is pinned to 1 by `tools/fix-face-model-shapes.sh`, like the embedder's.
|
||||
//!
|
||||
//! # The layout
|
||||
//!
|
||||
//! Checked by drawing the points on the reference faces rather than taken
|
||||
//! from a diagram: the subject's right eye (image-left) is points 33–42,
|
||||
//! the left 87–96, ten each round the lids.
|
||||
|
||||
use ndarray::Array4;
|
||||
|
||||
use crate::align::crop_box;
|
||||
use crate::{install_backend, FaceError, Pixels};
|
||||
|
||||
/// The graph's input edge, in pixels.
|
||||
pub const INPUT_EDGE: usize = 192;
|
||||
|
||||
/// How many points the model returns.
|
||||
pub const POINTS: usize = 106;
|
||||
|
||||
/// The crop's edge as a multiple of the detector box's longer edge.
|
||||
const CROP_SCALE: f32 = 1.5;
|
||||
|
||||
/// Point indices of the subject's right eye's lid contour (image-left).
|
||||
pub const RIGHT_EYE: [usize; 10] = [33, 34, 35, 36, 37, 38, 39, 40, 41, 42];
|
||||
/// Point indices of the subject's left eye's lid contour (image-right).
|
||||
pub const LEFT_EYE: [usize; 10] = [87, 88, 89, 90, 91, 92, 93, 94, 95, 96];
|
||||
|
||||
/// The 106 points of one face, in **source pixels**.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct Landmarks {
|
||||
pub points: [(f32, f32); POINTS],
|
||||
}
|
||||
|
||||
impl Landmarks {
|
||||
/// The lid contour of the subject's right eye.
|
||||
pub fn right_eye(&self) -> [(f32, f32); 10] {
|
||||
RIGHT_EYE.map(|i| self.points[i])
|
||||
}
|
||||
|
||||
/// The lid contour of the subject's left eye.
|
||||
pub fn left_eye(&self) -> [(f32, f32); 10] {
|
||||
LEFT_EYE.map(|i| self.points[i])
|
||||
}
|
||||
}
|
||||
|
||||
/// A loaded `2d106det` graph.
|
||||
pub struct Landmarker {
|
||||
session: ort::session::Session,
|
||||
}
|
||||
|
||||
impl Landmarker {
|
||||
pub fn from_path(path: impl AsRef<std::path::Path>) -> Result<Self, FaceError> {
|
||||
let bytes = std::fs::read(path).map_err(FaceError::ModelRead)?;
|
||||
Self::from_bytes(&bytes)
|
||||
}
|
||||
|
||||
pub fn from_bytes(bytes: &[u8]) -> Result<Self, FaceError> {
|
||||
install_backend();
|
||||
let session = ort::session::Session::builder()
|
||||
.map_err(FaceError::Inference)?
|
||||
.commit_from_memory(bytes)
|
||||
.map_err(FaceError::Inference)?;
|
||||
|
||||
let input = session.inputs().first().ok_or(FaceError::WrongModel {
|
||||
expected: "2d106det",
|
||||
detail: "model has no inputs".into(),
|
||||
})?;
|
||||
let shape: Option<Vec<i64>> = input.dtype().tensor_shape().map(|s| s.to_vec());
|
||||
let want = [1, 3, INPUT_EDGE as i64, INPUT_EDGE as i64];
|
||||
if shape.as_deref() != Some(&want[..]) {
|
||||
return Err(FaceError::WrongModel {
|
||||
expected: "2d106det",
|
||||
detail: format!(
|
||||
"input '{}' is {:?}, expected {:?} (batch pinned to 1)",
|
||||
input.name(),
|
||||
shape,
|
||||
want
|
||||
),
|
||||
});
|
||||
}
|
||||
let out = session.outputs().first().ok_or(FaceError::WrongModel {
|
||||
expected: "2d106det",
|
||||
detail: "model has no outputs".into(),
|
||||
})?;
|
||||
let last: Option<i64> = out.dtype().tensor_shape().and_then(|d| d.last().copied());
|
||||
if last != Some((POINTS * 2) as i64) {
|
||||
return Err(FaceError::WrongModel {
|
||||
expected: "2d106det",
|
||||
detail: format!(
|
||||
"output '{}' is {:?}-wide, expected {}",
|
||||
out.name(),
|
||||
last,
|
||||
POINTS * 2
|
||||
),
|
||||
});
|
||||
}
|
||||
Ok(Self { session })
|
||||
}
|
||||
|
||||
/// The landmarks of the face in `bbox` — `(x0, y0, x1, y1)` in source
|
||||
/// pixels, the detector's box — read from the source.
|
||||
///
|
||||
/// `None` for a box with no area or a buffer that is not the size it
|
||||
/// claims, as every crop here.
|
||||
pub fn landmarks(
|
||||
&mut self,
|
||||
px: Pixels<'_>,
|
||||
width: usize,
|
||||
height: usize,
|
||||
bbox: (f32, f32, f32, f32),
|
||||
) -> Result<Option<Landmarks>, FaceError> {
|
||||
let (w, h) = (bbox.2 - bbox.0, bbox.3 - bbox.1);
|
||||
let side = w.max(h) * CROP_SCALE;
|
||||
let (cx, cy) = ((bbox.0 + bbox.2) / 2.0, (bbox.1 + bbox.3) / 2.0);
|
||||
let (x0, y0) = (cx - side / 2.0, cy - side / 2.0);
|
||||
let Some(crop) = crop_box(
|
||||
px,
|
||||
width,
|
||||
height,
|
||||
(x0, y0, side, side),
|
||||
INPUT_EDGE,
|
||||
INPUT_EDGE,
|
||||
) else {
|
||||
return Ok(None);
|
||||
};
|
||||
|
||||
let e = INPUT_EDGE;
|
||||
let mut input = Array4::<f32>::zeros((1, 3, e, e));
|
||||
for y in 0..e {
|
||||
for x in 0..e {
|
||||
for c in 0..3 {
|
||||
input[[0, c, y, x]] = crop[(y * e + x) * 3 + c] * 255.0;
|
||||
}
|
||||
}
|
||||
}
|
||||
let outputs = self
|
||||
.session
|
||||
.run(ort::inputs![
|
||||
ort::value::Tensor::from_array(input).map_err(FaceError::Inference)?
|
||||
])
|
||||
.map_err(FaceError::Inference)?;
|
||||
let (_, data) = outputs[0]
|
||||
.try_extract_tensor::<f32>()
|
||||
.map_err(FaceError::Inference)?;
|
||||
if data.len() < POINTS * 2 {
|
||||
return Err(FaceError::WrongModel {
|
||||
expected: "2d106det",
|
||||
detail: format!("got {} values, expected {}", data.len(), POINTS * 2),
|
||||
});
|
||||
}
|
||||
|
||||
// −1..1 in the crop → crop pixels → source pixels.
|
||||
let scale = side / e as f32;
|
||||
let half = e as f32 / 2.0;
|
||||
let mut points = [(0.0_f32, 0.0_f32); POINTS];
|
||||
for (i, p) in points.iter_mut().enumerate() {
|
||||
let (u, v) = ((data[2 * i] + 1.0) * half, (data[2 * i + 1] + 1.0) * half);
|
||||
*p = (x0 + u * scale, y0 + v * scale);
|
||||
}
|
||||
Ok(Some(Landmarks { points }))
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user