dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Cut the eye box from a landmark contour, and refuse eyes that cannot be read

Browse Source 
SCRFD's eye point places a face, not an eye: on turned and smiling heads
the classifier's window had the eye in a corner, and two model-free ways
of re-centring it — the darkest blob, the most contrasty window — both
lost open eyes (19 → 15 and 19 → 9 of 25). Three landmark models were
then run over the same faces; Face Mesh V2 and InsightFace's 2d106det
tied at 22 of 25 and 2d106det ships, being the cheapest by far and under
the grant the detector and embedder already carry. The eye box is the
tight bounding box of its ten lid points, cut upright from the native
render, which is what the classifier was trained on.

The larger change is that the reading now carries, per eye, the source
pixels across the box and the sharpness of the patch — because the
commonest wrong answer on the reference library was a soft eye read as
closed, and a classifier shown a smear will always say something. An eye
under either floor, or narrower than six tenths of its partner (the far
eye of a turned head, whose contour collapses), is not asked; a face with
no readable eye is a fourth state, Unreadable, that no filter drops. On
twenty native renders the one real blink is caught, the laughing faces
are closed, the profiles are judged on the near eye, and the one thing
left beyond any floor is a face with a pot held over it.
This commit is contained in:
dtourolle
2026-09-19 14:04:08 +02:00
parent b908d861e0
commit f5956707e7
7 changed files with 690 additions and 311 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/dr-face/src/eyes.rs
+154 -30
View File
@@ -1,37 +1,55 @@
//! TRACES: FR-CULL-13
//! What a face's eyes are doing, and how the three numbers behind it are read.
//! What a face's eyes are doing, and how the numbers behind it are read.
//!
//! Model-free: the classifiers in [`crate::classify`] produce the numbers,
//! and everything that interprets them — the catalog's filter, the People
//! Model-free: the models in [`crate::classify`] produce the numbers, and
//! everything that interprets them — the catalog's filter, the People
//! screen's label — comes through here, so a threshold lives in exactly one
//! place.
//!
//! # Three numbers, one answer
//! # Seven numbers, one answer
//!
//! An eye classifier answers "open or closed" for whatever it is shown, and
//! shown a lens of dark glass it answers anyway. Its answer over sunglasses is
//! not *wrong* in any way it can report — it is a confident probability of a
//! state that cannot be seen — and a filter for "eyes open" that trusted it
//! would drop every photograph from the beach. So the reading carries a
//! third number, from a classifier that looks at the whole head, and it takes
//! precedence: a face behind sunglasses is [`EyeState::Sunglasses`], whatever
//! the eye classifier made of the glass.
//! it is shown three things it cannot answer for. **Dark glass**: over
//! sunglasses it answers anyway, confidently, for a state that cannot be
//! seen — so the reading carries P(sunglasses) from a classifier that looks
//! at the whole head, and that takes precedence. **A smear**: a soft eye is
//! not a closed one, but shown a blur the classifier says "closed" with the
//! same confidence it says anything, and on the reference library that was
//! the commonest wrong answer of all — small faces, motion, a proxy where
//! the native render should have been. So each eye carries how many source
//! pixels it spanned and how sharp the patch was, and an eye under either
//! floor is not asked. **A cheek**: a head turned far enough hides its far
//! eye, and the landmark contour of a hidden eye collapses to a sliver; an
//! eye much narrower than its partner is not asked either.
//!
//! The two eyes are kept apart rather than averaged. A wink is one eye
//! closed, and averaging it lands at 0.5 — the one value that says the least.
//! [`EyeState::Open`] requires both.
//! [`EyeState::Open`] requires every eye that *could be read* to be open;
//! a face with no readable eye is [`EyeState::Unreadable`], which is not a
//! blink and not open, and a filter for either leaves it alone.
/// The probabilities the classifiers produced for one face.
/// One eye's numbers.
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct Eye {
/// P(open), the classifier's sigmoid.
pub open: f32,
/// Source pixels across the eye box — [`crate::align::EyePatch::source_px`].
pub px: f32,
/// [`crate::align::EyePatch::sharpness`] of the patch the classifier saw.
pub sharpness: f32,
}
/// The numbers the models produced for one face.
///
/// Stored per face, nullable as a whole: a face indexed before the eye models
/// existed, or on a device without them, has no reading rather than a
/// reading of zeros.
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct EyeReading {
/// P(open) for the subject's **right** eye — image-left, landmark 0.
pub right_open: f32,
/// P(open) for the subject's **left** eye — image-right, landmark 1.
pub left_open: f32,
/// The subject's **right** eye — image-left.
pub right: Eye,
/// The subject's **left** eye — image-right.
pub left: Eye,
/// P(the head wears sunglasses).
pub sunglasses: f32,
}
@@ -44,35 +62,87 @@ pub const EYES_OPEN_THRESHOLD: f32 = 0.5;
/// Above this the head wears sunglasses and the eye readings are moot.
pub const SUNGLASSES_THRESHOLD: f32 = 0.5;
/// Fewest source pixels across an eye box for the eye to be read.
///
/// The classifier was trained on eyes down to about a dozen pixels wide
/// (its reference footage averaged 15–21); below that the 40-pixel patch is
/// an interpolation of nothing, and the answer is noise that reads as
/// "closed". docs/faces.md §17.3 has the measurement behind the number.
pub const MIN_EYE_PX: f32 = 12.0;
/// Least [`Eye::sharpness`] for the eye to be read.
///
/// The same measure as the face's `min_sharpness`, over the eye patch, and
/// chosen the same way: the value under which the open-eyed faces of the
/// reference sample were being called closed. docs/faces.md §17.3.
pub const MIN_EYE_SHARPNESS: f32 = 0.02;
/// An eye narrower than this fraction of its partner is the far eye of a
/// turned head, out of view behind the nose, and is not read.
///
/// A landmark model's contour for a hidden eye collapses towards the nose.
/// Measured on twenty native renders of the reference library
/// (docs/faces.md §17.4): profiles put the far eye at 0.02–0.43 of the near
/// one, two three-quarter faces whose far eye read closed sat at 0.54, and
/// every face looking at the camera — winks included, since a shut eye's
/// box keeps its width — sat at 0.78 or more. 0.6 splits the gap.
pub const HIDDEN_EYE_RATIO: f32 = 0.6;
/// What the reading says, for a screen or a filter.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum EyeState {
/// Both eyes open.
/// Every eye that could be read is open.
Open,
/// At least one eye closed — a blink, or a wink.
/// An eye that could be read is closed — a blink, or a wink.
Closed,
/// The eyes cannot be seen. Neither open nor closed, and a filter for
/// either leaves the face alone.
Sunglasses,
/// No eye was sharp enough, large enough and in view to read. Neither
/// open nor closed, like sunglasses, and left alone by every filter.
Unreadable,
}
impl Eye {
/// Whether this eye can be read at all: enough pixels, sharp enough,
/// and not the collapsed contour of a hidden eye — measured against
/// `other`, its partner.
pub fn readable(&self, other: &Eye) -> bool {
self.px >= MIN_EYE_PX
&& self.sharpness >= MIN_EYE_SHARPNESS
&& self.px >= other.px * HIDDEN_EYE_RATIO
}
}
impl EyeReading {
pub fn state(&self) -> EyeState {
if self.sunglasses >= SUNGLASSES_THRESHOLD {
EyeState::Sunglasses
} else if self.right_open >= EYES_OPEN_THRESHOLD && self.left_open >= EYES_OPEN_THRESHOLD {
return EyeState::Sunglasses;
}
let readable = [
self.right.readable(&self.left).then_some(self.right.open),
self.left.readable(&self.right).then_some(self.left.open),
];
let mut any = false;
for open in readable.into_iter().flatten() {
any = true;
if open < EYES_OPEN_THRESHOLD {
return EyeState::Closed;
}
}
if any {
EyeState::Open
} else {
EyeState::Closed
EyeState::Unreadable
}
}
/// Whether this is a face a "no one blinking" filter should drop.
///
/// The filter's question, rather than [`EyeState`]'s three-way answer,
/// because the two differ on exactly the case that matters: a face behind
/// sunglasses is not open, and it is not a blink either. Only
/// [`EyeState::Closed`] is one.
/// The filter's question, rather than [`EyeState`]'s four-way answer,
/// because the two differ on exactly the cases that matter: a face
/// behind sunglasses, or one whose eyes could not be read, is not open
/// — and it is not a blink either. Only [`EyeState::Closed`] is one.
pub fn is_blink(&self) -> bool {
self.state() == EyeState::Closed
}
@@ -85,6 +155,7 @@ impl EyeState {
EyeState::Open => "Eyes open",
EyeState::Closed => "Eyes closed",
EyeState::Sunglasses => "Sunglasses",
EyeState::Unreadable => "Eyes unclear",
}
}
}
@@ -93,10 +164,18 @@ impl EyeState {
mod tests {
use super::*;
fn eye(open: f32) -> Eye {
Eye {
open,
px: 40.0,
sharpness: 0.1,
}
}
fn reading(right: f32, left: f32, sunglasses: f32) -> EyeReading {
EyeReading {
right_open: right,
left_open: left,
right: eye(right),
left: eye(left),
sunglasses,
}
}
@@ -116,8 +195,8 @@ mod tests {
assert!(reading(0.2, 0.9, 0.1).is_blink());
}
/// The whole reason the third number exists: whatever the eye classifier
/// says over dark glass, it is not a reading of the eyes.
/// The whole reason the sunglasses number exists: whatever the eye
/// classifier says over dark glass, it is not a reading of the eyes.
#[test]
fn sunglasses_override_the_eye_readings_either_way() {
assert_eq!(reading(0.9, 0.9, 0.8).state(), EyeState::Sunglasses);
@@ -125,6 +204,46 @@ mod tests {
assert!(!reading(0.1, 0.1, 0.8).is_blink());
}
/// A soft or tiny eye is not asked; if neither can be, the face is
/// unreadable rather than closed.
#[test]
fn a_soft_or_tiny_eye_is_not_read() {
let mut r = reading(0.1, 0.9, 0.0);
r.right.sharpness = MIN_EYE_SHARPNESS / 2.0;
assert_eq!(r.state(), EyeState::Open, "the soft closed eye is ignored");
let mut r = reading(0.1, 0.9, 0.0);
r.right.px = MIN_EYE_PX - 1.0;
assert_eq!(r.state(), EyeState::Open, "the tiny closed eye is ignored");
let mut r = reading(0.1, 0.1, 0.0);
r.right.sharpness = 0.0;
r.left.px = 3.0;
assert_eq!(r.state(), EyeState::Unreadable);
assert!(!r.is_blink());
assert_eq!(r.state().label(), "Eyes unclear");
}
/// A profile: the far eye's contour collapses, and the sliver is not
/// read. The near eye still decides.
#[test]
fn a_turned_heads_collapsed_far_eye_is_not_read() {
let mut r = reading(0.05, 0.95, 0.0);
r.right.px = 40.0 * HIDDEN_EYE_RATIO - 1.0;
assert!(!r.right.readable(&r.left));
assert_eq!(r.state(), EyeState::Open);
let mut blink = reading(0.95, 0.05, 0.0);
blink.right.px = 40.0 * HIDDEN_EYE_RATIO - 1.0;
assert_eq!(blink.state(), EyeState::Closed);
// Both eyes narrow but alike is not a turned head: both count.
let mut small = reading(0.05, 0.95, 0.0);
small.right.px = 14.0;
small.left.px = 14.0;
assert_eq!(small.state(), EyeState::Closed);
}
#[test]
fn the_thresholds_are_inclusive_at_the_decision_point() {
assert_eq!(
@@ -135,5 +254,10 @@ mod tests {
reading(1.0, 1.0, SUNGLASSES_THRESHOLD).state(),
EyeState::Sunglasses
);
let mut r = reading(1.0, 1.0, 0.0);
r.right.px = MIN_EYE_PX;
r.left.px = MIN_EYE_PX;
r.right.sharpness = MIN_EYE_SHARPNESS;
assert!(r.right.readable(&r.left));
}
}