Name segmented people from the faces already recognised in them
The segmenter knows it found a person; the face index knows which person. Joining them turns "person" in the mask list into "Anna", which is the difference between a vocabulary of eighty COCO classes and one that includes the user's family. Selecting a subject in a group photograph stops being a guessing game between three identical rows. Containment, not IoU. A face is a small part of the person it belongs to, so a correct pairing has an IoU near zero and anything IoU-based would reject every true match. Confirmed names only. A suggestion is the system's guess, and printing a guessed name onto a mask region would launder it into a fact. Writing the tests corrected the design once: a tight head-and-shoulders portrait, where the face fills most of the person box, is the case where naming is most certain, not least. An earlier guard rejected exactly that and has been removed, with the reasoning left as a test because it is easy to get backwards a second time. The names hang on the develop session, set when the image opens because that is the one moment the catalog and the image id are both in reach. Every segmentation run afterwards picks them up for free, and a library with no face indexing behaves exactly as it did before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+57
-1
@@ -116,6 +116,14 @@ pub struct SegmentationJob {
|
||||
source: Arc<DemosaicedImage>,
|
||||
session: SessionId,
|
||||
abandon: Abandon,
|
||||
/// Confirmed faces in this photograph, **normalised to the long edge**.
|
||||
///
|
||||
/// Carried rather than looked up, because the job runs on a thread with no
|
||||
/// catalog in reach — the same reason it carries the pixels. Normalised
|
||||
/// rather than in pixels because the proxy size is only settled inside
|
||||
/// `run`, and these have to survive being scaled to whatever it turns out
|
||||
/// to be.
|
||||
names: Vec<crate::identity::NormalisedNamedBox>,
|
||||
}
|
||||
|
||||
impl SegmentationJob {
|
||||
@@ -158,7 +166,32 @@ impl SegmentationJob {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let seg = segmentation::compute(&self.ctx, &rgb, rw, rh, options)?;
|
||||
let mut seg = segmentation::compute(&self.ctx, &rgb, rw, rh, options)?;
|
||||
|
||||
// Put names on the people the segmenter found. Scaled here rather than
|
||||
// at the call site because this is where the proxy size is finally
|
||||
// known.
|
||||
if !self.names.is_empty() {
|
||||
let long_edge = rw.max(rh) as f32;
|
||||
let boxes: Vec<dr_face::NamedFace<'_>> = self
|
||||
.names
|
||||
.iter()
|
||||
.map(|(x, y, w, h, name)| dr_face::NamedFace {
|
||||
bbox: (
|
||||
x * long_edge,
|
||||
y * long_edge,
|
||||
(x + w) * long_edge,
|
||||
(y + h) * long_edge,
|
||||
),
|
||||
name,
|
||||
})
|
||||
.collect();
|
||||
let named = seg.apply_names(&boxes);
|
||||
if named > 0 {
|
||||
log::info!("named {named} segmented region(s) from known faces");
|
||||
}
|
||||
}
|
||||
|
||||
log::info!(
|
||||
"segmented {rw}×{rh}: {} subject(s), proxy {:.0} ms, total {:.0} ms",
|
||||
seg.instances().len(),
|
||||
@@ -478,6 +511,13 @@ fn bilinear_sample(mask: &[f32], w: usize, h: usize, x: f32, y: f32) -> f32 {
|
||||
pub struct DevelopSession {
|
||||
/// This session's name, for work that outlives the frame it started on.
|
||||
id: SessionId,
|
||||
/// TRACES: FR-CULL-10
|
||||
/// Confirmed faces in this photograph, normalised to the long edge.
|
||||
///
|
||||
/// Empty until [`DevelopSession::set_face_names`] is called, and empty for
|
||||
/// ever on a library with no face indexing — in which case segmentation
|
||||
/// behaves exactly as it did before, which is the point.
|
||||
face_names: Vec<crate::identity::NormalisedNamedBox>,
|
||||
/// Kept so the session can build GPU resources after construction.
|
||||
///
|
||||
/// The distance fields behind a subject mask are made when a layer is
|
||||
@@ -606,6 +646,7 @@ impl DevelopSession {
|
||||
let history = History::new(&graph);
|
||||
Self {
|
||||
id: SessionId::next(),
|
||||
face_names: Vec::new(),
|
||||
ctx: ctx.clone(),
|
||||
graph,
|
||||
history,
|
||||
@@ -1593,9 +1634,24 @@ impl DevelopSession {
|
||||
source: self.demosaiced.clone(),
|
||||
session: self.id,
|
||||
abandon: Abandon::default(),
|
||||
names: self.face_names.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
/// TRACES: FR-CULL-10 | FR-DEV-3
|
||||
/// The confirmed faces in this photograph, for naming segmented regions.
|
||||
///
|
||||
/// Set once when the image opens, because that is the only moment the
|
||||
/// catalog and the image id are both in reach — the develop session
|
||||
/// deliberately knows nothing about either, and every segmentation run
|
||||
/// after this point picks the names up for free.
|
||||
///
|
||||
/// Boxes are normalised to the long edge, as the catalog stores them, so
|
||||
/// they survive whatever proxy size a run settles on.
|
||||
pub fn set_face_names(&mut self, names: Vec<crate::identity::NormalisedNamedBox>) {
|
||||
self.face_names = names;
|
||||
}
|
||||
|
||||
/// TRACES: FR-DEV-3
|
||||
/// Take on a segmentation found elsewhere.
|
||||
///
|
||||
|
||||
@@ -240,6 +240,108 @@ fn decode_proxy(
|
||||
dr_thumbs::codec::decode_rgba(&thumb.bytes).ok()
|
||||
}
|
||||
|
||||
/// A named face box normalised to the image's long edge: `(x, y, w, h, name)`.
|
||||
///
|
||||
/// Named because it crosses three layers — catalog, develop session,
|
||||
/// segmentation job — and "the fourth float" is not something anyone should
|
||||
/// have to count out at each one.
|
||||
pub type NormalisedNamedBox = (f32, f32, f32, f32, String);
|
||||
|
||||
/// The confirmed, named faces in one image, **normalised to the long edge**.
|
||||
///
|
||||
/// The form a develop session carries, because the proxy a segmentation run
|
||||
/// settles on is not known when the image opens — so the scaling happens at
|
||||
/// the far end, in [`crate::develop::SegmentationJob`].
|
||||
///
|
||||
/// **Confirmed names only.** A suggestion is the system's guess, and printing
|
||||
/// a guessed name onto a mask region would launder it into a fact — the exact
|
||||
/// conflation the confirmed/suggested split exists to prevent (FR-CULL-10). An
|
||||
/// unrecognised or merely-suggested person leaves the region as "person",
|
||||
/// which is honest.
|
||||
pub fn named_boxes_normalised(
|
||||
catalog: &Catalog,
|
||||
image: ImageId,
|
||||
) -> Result<Vec<NormalisedNamedBox>, dr_catalog::CatalogError> {
|
||||
let conn = catalog.connection();
|
||||
|
||||
let mut names: std::collections::HashMap<PersonId, String> = std::collections::HashMap::new();
|
||||
for p in faces::people(conn)? {
|
||||
if !p.name.is_empty() {
|
||||
names.insert(p.id, p.name);
|
||||
}
|
||||
}
|
||||
if names.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
|
||||
Ok(faces::for_image(conn, image)?
|
||||
.into_iter()
|
||||
.filter(|f| f.confirmed)
|
||||
.filter_map(|f| {
|
||||
let name = f.person.and_then(|p| names.get(&p))?.clone();
|
||||
Some((f.x, f.y, f.w, f.h, name))
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// A named face box for one image, in proxy pixels.
|
||||
///
|
||||
/// Owned rather than borrowed because it crosses from a catalog read into a
|
||||
/// segmentation pass that outlives the query.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct NamedBox {
|
||||
pub bbox: (f32, f32, f32, f32),
|
||||
pub name: String,
|
||||
}
|
||||
|
||||
/// The named faces in one image, ready to label its segmented regions.
|
||||
///
|
||||
/// **Confirmed names only.** A suggestion is the system's guess, and printing
|
||||
/// a guessed name on a mask region would launder it into a fact — the exact
|
||||
/// conflation the confirmed/suggested split exists to prevent (FR-CULL-10).
|
||||
/// An unrecognised or merely-suggested person leaves the region as "person",
|
||||
/// which is honest.
|
||||
///
|
||||
/// Boxes come back in the proxy pixel space the caller names, because that is
|
||||
/// what `Segmentation` works in — the catalog stores them normalised to the
|
||||
/// long edge precisely so this conversion is possible at any resolution.
|
||||
pub fn named_boxes_for_image(
|
||||
catalog: &Catalog,
|
||||
image: ImageId,
|
||||
proxy_w: usize,
|
||||
proxy_h: usize,
|
||||
) -> Result<Vec<NamedBox>, dr_catalog::CatalogError> {
|
||||
let conn = catalog.connection();
|
||||
let long_edge = proxy_w.max(proxy_h) as f32;
|
||||
if long_edge <= 0.0 {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
|
||||
let mut names: std::collections::HashMap<PersonId, String> = std::collections::HashMap::new();
|
||||
for p in faces::people(conn)? {
|
||||
if !p.name.is_empty() {
|
||||
names.insert(p.id, p.name);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(faces::for_image(conn, image)?
|
||||
.into_iter()
|
||||
.filter(|f| f.confirmed)
|
||||
.filter_map(|f| {
|
||||
let name = f.person.and_then(|p| names.get(&p))?.clone();
|
||||
Some(NamedBox {
|
||||
bbox: (
|
||||
f.x * long_edge,
|
||||
f.y * long_edge,
|
||||
(f.x + f.w) * long_edge,
|
||||
(f.y + f.h) * long_edge,
|
||||
),
|
||||
name,
|
||||
})
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
// ── the user's decisions ──────────────────────────────────────────────────
|
||||
|
||||
/// Name a group, or rename a person.
|
||||
@@ -596,6 +698,62 @@ mod tests {
|
||||
assert_eq!(load_people(&c, "w600k_mbf").unwrap().people.len(), 1);
|
||||
}
|
||||
|
||||
/// The segmentation link: a confirmed face inside a `person` region should
|
||||
/// put that person's name on it.
|
||||
#[test]
|
||||
fn named_boxes_come_back_in_proxy_pixels() {
|
||||
let c = catalog();
|
||||
let img = image(&c, 1);
|
||||
let ids = faces::record_detections(c.connection(), img, "w600k_mbf", 1024, &[face(1)])
|
||||
.unwrap();
|
||||
let anna = faces::create_person(c.connection(), "Anna").unwrap();
|
||||
faces::confirm(c.connection(), ids[0], anna).unwrap();
|
||||
|
||||
// The fixture's face is x=0.1 y=0.1 w=0.2 h=0.3, normalised to the
|
||||
// long edge — so at 1024×683 it spans 102.4..307.2 across.
|
||||
let boxes = named_boxes_for_image(&c, img, 1024, 683).unwrap();
|
||||
assert_eq!(boxes.len(), 1);
|
||||
assert_eq!(boxes[0].name, "Anna");
|
||||
assert!((boxes[0].bbox.0 - 102.4).abs() < 0.1, "{:?}", boxes[0].bbox);
|
||||
assert!((boxes[0].bbox.2 - 307.2).abs() < 0.1, "{:?}", boxes[0].bbox);
|
||||
|
||||
// And the same face at another proxy size lands proportionally — the
|
||||
// reason the catalog stores these normalised.
|
||||
let bigger = named_boxes_for_image(&c, img, 2048, 1366).unwrap();
|
||||
assert!((bigger[0].bbox.0 - 204.8).abs() < 0.1);
|
||||
}
|
||||
|
||||
/// A suggestion is the system's guess. Printing a guessed name onto a mask
|
||||
/// region would launder it into a fact.
|
||||
#[test]
|
||||
fn a_suggested_person_does_not_name_a_region() {
|
||||
let c = catalog();
|
||||
let img = image(&c, 1);
|
||||
let ids = faces::record_detections(c.connection(), img, "w600k_mbf", 1024, &[face(1)])
|
||||
.unwrap();
|
||||
let anna = faces::create_person(c.connection(), "Anna").unwrap();
|
||||
faces::suggest(c.connection(), ids[0], anna, 0.95).unwrap();
|
||||
|
||||
assert!(named_boxes_for_image(&c, img, 1024, 683).unwrap().is_empty());
|
||||
|
||||
// Confirming it makes the name appear.
|
||||
faces::confirm(c.connection(), ids[0], anna).unwrap();
|
||||
assert_eq!(named_boxes_for_image(&c, img, 1024, 683).unwrap().len(), 1);
|
||||
}
|
||||
|
||||
/// An unnamed cluster has nothing to say about a region.
|
||||
#[test]
|
||||
fn an_unnamed_group_does_not_name_a_region() {
|
||||
let c = catalog();
|
||||
let img = image(&c, 1);
|
||||
let ids = faces::record_detections(c.connection(), img, "w600k_mbf", 1024, &[face(1)])
|
||||
.unwrap();
|
||||
let nameless = faces::create_person(c.connection(), "").unwrap();
|
||||
faces::confirm(c.connection(), ids[0], nameless).unwrap();
|
||||
|
||||
assert!(named_boxes_for_image(&c, img, 1024, 683).unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deleting_everything_empties_the_screen() {
|
||||
let c = catalog();
|
||||
|
||||
+32
-1
@@ -1685,6 +1685,10 @@ pub fn run(paths: Vec<PathBuf>) -> Result<()> {
|
||||
let redraw = redraw.clone();
|
||||
let rows = rows.clone();
|
||||
let gpu = gpu.clone();
|
||||
// Cloned for the timer closure: the outer callback is an `Fn` and
|
||||
// may run again for the next photograph.
|
||||
let library = library.clone();
|
||||
let path = path.clone();
|
||||
let sidecar_rx = Rc::new(sidecar_rx);
|
||||
let timer = Rc::new(slint::Timer::default());
|
||||
let held = timer.clone();
|
||||
@@ -1726,8 +1730,35 @@ pub fn run(paths: Vec<PathBuf>) -> Result<()> {
|
||||
w.set_exposure(describe_exposure(&l.meta).into());
|
||||
w.set_dimensions(format!("{} × {}", l.width, l.height).into());
|
||||
match l.session {
|
||||
Some(s) => {
|
||||
Some(mut s) => {
|
||||
w.set_adjust_enabled(true);
|
||||
// TRACES: FR-CULL-10
|
||||
// Who is in this photograph, so a
|
||||
// segmentation run can say "Anna" where
|
||||
// the model can only say "person". Read
|
||||
// here because this is the one moment the
|
||||
// catalog and the image id are both in
|
||||
// reach; every run afterwards gets them
|
||||
// for free.
|
||||
if let Some(id) = library.image_id_for_path(&path) {
|
||||
let cat = library.catalog();
|
||||
let borrowed = cat.borrow();
|
||||
if let Some(c) = borrowed.as_ref() {
|
||||
match crate::identity::named_boxes_normalised(c, id) {
|
||||
Ok(n) if !n.is_empty() => {
|
||||
log::info!(
|
||||
"{} known face(s) in this photograph",
|
||||
n.len()
|
||||
);
|
||||
s.set_face_names(n);
|
||||
}
|
||||
Ok(_) => {}
|
||||
Err(e) => {
|
||||
log::debug!("reading known faces: {e}")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
*session.borrow_mut() = Some(s);
|
||||
// TRACES: FR-CAT-8
|
||||
// The stored edit, if it has landed. It
|
||||
|
||||
@@ -293,6 +293,43 @@ pub fn compute(
|
||||
})
|
||||
}
|
||||
|
||||
/// Classes a recognised face may put a name on.
|
||||
///
|
||||
/// Only these. A face inside a `tv` or a `laptop` is a photograph of someone on
|
||||
/// a screen, and renaming the television to "Anna" would be worse than leaving
|
||||
/// it as the model found it.
|
||||
fn is_person_class(name: &str) -> bool {
|
||||
name == "person"
|
||||
}
|
||||
|
||||
impl Segmentation {
|
||||
/// Relabel `person` instances with the name of the face inside them.
|
||||
///
|
||||
/// The segmenter knows it found *a person*; the face index knows *which*
|
||||
/// person. Joining them turns "person" in the mask list into "Anna", which
|
||||
/// is the difference between COCO's eighty classes and a vocabulary that
|
||||
/// includes the user's family — and it is the same click either way, so the
|
||||
/// gain is entirely in being able to tell two people apart before clicking.
|
||||
///
|
||||
/// `faces` are in the same proxy pixels as [`InstanceSummary::bbox`]. The
|
||||
/// geometry lives in `dr_face::naming`, where it is testable without a
|
||||
/// model or a catalog.
|
||||
///
|
||||
/// Returns how many instances gained a name. An unrecognised person keeps
|
||||
/// the model's own label, which is the right default: "person" is merely
|
||||
/// unhelpful, where a wrong name is wrong and the user cannot tell which
|
||||
/// they are looking at.
|
||||
pub fn apply_names(&mut self, faces: &[dr_face::NamedFace<'_>]) -> usize {
|
||||
dr_face::name_instances(
|
||||
&mut self.instances,
|
||||
faces,
|
||||
|i| i.bbox,
|
||||
|i| is_person_class(&i.class_name),
|
||||
|i, name| i.class_name = name.into(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// Load the model and run it.
|
||||
///
|
||||
/// Loading is ~24 ms against the ~470 ms of inference that follows, and this
|
||||
|
||||
Reference in New Issue
Block a user