dtourolleandClaude Opus 5
d0b671d4db
The merge probability was `dr_face`'s constant and the smallest group was a bare `< 2` in the clustering pass. Both were tuned on one library — 1,813 faces of one photographer's family — and the quantity they optimise is a property of the population, not of the model. A household at close family resemblance and two thousand strangers at a wedding want different answers, and neither of them is the reference library. The doc comment already conceded the point and pointed at `face_index --tune`; a photographer does not have a terminal. So they are `FaceSettings` now, saved per device beside the cache budgets and edited from the People screen — beside the Regroup button that applies them and the rail that shows what they did, because a value changed three screens away from its effect is one nobody can tune. Moving them is safe by construction, which is why nothing asks for confirmation: a regroup writes only the suggested half, and confirmations, names and ignores enter as anchors and come back unchanged. The smallest-group rule is applied only to groups the system invented — a group the user named or set aside survives it whatever its size, because a display preference does not overrule a judgement. **Withdrawal, without which the setting does nothing visible.** Raising the smallest group stops the pass creating small groups; it does not remove the ones a previous pass made, because those still hold their suggestions, so they are not empty, so the prune leaves them. The pass now releases every unanchored face it did not place before pruning. And a dial you cannot see the effect of is not a dial. "What would this do?" runs the same population through the clusterer without opening a transaction and reports groups, faces grouped and largest group — one row of `--tune`'s table, on the user's own library, on a worker thread. The line leads with the group count because that is the number that says which side of the right setting you are on: it climbs as fragments are gathered into people and falls as separate people start being welded, while the grouped-face count rises straight through both. The preview parks its poll timer in a slot of its own. A preview and a regroup are allowed to be in flight together, and sharing the sweep's single slot would have the second to start drop the first's timer — visible as a Regroup that finished on its worker and never said so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1520 lines
57 KiB
Rust
1520 lines
57 KiB
Rust
//! TRACES: FR-CULL-8 | FR-CULL-9 | FR-CULL-10 | NFR-ARCH-2 | NFR-SEC-5
|
|
//! Face indexing and clustering, as background passes over the library.
|
|
//!
|
|
//! `dr-face` knows how to find a face in a buffer and `dr-catalog` knows how to
|
|
//! store one. This is the pass that connects them: read the proxy the grid
|
|
//! already built, detect, align, embed, write, and — once detection has gone
|
|
//! quiet — group what was found into people.
|
|
//!
|
|
//! # Detection runs on proxies, never on originals
|
|
//!
|
|
//! FR-CULL-8 pins this to the FR-CULL-2 ladder, and the consequence is the
|
|
//! thing that makes the feature affordable: a library that has been browsed has
|
|
//! already paid for its proxies, so face indexing adds **no RAW decodes that
|
|
//! were not already happening**. The tier is `ThumbSize::Large` — 1024 on the
|
|
//! long edge — and docs/faces.md §7 has the table of what the embedder actually
|
|
//! receives at that resolution.
|
|
//!
|
|
//! # Why clustering is a separate pass and not a job
|
|
//!
|
|
//! Detection is per image and parallel, so it is a job. Clustering is a
|
|
//! *whole-library* operation over the embeddings detection produced: it has no
|
|
//! natural `subject_id`, and running it per photograph would rebuild the world
|
|
//! on every one. It therefore runs debounced, when detection has been idle and
|
|
//! the face count has moved materially (catalog.md §10.2).
|
|
|
|
use std::path::PathBuf;
|
|
use std::sync::mpsc::{Receiver, Sender};
|
|
|
|
use dr_catalog::faces::{self, DetectedFace};
|
|
use dr_catalog::Catalog;
|
|
use dr_face::{align, Calibration, DetectOptions, Detection, Detector, Embedder, ModelId};
|
|
use dr_thumbs::{ThumbSize, ThumbStore};
|
|
use dr_types::settings::FaceSettings;
|
|
use dr_types::ImageId;
|
|
|
|
/// The tier faces are found on. See the module note.
|
|
pub const FACE_TIER: ThumbSize = ThumbSize::Large;
|
|
|
|
/// Progress from an indexing sweep.
|
|
#[derive(Debug, Clone, PartialEq)]
|
|
pub enum FaceSweepMessage {
|
|
/// How many images will be visited. Sent once, before any work.
|
|
Total(usize),
|
|
/// One image finished, with the faces found in it.
|
|
Indexed { image: ImageId, faces: usize },
|
|
/// The pass ended.
|
|
Finished {
|
|
images: usize,
|
|
faces: usize,
|
|
failed: usize,
|
|
},
|
|
}
|
|
|
|
/// An image waiting to be indexed.
|
|
#[derive(Debug, Clone, PartialEq)]
|
|
pub struct FaceRequest {
|
|
pub image_id: ImageId,
|
|
/// The thumbnail store's key — `oc:fileid`, stable across a server-side
|
|
/// move and the same id every other client sees (FR-NC-5).
|
|
pub file_id: u64,
|
|
}
|
|
|
|
/// Images that have a usable proxy and have not been through this model.
|
|
///
|
|
/// Asks `face_index` — the *run* marker — rather than asking whether the image
|
|
/// has any faces. Those are different questions, and confusing them is the
|
|
/// difference between a pass that converges and one that does not: a
|
|
/// photograph with no face in it would otherwise look identical to one never
|
|
/// examined, so every landscape in the library would be re-detected on every
|
|
/// run, for ever. See the V9 migration.
|
|
///
|
|
/// Keyed on the model, so a model upgrade re-indexes rather than leaving the
|
|
/// library half-described by weights that are no longer comparable.
|
|
pub fn faces_outstanding(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
model_id: &str,
|
|
) -> Result<Vec<FaceRequest>, dr_catalog::CatalogError> {
|
|
let mut stmt = catalog.connection().prepare(
|
|
"SELECT i.id, r.file_id
|
|
FROM images i
|
|
JOIN remote r ON r.image_id = i.id
|
|
WHERE r.file_id IS NOT NULL
|
|
AND i.trashed_at IS NULL
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM face_index fi
|
|
WHERE fi.image_id = i.id AND fi.model_id = ?1
|
|
)
|
|
ORDER BY i.id",
|
|
)?;
|
|
let rows = stmt
|
|
.query_map([model_id], |r| {
|
|
Ok(FaceRequest {
|
|
image_id: ImageId(r.get::<_, i64>(0)? as u64),
|
|
file_id: r.get::<_, i64>(1)? as u64,
|
|
})
|
|
})?
|
|
.filter_map(Result::ok)
|
|
// This pass reads the store and only the store, so an image without a
|
|
// proxy is not work it can do.
|
|
//
|
|
// **Not because FR-CULL-8 forbids it.** That requirement keeps indexing
|
|
// off the *full decode* and says the opposite about proxies — "where no
|
|
// proxy exists, the job requests one at background priority". An
|
|
// earlier comment here read it the other way round, and the result was
|
|
// a whole-library button that could only reach photographs the user had
|
|
// personally zoomed into. `library::spawn_face_sweep` is that
|
|
// requirement implemented; this one is the local-only variant.
|
|
.filter(|req| store.contains(req.file_id, FACE_TIER))
|
|
.collect();
|
|
Ok(rows)
|
|
}
|
|
|
|
/// What a coverage check found.
|
|
///
|
|
/// The catalog can say how many images have been through the model; only this
|
|
/// layer can say *why* the rest have not, because the reason usually lives in
|
|
/// the thumbnail store rather than the catalog.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
|
pub struct IndexAudit {
|
|
pub coverage: faces::Coverage,
|
|
/// Outstanding and ready: the proxy exists, so a sweep would do these now.
|
|
pub ready: u64,
|
|
/// Outstanding with no proxy on disk, so the whole-library pass will fetch
|
|
/// one. Still reported separately because it is the expensive half — these
|
|
/// cost a range request each, and the ones above cost nothing.
|
|
pub awaiting_proxy: u64,
|
|
}
|
|
|
|
impl IndexAudit {
|
|
/// One line, for a log or a status strip.
|
|
pub fn summary(&self) -> String {
|
|
let c = &self.coverage;
|
|
if c.images == 0 {
|
|
return "no images in the library".into();
|
|
}
|
|
// Whole numbers read fine at 40% and lie at 0.47%, which rounds to
|
|
// "0%" beside a count of 110 — a figure that says the feature is
|
|
// broken when it is merely early. One decimal below ten percent, and
|
|
// a floor so real progress never displays as none.
|
|
let pct = c.fraction() * 100.0;
|
|
let shown = if c.indexed > 0 && pct < 0.1 {
|
|
"<0.1%".to_string()
|
|
} else if pct < 10.0 {
|
|
format!("{pct:.1}%")
|
|
} else {
|
|
format!("{pct:.0}%")
|
|
};
|
|
let mut s = format!(
|
|
"{}/{} images indexed ({shown}), {} face(s), {} image(s) with none",
|
|
c.indexed, c.images, c.faces, c.without_faces,
|
|
);
|
|
if self.ready > 0 {
|
|
s.push_str(&format!("; {} ready to index", self.ready));
|
|
}
|
|
if self.awaiting_proxy > 0 {
|
|
// Not a blocker any more, and it must not read like one: the
|
|
// whole-library pass fetches these rather than skipping them.
|
|
s.push_str(&format!("; {} to fetch", self.awaiting_proxy));
|
|
}
|
|
s
|
|
}
|
|
}
|
|
|
|
/// Check every library image for a face-detection run marker.
|
|
///
|
|
/// The batch pass that answers "has face recognition been over all of this",
|
|
/// and the one to run before deciding whether to start a sweep. Cheap: two
|
|
/// counts and one indexed scan, no decoding and no inference.
|
|
pub fn audit(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
model_id: &str,
|
|
) -> Result<IndexAudit, dr_catalog::CatalogError> {
|
|
let conn = catalog.connection();
|
|
let coverage = faces::coverage(conn, model_id)?;
|
|
|
|
// Split the outstanding set by whether a proxy exists. This is the query
|
|
// `faces_outstanding` runs without the store filter, so the two cannot
|
|
// disagree about what is outstanding.
|
|
let mut stmt = conn.prepare(
|
|
"SELECT r.file_id
|
|
FROM images i
|
|
JOIN remote r ON r.image_id = i.id
|
|
WHERE r.file_id IS NOT NULL
|
|
AND i.trashed_at IS NULL
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM face_index fi
|
|
WHERE fi.image_id = i.id AND fi.model_id = ?1
|
|
)",
|
|
)?;
|
|
let (mut ready, mut awaiting) = (0u64, 0u64);
|
|
for file_id in stmt
|
|
.query_map([model_id], |r| r.get::<_, i64>(0))?
|
|
.filter_map(Result::ok)
|
|
{
|
|
if store.contains(file_id as u64, FACE_TIER) {
|
|
ready += 1;
|
|
} else {
|
|
awaiting += 1;
|
|
}
|
|
}
|
|
|
|
Ok(IndexAudit {
|
|
coverage,
|
|
ready,
|
|
awaiting_proxy: awaiting,
|
|
})
|
|
}
|
|
|
|
/// Detect and embed every face in one decoded proxy.
|
|
///
|
|
/// Coordinates come back **normalised to the long edge**, which is what the
|
|
/// catalog stores: a face must survive the proxy it was found on being evicted
|
|
/// and regenerated at a different size.
|
|
///
|
|
/// A face whose landmarks are degenerate is dropped rather than stored with a
|
|
/// junk embedding. That happens — a detector firing on a motion-blurred profile
|
|
/// can put all five landmarks on a line — and one junk embedding in the
|
|
/// clustering graph can bridge two real people.
|
|
pub fn index_proxy(
|
|
detector: &mut Detector,
|
|
embedder: &mut Embedder,
|
|
rgb: &[f32],
|
|
width: usize,
|
|
height: usize,
|
|
options: &DetectOptions,
|
|
) -> Result<Vec<DetectedFace>, dr_face::FaceError> {
|
|
let long_edge = width.max(height) as f32;
|
|
if long_edge <= 0.0 {
|
|
return Ok(Vec::new());
|
|
}
|
|
|
|
let dets = detector.detect(rgb, width, height, options)?;
|
|
let mut out = Vec::with_capacity(dets.len());
|
|
|
|
for d in &dets {
|
|
let Some(aligned) = align::warp(rgb, width, height, &d.landmarks) else {
|
|
log::debug!("face with degenerate landmarks skipped");
|
|
continue;
|
|
};
|
|
|
|
// The size floor, on the crop rather than the box. `min_face_px` has
|
|
// already thrown away the hopeless; this is the real gate, and it is
|
|
// here because `source_px` is only known once the warp has fixed the
|
|
// scale. A face below it was upsampled to reach the embedder, and no
|
|
// amount of upsampling puts back detail the sensor never recorded.
|
|
if aligned.source_px() < options.min_source_px {
|
|
log::debug!(
|
|
"face skipped: {:.0} source px below {:.0}",
|
|
aligned.source_px(),
|
|
options.min_source_px
|
|
);
|
|
continue;
|
|
}
|
|
|
|
// The blur gate, and the reason it is here rather than in the
|
|
// detector: sharpness is a property of the *aligned* crop, so it
|
|
// cannot be known until the warp has run.
|
|
//
|
|
// A motion-blurred face detects confidently, aligns cleanly and embeds
|
|
// to a perfectly ordinary-looking vector. Nothing downstream can tell
|
|
// it apart from a real one — and because blurs resemble each other
|
|
// more than they resemble the people they were, they cluster together
|
|
// and weld unrelated identities into one group. Dropping it costs a
|
|
// face the user could not have identified anyway.
|
|
let sharpness = aligned.sharpness();
|
|
if sharpness < options.min_sharpness {
|
|
log::debug!(
|
|
"face at {:.0}px skipped: sharpness {sharpness:.4} below {:.4}",
|
|
aligned.source_px(),
|
|
options.min_sharpness
|
|
);
|
|
continue;
|
|
}
|
|
|
|
let embedding = embedder.embed(&aligned)?;
|
|
|
|
out.push(DetectedFace {
|
|
x: d.bbox.0 / long_edge,
|
|
y: d.bbox.1 / long_edge,
|
|
w: d.width() / long_edge,
|
|
h: d.height() / long_edge,
|
|
landmarks: normalise_landmarks(&d.landmarks, long_edge),
|
|
confidence: d.confidence,
|
|
embedding: embedding.to_f16_bytes(),
|
|
crop_px: aligned.source_px(),
|
|
model_id: embedder.model().as_str().to_string(),
|
|
// Cut here, while the buffer is still in hand. This is the only
|
|
// moment in the whole pipeline where the pixels are free.
|
|
crop: cut_crop(rgb, width, height, d).unwrap_or_default(),
|
|
});
|
|
}
|
|
|
|
Ok(out)
|
|
}
|
|
|
|
fn normalise_landmarks(lm: &[(f32, f32); 5], long_edge: f32) -> [(f32, f32); 5] {
|
|
let mut out = [(0.0_f32, 0.0_f32); 5];
|
|
for (o, &(x, y)) in out.iter_mut().zip(lm.iter()) {
|
|
*o = (x / long_edge, y / long_edge);
|
|
}
|
|
out
|
|
}
|
|
|
|
/// Index every image whose proxy is **already on this disk**, in the background.
|
|
///
|
|
/// Not the whole-library pass — that is `library::spawn_face_sweep`, which
|
|
/// fetches what it has not got. This one never touches the network, which makes
|
|
/// it the right shape for a tool run against a local store (see
|
|
/// `examples/face_index.rs`) and the wrong shape for a user pressing "index my
|
|
/// library", because for an unbrowsed library the work list is nearly empty.
|
|
///
|
|
/// Strictly background work: it competes with thumbnailing, not with rendering
|
|
/// (NFR-ARCH-2), and it is interruptible simply by dropping the receiver — the
|
|
/// next run resumes from what is already in the catalog, because
|
|
/// [`faces_outstanding`] asks the catalog what is missing rather than keeping a
|
|
/// cursor. That is what makes it survive process death (FR-PLAT-AND-3) with no
|
|
/// repeated work beyond the in-flight image.
|
|
#[allow(clippy::too_many_arguments)]
|
|
pub fn spawn_store_face_sweep(
|
|
catalog_path: PathBuf,
|
|
store_dir: PathBuf,
|
|
detector_model: PathBuf,
|
|
embedder_model: PathBuf,
|
|
model_id: String,
|
|
options: DetectOptions,
|
|
) -> Receiver<FaceSweepMessage> {
|
|
let (tx, rx) = std::sync::mpsc::channel();
|
|
|
|
std::thread::spawn(move || {
|
|
let finish_empty = |tx: &Sender<FaceSweepMessage>| {
|
|
let _ = tx.send(FaceSweepMessage::Finished {
|
|
images: 0,
|
|
faces: 0,
|
|
failed: 0,
|
|
});
|
|
};
|
|
|
|
let catalog = match Catalog::open(&catalog_path) {
|
|
Ok(c) => c,
|
|
Err(e) => {
|
|
log::warn!("face sweep: cannot open catalog: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
let store = match ThumbStore::open(&store_dir) {
|
|
Ok(s) => s,
|
|
Err(e) => {
|
|
log::warn!("face sweep: cannot open the thumbnail store: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
// Models first: they are the expensive failure, and there is no point
|
|
// listing ten thousand images before discovering the weights are
|
|
// missing. This is also the path a library with face indexing enabled
|
|
// but no model downloaded takes (docs/faces.md §2.2), so it must be a
|
|
// quiet return rather than an error.
|
|
let mut detector = match Detector::from_path(&detector_model) {
|
|
Ok(d) => d,
|
|
Err(e) => {
|
|
log::warn!("face sweep: cannot load the detector: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
let mut embedder =
|
|
match Embedder::from_path(&embedder_model, ModelId::new(model_id.clone())) {
|
|
Ok(e) => e,
|
|
Err(e) => {
|
|
log::warn!("face sweep: cannot load the embedder: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
let wanted = match faces_outstanding(&catalog, &store, &model_id) {
|
|
Ok(w) => w,
|
|
Err(e) => {
|
|
log::warn!("face sweep: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
let total = wanted.len();
|
|
if total == 0 {
|
|
log::info!("face sweep: every image with a proxy is already indexed");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
log::info!("face sweep: {total} image(s) to index");
|
|
if tx.send(FaceSweepMessage::Total(total)).is_err() {
|
|
return;
|
|
}
|
|
|
|
let (mut images, mut found, mut failed) = (0usize, 0usize, 0usize);
|
|
|
|
for req in wanted {
|
|
let thumb = match store.get(req.file_id, FACE_TIER) {
|
|
Ok(Some(t)) => t,
|
|
// Evicted between the listing and now. Not a failure: the next
|
|
// pass will find it, or the thumbnail sweep will rebuild it.
|
|
Ok(None) => continue,
|
|
Err(e) => {
|
|
log::debug!("face sweep: reading proxy for {:?}: {e}", req.image_id);
|
|
failed += 1;
|
|
continue;
|
|
}
|
|
};
|
|
|
|
let (w, h, rgba) = match dr_thumbs::codec::decode_rgba(&thumb.bytes) {
|
|
Ok(v) => v,
|
|
Err(e) => {
|
|
log::debug!("face sweep: decoding proxy for {:?}: {e}", req.image_id);
|
|
failed += 1;
|
|
continue;
|
|
}
|
|
};
|
|
let rgb = rgba_to_rgb_f32(&rgba);
|
|
|
|
let faces = match index_proxy(
|
|
&mut detector,
|
|
&mut embedder,
|
|
&rgb,
|
|
w as usize,
|
|
h as usize,
|
|
&options,
|
|
) {
|
|
Ok(f) => f,
|
|
Err(e) => {
|
|
log::debug!("face sweep: indexing {:?}: {e}", req.image_id);
|
|
failed += 1;
|
|
continue;
|
|
}
|
|
};
|
|
|
|
if let Err(e) = faces::record_detections(
|
|
catalog.connection(),
|
|
req.image_id,
|
|
&model_id,
|
|
w.max(h),
|
|
&faces,
|
|
) {
|
|
log::warn!("face sweep: storing faces for {:?}: {e}", req.image_id);
|
|
failed += 1;
|
|
continue;
|
|
}
|
|
|
|
images += 1;
|
|
found += faces.len();
|
|
if tx
|
|
.send(FaceSweepMessage::Indexed {
|
|
image: req.image_id,
|
|
faces: faces.len(),
|
|
})
|
|
.is_err()
|
|
{
|
|
// Receiver dropped: the window closed, or the user turned face
|
|
// indexing off. Stop, leaving everything written so far.
|
|
log::info!("face sweep: cancelled after {images} image(s)");
|
|
return;
|
|
}
|
|
}
|
|
|
|
log::info!("face sweep: {found} face(s) across {images} image(s), {failed} failed");
|
|
let _ = tx.send(FaceSweepMessage::Finished {
|
|
images,
|
|
faces: found,
|
|
failed,
|
|
});
|
|
});
|
|
|
|
rx
|
|
}
|
|
|
|
/// Everything a grouping pass reads before it decides anything.
|
|
///
|
|
/// Split out because two callers need exactly this and must agree on it: the
|
|
/// pass that writes, and the preview that reports what the pass *would* do. A
|
|
/// preview built from a second, subtly different reading — anchors omitted,
|
|
/// say — would answer a question about a library nobody has.
|
|
struct Population {
|
|
cal: Calibration,
|
|
candidates: Vec<dr_face::Candidate>,
|
|
/// Catalog ids, parallel to `candidates`.
|
|
ids: Vec<faces::FaceId>,
|
|
/// Faces the user has ruled on, and who they are. See [`Population::read`].
|
|
anchors: std::collections::HashMap<faces::FaceId, faces::PersonId>,
|
|
}
|
|
|
|
impl Population {
|
|
/// Takes the catalog and not its connection: `rusqlite` is `dr-catalog`'s
|
|
/// dependency and not this crate's, and reaching for the connection type by
|
|
/// name here would drag it across a layer that has kept clear of it.
|
|
fn read(catalog: &Catalog, model_id: &str) -> Result<Self, dr_catalog::CatalogError> {
|
|
let conn = catalog.connection();
|
|
// No valid calibration is not a reason to refuse to cluster — it is a
|
|
// reason not to *display* a confidence (FR-CULL-9). `Calibration::
|
|
// default` is the reference implementation's fitted curve with `valid`
|
|
// false, which is a documented operating point rather than an invented
|
|
// one.
|
|
let cal = faces::calibration(conn, model_id)?
|
|
.map(|(c, _)| c)
|
|
.unwrap_or_else(Calibration::default);
|
|
|
|
// Which faces the user has already ruled on, so they enter as anchors.
|
|
//
|
|
// Anything the user has ruled on anchors, and there are three ways of
|
|
// ruling — only the first of which is obvious.
|
|
//
|
|
// A **confirmation** is the plain case. **Setting a group aside** is one
|
|
// too, and the faces it covers are only ever suggestions, so anchoring
|
|
// confirmations alone let every ignored group scatter into fresh unnamed
|
|
// groups that were not ignored, and the strangers came straight back.
|
|
//
|
|
// And so is **giving a group a name**. That was the omission that did
|
|
// the most damage, because it is silent. Naming a cluster does not
|
|
// confirm its faces — they stay suggestions — so the next Regroup cut
|
|
// them loose, regrouped them into a brand new person, and left the named
|
|
// one holding nothing. `prune_empty_unnamed` will not remove it, because
|
|
// it has a name. Name the new group the same thing and it happens again.
|
|
// That is how one library came to hold sixteen people called Catherine,
|
|
// fourteen of them empty, with her faces split across the two that were
|
|
// not.
|
|
//
|
|
// A name is a judgement about *this group* (FR-CULL-12), exactly as an
|
|
// ignore is. Anchoring them all also does one better: a newly indexed
|
|
// face that matches a named person now merges *into* them rather than
|
|
// arriving as a stranger.
|
|
let mut anchors = std::collections::HashMap::new();
|
|
for p in faces::people(conn)? {
|
|
let ruled_on = p.ignored || !p.name.trim().is_empty();
|
|
for f in faces::for_person(conn, p.id, ruled_on)? {
|
|
anchors.insert(f.id, p.id);
|
|
}
|
|
}
|
|
|
|
let stored = faces::embeddings(conn, model_id)?;
|
|
let model = ModelId::new(model_id.to_string());
|
|
let mut candidates = Vec::with_capacity(stored.len());
|
|
let mut ids = Vec::with_capacity(stored.len());
|
|
for (face_id, image_id, blob, crop_px) in stored {
|
|
let Some(emb) = dr_face::Embedding::from_f16_bytes(model.clone(), &blob) else {
|
|
log::warn!("face {face_id:?} has a malformed embedding, skipped");
|
|
continue;
|
|
};
|
|
candidates.push(dr_face::Candidate {
|
|
face: face_id.0,
|
|
image: image_id.0,
|
|
embedding: emb.v.to_vec(),
|
|
crop_px,
|
|
confirmed_person: anchors.get(&face_id).map(|p| p.0),
|
|
});
|
|
ids.push(face_id);
|
|
}
|
|
|
|
Ok(Self {
|
|
cal,
|
|
candidates,
|
|
ids,
|
|
anchors,
|
|
})
|
|
}
|
|
}
|
|
|
|
/// What a grouping pass would produce, without producing it.
|
|
///
|
|
/// The numbers `dr_face::cluster`'s own tuning table is built from, for one
|
|
/// setting rather than ten — because the question a photographer is actually
|
|
/// asking of the dials is "what does *my* library look like at this value", and
|
|
/// the doc-comment table answers it for a library that is not theirs.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
|
pub struct GroupingPreview {
|
|
/// Faces that went in.
|
|
pub faces: usize,
|
|
/// Groups that would survive the smallest-group rule.
|
|
pub groups: usize,
|
|
/// Faces those groups would hold.
|
|
pub grouped: usize,
|
|
/// The biggest group. The tell for over-merging: it is the number that runs
|
|
/// away when the confidence is set too low.
|
|
pub largest: usize,
|
|
}
|
|
|
|
/// Report what a grouping pass would do, writing nothing.
|
|
///
|
|
/// Read-only by construction — it never opens a transaction — which is what
|
|
/// makes it safe to run repeatedly while the user moves a slider. Comparing two
|
|
/// settings by *applying* both would leave the second one's answer polluted by
|
|
/// the first one's suggestions.
|
|
pub fn preview_grouping(
|
|
catalog: &Catalog,
|
|
model_id: &str,
|
|
grouping: &FaceSettings,
|
|
) -> Result<GroupingPreview, dr_catalog::CatalogError> {
|
|
let min_group = grouping.min_group_size.max(1) as usize;
|
|
let pop = Population::read(catalog, model_id)?;
|
|
if pop.candidates.is_empty() {
|
|
return Ok(GroupingPreview::default());
|
|
}
|
|
|
|
let clusters = dr_face::cluster(&pop.candidates, &pop.cal, grouping.merge_probability);
|
|
// The same rule the writing pass applies, so the preview and the result
|
|
// cannot disagree about what counts as a person.
|
|
let kept: Vec<_> = clusters
|
|
.iter()
|
|
.filter(|c| c.members.len() >= min_group || c.person.is_some())
|
|
.collect();
|
|
|
|
Ok(GroupingPreview {
|
|
faces: pop.candidates.len(),
|
|
groups: kept.len(),
|
|
grouped: kept.iter().map(|c| c.members.len()).sum(),
|
|
largest: kept.iter().map(|c| c.members.len()).max().unwrap_or(0),
|
|
})
|
|
}
|
|
|
|
/// Group the library's faces into people, writing suggestions.
|
|
///
|
|
/// Confirmations are never touched: they enter the clusterer as anchors and
|
|
/// come back out unchanged, which is the invariant FR-CULL-10 turns on. What
|
|
/// this writes is the *suggested* half, and it may be re-run at any time.
|
|
///
|
|
/// Returns how many suggestions were written and how many new unnamed people
|
|
/// were created.
|
|
pub fn recluster(
|
|
catalog: &Catalog,
|
|
model_id: &str,
|
|
grouping: &FaceSettings,
|
|
) -> Result<(usize, usize), dr_catalog::CatalogError> {
|
|
let min_probability = grouping.merge_probability;
|
|
let min_group = grouping.min_group_size.max(1) as usize;
|
|
let conn = catalog.connection();
|
|
|
|
let Population {
|
|
cal,
|
|
candidates,
|
|
ids,
|
|
anchors: confirmed,
|
|
} = Population::read(catalog, model_id)?;
|
|
if candidates.is_empty() {
|
|
return Ok((0, 0));
|
|
}
|
|
|
|
let dr_face::Grouping {
|
|
clusters,
|
|
confidence,
|
|
} = dr_face::cluster_scored(&candidates, &cal, min_probability);
|
|
|
|
let mut suggested = 0usize;
|
|
let mut created = 0usize;
|
|
// Every face this pass actually placed. What is *not* in here at the end is
|
|
// a face the previous pass had an opinion about and this one does not, and
|
|
// it has to be let go — see below.
|
|
let mut placed = std::collections::HashSet::with_capacity(ids.len());
|
|
for c in &clusters {
|
|
// A group of one is not a person, and the user says how much bigger
|
|
// than one it has to be (`FaceSettings::min_group_size`). Naming every
|
|
// stray face would fill the People view with noise they then have to
|
|
// dismiss one entry at a time.
|
|
//
|
|
// Only ever applied to a group the system invented. A group with a
|
|
// `person` is one the user has already confirmed, named or set aside,
|
|
// and a display preference does not overrule a judgement (FR-CULL-12).
|
|
if c.members.len() < min_group && c.person.is_none() {
|
|
continue;
|
|
}
|
|
|
|
let person = match c.person {
|
|
Some(p) => faces::PersonId(p),
|
|
None => {
|
|
created += 1;
|
|
// Unnamed: FR-CULL-10 has the user name a group, and a group
|
|
// the system named would be a guess wearing a fact's clothes.
|
|
faces::create_person(conn, "")?
|
|
}
|
|
};
|
|
|
|
for &m in &c.members {
|
|
let face = ids[m];
|
|
placed.insert(face);
|
|
if confirmed.contains_key(&face) {
|
|
continue;
|
|
}
|
|
// The probability the user is shown is this identity's share of
|
|
// the evidence for the face, against every other identity that
|
|
// could plausibly claim it (dr_face::assign) — not the single best
|
|
// edge, which cannot tell a sole match from a coin toss between
|
|
// two siblings.
|
|
let p = confidence[m];
|
|
if faces::suggest(conn, face, person, p)? {
|
|
suggested += 1;
|
|
}
|
|
}
|
|
}
|
|
|
|
// Faces the previous pass placed and this one did not.
|
|
//
|
|
// Without this the parameters above are only half connected to the screen.
|
|
// Raise the smallest group to three and the pass stops *creating* groups of
|
|
// two — but last pass's group of two still holds its two suggestions, so it
|
|
// is not empty, so the prune below leaves it, and the rail does not change.
|
|
// The setting would appear to do nothing until the library was reindexed.
|
|
//
|
|
// Suggestions are this pass's own output (FR-CULL-10), so withdrawing one it
|
|
// no longer stands behind is exactly what it is entitled to do. Anchors are
|
|
// skipped by construction: a confirmed, named or ignored face is in
|
|
// `confirmed`, enters as an anchor, and comes back out inside a group that
|
|
// is never dropped.
|
|
let mut released = 0usize;
|
|
for face in &ids {
|
|
if placed.contains(face) || confirmed.contains_key(face) {
|
|
continue;
|
|
}
|
|
faces::unassign(conn, *face)?;
|
|
released += 1;
|
|
}
|
|
if released > 0 {
|
|
log::info!("reclustering released {released} face(s) it no longer groups");
|
|
}
|
|
|
|
// Groups the *previous* pass created that this one left empty. Without
|
|
// this, every press of Regroup adds a rail entry per group it no longer
|
|
// believes in, and the screen fills with "Unnamed (0 faces)" — which is
|
|
// what made pressing the button twice look like it had broken something.
|
|
match faces::prune_empty_unnamed(conn) {
|
|
Ok(0) => {}
|
|
Ok(n) => log::info!("reclustering removed {n} empty group(s) from the previous pass"),
|
|
Err(e) => log::warn!("pruning empty groups: {e}"),
|
|
}
|
|
|
|
log::info!(
|
|
"reclustered {} face(s) into {} group(s): {suggested} suggestion(s), {created} new",
|
|
candidates.len(),
|
|
clusters.len()
|
|
);
|
|
Ok((suggested, created))
|
|
}
|
|
|
|
/// The answer from a preview pass.
|
|
#[derive(Debug, Clone, PartialEq)]
|
|
pub enum PreviewMessage {
|
|
Ready(GroupingPreview),
|
|
Failed(String),
|
|
}
|
|
|
|
/// Report what a grouping pass would do, **on a worker thread**.
|
|
///
|
|
/// Same reasoning as [`spawn_recluster`], and the same cost: a preview is a
|
|
/// clustering pass that throws its answer away, so it is exactly as unbounded
|
|
/// as the pass it is previewing and exactly as unwelcome on the UI thread.
|
|
///
|
|
/// Cancellation is dropping the receiver — so moving the slider again while one
|
|
/// is in flight abandons it, which is the behaviour a dial with a preview
|
|
/// button needs.
|
|
pub fn spawn_grouping_preview(
|
|
catalog_path: PathBuf,
|
|
model_id: String,
|
|
grouping: FaceSettings,
|
|
) -> Receiver<PreviewMessage> {
|
|
let (tx, rx) = std::sync::mpsc::channel();
|
|
|
|
std::thread::spawn(move || {
|
|
let msg = match Catalog::open(&catalog_path) {
|
|
Ok(catalog) => match preview_grouping(&catalog, &model_id, &grouping) {
|
|
Ok(p) => PreviewMessage::Ready(p),
|
|
Err(e) => PreviewMessage::Failed(e.to_string()),
|
|
},
|
|
Err(e) => PreviewMessage::Failed(format!("cannot open catalog: {e}")),
|
|
};
|
|
let _ = tx.send(msg);
|
|
});
|
|
|
|
rx
|
|
}
|
|
|
|
/// Progress from a regrouping pass.
|
|
#[derive(Debug, Clone, PartialEq)]
|
|
pub enum ReclusterMessage {
|
|
/// How many faces went in. Sent once, before the arithmetic starts.
|
|
Started { faces: usize },
|
|
/// It finished.
|
|
Finished { suggested: usize, created: usize },
|
|
/// It did not.
|
|
Failed(String),
|
|
}
|
|
|
|
/// Group the library's faces into people, **on a worker thread**.
|
|
///
|
|
/// The reason this exists rather than callers just invoking [`recluster`]: it
|
|
/// used to run inside the Slint callback, on the UI thread, and clustering a
|
|
/// real library is not something a callback can do. The window froze for as
|
|
/// long as it took, with no progress, no cancel and no repaint — the button
|
|
/// looked broken because from the outside it was indistinguishable from broken.
|
|
///
|
|
/// It is much faster now (see [`dr_face::cluster`]), but *fast* is not the same
|
|
/// as *bounded*: the work grows with the library and the one thing that must
|
|
/// not grow with the library is how long the window stops answering. So it runs
|
|
/// where every other long pass in this module runs.
|
|
///
|
|
/// Cancellation is dropping the receiver, exactly as with the indexing sweep.
|
|
/// Nothing is left half-written: [`recluster`] does its work in the catalog's
|
|
/// own transactions, and a pass abandoned partway simply leaves the previous
|
|
/// grouping in place to be redone.
|
|
pub fn spawn_recluster(
|
|
catalog_path: PathBuf,
|
|
model_id: String,
|
|
grouping: FaceSettings,
|
|
) -> Receiver<ReclusterMessage> {
|
|
let (tx, rx) = std::sync::mpsc::channel();
|
|
|
|
std::thread::spawn(move || {
|
|
let catalog = match Catalog::open(&catalog_path) {
|
|
Ok(c) => c,
|
|
Err(e) => {
|
|
let _ = tx.send(ReclusterMessage::Failed(format!(
|
|
"cannot open catalog: {e}"
|
|
)));
|
|
return;
|
|
}
|
|
};
|
|
|
|
// Announced before the work so the screen can say what it is chewing
|
|
// on. Cheap: it is a count, not the embeddings themselves.
|
|
let count = faces::embeddings(catalog.connection(), &model_id)
|
|
.map(|e| e.len())
|
|
.unwrap_or(0);
|
|
if tx.send(ReclusterMessage::Started { faces: count }).is_err() {
|
|
return;
|
|
}
|
|
|
|
let msg = match recluster(&catalog, &model_id, &grouping) {
|
|
Ok((suggested, created)) => ReclusterMessage::Finished { suggested, created },
|
|
Err(e) => ReclusterMessage::Failed(e.to_string()),
|
|
};
|
|
let _ = tx.send(msg);
|
|
});
|
|
|
|
rx
|
|
}
|
|
|
|
/// A face cut out of its photograph, ready to draw.
|
|
#[derive(Debug, Clone, PartialEq)]
|
|
pub struct FaceCrop {
|
|
pub width: u32,
|
|
pub height: u32,
|
|
/// Tightly packed RGBA.
|
|
pub rgba: Vec<u8>,
|
|
}
|
|
|
|
/// How much of the surrounding frame a face crop keeps, per side.
|
|
///
|
|
/// A face cut exactly to its detection box reads as a mugshot: no hair, no
|
|
/// chin, no context, and a row of them is genuinely hard to tell apart — which
|
|
/// matters, because telling them apart is the entire task the People screen
|
|
/// asks of the user. A third on each side gives back the head.
|
|
const CROP_MARGIN: f32 = 0.35;
|
|
|
|
/// Cut one face out of its proxy.
|
|
///
|
|
/// The box is normalised to the long edge (the catalog's convention), so this
|
|
/// works whatever size the proxy happens to be now — the property that made
|
|
/// normalising worth the trouble. The thumbnail cache is entitled to evict a
|
|
/// proxy and regenerate it at another resolution, and a face stored in pixels
|
|
/// would then point at the wrong part of the picture.
|
|
pub fn crop_face(
|
|
rgba: &[u8],
|
|
width: u32,
|
|
height: u32,
|
|
face: &faces::Face,
|
|
out_edge: u32,
|
|
) -> Option<FaceCrop> {
|
|
if width == 0 || height == 0 || out_edge == 0 {
|
|
return None;
|
|
}
|
|
let long_edge = width.max(height) as f32;
|
|
|
|
// Square, centred on the face: the grid draws square cells, and cropping to
|
|
// a square here rather than letterboxing there means the face fills the
|
|
// cell instead of floating in it.
|
|
let cx = (face.x + face.w * 0.5) * long_edge;
|
|
let cy = (face.y + face.h * 0.5) * long_edge;
|
|
let half = (face.w.max(face.h) * long_edge * 0.5) * (1.0 + CROP_MARGIN);
|
|
if !(half.is_finite() && half > 0.5) {
|
|
return None;
|
|
}
|
|
|
|
let mut out = vec![0u8; (out_edge * out_edge * 4) as usize];
|
|
let step = (half * 2.0) / out_edge as f32;
|
|
|
|
for oy in 0..out_edge {
|
|
let sy = cy - half + (oy as f32 + 0.5) * step;
|
|
for ox in 0..out_edge {
|
|
let sx = cx - half + (ox as f32 + 0.5) * step;
|
|
let o = ((oy * out_edge + ox) * 4) as usize;
|
|
// Nearest neighbour: this is a downscale of an already-small proxy
|
|
// shown at ~96 px, and a bilinear tap would cost four reads per
|
|
// pixel for a difference nobody can see at that size. Outside the
|
|
// frame stays transparent, so a face at the very edge of the
|
|
// picture is drawn short rather than smeared.
|
|
if sx < 0.0 || sy < 0.0 || sx >= width as f32 || sy >= height as f32 {
|
|
continue;
|
|
}
|
|
let i = ((sy as u32 * width + sx as u32) * 4) as usize;
|
|
if i + 4 <= rgba.len() {
|
|
out[o..o + 4].copy_from_slice(&rgba[i..i + 4]);
|
|
}
|
|
}
|
|
}
|
|
|
|
Some(FaceCrop {
|
|
width: out_edge,
|
|
height: out_edge,
|
|
rgba: out,
|
|
})
|
|
}
|
|
|
|
/// Edge of the crop stored with a face.
|
|
///
|
|
/// Above both [`crate::identity::FACE_CROP_EDGE`] (128) and
|
|
/// `COVER_CROP_EDGE` (80), so the stored image is downsampled to draw and never
|
|
/// upsampled — a stored crop the same size as the grid cell would go soft the
|
|
/// moment either constant grew. 160 px of JPEG is a few KB, which is nothing
|
|
/// beside the 250 KB proxy it saves decoding.
|
|
pub const STORED_CROP_EDGE: u32 = 160;
|
|
|
|
/// Cut one detected face out of the buffer it was found in, as a JPEG.
|
|
///
|
|
/// The same square [`crop_face`] would cut — centred on the box, widened by
|
|
/// [`CROP_MARGIN`] — so a face drawn from its stored crop and one drawn the old
|
|
/// way from the proxy are the same picture. Working in pixels rather than
|
|
/// normalised coordinates because at this point in the pipeline that is what
|
|
/// there is; the normalising happens afterwards.
|
|
///
|
|
/// **Out-of-frame samples clamp to the edge rather than going transparent.**
|
|
/// [`crop_face`] leaves them clear, which is right when the caller can composite
|
|
/// them; JPEG has no alpha, so the same choice here would bake a black bar into
|
|
/// every face near the edge of its photograph — and then drag that face's mean
|
|
/// luma down far enough for `load_cover` to reject it as too dark.
|
|
///
|
|
/// `None` where the geometry is degenerate, which is the caller's cue to store
|
|
/// nothing and fall back to the proxy.
|
|
fn cut_crop(rgb: &[f32], width: usize, height: usize, d: &Detection) -> Option<Vec<u8>> {
|
|
if width == 0 || height == 0 {
|
|
return None;
|
|
}
|
|
let cx = d.bbox.0 + d.width() * 0.5;
|
|
let cy = d.bbox.1 + d.height() * 0.5;
|
|
let half = d.width().max(d.height()) * 0.5 * (1.0 + CROP_MARGIN);
|
|
if !(half.is_finite() && half > 0.5 && cx.is_finite() && cy.is_finite()) {
|
|
return None;
|
|
}
|
|
|
|
let edge = STORED_CROP_EDGE;
|
|
let mut out = vec![0u8; (edge * edge * 4) as usize];
|
|
let step = (half * 2.0) / edge as f32;
|
|
|
|
for oy in 0..edge {
|
|
let sy = cy - half + (oy as f32 + 0.5) * step;
|
|
let sy = (sy.max(0.0) as usize).min(height - 1);
|
|
for ox in 0..edge {
|
|
let sx = cx - half + (ox as f32 + 0.5) * step;
|
|
let sx = (sx.max(0.0) as usize).min(width - 1);
|
|
|
|
let i = (sy * width + sx) * 3;
|
|
let o = ((oy * edge + ox) * 4) as usize;
|
|
if i + 3 > rgb.len() {
|
|
continue;
|
|
}
|
|
for c in 0..3 {
|
|
out[o + c] = (rgb[i + c].clamp(0.0, 1.0) * 255.0).round() as u8;
|
|
}
|
|
out[o + 3] = 255;
|
|
}
|
|
}
|
|
|
|
match dr_thumbs::encode_rgba(edge, edge, &out) {
|
|
Ok(bytes) => Some(bytes),
|
|
Err(e) => {
|
|
log::debug!("encoding a face crop: {e}");
|
|
None
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Detect and embed every face in a decoded preview.
|
|
///
|
|
/// The counterpart to [`index_proxy`] for the fetching sweep, which holds a
|
|
/// `dr_decode::Preview` rather than a thumbnail out of the store. The preview
|
|
/// arrives **already turned the right way up** — `fetch_preview` applies the
|
|
/// orientation before returning — so the coordinates this produces are in the
|
|
/// photograph's space, which is the space the catalog stores and the develop
|
|
/// overlay draws in. Nothing here has to know about the sensor.
|
|
///
|
|
/// Returns the faces and the long edge they were normalised against, which is
|
|
/// what `record_detections` stores so a proxy regenerated at another size does
|
|
/// not move them.
|
|
pub fn index_preview(
|
|
detector: &mut Detector,
|
|
embedder: &mut Embedder,
|
|
preview: &dr_decode::Preview,
|
|
options: &DetectOptions,
|
|
) -> Result<(Vec<DetectedFace>, u32), dr_face::FaceError> {
|
|
let rgb = rgba_to_rgb_f32(&preview.rgba);
|
|
let faces = index_proxy(
|
|
detector,
|
|
embedder,
|
|
&rgb,
|
|
preview.width as usize,
|
|
preview.height as usize,
|
|
options,
|
|
)?;
|
|
Ok((faces, preview.width.max(preview.height)))
|
|
}
|
|
|
|
/// `dr-thumbs` decodes to RGBA; `dr-face` reads packed `f32` RGB.
|
|
fn rgba_to_rgb_f32(rgba: &[u8]) -> Vec<f32> {
|
|
let mut out = Vec::with_capacity(rgba.len() / 4 * 3);
|
|
for px in rgba.chunks_exact(4) {
|
|
out.push(px[0] as f32 / 255.0);
|
|
out.push(px[1] as f32 / 255.0);
|
|
out.push(px[2] as f32 / 255.0);
|
|
}
|
|
out
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn an_audit_summary_names_both_kinds_of_outstanding() {
|
|
let a = IndexAudit {
|
|
coverage: faces::Coverage {
|
|
images: 100,
|
|
indexed: 60,
|
|
without_faces: 40,
|
|
faces: 35,
|
|
},
|
|
ready: 30,
|
|
awaiting_proxy: 10,
|
|
};
|
|
let s = a.summary();
|
|
assert!(s.contains("60/100"), "{s}");
|
|
assert!(s.contains("60%"), "{s}");
|
|
assert!(!s.contains("60.0%"), "whole numbers above ten percent: {s}");
|
|
assert!(s.contains("30 ready"), "{s}");
|
|
// "to fetch", not "awaiting a proxy": the whole-library pass fetches
|
|
// these rather than being blocked by them, and the line must not send
|
|
// the user off to run the thumbnail sweep first.
|
|
assert!(s.contains("10 to fetch"), "{s}");
|
|
}
|
|
|
|
#[test]
|
|
fn a_complete_audit_mentions_neither() {
|
|
let a = IndexAudit {
|
|
coverage: faces::Coverage {
|
|
images: 10,
|
|
indexed: 10,
|
|
without_faces: 7,
|
|
faces: 4,
|
|
},
|
|
ready: 0,
|
|
awaiting_proxy: 0,
|
|
};
|
|
let s = a.summary();
|
|
assert!(!s.contains("ready"), "{s}");
|
|
assert!(!s.contains("awaiting"), "{s}");
|
|
assert!(s.contains("10/10"), "{s}");
|
|
}
|
|
|
|
/// The figure the real library actually produced: 110 of 23,528 rounds to
|
|
/// "0%" at whole-number precision, which reads as nothing having happened.
|
|
#[test]
|
|
fn early_progress_does_not_display_as_zero() {
|
|
let a = IndexAudit {
|
|
coverage: faces::Coverage {
|
|
images: 23_528,
|
|
indexed: 110,
|
|
without_faces: 64,
|
|
faces: 125,
|
|
},
|
|
ready: 69,
|
|
awaiting_proxy: 23_349,
|
|
};
|
|
let s = a.summary();
|
|
assert!(s.contains("0.5%"), "{s}");
|
|
assert!(!s.contains("(0%)"), "{s}");
|
|
}
|
|
|
|
#[test]
|
|
fn a_single_image_in_a_huge_library_still_shows_something() {
|
|
let a = IndexAudit {
|
|
coverage: faces::Coverage {
|
|
images: 100_000,
|
|
indexed: 1,
|
|
without_faces: 1,
|
|
faces: 0,
|
|
},
|
|
ready: 99_999,
|
|
awaiting_proxy: 0,
|
|
};
|
|
assert!(a.summary().contains("<0.1%"), "{}", a.summary());
|
|
}
|
|
|
|
#[test]
|
|
fn an_empty_library_says_so_rather_than_reporting_zero_of_zero() {
|
|
assert_eq!(IndexAudit::default().summary(), "no images in the library");
|
|
}
|
|
|
|
#[test]
|
|
fn landmarks_normalise_against_the_long_edge() {
|
|
let lm = [
|
|
(512.0, 256.0),
|
|
(0.0, 0.0),
|
|
(1024.0, 512.0),
|
|
(10.0, 20.0),
|
|
(5.0, 5.0),
|
|
];
|
|
let n = normalise_landmarks(&lm, 1024.0);
|
|
assert!((n[0].0 - 0.5).abs() < 1e-6);
|
|
assert!((n[0].1 - 0.25).abs() < 1e-6);
|
|
assert!((n[2].0 - 1.0).abs() < 1e-6);
|
|
}
|
|
|
|
fn gradient(width: u32, height: u32) -> Vec<u8> {
|
|
let mut v = vec![0u8; (width * height * 4) as usize];
|
|
for (i, px) in v.chunks_exact_mut(4).enumerate() {
|
|
// A gradient, so a mis-placed crop shows up as the wrong value
|
|
// rather than as more of the same colour.
|
|
px[0] = (i % 251) as u8;
|
|
px[1] = 40;
|
|
px[2] = 90;
|
|
px[3] = 255;
|
|
}
|
|
v
|
|
}
|
|
|
|
fn stored_face(x: f32, y: f32, w: f32, h: f32) -> faces::Face {
|
|
faces::Face {
|
|
id: faces::FaceId(1),
|
|
image_id: ImageId(1),
|
|
x,
|
|
y,
|
|
w,
|
|
h,
|
|
landmarks: [(0.0, 0.0); 5],
|
|
confidence: 0.9,
|
|
crop_px: 120.0,
|
|
model_id: "w600k_mbf".into(),
|
|
person: None,
|
|
probability: 0.0,
|
|
confirmed: false,
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn a_face_crop_is_square_and_the_size_asked_for() {
|
|
let px = gradient(1024, 683);
|
|
let c = crop_face(&px, 1024, 683, &stored_face(0.3, 0.2, 0.1, 0.15), 96).unwrap();
|
|
assert_eq!((c.width, c.height), (96, 96));
|
|
assert_eq!(c.rgba.len(), 96 * 96 * 4);
|
|
}
|
|
|
|
#[test]
|
|
fn a_degenerate_box_yields_no_crop_rather_than_a_panic() {
|
|
let px = gradient(64, 64);
|
|
assert!(crop_face(&px, 64, 64, &stored_face(0.5, 0.5, 0.0, 0.0), 96).is_none());
|
|
assert!(crop_face(&px, 0, 0, &stored_face(0.1, 0.1, 0.2, 0.2), 96).is_none());
|
|
assert!(crop_face(&px, 64, 64, &stored_face(0.1, 0.1, 0.2, 0.2), 0).is_none());
|
|
}
|
|
|
|
/// A face at the very edge of the frame is drawn short, not smeared: the
|
|
/// out-of-frame margin stays transparent.
|
|
#[test]
|
|
fn a_face_at_the_edge_keeps_a_transparent_margin() {
|
|
let px = gradient(200, 200);
|
|
let c = crop_face(&px, 200, 200, &stored_face(0.0, 0.0, 0.1, 0.1), 32).unwrap();
|
|
assert_eq!(c.rgba[3], 0, "outside the frame should be transparent");
|
|
let centre = ((16 * 32 + 16) * 4 + 3) as usize;
|
|
assert_eq!(c.rgba[centre], 255, "the face itself should be opaque");
|
|
}
|
|
|
|
/// The normalised box means a proxy regenerated at another resolution still
|
|
/// crops the same part of the picture — what the catalog's normalisation is
|
|
/// for.
|
|
#[test]
|
|
fn the_same_face_crops_the_same_region_at_two_proxy_sizes() {
|
|
let face = stored_face(0.25, 0.25, 0.2, 0.2);
|
|
let small = crop_face(&gradient(400, 400), 400, 400, &face, 16).unwrap();
|
|
let large = crop_face(&gradient(800, 800), 800, 800, &face, 16).unwrap();
|
|
assert!(small.rgba.chunks_exact(4).all(|p| p[3] == 255));
|
|
assert!(large.rgba.chunks_exact(4).all(|p| p[3] == 255));
|
|
}
|
|
|
|
#[test]
|
|
fn the_crop_keeps_margin_around_the_detection_box() {
|
|
// A 0.1-wide face in a 1000px frame is 100px; with the margin the crop
|
|
// spans 100 * 1.35 = 135px of source.
|
|
let face = stored_face(0.4, 0.4, 0.1, 0.1);
|
|
let c = crop_face(&gradient(1000, 1000), 1000, 1000, &face, 135).unwrap();
|
|
assert_eq!(c.width, 135);
|
|
// Fully inside the frame, so nothing is transparent.
|
|
assert!(c.rgba.chunks_exact(4).all(|p| p[3] == 255));
|
|
}
|
|
|
|
#[test]
|
|
fn rgba_becomes_packed_rgb_dropping_alpha() {
|
|
let rgba = [255u8, 128, 0, 255, 0, 0, 0, 128];
|
|
let rgb = rgba_to_rgb_f32(&rgba);
|
|
assert_eq!(rgb.len(), 6);
|
|
assert!((rgb[0] - 1.0).abs() < 1e-6);
|
|
assert!((rgb[1] - 128.0 / 255.0).abs() < 1e-6);
|
|
assert!((rgb[2] - 0.0).abs() < 1e-6);
|
|
assert!(rgb[3..6].iter().all(|&v| v == 0.0));
|
|
}
|
|
|
|
// ── setting a group aside has to survive regrouping ───────────────────
|
|
|
|
const TEST_MODEL: &str = "w600k_mbf";
|
|
|
|
/// A catalog holding `n` images and nothing else.
|
|
fn catalog_with(n: usize) -> Catalog {
|
|
let c = Catalog::in_memory().unwrap();
|
|
let conn = c.connection();
|
|
conn.execute(
|
|
"INSERT OR IGNORE INTO roots(id, kind, label) VALUES (1, 'local', 'lib')",
|
|
[],
|
|
)
|
|
.unwrap();
|
|
for i in 0..n {
|
|
conn.execute(
|
|
&format!(
|
|
"INSERT INTO images(id, root_id, source_ref, added_at)
|
|
VALUES ({}, 1, 'IMG_{i}.CR3', 0)",
|
|
i + 1
|
|
),
|
|
[],
|
|
)
|
|
.unwrap();
|
|
}
|
|
c
|
|
}
|
|
|
|
/// A unit embedding pointing at `identity`, `cosine` of the way there.
|
|
fn embedding(identity: usize, cosine: f32) -> Vec<u8> {
|
|
let mut v = Box::new([0.0_f32; dr_face::EMBEDDING_DIM]);
|
|
v[identity * 2] = cosine;
|
|
v[identity * 2 + 1] = (1.0 - cosine * cosine).max(0.0).sqrt();
|
|
dr_face::Embedding {
|
|
model: ModelId::new(TEST_MODEL.to_string()),
|
|
v,
|
|
}
|
|
.to_f16_bytes()
|
|
}
|
|
|
|
fn put_face(catalog: &Catalog, image: u64, identity: usize, cosine: f32) {
|
|
let f = DetectedFace {
|
|
x: 0.1,
|
|
y: 0.1,
|
|
w: 0.2,
|
|
h: 0.2,
|
|
landmarks: [(0.0, 0.0); 5],
|
|
confidence: 0.9,
|
|
embedding: embedding(identity, cosine),
|
|
crop_px: 150.0,
|
|
model_id: TEST_MODEL.to_string(),
|
|
crop: Vec::new(),
|
|
};
|
|
faces::record_detections(
|
|
catalog.connection(),
|
|
ImageId(image),
|
|
TEST_MODEL,
|
|
1024,
|
|
std::slice::from_ref(&f),
|
|
)
|
|
.unwrap();
|
|
}
|
|
|
|
/// The bug this pins: "not interested" only ever covered *suggested* faces,
|
|
/// and reclustering anchored confirmations alone. So the next Regroup cut
|
|
/// the ignored group's faces loose, built fresh unnamed groups out of them,
|
|
/// and every stranger the user had dismissed came straight back.
|
|
#[test]
|
|
fn a_group_set_aside_does_not_come_back_on_the_next_regroup() {
|
|
let catalog = catalog_with(3);
|
|
put_face(&catalog, 1, 0, 1.0);
|
|
put_face(&catalog, 2, 0, 0.99);
|
|
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
let people = faces::people(catalog.connection()).unwrap();
|
|
assert_eq!(people.len(), 1, "the two faces should have grouped");
|
|
let stranger = people[0].id;
|
|
assert_eq!(people[0].suggested_faces, 2);
|
|
|
|
faces::set_ignored(catalog.connection(), stranger, true).unwrap();
|
|
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
|
|
let after = faces::people(catalog.connection()).unwrap();
|
|
assert_eq!(
|
|
after.len(),
|
|
1,
|
|
"regrouping resurrected the group that was set aside: {after:?}"
|
|
);
|
|
assert_eq!(after[0].id, stranger);
|
|
assert!(after[0].ignored, "the group stopped being set aside");
|
|
assert_eq!(
|
|
after[0].suggested_faces, 2,
|
|
"the faces left the group they were set aside in"
|
|
);
|
|
}
|
|
|
|
/// And it holds as the library grows: a stranger photographed again joins
|
|
/// the group that was set aside rather than arriving as somebody new.
|
|
#[test]
|
|
fn a_new_face_joins_the_group_it_matches_even_when_that_group_is_set_aside() {
|
|
let catalog = catalog_with(3);
|
|
put_face(&catalog, 1, 0, 1.0);
|
|
put_face(&catalog, 2, 0, 0.99);
|
|
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
let stranger = faces::people(catalog.connection()).unwrap()[0].id;
|
|
faces::set_ignored(catalog.connection(), stranger, true).unwrap();
|
|
|
|
// The same person turns up in a third photograph.
|
|
put_face(&catalog, 3, 0, 0.98);
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
|
|
let after = faces::people(catalog.connection()).unwrap();
|
|
assert_eq!(after.len(), 1, "a new face made a second group: {after:?}");
|
|
assert!(after[0].ignored);
|
|
assert_eq!(after[0].suggested_faces, 3);
|
|
}
|
|
|
|
/// The other half of the promise: bringing them back really does.
|
|
#[test]
|
|
fn bringing_a_group_back_makes_it_ordinary_again() {
|
|
let catalog = catalog_with(3);
|
|
put_face(&catalog, 1, 0, 1.0);
|
|
put_face(&catalog, 2, 0, 0.99);
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
|
|
let id = faces::people(catalog.connection()).unwrap()[0].id;
|
|
faces::set_ignored(catalog.connection(), id, true).unwrap();
|
|
faces::set_ignored(catalog.connection(), id, false).unwrap();
|
|
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
let after = faces::people(catalog.connection()).unwrap();
|
|
assert_eq!(after.len(), 1);
|
|
assert!(!after[0].ignored);
|
|
}
|
|
|
|
/// Naming a group does not confirm its faces, so before this they were
|
|
/// still only suggestions — and the next Regroup cut them loose, built a
|
|
/// new person out of them, and left the named one empty. Do that a few
|
|
/// times and the rail fills with same-named people holding nothing while
|
|
/// the faces sit under whichever one was made last.
|
|
#[test]
|
|
fn a_named_group_keeps_its_faces_through_the_next_regroup() {
|
|
let catalog = catalog_with(3);
|
|
put_face(&catalog, 1, 0, 1.0);
|
|
put_face(&catalog, 2, 0, 0.99);
|
|
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
let people = faces::people(catalog.connection()).unwrap();
|
|
assert_eq!(people.len(), 1);
|
|
let her = people[0].id;
|
|
assert_eq!(people[0].suggested_faces, 2);
|
|
|
|
// Named, and nothing else — no confirmations, which is what a user who
|
|
// types a name and moves on has done.
|
|
faces::rename_person(catalog.connection(), her, "Catherine").unwrap();
|
|
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
|
|
let after = faces::people(catalog.connection()).unwrap();
|
|
assert_eq!(
|
|
after.len(),
|
|
1,
|
|
"regrouping left a second person behind: {after:?}"
|
|
);
|
|
assert_eq!(after[0].id, her);
|
|
assert_eq!(after[0].name, "Catherine");
|
|
assert_eq!(
|
|
after[0].suggested_faces, 2,
|
|
"the named group lost the faces it was named for"
|
|
);
|
|
}
|
|
|
|
/// And a face found later joins the person it matches rather than arriving
|
|
/// as somebody new — the same benefit anchoring gives an ignored group.
|
|
#[test]
|
|
fn a_new_face_joins_a_named_person_rather_than_starting_a_rival() {
|
|
let catalog = catalog_with(3);
|
|
put_face(&catalog, 1, 0, 1.0);
|
|
put_face(&catalog, 2, 0, 0.99);
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
let her = faces::people(catalog.connection()).unwrap()[0].id;
|
|
faces::rename_person(catalog.connection(), her, "Catherine").unwrap();
|
|
|
|
put_face(&catalog, 3, 0, 0.98);
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
|
|
let after = faces::people(catalog.connection()).unwrap();
|
|
assert_eq!(after.len(), 1, "a second Catherine appeared: {after:?}");
|
|
assert_eq!(after[0].suggested_faces, 3);
|
|
}
|
|
|
|
/// `dr-types` sits below the face engine and cannot name its constant, so
|
|
/// it restates the number. This is the thing that stops the two drifting:
|
|
/// a settings page marking 0.80 as "default" while the engine had moved to
|
|
/// 0.75 would put the reset dot on a value nothing else agreed with.
|
|
#[test]
|
|
fn the_settings_default_is_the_engines_own_tuned_value() {
|
|
assert_eq!(
|
|
FaceSettings::default().merge_probability,
|
|
dr_face::DEFAULT_MERGE_PROBABILITY
|
|
);
|
|
}
|
|
|
|
/// The smallest-group setting has to change what is on the rail, not just
|
|
/// what the *next* pass would build. Before the release step in
|
|
/// [`recluster`], raising it left the previous pass's small groups sitting
|
|
/// there full of suggestions — not empty, so not pruned — and the control
|
|
/// looked broken.
|
|
#[test]
|
|
fn raising_the_smallest_group_takes_the_small_groups_off_the_rail() {
|
|
let catalog = catalog_with(5);
|
|
// One pair, and one trio: two groups at the default of two.
|
|
put_face(&catalog, 1, 0, 1.0);
|
|
put_face(&catalog, 2, 0, 0.99);
|
|
put_face(&catalog, 3, 1, 1.0);
|
|
put_face(&catalog, 4, 1, 0.99);
|
|
put_face(&catalog, 5, 1, 0.98);
|
|
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
assert_eq!(
|
|
faces::people(catalog.connection()).unwrap().len(),
|
|
2,
|
|
"the two groups should have formed at the default"
|
|
);
|
|
|
|
recluster(
|
|
&catalog,
|
|
TEST_MODEL,
|
|
&FaceSettings {
|
|
min_group_size: 3,
|
|
..FaceSettings::default()
|
|
},
|
|
)
|
|
.unwrap();
|
|
|
|
let after = faces::people(catalog.connection()).unwrap();
|
|
assert_eq!(
|
|
after.len(),
|
|
1,
|
|
"the pair survived a smallest-group of three: {after:?}"
|
|
);
|
|
assert_eq!(after[0].suggested_faces, 3, "the trio lost members");
|
|
}
|
|
|
|
/// And the setting does not overrule the user. A group they named is theirs
|
|
/// (FR-CULL-12), however few faces it holds.
|
|
#[test]
|
|
fn a_named_group_survives_a_smallest_group_it_is_under() {
|
|
let catalog = catalog_with(3);
|
|
put_face(&catalog, 1, 0, 1.0);
|
|
put_face(&catalog, 2, 0, 0.99);
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
let her = faces::people(catalog.connection()).unwrap()[0].id;
|
|
faces::rename_person(catalog.connection(), her, "Catherine").unwrap();
|
|
|
|
recluster(
|
|
&catalog,
|
|
TEST_MODEL,
|
|
&FaceSettings {
|
|
min_group_size: 6,
|
|
..FaceSettings::default()
|
|
},
|
|
)
|
|
.unwrap();
|
|
|
|
let after = faces::people(catalog.connection()).unwrap();
|
|
assert_eq!(after.len(), 1, "Catherine was dropped: {after:?}");
|
|
assert_eq!(after[0].name, "Catherine");
|
|
assert_eq!(after[0].suggested_faces, 2);
|
|
}
|
|
|
|
/// Down at one, every stray becomes a group of its own — which is what a
|
|
/// user hunting for a face the grouping missed has asked for.
|
|
#[test]
|
|
fn a_smallest_group_of_one_shows_the_strays() {
|
|
let catalog = catalog_with(2);
|
|
put_face(&catalog, 1, 0, 1.0);
|
|
put_face(&catalog, 2, 1, 1.0);
|
|
|
|
recluster(&catalog, TEST_MODEL, &FaceSettings::default()).unwrap();
|
|
assert!(
|
|
faces::people(catalog.connection()).unwrap().is_empty(),
|
|
"two unrelated faces made a person at the default"
|
|
);
|
|
|
|
recluster(
|
|
&catalog,
|
|
TEST_MODEL,
|
|
&FaceSettings {
|
|
min_group_size: 1,
|
|
..FaceSettings::default()
|
|
},
|
|
)
|
|
.unwrap();
|
|
assert_eq!(faces::people(catalog.connection()).unwrap().len(), 2);
|
|
}
|
|
}
|