Keep each face's quality, and never compare against a poor one
The embedder's raw output has a length, and the length is a reading of how recognisable the crop was: a blur, an occlusion or a hard profile comes out short. Normalising threw it away. A short vector sits near the middle of the sphere and matches a little of everyone, which is how one bad crop bridges two people in a grouping pass. So the length is kept — the store now holds the raw vector, re-normalised on load, with the length beside it as `faces.quality` — and a face under MIN_GALLERY_QUALITY (14) is a probe: measured against the gallery and placed where it fits, but never what another face is measured against. Two probes are never paired, and a probe is nobody's evidence for a confidence. The People screen shows the number as "Quality 17.3", dimmed below the floor. Faces indexed before this stored unit vectors and have no reading; they are admitted to the gallery, and schema V14 forgets the run marker of every image holding one so the next indexing pass measures them. A peer's unmeasured shard faces are not adopted, or a sync would write that marker back.
This commit is contained in:
+307
-3
@@ -19,6 +19,23 @@
|
||||
//! and clustering never moves it. Two groups holding confirmations of
|
||||
//! *different* people cannot merge, whatever their similarity says.
|
||||
//!
|
||||
//! # The gallery, and the faces that are only ever compared against it
|
||||
//!
|
||||
//! A third defence, and the cheapest of all: **a short embedding is never a
|
||||
//! reference.** The length of the raw vector is the model's own reading of
|
||||
//! how recognisable the crop was ([`crate::embedding::MIN_GALLERY_QUALITY`]),
|
||||
//! and a short one sits near the centre of the sphere, matching a little of
|
||||
//! everybody. One of those in a group is a bridge to the next group over.
|
||||
//!
|
||||
//! So the population is split. Faces at or above the floor are the
|
||||
//! **gallery**, and they cluster exactly as described below. Faces under it
|
||||
//! are **probes**: each is measured against the finished groups and joins the
|
||||
//! one it fits, by the same average-link rule and under the same constraints
|
||||
//! — but it is measured against the gallery members only, never against
|
||||
//! another probe, and once placed it is never part of what the next face is
|
||||
//! measured against. A blurred photograph of a known person is still named;
|
||||
//! it just cannot vouch for anyone else.
|
||||
//!
|
||||
//! # Average link, not single link
|
||||
//!
|
||||
//! Single-link chains: one bad edge welds two identities together, and it is
|
||||
@@ -117,6 +134,11 @@ pub struct Candidate {
|
||||
pub embedding: Vec<f32>,
|
||||
/// Source pixels across the aligned crop, for the calibration's size term.
|
||||
pub crop_px: f32,
|
||||
/// Length of the raw embedding, where it was recorded
|
||||
/// ([`crate::embedding::MIN_GALLERY_QUALITY`]). `None` for a face indexed
|
||||
/// before it was kept, which is admitted to the gallery — see
|
||||
/// [`Candidate::in_gallery`].
|
||||
pub quality: Option<f32>,
|
||||
/// The person this face is *confirmed* to be, if any.
|
||||
///
|
||||
/// Suggestions are deliberately not passed here. They are this function's
|
||||
@@ -125,6 +147,13 @@ pub struct Candidate {
|
||||
pub confirmed_person: Option<u64>,
|
||||
}
|
||||
|
||||
impl Candidate {
|
||||
/// Whether this face may be compared *against*, as well as compared.
|
||||
pub fn in_gallery(&self) -> bool {
|
||||
crate::embedding::in_gallery(self.quality)
|
||||
}
|
||||
}
|
||||
|
||||
/// One group of faces the clusterer believes are one person.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct Cluster {
|
||||
@@ -202,7 +231,7 @@ pub fn cluster_scored(faces: &[Candidate], cal: &Calibration, min_probability: f
|
||||
|
||||
let clusters = build(faces, cal, min_probability, &merges);
|
||||
let confidence = crate::assign::identity_shares(
|
||||
faces.len(),
|
||||
&columns.gallery,
|
||||
&clusters,
|
||||
&evidence,
|
||||
crate::assign::TOP_MATCHES,
|
||||
@@ -225,6 +254,7 @@ struct Columns {
|
||||
dim: usize,
|
||||
crop_px: Vec<f32>,
|
||||
images: Vec<u64>,
|
||||
gallery: Vec<bool>,
|
||||
}
|
||||
|
||||
impl Columns {
|
||||
@@ -245,6 +275,7 @@ impl Columns {
|
||||
dim,
|
||||
crop_px: faces.iter().map(|f| f.crop_px).collect(),
|
||||
images: faces.iter().map(|f| f.image).collect(),
|
||||
gallery: faces.iter().map(Candidate::in_gallery).collect(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -254,22 +285,171 @@ impl Columns {
|
||||
dim: self.dim,
|
||||
crop_px: &self.crop_px,
|
||||
images: &self.images,
|
||||
gallery: &self.gallery,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Agglomerate the gallery over its pairs, then place the probes.
|
||||
///
|
||||
/// `pairs` is what [`neighbours::above_threshold`] returned: every pair has a
|
||||
/// gallery side, but a pair with a probe on the other side is not a merge —
|
||||
/// it is the evidence [`place_probes`] works from. Only the gallery-to-gallery
|
||||
/// pairs reach the engine, so a probe enters it as a singleton with no edges
|
||||
/// and comes out exactly as it went in.
|
||||
fn build(
|
||||
faces: &[Candidate],
|
||||
cal: &Calibration,
|
||||
min_probability: f32,
|
||||
pairs: &[neighbours::Pair],
|
||||
) -> Vec<Cluster> {
|
||||
let gallery: Vec<bool> = faces.iter().map(Candidate::in_gallery).collect();
|
||||
let (merges, probe_pairs): (Vec<_>, Vec<_>) = pairs
|
||||
.iter()
|
||||
.copied()
|
||||
.partition(|p| gallery[p.i] && gallery[p.j]);
|
||||
|
||||
let mut engine = Engine::new(faces, cal, min_probability);
|
||||
let parts = components(faces.len(), pairs);
|
||||
let parts = components(faces.len(), &merges);
|
||||
for (component, edges) in parts.members.iter().zip(&parts.edges) {
|
||||
engine.agglomerate(component, edges);
|
||||
}
|
||||
engine.finish()
|
||||
let dot = engine.dot;
|
||||
let clusters = engine.finish();
|
||||
if probe_pairs.is_empty() {
|
||||
return clusters;
|
||||
}
|
||||
place_probes(
|
||||
faces,
|
||||
cal,
|
||||
min_probability,
|
||||
dot,
|
||||
&gallery,
|
||||
clusters,
|
||||
&probe_pairs,
|
||||
)
|
||||
}
|
||||
|
||||
/// Put each probe into the finished group it fits, or leave it alone.
|
||||
///
|
||||
/// The same decision the engine makes for a singleton — average link over the
|
||||
/// group, at or above `min_probability`, subject to [`Engine::can_link`]'s two
|
||||
/// constraints — with one difference that is the whole point: the average is
|
||||
/// over the group's **gallery** members. A probe already placed is not part of
|
||||
/// what the next one is measured against, so a run of short vectors cannot
|
||||
/// pull each other in one after another.
|
||||
///
|
||||
/// Probes are placed in index order and each placement is final, which is
|
||||
/// what keeps this deterministic. The group a probe joins gains its
|
||||
/// photograph, so a second face from the same frame cannot follow it — the
|
||||
/// co-occurrence rule, applied exactly as the engine applies it.
|
||||
fn place_probes(
|
||||
faces: &[Candidate],
|
||||
cal: &Calibration,
|
||||
min_probability: f32,
|
||||
dot: neighbours::DotFn,
|
||||
gallery: &[bool],
|
||||
mut clusters: Vec<Cluster>,
|
||||
probe_pairs: &[neighbours::Pair],
|
||||
) -> Vec<Cluster> {
|
||||
// Where each face sits, and what each group's photographs and gallery
|
||||
// members are. The probe's own singleton is here too, and is dropped once
|
||||
// it has moved.
|
||||
let mut group_of = vec![usize::MAX; faces.len()];
|
||||
for (g, c) in clusters.iter().enumerate() {
|
||||
for &m in &c.members {
|
||||
group_of[m] = g;
|
||||
}
|
||||
}
|
||||
let mut images: Vec<HashSet<u64>> = clusters
|
||||
.iter()
|
||||
.map(|c| c.members.iter().map(|&m| faces[m].image).collect())
|
||||
.collect();
|
||||
let references: Vec<Vec<usize>> = clusters
|
||||
.iter()
|
||||
.map(|c| c.members.iter().copied().filter(|&m| gallery[m]).collect())
|
||||
.collect();
|
||||
|
||||
// Which groups each probe has any above-threshold pair into. Only those
|
||||
// can average above the threshold — the argument the module note makes
|
||||
// for the engine holds here unchanged.
|
||||
let mut candidates: Vec<Vec<usize>> = vec![Vec::new(); faces.len()];
|
||||
for p in probe_pairs {
|
||||
let (probe, reference) = if gallery[p.i] { (p.j, p.i) } else { (p.i, p.j) };
|
||||
candidates[probe].push(group_of[reference]);
|
||||
}
|
||||
|
||||
let mut moved: Vec<usize> = Vec::new();
|
||||
for probe in 0..faces.len() {
|
||||
if gallery[probe] || candidates[probe].is_empty() {
|
||||
continue;
|
||||
}
|
||||
let mut groups = std::mem::take(&mut candidates[probe]);
|
||||
groups.sort_unstable();
|
||||
groups.dedup();
|
||||
|
||||
let face = &faces[probe];
|
||||
let mut best: Option<(f32, usize)> = None;
|
||||
for g in groups {
|
||||
let target = &clusters[g];
|
||||
if let (Some(mine), Some(theirs)) = (face.confirmed_person, target.person) {
|
||||
if mine != theirs {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if images[g].contains(&face.image) {
|
||||
continue;
|
||||
}
|
||||
let (mut sum, mut count) = (0.0_f64, 0.0_f64);
|
||||
for &r in &references[g] {
|
||||
let cos = dot(&face.embedding, &faces[r].embedding);
|
||||
let min_crop = face.crop_px.min(faces[r].crop_px);
|
||||
sum += cal.probability(cos, min_crop, 0.0) as f64;
|
||||
count += 1.0;
|
||||
}
|
||||
if count == 0.0 {
|
||||
continue;
|
||||
}
|
||||
let p = (sum / count) as f32;
|
||||
// Strictly better wins; on a tie the lowest group index, which is
|
||||
// the engine's own tiebreak.
|
||||
if p >= min_probability && best.is_none_or(|(bp, _)| p > bp) {
|
||||
best = Some((p, g));
|
||||
}
|
||||
}
|
||||
|
||||
let Some((_, g)) = best else { continue };
|
||||
let own = group_of[probe];
|
||||
clusters[g].members.push(probe);
|
||||
clusters[g].members.sort_unstable();
|
||||
clusters[g].person = clusters[g].person.or(face.confirmed_person);
|
||||
images[g].insert(face.image);
|
||||
group_of[probe] = g;
|
||||
moved.push(own);
|
||||
}
|
||||
|
||||
if moved.is_empty() {
|
||||
return clusters;
|
||||
}
|
||||
// The singletons the probes left behind, then the order `Engine::finish`
|
||||
// promises: largest first, lowest member first among equals.
|
||||
let mut vacated = vec![false; clusters.len()];
|
||||
for g in moved {
|
||||
vacated[g] = true;
|
||||
}
|
||||
let mut out: Vec<Cluster> = clusters
|
||||
.into_iter()
|
||||
.zip(vacated)
|
||||
.filter(|(_, gone)| !gone)
|
||||
.map(|(c, _)| c)
|
||||
.collect();
|
||||
out.sort_by(|x, y| {
|
||||
y.members
|
||||
.len()
|
||||
.cmp(&x.members.len())
|
||||
.then(x.members[0].cmp(&y.members[0]))
|
||||
});
|
||||
out
|
||||
}
|
||||
|
||||
/// Split one person's faces into the groups a raised threshold separates them
|
||||
@@ -726,10 +906,19 @@ mod tests {
|
||||
image,
|
||||
embedding: at_cosine(identity, cosine),
|
||||
crop_px: 150.0,
|
||||
quality: None,
|
||||
confirmed_person: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// A face too short to be a reference: compared, never compared against.
|
||||
fn probe(face: u64, image: u64, identity: usize, cosine: f32) -> Candidate {
|
||||
Candidate {
|
||||
quality: Some(crate::embedding::MIN_GALLERY_QUALITY - 5.0),
|
||||
..candidate(face, image, identity, cosine)
|
||||
}
|
||||
}
|
||||
|
||||
/// A calibration steep enough that the test's cosines are unambiguous:
|
||||
/// 0.6 is near-certain, 0.1 is near-impossible.
|
||||
fn cal() -> Calibration {
|
||||
@@ -1099,6 +1288,7 @@ mod tests {
|
||||
image,
|
||||
embedding: at_cosine(p, cosine),
|
||||
crop_px: 60.0 + ((out.len() % 11) as f32) * 25.0,
|
||||
quality: None,
|
||||
confirmed_person: None,
|
||||
});
|
||||
image += 1;
|
||||
@@ -1182,6 +1372,7 @@ mod tests {
|
||||
image: 5_000,
|
||||
embedding: at_cosine(200, 1.0),
|
||||
crop_px: 150.0,
|
||||
quality: None,
|
||||
confirmed_person: None,
|
||||
});
|
||||
let out = cluster(&faces, &cal(), DEFAULT_MERGE_PROBABILITY);
|
||||
@@ -1191,4 +1382,117 @@ mod tests {
|
||||
"the outlier was absorbed"
|
||||
);
|
||||
}
|
||||
|
||||
// ── the gallery ───────────────────────────────────────────────────────
|
||||
|
||||
/// A short vector is still somebody: it joins the group it matches.
|
||||
#[test]
|
||||
fn a_probe_joins_the_group_it_matches() {
|
||||
let faces = vec![
|
||||
candidate(1, 10, 0, 1.0),
|
||||
candidate(2, 11, 0, 0.95),
|
||||
probe(3, 12, 0, 0.92),
|
||||
];
|
||||
let out = cluster(&faces, &cal(), DEFAULT_MERGE_PROBABILITY);
|
||||
assert_eq!(out.len(), 1);
|
||||
assert_eq!(out[0].members, vec![0, 1, 2]);
|
||||
}
|
||||
|
||||
/// Two short vectors that resemble each other are noise agreeing with
|
||||
/// noise, and there is nothing in the gallery for either to be measured
|
||||
/// against.
|
||||
#[test]
|
||||
fn two_probes_are_never_grouped_with_each_other() {
|
||||
let faces = vec![probe(1, 10, 0, 1.0), probe(2, 11, 0, 0.98)];
|
||||
let out = cluster(&faces, &cal(), DEFAULT_MERGE_PROBABILITY);
|
||||
assert_eq!(out.len(), 2, "two probes were grouped: {out:?}");
|
||||
}
|
||||
|
||||
/// The point of measuring against the gallery only: a probe that has been
|
||||
/// placed is not a stepping stone for the next one.
|
||||
#[test]
|
||||
fn a_placed_probe_is_not_what_the_next_probe_is_measured_against() {
|
||||
let mut first = probe(2, 11, 0, 0.6);
|
||||
// 0.6 along identity 0 and 0.8 along its perpendicular: near enough to
|
||||
// the reference to join it, and much nearer to the face below.
|
||||
first.embedding = at_cosine(0, 0.6);
|
||||
let mut second = probe(3, 12, 0, 0.0);
|
||||
second.embedding = at_cosine(0, 0.0);
|
||||
let faces = vec![candidate(1, 10, 0, 1.0), first, second];
|
||||
|
||||
let out = cluster(&faces, &cal(), DEFAULT_MERGE_PROBABILITY);
|
||||
let group = out.iter().find(|c| c.members.contains(&0)).unwrap();
|
||||
assert_eq!(
|
||||
group.members,
|
||||
vec![0, 1],
|
||||
"the first probe should have joined"
|
||||
);
|
||||
assert!(
|
||||
out.iter().any(|c| c.members == vec![2]),
|
||||
"the second probe reached the group through the first: {out:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// A confirmation on a probe is still the user's word: the group it joins
|
||||
/// becomes that person, and a group already someone else's is closed to it.
|
||||
#[test]
|
||||
fn a_probe_carries_its_confirmation_and_respects_others() {
|
||||
let mut anchored = probe(3, 12, 0, 0.92);
|
||||
anchored.confirmed_person = Some(7);
|
||||
let faces = vec![
|
||||
candidate(1, 10, 0, 1.0),
|
||||
candidate(2, 11, 0, 0.95),
|
||||
anchored,
|
||||
];
|
||||
let out = cluster(&faces, &cal(), DEFAULT_MERGE_PROBABILITY);
|
||||
assert_eq!(out.len(), 1);
|
||||
assert_eq!(out[0].person, Some(7));
|
||||
|
||||
let mut theirs = candidate(1, 10, 0, 1.0);
|
||||
theirs.confirmed_person = Some(8);
|
||||
let faces = vec![theirs, candidate(2, 11, 0, 0.95), {
|
||||
let mut a = probe(3, 12, 0, 0.92);
|
||||
a.confirmed_person = Some(7);
|
||||
a
|
||||
}];
|
||||
let out = cluster(&faces, &cal(), DEFAULT_MERGE_PROBABILITY);
|
||||
assert!(
|
||||
out.iter()
|
||||
.any(|c| c.members == vec![2] && c.person == Some(7)),
|
||||
"a probe confirmed as one person joined another's group: {out:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The co-occurrence rule follows a probe in: once it has joined, its
|
||||
/// photograph is the group's.
|
||||
#[test]
|
||||
fn a_probe_cannot_join_a_group_holding_a_face_from_its_own_photograph() {
|
||||
let faces = vec![
|
||||
candidate(1, 10, 0, 1.0),
|
||||
candidate(2, 11, 0, 0.95),
|
||||
probe(3, 10, 0, 0.92),
|
||||
];
|
||||
let out = cluster(&faces, &cal(), DEFAULT_MERGE_PROBABILITY);
|
||||
assert!(out.iter().any(|c| c.members == vec![2]), "{out:?}");
|
||||
}
|
||||
|
||||
/// A probe's placement is scored like anyone else's, from the references
|
||||
/// it matched — and the references' own scores do not hear from it.
|
||||
#[test]
|
||||
fn a_probe_is_scored_but_is_not_evidence() {
|
||||
let gallery_only = vec![candidate(1, 10, 0, 1.0), candidate(2, 11, 0, 0.95)];
|
||||
let without = cluster_scored(&gallery_only, &cal(), DEFAULT_MERGE_PROBABILITY);
|
||||
|
||||
let mut with_probe = gallery_only.clone();
|
||||
with_probe.push(probe(3, 12, 0, 0.99));
|
||||
let with = cluster_scored(&with_probe, &cal(), DEFAULT_MERGE_PROBABILITY);
|
||||
|
||||
assert_eq!(with.clusters[0].members, vec![0, 1, 2]);
|
||||
assert!(with.confidence[2] > 0.9, "{}", with.confidence[2]);
|
||||
assert_eq!(
|
||||
&with.confidence[..2],
|
||||
&without.confidence[..],
|
||||
"a probe changed what the references were sure of"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user