dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Score a suggestion against the people the user has named

Browse Source 
The number beside a suggestion was the mean calibrated probability
between the face and the rest of its group, which measures the wrong
thing twice. It punishes coverage: a person with two hundred faces over
fifteen years is *meant* to have members a given photograph is
orthogonal to, so a correct suggestion onto a well-photographed person
scored low for being well photographed. And it never asked who else the
face might be — a face matching Anna at 0.95 and nobody else, and one
matching Anna at 0.95 and her sister at 0.93, came out identical, when
the second is the only one worth the user's attention.

dr_face::assign answers both, and multiplies them: the mean of the best
ten calibrated matches into the identity (the old mean, capped, which is
what stops coverage counting against it), times that identity's share of
the evidence against every *named* rival.

Only named people compete, and per person rather than per group. Both
halves of that had to be measured on a real 18,000-face library rather
than reasoned about. Normalising across every group made the number
useless — median suggestion 21%, four in five under half — because
clustering leaves one person spread over many groups, so a face competed
against itself; and keying rivals by group left Catherine competing with
Catherine, median 39%. Per named person: median 99.5%.

Rivals are gathered below the merge threshold, down to even odds: a
named person matching at 0.6 will never be merged into but is exactly
the competition to discount for. That would be a second similarity scan,
the expensive half of regrouping a library, so cluster_scored scans once
at the looser floor and hands the merge engine the subset at or above
the threshold — pair for pair what it would have scanned for itself,
held to that by a test.

Leave-one-out over that library's 2,702 confirmations across 54 named
people: 99.33% of faces placed on the right person against the old
mean's 99.15%, and the number shown for the right person moves from a
median of 90.4% to 99.3%. It errs low — 100% correct wherever it states
80% or more — which is the safe direction, and docs/faces.md §9.1 says
plainly that the low bands are not calibrated.

The example that measures it comes too: this is a claim about a
library's numbers, and nobody should have to take it on faith.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dtourolle
2026-08-29 10:44:01 +02:00
co-authored by Claude Opus 5
parent c8e05831f4
commit ebb7d3cf5c
7 changed files with 971 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/dr-face/src/assign.rs
+360
View File
@@ -0,0 +1,360 @@
//! TRACES: FR-CULL-9 | FR-CULL-10
//! How sure a *suggestion* is: an identity's share of the evidence for a face.
//!
//! [`crate::cluster`] decides which people exist; this decides what number to
//! put beside "we think this is Anna". They are not the same question, and the
//! answer to the second used to be a by-product of the first — the mean
//! calibrated probability between a face and *every* other member of its group.
//!
//! # Why a mean over the group is the wrong number
//!
//! It measures the wrong thing twice over.
//!
//! **It punishes large, well-photographed people.** Anna has two hundred faces
//! spanning fifteen years; a new photograph of her matches thirty of them
//! strongly and is near-orthogonal to the rest, because a face at 8 and a face
//! at 23 genuinely are. The mean lands around 0.2 and the interface reports a
//! correct suggestion as a doubtful one. The better a person is covered, the
//! worse their confidences get, which is exactly backwards.
//!
//! **It never asks who else it could be.** A face that matches Anna at 0.95 and
//! matches nobody else at all, and a face that matches Anna at 0.95 *and her
//! sister at 0.93*, are the same number under a within-group mean. The second
//! is the one the user actually needs to look at, and it was indistinguishable
//! from the first.
//!
//! # Coherence, times uniqueness
//!
//! Two questions, and the number is their product because they are genuinely
//! independent: *is this the same person at all*, and *of the people we know,
//! is it uniquely this one*.
//!
//! ```text
//! evidence(P) = Σ of the top n of { P(same | this face, f) : f ∈ P }
//! coherence = evidence(own) / (however many of the top n there were)
//! uniqueness = evidence(own) / (evidence(own) + Σ evidence(named rivals))
//! confidence = coherence × uniqueness
//! ```
//!
//! **Coherence** is a mean, like the old number, but over the face's best
//! [`TOP_MATCHES`] matches into the identity rather than over all of them. That single cap is what
//! stops a well-photographed person scoring worse than a thin one: the two
//! hundred faces a given photograph is legitimately orthogonal to no longer
//! count against it.
//!
//! **Uniqueness** is the competition. A sole strong match leaves it at 1 and
//! the confidence is the coherence; two identities matching equally well pull
//! it to 0.5 each, and the screen has told the user the truth, which is that
//! this face is a coin toss between two people.
//!
//! # Only the people the user has named compete
//!
//! Measured on a real 18,000-face library, normalising across *every* group
//! made the number useless: the median suggestion read 21% and four in five
//! read under half. The cause is not a bug in the arithmetic but a fact about
//! clustering — one person is spread across many groups, since the pairs that
//! would have joined them are the ones that fell short of the merge threshold.
//! Normalising over groups therefore makes a face compete against *itself*,
//! and the better covered the person, the more fragments there are to lose to.
//!
//! A fragment is not a rival. An identity the user has actually asserted is, so
//! the denominator counts only people a confirmation names ([`Cluster::person`])
//! — and counts them **per person, not per group**, since a named person is
//! left in several anchored groups for the same reason. Keying it by group had
//! Catherine competing with Catherine and put the median suggestion onto a
//! named person at 39%; keying it by person put it at 99.5%.
//!
//! Leave-one-out over that library's 2,702 confirmations across 54 named
//! people, this is the regime where the number is worth having: 99.3% of faces
//! are placed on the right person (the old mean managed 99.15%), the stated
//! percentage is monotone in being right, and it errs low — 100% correct
//! wherever it states 80% or more, 84% correct where it states under half.
//! Understating is the safe direction for a screen whose whole purpose is
//! deciding what to look at first, but it is *not* calibrated in the low bands
//! and should not be read as though it were.
//!
//! Two faces of one unnamed group cannot be told from two fragments of one
//! person by similarity alone; that is exactly why clustering stopped where it
//! did. So the module does not pretend to: where nobody is named, uniqueness is
//! 1 and the number falls back to plain coherence.
//!
//! # What it does not do
//!
//! It is not a merge threshold and must not become one. Clustering keeps
//! deciding on the pairwise calibrated probability: uniqueness is *relative*,
//! so a library with one named person in it would hand every stray face a
//! uniqueness of 1. The absolute question ("is this the same person at all")
//! and the comparative one ("of the people we know, which") are different, and
//! the product is what keeps both in the answer.
use std::collections::BTreeMap;
use crate::cluster::Cluster;
use crate::neighbours::Pair;
/// Who the evidence is for.
///
/// Ordered rather than hashed so the sums below are reproducible; the ordering
/// itself carries no meaning.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
enum Identity {
/// A person the user has confirmed a face onto. Every group anchored to
/// them is the same identity, however many of them the clusterer left.
Person(u64),
/// A group nobody has ruled on. It stands for itself and competes with
/// nothing.
Group(usize),
}
/// How many of an identity's best matches count as its evidence.
///
/// The cap is the whole reason the sum works: uncapped, evidence would grow
/// with a person's face count and the largest group in the library would win
/// every contest. Ten is enough that a person photographed from several angles
/// contributes more than one lucky frame, and small enough that the hundred
/// mediocre matches inside a well-covered identity cannot add up to a strong
/// one. It is a starting point, not a measured optimum — M7's corpus is where
/// it would be tuned.
pub const TOP_MATCHES: usize = 10;
/// The weakest match that counts as evidence for an identity.
///
/// Rivals are half the point of this module, so the evidence scan has to reach
/// *below* the merge threshold — a named person who matches at 0.6 will never
/// be merged into but is precisely the competition a suggestion should be
/// discounted for. Even odds is the natural floor: below it a pair is more
/// likely different people than the same, and it is not a small distinction —
/// summing the near-orthogonal pairs instead of dropping them lets fifty
/// identities' worth of noise, each contributing its *upper tail*, outweigh one
/// real match. Measured on a real library, that alone moved the median stated
/// confidence from 100% to 31%.
pub const RIVAL_FLOOR: f32 = 0.5;
/// How confident each face's placement is, indexed like the face slice the
/// clusters came from.
///
/// A face in no group, or one with no evidence for anybody, scores 0.
///
/// `pairs` must be the *evidence* list — scanned at [`RIVAL_FLOOR`], not at the
/// merge threshold. Passing the merge list still works but silently removes
/// every rival weaker than a merge, which is most of them, and every uniqueness
/// collapses to 1.
pub fn identity_shares(faces: usize, clusters: &[Cluster], pairs: &[Pair], top: usize) -> Vec<f32> {
// An identity is a *person*, not a group. One person routinely holds
// several anchored groups — the same reason they hold several unnamed ones
// — and keying this by group had Catherine competing with Catherine, which
// on the library it was measured against put the median suggestion onto a
// named person at 39%.
let key_of: Vec<Identity> = clusters
.iter()
.enumerate()
.map(|(g, c)| match c.person {
Some(p) => Identity::Person(p),
None => Identity::Group(g),
})
.collect();
let mut group_of = vec![usize::MAX; faces];
for (g, c) in clusters.iter().enumerate() {
for &m in &c.members {
if m < faces {
group_of[m] = g;
}
}
}
// Ordered, not hashed: the numbers are sums of floats over these buckets
// and this module inherits [`crate::cluster`]'s promise that the same input
// yields the same output, bit for bit.
let mut evidence: Vec<BTreeMap<Identity, Vec<f32>>> = vec![BTreeMap::new(); faces];
for p in pairs {
if p.i >= faces || p.j >= faces {
continue;
}
// A pair is evidence in both directions: j's identity hears about i,
// and i's identity hears about j. The pair list holds each unordered
// pair once, so both have to be recorded here.
let (gi, gj) = (group_of[p.i], group_of[p.j]);
if gj != usize::MAX {
evidence[p.i]
.entry(key_of[gj])
.or_default()
.push(p.probability);
}
if gi != usize::MAX {
evidence[p.j]
.entry(key_of[gi])
.or_default()
.push(p.probability);
}
}
let mut out = vec![0.0; faces];
for (i, buckets) in evidence.iter_mut().enumerate() {
let mine = group_of[i];
if mine == usize::MAX {
continue;
}
let mine = key_of[mine];
let mut coherence = 0.0;
let mut ours = 0.0;
let mut rivals = 0.0;
for (&who, probabilities) in buckets.iter_mut() {
// Descending, and the ties broken by nothing: equal probabilities
// sum the same whichever order they land in.
probabilities.sort_by(|a, b| b.total_cmp(a));
let counted = probabilities.len().min(top);
let score: f32 = probabilities.iter().take(top).sum();
if who == mine {
ours = score;
coherence = score / counted as f32;
} else if matches!(who, Identity::Person(_)) {
// Only an identity the user has asserted competes. An unnamed
// group that matches this face is far more likely to be another
// fragment of the same person than a different one — see the
// module note, and the library it was measured on.
rivals += score;
}
}
let total = ours + rivals;
// No evidence at all: a face anchored into a group it has no measured
// similarity to. Nothing honest to report, so nothing is claimed.
out[i] = if total > 0.0 {
coherence * (ours / total)
} else {
0.0
};
}
out
}
#[cfg(test)]
mod tests {
use super::*;
/// An unnamed group: nobody has ruled on it, so it competes with nothing.
fn cluster(members: &[usize]) -> Cluster {
Cluster {
members: members.to_vec(),
person: None,
}
}
/// A group the user has confirmed a face onto — an identity, and therefore
/// a rival.
fn named(members: &[usize], person: u64) -> Cluster {
Cluster {
members: members.to_vec(),
person: Some(person),
}
}
fn pair(i: usize, j: usize, probability: f32) -> Pair {
Pair { i, j, probability }
}
/// The failure the module exists to fix: face 0 matches its own group's
/// three members strongly, and the group has forty more it is unrelated to.
/// The old within-group mean reported ~0.07 for this.
#[test]
fn a_large_group_does_not_dilute_a_strong_match() {
let members: Vec<usize> = (0..44).collect();
let clusters = vec![cluster(&members)];
let pairs = vec![pair(0, 1, 0.99), pair(0, 2, 0.97), pair(0, 3, 0.95)];
let shares = identity_shares(44, &clusters, &pairs, TOP_MATCHES);
assert!(
(shares[0] - 0.97).abs() < 1e-6,
"the mean of its three real matches, undiluted: {}",
shares[0]
);
}
/// Two named people matching equally well is a coin toss, and saying so is
/// the point — this is the sibling case FR-CULL-10 warns about.
#[test]
fn an_ambiguous_face_splits_its_confidence_between_the_rivals() {
let clusters = vec![named(&[0, 1, 2], 1), named(&[3, 4], 2)];
let pairs = vec![
pair(0, 1, 0.90),
pair(0, 2, 0.90),
pair(0, 3, 0.90),
pair(0, 4, 0.90),
];
let shares = identity_shares(5, &clusters, &pairs, TOP_MATCHES);
// Coherent at 0.90, and only half of the evidence is its own.
assert!(
(shares[0] - 0.45).abs() < 1e-6,
"even evidence both ways: {}",
shares[0]
);
}
/// A rival below the merge threshold still has to count, which is why the
/// evidence scan reaches down to [`RIVAL_FLOOR`].
#[test]
fn a_rival_too_weak_to_merge_still_lowers_the_confidence() {
let clusters = vec![named(&[0, 1], 1), named(&[2, 3], 2)];
let sure = identity_shares(4, &clusters, &[pair(0, 1, 0.95)], TOP_MATCHES);
let contested = identity_shares(
4,
&clusters,
&[pair(0, 1, 0.95), pair(0, 2, 0.60)],
TOP_MATCHES,
);
assert_eq!(sure[0], 0.95, "nobody else to be: its coherence stands");
assert!(
contested[0] < 0.59 && contested[0] > 0.57,
"0.95 coherent, but 0.95 against 0.60: {}",
contested[0]
);
}
/// A fragment of the same person is not a rival. Measured on a real
/// library, counting unnamed groups as competition put four suggestions in
/// five under half — see the module note.
#[test]
fn an_unnamed_group_is_not_treated_as_competition() {
let clusters = vec![cluster(&[0, 1]), cluster(&[2, 3])];
let shares = identity_shares(
4,
&clusters,
&[pair(0, 1, 0.95), pair(0, 2, 0.90)],
TOP_MATCHES,
);
assert_eq!(
shares[0], 0.95,
"an unnamed group took evidence off a suggestion"
);
}
/// The cap, doing its job: an identity with fifty mediocre matches must not
/// beat one with ten strong ones on volume alone.
#[test]
fn evidence_is_capped_so_the_biggest_group_cannot_win_on_volume() {
let small: Vec<usize> = (0..11).collect();
let large: Vec<usize> = (11..62).collect();
let clusters = vec![named(&small, 1), named(&large, 2)];
let mut pairs: Vec<Pair> = (1..11).map(|j| pair(0, j, 0.90)).collect();
pairs.extend((11..62).map(|j| pair(0, j, 0.55)));
let shares = identity_shares(62, &clusters, &pairs, TOP_MATCHES);
// Ten at 0.90 against ten at 0.55 — not fifty-one at 0.55.
assert!(
(shares[0] - 0.90 * (9.0 / 14.5)).abs() < 1e-5,
"capped at ten either side: {}",
shares[0]
);
}
/// A face nothing has any evidence about claims nothing.
#[test]
fn a_face_with_no_evidence_reports_no_confidence() {
let clusters = vec![cluster(&[0, 1])];
let shares = identity_shares(2, &clusters, &[], TOP_MATCHES);
assert_eq!(shares, vec![0.0, 0.0]);
}
}