Match synced faces by embedding, not only by box #77

New Issue

2026-09-26T19:42:45Z

dtourolle commented

2026-09-26 19:42:45 +00:00

The catalog merge matches a synced face to a local one by box alone, and never reads the embedding it carries. merge::match_faces pairs faces on the photograph's file_id, the embedder half of model_id, and a box IoU of at least 0.5. Every synced face row already carries its 512×f16 embedding (19 MB of the ~50 MB snapshot), and the merge ignores all of it.

Why

A face that both devices found, but whose boxes disagree (a detector change or an alignment shift), or that sits among overlapping faces, either goes unmatched or is matched by geometry alone. An unmatched face strands its confirmation on one device. That is the "faces don't sync" failure of 2026-09-14 and 2026-09-20 in a new form, against the rule that a re-found face keeps its identity and a new face is new. Two embeddings from the same embedder for the same face in the same photograph are near-identical, so they are the stronger evidence of all.

Deliverable

match_faces reads the remote embeddings. For the same file and embedder, a pair counts as the same face when the embeddings agree, and the box is corroboration, not the only test.
Measured first: on the desktop catalog against the server snapshot (the tablet's catalog after its last push), how many remote faces go unmatched or are matched ambiguously today, and how many embeddings recover.
No false merges: a match must be unique and mutual, and the threshold must be justified by the measured distribution of same-face and different-face similarities.

Acceptance

Before and after counts of matched, unmatched and ambiguous faces on the real pair of catalogs
The similarity distribution for same-face and different-face pairs, and the chosen threshold
Tests: a shifted box with the same embedding matches; two overlapping faces are told apart by embedding; a similar-looking different person in another photograph never matches
Merge time on the reference catalog not materially worse (currently ~135 ms)
No schema bump (0.16.0 tablets must keep syncing)

**The catalog merge matches a synced face to a local one by box alone, and never reads the embedding it carries.** `merge::match_faces` pairs faces on the photograph's `file_id`, the embedder half of `model_id`, and a box IoU of at least 0.5. Every synced face row already carries its 512×f16 embedding (19 MB of the ~50 MB snapshot), and the merge ignores all of it. ## Why A face that both devices found, but whose boxes disagree (a detector change or an alignment shift), or that sits among overlapping faces, either goes unmatched or is matched by geometry alone. An unmatched face strands its confirmation on one device. That is the "faces don't sync" failure of 2026-09-14 and 2026-09-20 in a new form, against the rule that a re-found face keeps its identity and a new face is new. Two embeddings from the same embedder for the same face in the same photograph are near-identical, so they are the stronger evidence of all. ## Deliverable - `match_faces` reads the remote embeddings. For the same file and embedder, a pair counts as the same face when the embeddings agree, and the box is corroboration, not the only test. - Measured first: on the desktop catalog against the server snapshot (the tablet's catalog after its last push), how many remote faces go unmatched or are matched ambiguously today, and how many embeddings recover. - No false merges: a match must be unique and mutual, and the threshold must be justified by the measured distribution of same-face and different-face similarities. ## Acceptance - [ ] Before and after counts of matched, unmatched and ambiguous faces on the real pair of catalogs - [ ] The similarity distribution for same-face and different-face pairs, and the chosen threshold - [ ] Tests: a shifted box with the same embedding matches; two overlapping faces are told apart by embedding; a similar-looking different person in another photograph never matches - [ ] Merge time on the reference catalog not materially worse (currently ~135 ms) - [ ] No schema bump (0.16.0 tablets must keep syncing)

dtourolle added the catalog size:M sync labels 2026-09-26 19:42:45 +00:00

dtourolle referenced this issue

2026-09-26 20:20:15 +00:00

Sync snapshot copies every face crop: the largest remaining cost of a pass #71

dtourolle referenced this issue

2026-09-26 21:26:48 +00:00

Deduplicate people with the same name and agreeing faces, and duplicate face rows #78

dtourolle referenced this issue from a commit

2026-09-26 23:13:18 +00:00

Time a merge with another device's catalog in catalog_bench

dtourolle referenced this issue from a commit

2026-09-26 23:13:18 +00:00

Match synced faces by embedding where the boxes cannot (#77)

dtourolle referenced this issue from a commit

2026-09-26 23:13:18 +00:00

Say how synced faces and same-name people merge since #77 and #78

dtourolle commented

2026-09-26 23:14:03 +00:00

Done in 884b681, b5b30e3 and 1479e45, released in v0.18.0; docs in b2f3936 (faces.md §18.2).

Measured on the desktop catalog against the server snapshot (the tablet), one embedder (w600k_mbf):

Box matching alone: 18,348 matched uniquely, 0 ambiguous, 631 unmatched.
Cosine distributions: same face across devices has p5 0.79 and median 1.00; different faces in the same photo have p99.9 0.55 and max 0.82; the same confirmed person in different photos reaches 1.00, so matching stays within one file_id.
Rule: box IoU ≥ 0.5 and unique both ways first; then, only where a remote face is left over and a local face is free, cosine ≥ 0.7, mutual best, with a lead of ≥ 0.2 over the runner-up. A low cosine never overrules a box, since ~150 genuine pairs of tiny faces score below 0.45.
Result: 20 faces recovered (2 of them confirmed), 18,348 → 18,368 matched, no false matches. The other 611 are faces only one device found.

The duplicate people reported alongside turned out to be one person assigned to two different faces in a photograph, not duplicate face rows (there are 0 of those): 67 photographs on the desktop and 75 on the tablet, nearly all unnamed set-aside groups. The merge now keeps one person to one face per photograph; a remote confirmation moves a local suggestion rather than doubling it. One genuine conflict (Oskar Boldre on two faces of image 10309) is left for the user. Merge time against the real peer is not worse.

Done in 884b681, b5b30e3 and 1479e45, released in v0.18.0; docs in b2f3936 (faces.md §18.2). Measured on the desktop catalog against the server snapshot (the tablet), one embedder (w600k_mbf): - Box matching alone: 18,348 matched uniquely, 0 ambiguous, 631 unmatched. - Cosine distributions: same face across devices has p5 0.79 and median 1.00; different faces in the same photo have p99.9 0.55 and max 0.82; the same confirmed person in different photos reaches 1.00, so matching stays within one `file_id`. - Rule: box IoU ≥ 0.5 and unique both ways first; then, only where a remote face is left over and a local face is free, cosine ≥ 0.7, mutual best, with a lead of ≥ 0.2 over the runner-up. A low cosine never overrules a box, since ~150 genuine pairs of tiny faces score below 0.45. - Result: 20 faces recovered (2 of them confirmed), 18,348 → 18,368 matched, no false matches. The other 611 are faces only one device found. The duplicate people reported alongside turned out to be one person assigned to two different faces in a photograph, not duplicate face rows (there are 0 of those): 67 photographs on the desktop and 75 on the tablet, nearly all unnamed set-aside groups. The merge now keeps one person to one face per photograph; a remote confirmation moves a local suggestion rather than doubling it. One genuine conflict (Oskar Boldre on two faces of image 10309) is left for the user. Merge time against the real peer is not worse.

dtourolle closed this issue

2026-09-26 23:14:03 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dtourolle/DarkRoom#77