Fuse every detector's faces into one population per embedder
Choosing "Thorough" made the library look empty. The detector setting writes under its own faces.model_id, and every reader of "the faces" keyed on that exact id: the clustering pass, the coverage figure, the sweep's work list, the shard export and import, and the sync merge's face matching. On the reference library that restarted coverage at 1,834 of 19,140, drew a People rail of 36 faces for a person with 520, queued a ~400 GB re-fetch on each device, and stranded the desktop's 3,583 confirmations under the old id: the tablet held the same faces under the new one and the merge refused to match them. Same photograph, same box, same embedder, two ids — that is one face, not two libraries. The embedder half of the id is now the key. embedder_of and embedder_sql give it to every query; writes keep the full id, so which detector drew a box stays on record. record_detections is unchanged and is where the generations meet: an image holds one pipeline's faces at a time, and a re-detection carries confirmations across by box overlap. The merge's match_faces applies the same rule within an embedder. The calibration is keyed on the embedder too, since the similarity space did not change. Shards travel every generation, each under its own id, and a peer adopts whichever it is sent — including a stronger detector's pass over an image it indexed itself with a weaker one, which is the re-detection its own sweep would otherwise queue, already done. Never downwards: a tablet on Fast keeps the desktop's Thorough faces. The sweep gains the same tail — images a weaker detector indexed, after the ones nothing has — driven by FaceDetector::supersedes, so choosing a stronger detector still improves the library over time without first making it disappear.
This commit is contained in:
+13
-10
@@ -91,14 +91,15 @@ pub struct FaceRequest {
|
||||
/// examined, so every landscape in the library would be re-detected on every
|
||||
/// run, for ever. See the V9 migration.
|
||||
///
|
||||
/// Keyed on the model, so a model upgrade re-indexes rather than leaving the
|
||||
/// library half-described by weights that are no longer comparable.
|
||||
/// Keyed on the embedder, so an embedder upgrade re-indexes rather than
|
||||
/// leaving the library half-described by weights that are no longer
|
||||
/// comparable — and a detector change, which keeps the embedder, does not.
|
||||
pub fn faces_outstanding(
|
||||
catalog: &Catalog,
|
||||
store: &ThumbStore,
|
||||
model_id: &str,
|
||||
) -> Result<Vec<FaceRequest>, dr_catalog::CatalogError> {
|
||||
let mut stmt = catalog.connection().prepare(
|
||||
let mut stmt = catalog.connection().prepare(&format!(
|
||||
"SELECT i.id, r.file_id
|
||||
FROM images i
|
||||
JOIN remote r ON r.image_id = i.id
|
||||
@@ -107,12 +108,13 @@ pub fn faces_outstanding(
|
||||
AND i.shadowed_by IS NULL
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM face_index fi
|
||||
WHERE fi.image_id = i.id AND fi.model_id = ?1
|
||||
WHERE fi.image_id = i.id AND {} = ?1
|
||||
)
|
||||
ORDER BY i.id",
|
||||
)?;
|
||||
faces::embedder_sql("fi.model_id")
|
||||
))?;
|
||||
let rows = stmt
|
||||
.query_map([model_id], |r| {
|
||||
.query_map([faces::embedder_of(model_id)], |r| {
|
||||
Ok(FaceRequest {
|
||||
image_id: ImageId(r.get::<_, i64>(0)? as u64),
|
||||
file_id: r.get::<_, i64>(1)? as u64,
|
||||
@@ -209,7 +211,7 @@ pub fn audit(
|
||||
// for the same reason `faces::coverage` excludes them: they are the JPEG
|
||||
// half of a pair, no sweep will ever index one, and counting them makes
|
||||
// the outstanding figure a number that cannot reach zero.
|
||||
let mut stmt = conn.prepare(
|
||||
let mut stmt = conn.prepare(&format!(
|
||||
"SELECT r.file_id
|
||||
FROM images i
|
||||
JOIN remote r ON r.image_id = i.id
|
||||
@@ -218,12 +220,13 @@ pub fn audit(
|
||||
AND i.shadowed_by IS NULL
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM face_index fi
|
||||
WHERE fi.image_id = i.id AND fi.model_id = ?1
|
||||
WHERE fi.image_id = i.id AND {} = ?1
|
||||
)",
|
||||
)?;
|
||||
faces::embedder_sql("fi.model_id")
|
||||
))?;
|
||||
let (mut ready, mut awaiting) = (0u64, 0u64);
|
||||
for file_id in stmt
|
||||
.query_map([model_id], |r| r.get::<_, i64>(0))?
|
||||
.query_map([faces::embedder_of(model_id)], |r| r.get::<_, i64>(0))?
|
||||
.filter_map(Result::ok)
|
||||
{
|
||||
if store.contains(file_id as u64, FACE_TIER) {
|
||||
|
||||
Reference in New Issue
Block a user