Count the same images in the progress figure that the sweeps index

The Identity screen said 4,593 images were left to index and stayed
there for hours across repeated runs, which is what a stuck job looks
like. It was not stuck. 4,424 of those 4,593 are shadowed -- the JPEG
half of a RAW+JPEG pair -- and no sweep will ever index one, because
every work list is built on VISIBLE, which excludes them. They are not
separate photographs and the grid does not show them either.

But faces::coverage counted them: its denominator was "images WHERE
trashed_at IS NULL", with no shadowed_by clause. So the outstanding
figure had a floor of 4,424 that no amount of work could bring down, and
Coverage::is_complete could never once return true no matter how
completely the library had been indexed. A progress number that cannot
reach its own target is worse than no progress number.

The fix is to count the population the sweeps actually draw from, in all
three places that were describing it differently: coverage's denominator
and its indexed join, and audit's split of the outstanding set, which
had the same gap and fed the same status line.

On the reference library the denominator goes from 23,531 to 19,107 and
outstanding from 4,593 to 169 -- the second of which is a number the
user can watch go down, and which turns out to be a real and separate
fetch failure worth chasing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-30 19:41:04 +02:00
co-authored by Claude Opus 5
parent 6783d0c723
commit d0ebc9f571
2 changed files with 48 additions and 4 deletions
+42 -3
View File
@@ -254,7 +254,9 @@ pub fn record_detections(
/// How much of the library has been through face detection. /// How much of the library has been through face detection.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct Coverage { pub struct Coverage {
/// Images that are candidates at all — present, not trashed. /// Images that are candidates at all — present, not trashed, and not the
/// shadowed half of a RAW+JPEG pair. The same population every sweep's
/// work list is drawn from; see [`coverage`] for why that matters.
pub images: u64, pub images: u64,
/// Images this model has actually looked at. /// Images this model has actually looked at.
pub indexed: u64, pub indexed: u64,
@@ -292,9 +294,21 @@ impl Coverage {
/// progress figure. Answerable only because [`record_detections`] writes a run /// progress figure. Answerable only because [`record_detections`] writes a run
/// marker: counting `faces` rows would report how many faces exist, which is a /// marker: counting `faces` rows would report how many faces exist, which is a
/// different number and never reaches the image count. /// different number and never reaches the image count.
///
/// # Shadowed images are not candidates, and the denominator has to agree
///
/// A shadowed image is the JPEG half of a RAW+JPEG pair. It is not a separate
/// photograph — the grid does not show it, and every sweep that builds a work
/// list excludes it. Counting it here anyway is not a rounding error: on the
/// reference library it put 4,424 images into the denominator that no pass is
/// permitted to touch, so "4,593 outstanding" had a floor of 4,424 that no
/// amount of indexing could ever bring down, and [`Coverage::is_complete`]
/// could never once return true. A progress figure that cannot reach its own
/// target reads exactly like a stuck job, which is what it was taken for.
pub fn coverage(conn: &Connection, model_id: &str) -> Result<Coverage, CatalogError> { pub fn coverage(conn: &Connection, model_id: &str) -> Result<Coverage, CatalogError> {
let images: i64 = conn.query_row( let images: i64 = conn.query_row(
"SELECT COUNT(*) FROM images WHERE trashed_at IS NULL", "SELECT COUNT(*) FROM images
WHERE trashed_at IS NULL AND shadowed_by IS NULL",
[], [],
|r| r.get(0), |r| r.get(0),
)?; )?;
@@ -302,7 +316,8 @@ pub fn coverage(conn: &Connection, model_id: &str) -> Result<Coverage, CatalogEr
"SELECT COUNT(*), COALESCE(SUM(faces_found = 0), 0), COALESCE(SUM(faces_found), 0) "SELECT COUNT(*), COALESCE(SUM(faces_found = 0), 0), COALESCE(SUM(faces_found), 0)
FROM face_index fi FROM face_index fi
JOIN images i ON i.id = fi.image_id JOIN images i ON i.id = fi.image_id
WHERE fi.model_id = ?1 AND i.trashed_at IS NULL", WHERE fi.model_id = ?1
AND i.trashed_at IS NULL AND i.shadowed_by IS NULL",
[model_id], [model_id],
|r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)), |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)),
)?; )?;
@@ -1345,6 +1360,30 @@ mod tests {
assert!((cov.fraction() - 2.0 / 3.0).abs() < 1e-6); assert!((cov.fraction() - 2.0 / 3.0).abs() < 1e-6);
} }
#[test]
fn coverage_leaves_out_the_shadowed_half_of_a_raw_jpeg_pair() {
let c = db();
let raw = image(&c, 1);
let jpeg = image(&c, 2);
// The JPEG beside a RAW is not a separate photograph, and no sweep
// will ever index one.
c.execute(
"UPDATE images SET shadowed_by = ?1 WHERE id = ?2",
rusqlite::params![raw.0 as i64, jpeg.0 as i64],
)
.unwrap();
record_detections(&c, raw, "w600k_mbf", 2560, &[face(1)]).unwrap();
let cov = coverage(&c, "w600k_mbf").unwrap();
// Counting the shadowed one would leave a permanent remainder that no
// amount of indexing could bring down -- 4,424 of them on the
// reference library, which read as a stuck job.
assert_eq!(cov.images, 1);
assert_eq!(cov.outstanding(), 0);
assert!(cov.is_complete());
}
#[test] #[test]
fn re_indexing_one_image_updates_its_marker_rather_than_adding_a_second() { fn re_indexing_one_image_updates_its_marker_rather_than_adding_a_second() {
let c = db(); let c = db();
+6 -1
View File
@@ -82,6 +82,7 @@ pub fn faces_outstanding(
JOIN remote r ON r.image_id = i.id JOIN remote r ON r.image_id = i.id
WHERE r.file_id IS NOT NULL WHERE r.file_id IS NOT NULL
AND i.trashed_at IS NULL AND i.trashed_at IS NULL
AND i.shadowed_by IS NULL
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM face_index fi SELECT 1 FROM face_index fi
WHERE fi.image_id = i.id AND fi.model_id = ?1 WHERE fi.image_id = i.id AND fi.model_id = ?1
@@ -177,13 +178,17 @@ pub fn audit(
// Split the outstanding set by whether a proxy exists. This is the query // Split the outstanding set by whether a proxy exists. This is the query
// `faces_outstanding` runs without the store filter, so the two cannot // `faces_outstanding` runs without the store filter, so the two cannot
// disagree about what is outstanding. // disagree about what is outstanding. Shadowed images are excluded here
// for the same reason `faces::coverage` excludes them: they are the JPEG
// half of a pair, no sweep will ever index one, and counting them makes
// the outstanding figure a number that cannot reach zero.
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT r.file_id "SELECT r.file_id
FROM images i FROM images i
JOIN remote r ON r.image_id = i.id JOIN remote r ON r.image_id = i.id
WHERE r.file_id IS NOT NULL WHERE r.file_id IS NOT NULL
AND i.trashed_at IS NULL AND i.trashed_at IS NULL
AND i.shadowed_by IS NULL
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM face_index fi SELECT 1 FROM face_index fi
WHERE fi.image_id = i.id AND fi.model_id = ?1 WHERE fi.image_id = i.id AND fi.model_id = ?1