Count the same images in the progress figure that the sweeps index
The Identity screen said 4,593 images were left to index and stayed there for hours across repeated runs, which is what a stuck job looks like. It was not stuck. 4,424 of those 4,593 are shadowed -- the JPEG half of a RAW+JPEG pair -- and no sweep will ever index one, because every work list is built on VISIBLE, which excludes them. They are not separate photographs and the grid does not show them either. But faces::coverage counted them: its denominator was "images WHERE trashed_at IS NULL", with no shadowed_by clause. So the outstanding figure had a floor of 4,424 that no amount of work could bring down, and Coverage::is_complete could never once return true no matter how completely the library had been indexed. A progress number that cannot reach its own target is worse than no progress number. The fix is to count the population the sweeps actually draw from, in all three places that were describing it differently: coverage's denominator and its indexed join, and audit's split of the outstanding set, which had the same gap and fed the same status line. On the reference library the denominator goes from 23,531 to 19,107 and outstanding from 4,593 to 169 -- the second of which is a number the user can watch go down, and which turns out to be a real and separate fetch failure worth chasing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -254,7 +254,9 @@ pub fn record_detections(
|
||||
/// How much of the library has been through face detection.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub struct Coverage {
|
||||
/// Images that are candidates at all — present, not trashed.
|
||||
/// Images that are candidates at all — present, not trashed, and not the
|
||||
/// shadowed half of a RAW+JPEG pair. The same population every sweep's
|
||||
/// work list is drawn from; see [`coverage`] for why that matters.
|
||||
pub images: u64,
|
||||
/// Images this model has actually looked at.
|
||||
pub indexed: u64,
|
||||
@@ -292,9 +294,21 @@ impl Coverage {
|
||||
/// progress figure. Answerable only because [`record_detections`] writes a run
|
||||
/// marker: counting `faces` rows would report how many faces exist, which is a
|
||||
/// different number and never reaches the image count.
|
||||
///
|
||||
/// # Shadowed images are not candidates, and the denominator has to agree
|
||||
///
|
||||
/// A shadowed image is the JPEG half of a RAW+JPEG pair. It is not a separate
|
||||
/// photograph — the grid does not show it, and every sweep that builds a work
|
||||
/// list excludes it. Counting it here anyway is not a rounding error: on the
|
||||
/// reference library it put 4,424 images into the denominator that no pass is
|
||||
/// permitted to touch, so "4,593 outstanding" had a floor of 4,424 that no
|
||||
/// amount of indexing could ever bring down, and [`Coverage::is_complete`]
|
||||
/// could never once return true. A progress figure that cannot reach its own
|
||||
/// target reads exactly like a stuck job, which is what it was taken for.
|
||||
pub fn coverage(conn: &Connection, model_id: &str) -> Result<Coverage, CatalogError> {
|
||||
let images: i64 = conn.query_row(
|
||||
"SELECT COUNT(*) FROM images WHERE trashed_at IS NULL",
|
||||
"SELECT COUNT(*) FROM images
|
||||
WHERE trashed_at IS NULL AND shadowed_by IS NULL",
|
||||
[],
|
||||
|r| r.get(0),
|
||||
)?;
|
||||
@@ -302,7 +316,8 @@ pub fn coverage(conn: &Connection, model_id: &str) -> Result<Coverage, CatalogEr
|
||||
"SELECT COUNT(*), COALESCE(SUM(faces_found = 0), 0), COALESCE(SUM(faces_found), 0)
|
||||
FROM face_index fi
|
||||
JOIN images i ON i.id = fi.image_id
|
||||
WHERE fi.model_id = ?1 AND i.trashed_at IS NULL",
|
||||
WHERE fi.model_id = ?1
|
||||
AND i.trashed_at IS NULL AND i.shadowed_by IS NULL",
|
||||
[model_id],
|
||||
|r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)),
|
||||
)?;
|
||||
@@ -1345,6 +1360,30 @@ mod tests {
|
||||
assert!((cov.fraction() - 2.0 / 3.0).abs() < 1e-6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn coverage_leaves_out_the_shadowed_half_of_a_raw_jpeg_pair() {
|
||||
let c = db();
|
||||
let raw = image(&c, 1);
|
||||
let jpeg = image(&c, 2);
|
||||
// The JPEG beside a RAW is not a separate photograph, and no sweep
|
||||
// will ever index one.
|
||||
c.execute(
|
||||
"UPDATE images SET shadowed_by = ?1 WHERE id = ?2",
|
||||
rusqlite::params![raw.0 as i64, jpeg.0 as i64],
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
record_detections(&c, raw, "w600k_mbf", 2560, &[face(1)]).unwrap();
|
||||
|
||||
let cov = coverage(&c, "w600k_mbf").unwrap();
|
||||
// Counting the shadowed one would leave a permanent remainder that no
|
||||
// amount of indexing could bring down -- 4,424 of them on the
|
||||
// reference library, which read as a stuck job.
|
||||
assert_eq!(cov.images, 1);
|
||||
assert_eq!(cov.outstanding(), 0);
|
||||
assert!(cov.is_complete());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn re_indexing_one_image_updates_its_marker_rather_than_adding_a_second() {
|
||||
let c = db();
|
||||
|
||||
Reference in New Issue
Block a user