Give the face sweep a size budget, so a panorama is never fetched

The sweep fetches the whole original before it can learn anything
about it, and the one file in the reference library the decoder
refuses on sight is a 521 MB stitched panorama — so every pass on the
tablet spent half a gigabyte of Wi-Fi to find that out again. The
catalog already knows the byte count, and that is enough to decide
before the fetch: originals over 256 MB are marked examined with
nothing found and a zero edge, counted as failed, and named in the
log. Below the line is every camera RAW the library holds; above it,
four files, all panoramas.

A budget and not a verdict on panoramas. The right treatment for one
is a tiled pass — read it in strips, detect in each, stitch the boxes
back — and the zero edge is what that pass would select on. Until it
exists, this is what keeps a background sweep on a phone from paying
for the decision the decoder cannot make.
This commit is contained in:
2026-09-19 12:14:05 +02:00
parent 7db999c1f6
commit e6ac31d39d
2 changed files with 133 additions and 24 deletions
+111 -2
View File
@@ -3528,6 +3528,26 @@ const SWEEP_CHUNK: usize = 96;
/// without provoking it; the retry above covers what still slips through.
const SWEEP_LANES: usize = 6;
/// TRACES: FR-CULL-8 | NFR-RES-2
/// The largest original the face sweep will fetch, in bytes.
///
/// A budget, not a correctness rule: nothing about a byte count says whether
/// a file decodes. It exists because the sweep fetches the whole original
/// before it can learn anything about it, and the one file in the reference
/// library above this line is a 521 MB stitched panorama the decoder refuses
/// on sight — so every pass on the tablet spent half a gigabyte of Wi-Fi to
/// find that out again. Below the line: every camera RAW this library holds,
/// the largest a 60 MB medium-format file; above it, four files, all
/// panoramas.
///
/// A file over budget is marked examined with nothing found and a zero
/// edge, the same mark a file the decoder cannot open gets, so the count in
/// the sweep's report says it was skipped and a later pass can select it.
/// That later pass is the real answer for a panorama — read it in tiles,
/// detect in each, and stitch the boxes back — and this constant is the
/// placeholder for it, not a decision that panoramas hold no faces.
const SWEEP_MAX_ORIGINAL_BYTES: u64 = 256 * 1024 * 1024;
/// Date **every** image in the library, not just the ones on screen.
///
/// Thumbnails are deliberately *not* fetched here. A thumbnail needs the
@@ -3980,6 +4000,45 @@ fn faces_superseded(
Ok(rows)
}
/// Take the originals over [`SWEEP_MAX_ORIGINAL_BYTES`] out of the work list,
/// marking each as examined so no later pass fetches it either.
///
/// Before the fetch loop and on the byte count the catalog already holds,
/// which is the whole point: the decision costs nothing, where fetching the
/// file to decide would cost the file. Returns how many were set aside, which
/// the sweep reports as failed — not indexed, and said so.
fn set_aside_oversized(
catalog: &Catalog,
model_id: &str,
wanted: &mut Vec<(ThumbnailRequest, SweepWork)>,
) -> usize {
let mut skipped = 0usize;
wanted.retain(|(req, _)| {
if req.size <= SWEEP_MAX_ORIGINAL_BYTES {
return true;
}
log::warn!(
"face sweep: {} is {} MB, over the {} MB budget for a background fetch; \
marked as examined and skipped",
req.path,
req.size >> 20,
SWEEP_MAX_ORIGINAL_BYTES >> 20
);
if let Err(e) = dr_catalog::faces::record_detections(
catalog.connection(),
dr_types::ImageId(req.image_id as u64),
model_id,
0,
&[],
) {
log::warn!("face sweep: marking {} as skipped: {e}", req.image_id);
}
skipped += 1;
false
});
skipped
}
/// What the sweep does with one fetched original.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum SweepWork {
@@ -4184,10 +4243,16 @@ pub fn spawn_face_sweep(
Err(e) => log::warn!("face sweep: looking for images under a weaker detector: {e}"),
}
let skipped = set_aside_oversized(&catalog, &model_id, &mut wanted);
let total = wanted.len();
if total == 0 {
log::info!("face sweep: every image has been through this model");
finish_empty(&tx);
let _ = tx.send(FaceSweepMessage::Finished {
images: 0,
faces: 0,
failed: skipped,
});
return;
}
if upgrades > 0 {
@@ -4222,7 +4287,10 @@ pub fn spawn_face_sweep(
}
};
let (mut images, mut found, mut failed) = (0usize, 0usize, 0usize);
// `failed` starts at what the size budget already set aside, so
// the report counts them the way it counts a file that would not
// decode: not indexed, and said so.
let (mut images, mut found, mut failed) = (0usize, 0usize, skipped);
let mut offline = false;
// TRACES: FR-NC-6c
@@ -6784,6 +6852,47 @@ mod tests {
let _ = std::fs::remove_dir_all(&store_dir);
}
/// TRACES: FR-CULL-8 | NFR-RES-2
/// A panorama over the fetch budget leaves the work list before anything
/// is fetched, and does not come back on the next pass.
#[test]
fn an_original_over_the_size_budget_is_set_aside_once() {
let catalog = Catalog::in_memory().unwrap();
let images = vec![
entry("PhotosRaw/IMG_4181.dng", 1, 23_230_634),
entry("PhotosRaw/IMG_4181-Pano.dng", 2, 521_218_956),
entry("PhotosRaw/IMG_4182.dng", 3, 22_841_218),
];
let result = dr_sync::ScanResult {
images,
directories: vec![(RemotePath::new("PhotosRaw"), dr_sync::Validator::new("e"))],
progress: Default::default(),
sidecars: Vec::new(),
};
persist(&catalog, "PhotosRaw", &result).unwrap();
let mut wanted: Vec<_> = faces_unindexed(&catalog, "w600k_mbf")
.unwrap()
.into_iter()
.map(|r| (r, SweepWork::Detect))
.collect();
assert_eq!(wanted.len(), 3);
let skipped = set_aside_oversized(&catalog, "w600k_mbf", &mut wanted);
assert_eq!(skipped, 1);
assert_eq!(wanted.len(), 2, "the two camera files are still work");
assert!(
wanted.iter().all(|(r, _)| !r.path.contains("Pano")),
"the panorama must not be fetched"
);
// Marked, so the next pass does not offer it again: the point is to
// spend the half gigabyte never, not once per sweep.
let again = faces_unindexed(&catalog, "w600k_mbf").unwrap();
assert_eq!(again.len(), 2);
assert!(again.iter().all(|r| !r.path.contains("Pano")));
}
/// The regression this whole pass exists for.
///
/// The previous work list intersected with the thumbnail store, so a