dtourolle
5c00942b84
A library's records are never all complete at once. A face found before its quality was kept has no quality; one found before the eye models existed has no reading; one adopted from a peer's shard has no crop; an image the fast detector examined on a 1024 px proxy has boxes the current detector would not have drawn; an image the scan stat'ed has no capture date. On the reference library that is 17,762 faces under the bare w600k_mbf id with no quality, no reading and no dense landmarks, 4,144 of them without a crop, beside 12,217 images the fast detector examined and found nothing in. Every one of those gaps was its own pass — V14's measuring pass, §17.5's eye pass, the sweep's proxy repair, the sweep's detector upgrade — with its own work list, its own count and its own idea of done, and adding a per-face field meant adding a pass. There was no pass at all for the case the library is actually in: boxes and landmarks drawn by a weaker detector on a proxy, which every later per-face pass would have read from. dr_ui::repairs replaces them with one job over a registry. A Repair names one thing a record can lack — the predicate that says which images still owe it, the input its handler needs (a header, the original, or a native render), the handler, and what to record for an image that can never be done. The job unions the predicates into one work list, fetches each image once at the most any claimant asks for, renders it at most once, and runs every handler whose predicate that image still matches, checked again before each because a detection writes every field a per-face handler would fill. The registry today: face-proxy, face-quality, face-eyes, face-crop, face-detection, face-upgrade, metadata — the last there to say that this is not a face job. Adding a field is one entry. A repair's predicate is the only definition of its work: the count the settings page shows, the list the job fetches and the check before its handler run are one predicate, so the job converges. That is why the registry is cut to what the device can do rather than listing what it skips — an entry is a count and a set of originals to fetch — and why an eye reading that cannot be cut is not a criterion. The catalog side is generic to match: record_updates writes whichever fields a FaceUpdate carries and re-marks the image so the shards export it; faces_needing and count_needing answer a predicate the caller supplies, replacing the measuring pass's three special cases. Two buttons on the settings page run the job and differ in one predicate. "Index faces" converges on coverage: has anything examined this image. "Re-index every face" converges on provenance: face-detection claims every image with no marker under the chosen detector, in either of its forms (FaceDetector::model_ids, so a desktop in f32 and a tablet on the Hexagon do not re-index each other's work), and a marker saying a weaker one looked is not that. An original over the fetch budget is left exactly as it was under the re-index, where the sweep marks it examined: a re-detection with nothing found would delete the faces, and "cannot fetch" is not "no faces".
1617 lines
59 KiB
Rust
1617 lines
59 KiB
Rust
//! TRACES: FR-CULL-8 | FR-CULL-10 | FR-CAT-3 | NFR-ARCH-2 | NFR-RES-2
|
|
//! The completeness job: what a catalog record can lack, and how to fill it.
|
|
//!
|
|
//! A library's records are never all complete at once. A face found before
|
|
//! its quality was kept has no quality; one found before the eye models
|
|
//! existed has no reading; one adopted from a peer's shard has no crop; an
|
|
//! image the fast detector examined on a 1024 px proxy has boxes the current
|
|
//! detector would not have drawn; an image the scan stat'ed has no capture
|
|
//! date. Every one of those used to be its own pass with its own work list,
|
|
//! its own button and its own idea of "done", and adding a field to a record
|
|
//! meant adding a pass.
|
|
//!
|
|
//! This module is one job over a **registry**. A [`Repair`] names one thing
|
|
//! a record can lack: the predicate that says which images still owe it,
|
|
//! the input its handler needs (a header, the original, or a native
|
|
//! render), and the handler that fills it. The job unions the predicates
|
|
//! into one work list, fetches each image once and renders it once, and
|
|
//! runs every handler whose predicate that image still matches — checked
|
|
//! again before each handler, because one handler's write satisfies the
|
|
//! next's predicate (a re-detection writes every field a per-face handler
|
|
//! would have filled). Adding a per-face field, or a per-image one, is one
|
|
//! entry in [`registry`].
|
|
//!
|
|
//! # Convergence is the property, and the predicate is what carries it
|
|
//!
|
|
//! A repair's `needs` is the *only* definition of its work: the count the
|
|
//! settings page shows, the list the job fetches and the check before its
|
|
//! handler runs are one predicate, so a record the count reports is one the
|
|
//! job fetches, and a record the job fetches is one the handler fills — and
|
|
//! the job ends. A handler that cannot fill what its predicate lists is a
|
|
//! job that fetches the same originals on every press, which is why an eye
|
|
//! reading that cannot be cut is *not* a criterion (a face stays unread
|
|
//! however often it is detected) and why a degenerate face is dropped
|
|
//! rather than left.
|
|
//!
|
|
//! # Two scopes
|
|
//!
|
|
//! [`Scope::Outstanding`] is the converging pass behind "Index faces":
|
|
//! detection runs over what nothing has examined, and every other repair
|
|
//! over what it lists. [`Scope::Reindex`] is "Re-index every face": detection
|
|
//! runs over everything the *chosen detector* has not been over, whatever a
|
|
//! weaker one found there. Same job, one predicate differs.
|
|
|
|
use std::collections::HashSet;
|
|
use std::path::PathBuf;
|
|
use std::sync::mpsc::{Receiver, Sender};
|
|
|
|
use dr_catalog::faces::{self, FaceUpdate};
|
|
use dr_catalog::Catalog;
|
|
use dr_sync::{Connection, RemoteId, RemotePath};
|
|
use dr_thumbs::ThumbStore;
|
|
use dr_types::{FaceDetector, ImageId};
|
|
|
|
use crate::faces::FaceSweepMessage;
|
|
use crate::library::{FaceModelPaths, MetadataFound};
|
|
|
|
/// What a handler needs in hand for one image, cheapest first.
|
|
///
|
|
/// Ordered, because an image several repairs claim is fetched once at the
|
|
/// most a handler asks for: a header serves the metadata handler, and a
|
|
/// native render serves every face handler and the metadata one besides.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
|
|
pub enum Input {
|
|
/// The first `dr_decode::HEADER_BYTES` of the file: EXIF, and the
|
|
/// embedded preview's offset.
|
|
Header,
|
|
/// The whole original, undecoded.
|
|
Original,
|
|
/// The whole original, rendered at native resolution through the export
|
|
/// path (FR-CULL-8's first resolution).
|
|
NativeRender,
|
|
}
|
|
|
|
/// A set of image ids the catalog cannot compute alone -- one that asks the
|
|
/// thumbnail store.
|
|
pub type SetFn =
|
|
Box<dyn Fn(&Catalog, &ThumbStore) -> Result<HashSet<i64>, dr_catalog::CatalogError>>;
|
|
|
|
/// Which images still owe a repair.
|
|
pub enum Needs {
|
|
/// SQL over `images i JOIN remote r ON r.image_id = i.id`, true where the
|
|
/// image still owes it. Evaluated for the list, for the count, and again
|
|
/// per image before the handler runs.
|
|
Sql(String),
|
|
/// Evaluated once, at the start of the job.
|
|
Set(SetFn),
|
|
}
|
|
|
|
/// A handler: fill one image, given what was fetched for it.
|
|
pub type ApplyFn = fn(&mut Toolkit, &Catalog, &Target, &mut Fetched) -> Result<usize, Failure>;
|
|
|
|
/// What to record for an image that can never be done.
|
|
pub type GiveUpFn = fn(&Toolkit, &Catalog, &Target) -> Result<(), String>;
|
|
|
|
/// Why a handler could not do its work on one image.
|
|
#[derive(Debug)]
|
|
pub enum Failure {
|
|
/// The decoder refused the file. It will refuse it on every pass, so
|
|
/// the repair's `give_up` is called and the image is not fetched again.
|
|
Unreadable(String),
|
|
/// Anything else -- a catalog error, a model error. Left for the next
|
|
/// pass.
|
|
Other(String),
|
|
}
|
|
|
|
impl From<String> for Failure {
|
|
fn from(s: String) -> Self {
|
|
Failure::Other(s)
|
|
}
|
|
}
|
|
|
|
/// One thing the catalog can be missing for an image, and how to fill it.
|
|
pub struct Repair {
|
|
/// For the log.
|
|
pub name: &'static str,
|
|
/// For the settings line, after a count of images: "images with faces to
|
|
/// read for quality".
|
|
pub label: &'static str,
|
|
pub needs: Needs,
|
|
pub input: Input,
|
|
/// Fill it for one image. Returns how many records it wrote -- faces,
|
|
/// for a face handler; one, for a per-image one.
|
|
pub apply: ApplyFn,
|
|
/// What to record for an image that can never be done -- over the
|
|
/// fetch budget, or refused by the decoder -- so the job does not offer
|
|
/// it again. `None` leaves the image exactly as it was, which is right
|
|
/// for a repair over records that already exist.
|
|
pub give_up: Option<GiveUpFn>,
|
|
}
|
|
|
|
/// One image on the work list.
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
pub struct Target {
|
|
pub image: ImageId,
|
|
pub path: String,
|
|
/// The thumbnail store's key (`oc:fileid`).
|
|
pub file_id: Option<u64>,
|
|
pub size: u64,
|
|
}
|
|
|
|
/// What was fetched for one image, and what has been made of it so far.
|
|
///
|
|
/// The render is lazy and cached: the first handler that asks pays for it,
|
|
/// the rest read it. That is the whole reason the job runs handlers per
|
|
/// image rather than per repair -- a native render is the expensive step,
|
|
/// and the reference library holds three per-face fields to fill from one.
|
|
pub struct Fetched {
|
|
bytes: Vec<u8>,
|
|
input: Input,
|
|
session: Option<crate::develop::DevelopSession>,
|
|
frame: Option<dr_export::Frame>,
|
|
}
|
|
|
|
impl Fetched {
|
|
fn new(bytes: Vec<u8>, input: Input) -> Self {
|
|
Self {
|
|
bytes,
|
|
input,
|
|
session: None,
|
|
frame: None,
|
|
}
|
|
}
|
|
|
|
/// The bytes fetched -- the header, or the whole file, per [`Input`].
|
|
pub fn bytes(&self) -> &[u8] {
|
|
&self.bytes
|
|
}
|
|
|
|
/// The open session, for a handler that wants a second render out of
|
|
/// it -- the proxy the People screen crops from.
|
|
pub fn session(
|
|
&mut self,
|
|
gpu: &dr_gpu::GpuContext,
|
|
) -> Result<&mut crate::develop::DevelopSession, Failure> {
|
|
if self.input < Input::Original {
|
|
return Err(Failure::Other("only the header was fetched".into()));
|
|
}
|
|
if self.session.is_none() {
|
|
self.session =
|
|
Some(crate::library::open_native(gpu, &self.bytes).map_err(Failure::Unreadable)?);
|
|
}
|
|
Ok(self.session.as_mut().expect("just opened"))
|
|
}
|
|
|
|
/// The native render, made once.
|
|
pub fn frame(&mut self, gpu: &dr_gpu::GpuContext) -> Result<&dr_export::Frame, Failure> {
|
|
if self.frame.is_none() {
|
|
let frame = self
|
|
.session(gpu)?
|
|
.render_for_export(dr_types::ColourSpace::Srgb)
|
|
.map_err(Failure::Unreadable)?;
|
|
self.frame = Some(frame);
|
|
}
|
|
Ok(self.frame.as_ref().expect("just rendered"))
|
|
}
|
|
}
|
|
|
|
/// The models this device has loaded, handed to every face handler.
|
|
pub struct FaceModels {
|
|
pub detector: dr_face::Detector,
|
|
pub embedder: dr_face::Embedder,
|
|
/// `None` on a device without them: it detects and embeds, and its
|
|
/// faces have no eye reading until a device that has them measures.
|
|
pub eyes: Option<dr_face::EyeModels>,
|
|
}
|
|
|
|
/// Everything a handler may need besides the catalog and the image.
|
|
pub struct Toolkit {
|
|
/// `None` on a build with no adapter. Every handler that renders needs
|
|
/// it, and says so by failing rather than by being left out of the
|
|
/// registry: the registry is built from [`Capabilities`], which is
|
|
/// where a missing GPU takes those repairs out.
|
|
pub gpu: Option<dr_gpu::GpuContext>,
|
|
pub models: Option<FaceModels>,
|
|
/// The pipeline id this device indexes under.
|
|
pub model_id: String,
|
|
pub store: ThumbStore,
|
|
pub options: dr_face::DetectOptions,
|
|
}
|
|
|
|
impl Toolkit {
|
|
fn gpu(&self) -> Result<&dr_gpu::GpuContext, Failure> {
|
|
self.gpu
|
|
.as_ref()
|
|
.ok_or_else(|| Failure::Other("no GPU to render with".into()))
|
|
}
|
|
|
|
fn models(&mut self) -> Result<&mut FaceModels, Failure> {
|
|
self.models
|
|
.as_mut()
|
|
.ok_or_else(|| Failure::Other("face models not loaded".into()))
|
|
}
|
|
}
|
|
|
|
/// What this device can do, which decides which repairs are registered.
|
|
///
|
|
/// A repair a device cannot perform is left out rather than listed and
|
|
/// skipped, and that is not tidiness: a repair in the registry is a count
|
|
/// on the settings page and a set of originals the job will fetch, and a
|
|
/// device without the eye models must not fetch every original in the
|
|
/// library to do nothing to it.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
|
pub struct Capabilities {
|
|
pub gpu: bool,
|
|
pub face_models: bool,
|
|
pub eye_models: bool,
|
|
}
|
|
|
|
/// Which images the job visits. See the module note.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub enum Scope {
|
|
Outstanding,
|
|
Reindex,
|
|
}
|
|
|
|
const VISIBLE: &str = "i.shadowed_by IS NULL AND i.trashed_at IS NULL";
|
|
|
|
/// The registry: every repair this device can do, in the order the job
|
|
/// runs them over one image -- and the order the work list is built in,
|
|
/// so an image only the last repair lists comes after one the first does.
|
|
///
|
|
/// The order is deliberate. Under [`Scope::Outstanding`]: the faces the
|
|
/// People screen is drawing now and cannot (no proxy) first, then the ones
|
|
/// it is grouping on vectors the gallery rule cannot act on, then what
|
|
/// nothing has examined, then what a weaker detector examined; metadata
|
|
/// last, since the timeline is the grid's own concern and the grid fills
|
|
/// it as it browses. Under [`Scope::Reindex`] detection leads, because
|
|
/// after it the per-face repairs have nothing left to do on that image.
|
|
///
|
|
/// Within the list, detection runs before the per-face repairs on any one
|
|
/// image, for the same reason: a detection writes every field they fill.
|
|
pub fn registry(
|
|
scope: Scope,
|
|
model_id: &str,
|
|
detector: FaceDetector,
|
|
caps: Capabilities,
|
|
) -> Vec<Repair> {
|
|
let mut out = Vec::new();
|
|
let embedder = faces::embedder_of(model_id).to_string();
|
|
let fi_embedder = faces::embedder_sql("fi.model_id");
|
|
let f_embedder = faces::embedder_sql("f.model_id");
|
|
let faces_here = format!(
|
|
"EXISTS (SELECT 1 FROM faces f WHERE f.image_id = i.id AND {f_embedder} = '{embedder}')"
|
|
);
|
|
let marker_under = |ids: &[&str]| {
|
|
let list = ids
|
|
.iter()
|
|
.map(|id| format!("'{id}'"))
|
|
.collect::<Vec<_>>()
|
|
.join(", ");
|
|
format!("EXISTS (SELECT 1 FROM face_index fi WHERE fi.image_id = i.id AND fi.model_id IN ({list}))")
|
|
};
|
|
let face_needing = |pred: &str| {
|
|
format!(
|
|
"EXISTS (SELECT 1 FROM faces f WHERE f.image_id = i.id AND {f_embedder} = '{embedder}' AND ({pred}))"
|
|
)
|
|
};
|
|
|
|
let can_detect = caps.gpu && caps.face_models;
|
|
|
|
if can_detect && scope == Scope::Reindex {
|
|
// Keyed on the chosen detector, in both its forms, where everything
|
|
// else in this subsystem keys on the embedder. The sweep converges
|
|
// on coverage -- has anything looked -- and treats a face a weaker
|
|
// detector found on a proxy as found. The re-index converges on
|
|
// provenance: every box, landmark, crop and vector from the current
|
|
// detector over the native render, because those are what every
|
|
// later per-face pass reads (docs/faces.md §17.4a).
|
|
out.push(Repair {
|
|
name: "face-detection",
|
|
label: "images to detect faces in with the chosen detector",
|
|
needs: Needs::Sql(format!("NOT {}", marker_under(&detector.model_ids()))),
|
|
input: Input::NativeRender,
|
|
apply: detect,
|
|
// Faces already there stay: a re-detection with nothing found
|
|
// would delete them, and "cannot fetch" is not "no faces".
|
|
give_up: None,
|
|
});
|
|
}
|
|
|
|
if can_detect && scope == Scope::Outstanding {
|
|
// Faces with nothing left to be cut out of: the proxy an indexing
|
|
// pass fetched and did not keep, or an ordinary eviction. The
|
|
// People screen draws "no preview" for every cell and cannot repair
|
|
// itself, because the image has its marker.
|
|
let embedder_for_set = embedder.clone();
|
|
out.push(Repair {
|
|
name: "face-proxy",
|
|
label: "images whose faces have no proxy to draw",
|
|
needs: Needs::Set(Box::new(move |catalog, store| {
|
|
faces_without_proxy(catalog, store, &embedder_for_set)
|
|
})),
|
|
input: Input::NativeRender,
|
|
apply: detect,
|
|
give_up: None,
|
|
});
|
|
}
|
|
|
|
if can_detect {
|
|
out.push(Repair {
|
|
name: "face-quality",
|
|
label: "images with faces to read for quality",
|
|
needs: Needs::Sql(face_needing(NEEDS_QUALITY)),
|
|
input: Input::NativeRender,
|
|
apply: quality,
|
|
give_up: None,
|
|
});
|
|
}
|
|
if can_detect && caps.eye_models {
|
|
out.push(Repair {
|
|
name: "face-eyes",
|
|
label: "images with faces to read for eye state",
|
|
needs: Needs::Sql(face_needing(NEEDS_EYES)),
|
|
input: Input::NativeRender,
|
|
apply: eyes,
|
|
give_up: None,
|
|
});
|
|
}
|
|
if caps.gpu {
|
|
out.push(Repair {
|
|
name: "face-crop",
|
|
label: "images with faces without a crop",
|
|
needs: Needs::Sql(face_needing(NEEDS_CROP)),
|
|
input: Input::NativeRender,
|
|
apply: crop,
|
|
give_up: None,
|
|
});
|
|
}
|
|
|
|
if can_detect && scope == Scope::Outstanding {
|
|
// What nothing has examined. `face_index` records that detection
|
|
// *ran*, and an image with no face in it must not come back on the
|
|
// next pass -- otherwise a personal library, which is mostly
|
|
// landscapes and documents, never finishes. An image holding this
|
|
// embedder's faces but no marker is the state V14 left, and it is
|
|
// `face-quality`'s work, not this one's: a re-detection would carry
|
|
// the identities across by matching, where the update keeps them.
|
|
out.push(Repair {
|
|
name: "face-detection",
|
|
label: "images to index",
|
|
needs: Needs::Sql(format!(
|
|
"NOT EXISTS (SELECT 1 FROM face_index fi WHERE fi.image_id = i.id AND {fi_embedder} = '{embedder}')
|
|
AND NOT {faces_here}"
|
|
)),
|
|
input: Input::NativeRender,
|
|
apply: detect,
|
|
// An examination that found nothing is the honest record for an
|
|
// image that cannot be examined, and it is what stops the half
|
|
// gigabyte being spent once per sweep.
|
|
give_up: Some(mark_examined_empty),
|
|
});
|
|
let weaker = detector.supersedes();
|
|
if !weaker.is_empty() {
|
|
// Images a weaker detector indexed. Only ever upwards: a device
|
|
// set to the fast detector leaves a peer's thorough pass alone.
|
|
out.push(Repair {
|
|
name: "face-upgrade",
|
|
label: "images indexed by a weaker detector",
|
|
needs: Needs::Sql(marker_under(weaker)),
|
|
input: Input::NativeRender,
|
|
apply: detect,
|
|
give_up: None,
|
|
});
|
|
}
|
|
}
|
|
|
|
// Not a face at all, and here to say that this is not a face job. The
|
|
// grid dates images as it browses and the thumbnail sweep dates the
|
|
// rest; an image neither has reached is one this pass has the header
|
|
// of anyway, or can fetch for 256 KB.
|
|
out.push(Repair {
|
|
name: "metadata",
|
|
label: "images without capture metadata",
|
|
needs: Needs::Sql("i.metadata_state < 2".into()),
|
|
input: Input::Header,
|
|
apply: metadata,
|
|
give_up: None,
|
|
});
|
|
|
|
out
|
|
}
|
|
|
|
/// The per-face predicates, over `faces f`. Named once each because the
|
|
/// registry lists by them and the handler selects by them, and the two
|
|
/// agreeing is what makes a repair converge.
|
|
const NEEDS_QUALITY: &str = "f.quality IS NULL";
|
|
const NEEDS_EYES: &str = "(f.eye_right IS NULL OR f.landmarks_dense IS NULL)";
|
|
const NEEDS_CROP: &str = "f.crop IS NULL";
|
|
|
|
// ── the handlers ──────────────────────────────────────────────────────────
|
|
|
|
/// Detect and embed from scratch, replacing the image's faces and carrying
|
|
/// their identities across (`faces::record_detections`).
|
|
fn detect(
|
|
tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let gpu = tk.gpu()?.clone();
|
|
let options = tk.options;
|
|
let model_id = tk.model_id.clone();
|
|
let frame = fetched.frame(&gpu)?;
|
|
let edge = frame.width.max(frame.height);
|
|
let models = tk.models()?;
|
|
let found = crate::faces::index_native(
|
|
&mut models.detector,
|
|
&mut models.embedder,
|
|
models.eyes.as_mut(),
|
|
&frame.rgba,
|
|
frame.width as usize,
|
|
frame.height as usize,
|
|
&options,
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
|
|
// The proxy the People screen crops from, only where there is a face to
|
|
// cut out of it, and before the detections: a kill between the two
|
|
// leaves a proxy with no faces recorded -- which the next pass simply
|
|
// re-indexes -- rather than faces with no proxy, which is the state
|
|
// that draws an empty grid and cannot repair itself.
|
|
if !found.is_empty() {
|
|
if let Some(file_id) = target.file_id {
|
|
match fetched
|
|
.session(&gpu)?
|
|
.render_thumbnail(dr_thumbs::ThumbSize::Large.edge())
|
|
{
|
|
Ok((w, h, rgba)) => match dr_thumbs::encode_rgba(w, h, &rgba) {
|
|
Ok(bytes) => {
|
|
crate::library::store_thumbnail(
|
|
&mut tk.store,
|
|
file_id,
|
|
dr_thumbs::ThumbSize::Large,
|
|
&dr_thumbs::Thumbnail {
|
|
width: w,
|
|
height: h,
|
|
bytes,
|
|
},
|
|
);
|
|
}
|
|
Err(e) => log::debug!("encoding a face proxy: {e}"),
|
|
},
|
|
Err(e) => log::debug!("rendering a face proxy: {e}"),
|
|
}
|
|
}
|
|
}
|
|
|
|
faces::record_detections(catalog.connection(), target.image, &model_id, edge, &found)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
Ok(found.len())
|
|
}
|
|
|
|
/// Mark an image examined with nothing found, at a zero edge that says why.
|
|
fn mark_examined_empty(tk: &Toolkit, catalog: &Catalog, target: &Target) -> Result<(), String> {
|
|
faces::record_detections(catalog.connection(), target.image, &tk.model_id, 0, &[])
|
|
.map(|_| ())
|
|
.map_err(|e| e.to_string())
|
|
}
|
|
|
|
/// TRACES: FR-CULL-9
|
|
/// Embed a face again from the native render, for the raw vector and its
|
|
/// length (schema V14).
|
|
///
|
|
/// The size and sharpness gates are deliberately not re-applied. They
|
|
/// decide whether a face is worth *storing*, and these are stored; what is
|
|
/// being established now is how much the model can make of each, which is
|
|
/// the quality itself, and a face that would have failed a gate is precisely
|
|
/// one that should come out short and stop vouching for anyone.
|
|
fn quality(
|
|
tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let owed = faces::faces_needing(
|
|
catalog.connection(),
|
|
target.image,
|
|
&tk.model_id,
|
|
NEEDS_QUALITY,
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
if owed.is_empty() {
|
|
return Ok(0);
|
|
}
|
|
let gpu = tk.gpu()?.clone();
|
|
let frame = fetched.frame(&gpu)?;
|
|
let edge = frame.width.max(frame.height);
|
|
let models = tk.models()?;
|
|
// The eyes are read on the same warp where the device can, for the
|
|
// faces that have none: the pixels are in hand, and it spares the eye
|
|
// repair a second claim on this image.
|
|
let measured = crate::faces::measure_native(
|
|
&mut models.embedder,
|
|
models.eyes.as_mut(),
|
|
&frame.rgba,
|
|
frame.width as usize,
|
|
frame.height as usize,
|
|
&owed,
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
let n = measured.measured.len();
|
|
faces::record_updates(
|
|
catalog.connection(),
|
|
target.image,
|
|
&tk.model_id,
|
|
edge,
|
|
&measured.measured,
|
|
&measured.dropped,
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
Ok(n)
|
|
}
|
|
|
|
/// TRACES: FR-CULL-8a
|
|
/// Read a face's eyes and dense landmarks from the native render, with the
|
|
/// box and five landmarks it already has (schema V16, V18).
|
|
fn eyes(
|
|
tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let owed = faces::faces_needing(catalog.connection(), target.image, &tk.model_id, NEEDS_EYES)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
if owed.is_empty() {
|
|
return Ok(0);
|
|
}
|
|
let gpu = tk.gpu()?.clone();
|
|
let frame = fetched.frame(&gpu)?;
|
|
let edge = frame.width.max(frame.height);
|
|
let long_edge = edge as f32;
|
|
let Some(models) = tk.models()?.eyes.as_mut() else {
|
|
return Err(Failure::Other("eye models not loaded".into()));
|
|
};
|
|
let mut updates = Vec::new();
|
|
for f in &owed {
|
|
let mut landmarks = [(0.0_f32, 0.0_f32); 5];
|
|
for (o, &(x, y)) in landmarks.iter_mut().zip(f.landmarks.iter()) {
|
|
*o = (x * long_edge, y * long_edge);
|
|
}
|
|
let bbox = (
|
|
f.x * long_edge,
|
|
f.y * long_edge,
|
|
(f.x + f.w) * long_edge,
|
|
(f.y + f.h) * long_edge,
|
|
);
|
|
match models.read(
|
|
dr_face::Pixels::Rgba8(&frame.rgba),
|
|
frame.width as usize,
|
|
frame.height as usize,
|
|
bbox,
|
|
&landmarks,
|
|
) {
|
|
Ok(Some((reading, dense))) => updates.push(FaceUpdate {
|
|
eyes: Some((reading, dense.to_packed_bytes(long_edge))),
|
|
..FaceUpdate::for_face(f.id)
|
|
}),
|
|
// Nothing could be cut: the face stays unread, and is not
|
|
// listed again by anything -- see the module note.
|
|
Ok(None) => {}
|
|
Err(e) => log::debug!("eye reading failed: {e}"),
|
|
}
|
|
}
|
|
let n = updates.len();
|
|
if n > 0 {
|
|
faces::record_updates(
|
|
catalog.connection(),
|
|
target.image,
|
|
&tk.model_id,
|
|
edge,
|
|
&updates,
|
|
&[],
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
}
|
|
Ok(n)
|
|
}
|
|
|
|
/// Cut a face's crop out of the native render, for the People screen.
|
|
fn crop(
|
|
tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let owed = faces::faces_needing(catalog.connection(), target.image, &tk.model_id, NEEDS_CROP)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
if owed.is_empty() {
|
|
return Ok(0);
|
|
}
|
|
let gpu = tk.gpu()?.clone();
|
|
let frame = fetched.frame(&gpu)?;
|
|
let edge = frame.width.max(frame.height);
|
|
let long_edge = edge as f32;
|
|
let updates: Vec<FaceUpdate> = owed
|
|
.iter()
|
|
.filter_map(|f| {
|
|
let cut = crate::faces::cut_crop_native(
|
|
dr_face::Pixels::Rgba8(&frame.rgba),
|
|
frame.width as usize,
|
|
frame.height as usize,
|
|
(
|
|
f.x * long_edge,
|
|
f.y * long_edge,
|
|
f.w * long_edge,
|
|
f.h * long_edge,
|
|
),
|
|
)?;
|
|
Some(FaceUpdate {
|
|
crop: Some(cut),
|
|
..FaceUpdate::for_face(f.id)
|
|
})
|
|
})
|
|
.collect();
|
|
let n = updates.len();
|
|
if n > 0 {
|
|
faces::record_updates(
|
|
catalog.connection(),
|
|
target.image,
|
|
&tk.model_id,
|
|
edge,
|
|
&updates,
|
|
&[],
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
}
|
|
Ok(n)
|
|
}
|
|
|
|
/// TRACES: FR-CAT-3
|
|
/// Read the capture metadata out of the header and promote the image.
|
|
///
|
|
/// What the thumbnail sweep does per image, as a repair: a real date takes
|
|
/// the image to `metadata_state = 2`, and a file that genuinely has none is
|
|
/// marked examined too, so it is not fetched again -- the state the sweep
|
|
/// records for the dateless, for the same reason.
|
|
fn metadata(
|
|
_tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let found = match dr_decode::metadata(fetched.bytes()) {
|
|
Ok(md) => MetadataFound {
|
|
image_id: target.image.0 as i64,
|
|
captured_at: md.captured_at,
|
|
captured_offset: md.captured_offset,
|
|
camera: crate::library::camera_label(md.make.as_deref(), md.model.as_deref()),
|
|
lens: md.lens.map(|l| l.trim().to_string()),
|
|
iso: md.iso,
|
|
},
|
|
Err(e) => {
|
|
log::debug!("metadata for {}: {e}", target.path);
|
|
MetadataFound {
|
|
image_id: target.image.0 as i64,
|
|
captured_at: None,
|
|
captured_offset: None,
|
|
camera: None,
|
|
lens: None,
|
|
iso: None,
|
|
}
|
|
}
|
|
};
|
|
let dated = found.captured_at.is_some();
|
|
crate::library::write_metadata(catalog, std::slice::from_ref(&found))
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
if !dated {
|
|
catalog
|
|
.connection()
|
|
.execute(
|
|
"UPDATE images SET metadata_state = 2 WHERE id = ?1",
|
|
[target.image.0 as i64],
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
}
|
|
Ok(1)
|
|
}
|
|
|
|
// ── the scan ──────────────────────────────────────────────────────────────
|
|
|
|
/// Images holding this embedder's faces whose proxy is not in the store.
|
|
fn faces_without_proxy(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
embedder: &str,
|
|
) -> Result<HashSet<i64>, dr_catalog::CatalogError> {
|
|
let mut stmt = catalog.connection().prepare(&format!(
|
|
"SELECT DISTINCT i.id, r.file_id
|
|
FROM images i
|
|
JOIN remote r ON r.image_id = i.id
|
|
JOIN faces f ON f.image_id = i.id
|
|
WHERE r.file_id IS NOT NULL AND {VISIBLE} AND {} = ?1",
|
|
faces::embedder_sql("f.model_id"),
|
|
))?;
|
|
let rows = stmt
|
|
.query_map([embedder], |r| {
|
|
Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?))
|
|
})?
|
|
.filter_map(Result::ok)
|
|
.filter(|(_, file_id)| !store.contains(*file_id as u64, dr_thumbs::ThumbSize::Large))
|
|
.map(|(id, _)| id)
|
|
.collect();
|
|
Ok(rows)
|
|
}
|
|
|
|
/// The images one repair lists, in the order the job visits them: the ones
|
|
/// holding faces first -- they are what the People screen is drawing and
|
|
/// what a re-detection carries names across -- then the rest, by id.
|
|
fn listed(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
repair: &Repair,
|
|
) -> Result<Vec<Target>, dr_catalog::CatalogError> {
|
|
let (predicate, set) = match &repair.needs {
|
|
Needs::Sql(sql) => (sql.clone(), None),
|
|
Needs::Set(f) => ("1".to_string(), Some(f(catalog, store)?)),
|
|
};
|
|
let mut stmt = catalog.connection().prepare(&format!(
|
|
"SELECT i.id, i.source_ref, r.file_id, i.file_size
|
|
FROM images i
|
|
JOIN remote r ON r.image_id = i.id
|
|
WHERE r.file_id IS NOT NULL AND {VISIBLE} AND ({predicate})
|
|
ORDER BY EXISTS (SELECT 1 FROM faces f WHERE f.image_id = i.id) DESC, i.id"
|
|
))?;
|
|
let rows = stmt
|
|
.query_map([], |r| {
|
|
Ok(Target {
|
|
image: ImageId(r.get::<_, i64>(0)? as u64),
|
|
path: r.get(1)?,
|
|
file_id: r.get::<_, Option<i64>>(2)?.map(|v| v as u64),
|
|
size: r.get::<_, Option<i64>>(3)?.unwrap_or(0) as u64,
|
|
})
|
|
})?
|
|
.filter_map(Result::ok)
|
|
.filter(|t| set.as_ref().is_none_or(|s| s.contains(&(t.image.0 as i64))))
|
|
.collect();
|
|
Ok(rows)
|
|
}
|
|
|
|
/// Whether one image still owes a repair -- asked again before each
|
|
/// handler runs, because an earlier handler's write may have answered it.
|
|
fn still_owed(
|
|
catalog: &Catalog,
|
|
repair: &Repair,
|
|
set: Option<&HashSet<i64>>,
|
|
image: ImageId,
|
|
) -> bool {
|
|
match (&repair.needs, set) {
|
|
(Needs::Set(_), Some(s)) => s.contains(&(image.0 as i64)),
|
|
(Needs::Set(_), None) => false,
|
|
(Needs::Sql(sql), _) => catalog
|
|
.connection()
|
|
.query_row(
|
|
&format!(
|
|
"SELECT EXISTS (SELECT 1 FROM images i JOIN remote r ON r.image_id = i.id
|
|
WHERE i.id = ?1 AND ({sql}))"
|
|
),
|
|
[image.0 as i64],
|
|
|r| r.get::<_, bool>(0),
|
|
)
|
|
.unwrap_or(false),
|
|
}
|
|
}
|
|
|
|
/// How many images each repair still lists, for the settings line.
|
|
pub fn counts(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
repairs: &[Repair],
|
|
) -> Result<Vec<(&'static str, u64)>, dr_catalog::CatalogError> {
|
|
repairs
|
|
.iter()
|
|
.map(|r| Ok((r.label, listed(catalog, store, r)?.len() as u64)))
|
|
.collect()
|
|
}
|
|
|
|
/// One image on the work list, with the most any repair claiming it asks
|
|
/// for.
|
|
struct Planned {
|
|
target: Target,
|
|
input: Input,
|
|
}
|
|
|
|
/// The work list, and each [`Needs::Set`] repair's set (`None` for a SQL
|
|
/// one), indexed like the registry.
|
|
type Plan = (Vec<Planned>, Vec<Option<HashSet<i64>>>);
|
|
|
|
/// The union of every repair's list, first claim first.
|
|
///
|
|
/// The order across repairs is the registry's, so the images the first
|
|
/// repair lists come first however many the last lists -- which is what
|
|
/// puts a few hundred proxy repairs ahead of twenty thousand un-indexed
|
|
/// images, where appended they would sit two hours down the queue.
|
|
fn plan(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
repairs: &[Repair],
|
|
) -> Result<Plan, dr_catalog::CatalogError> {
|
|
let mut out: Vec<Planned> = Vec::new();
|
|
let mut at: std::collections::HashMap<u64, usize> = std::collections::HashMap::new();
|
|
let mut sets = Vec::with_capacity(repairs.len());
|
|
for repair in repairs {
|
|
let set = match &repair.needs {
|
|
Needs::Set(f) => Some(f(catalog, store)?),
|
|
Needs::Sql(_) => None,
|
|
};
|
|
let listed = listed(catalog, store, repair)?;
|
|
if !listed.is_empty() {
|
|
log::info!("repairs: {}: {} image(s)", repair.name, listed.len());
|
|
}
|
|
for target in listed {
|
|
match at.get(&target.image.0) {
|
|
Some(&i) => out[i].input = out[i].input.max(repair.input),
|
|
None => {
|
|
at.insert(target.image.0, out.len());
|
|
out.push(Planned {
|
|
target,
|
|
input: repair.input,
|
|
});
|
|
}
|
|
}
|
|
}
|
|
sets.push(set);
|
|
}
|
|
Ok((out, sets))
|
|
}
|
|
|
|
/// Originals over this are not fetched for a repair that wants the whole
|
|
/// file. See `library::SWEEP_MAX_ORIGINAL_BYTES`.
|
|
const MAX_ORIGINAL_BYTES: u64 = crate::library::SWEEP_MAX_ORIGINAL_BYTES;
|
|
|
|
/// How many originals are in flight at once. See `library::SWEEP_LANES`.
|
|
const LANES: usize = crate::library::SWEEP_LANES;
|
|
|
|
// ── the job ───────────────────────────────────────────────────────────────
|
|
|
|
/// Run the registry over the library, at native resolution, in the
|
|
/// background.
|
|
///
|
|
/// Every image on the union work list is fetched once -- the header, or
|
|
/// the whole original, per the most any repair claiming it asks for --
|
|
/// rendered at most once, and handed to each repair whose predicate it
|
|
/// still matches. The receiver is the cancellation handle: dropping it
|
|
/// stops the job at the next image, and everything written stays written.
|
|
///
|
|
/// Resumable by construction, because the work list is what the catalog
|
|
/// says is incomplete: a kill costs the images in flight and nothing else.
|
|
///
|
|
/// # Cost, stated plainly
|
|
///
|
|
/// One whole original per image any face repair claims, and one full
|
|
/// render. On the reference library that is 412 GB and roughly a hundred
|
|
/// minutes of decode for a whole-library pass -- which is why FR-CULL-8
|
|
/// makes it a transfer under FR-NC-6 that starts when the user says so.
|
|
/// Nothing is kept that was not already wanted: the original is borrowed
|
|
/// and given back (ARCH §9.0a), and the only thing written per image
|
|
/// besides the catalog is the proxy the People screen crops from.
|
|
#[allow(clippy::too_many_arguments)]
|
|
pub fn spawn(
|
|
conn: Connection,
|
|
catalog_path: PathBuf,
|
|
store_dir: PathBuf,
|
|
models: Option<FaceModelPaths>,
|
|
model_id: String,
|
|
detector: FaceDetector,
|
|
scope: Scope,
|
|
options: dr_face::DetectOptions,
|
|
gpu: Option<dr_gpu::GpuContext>,
|
|
) -> Receiver<FaceSweepMessage> {
|
|
let (tx, rx) = std::sync::mpsc::channel();
|
|
|
|
std::thread::spawn(move || {
|
|
let finish_empty = |tx: &Sender<FaceSweepMessage>| {
|
|
let _ = tx.send(FaceSweepMessage::Finished {
|
|
images: 0,
|
|
faces: 0,
|
|
failed: 0,
|
|
});
|
|
};
|
|
|
|
let catalog = match Catalog::open(&catalog_path) {
|
|
Ok(c) => c,
|
|
Err(e) => {
|
|
log::warn!("repairs: cannot open catalog: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
let store = match ThumbStore::open(&store_dir) {
|
|
Ok(s) => s,
|
|
Err(e) => {
|
|
log::warn!("repairs: cannot open the thumbnail store: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
// Models before the work list: they are the expensive failure, and
|
|
// listing twenty thousand images before discovering the weights are
|
|
// missing helps nobody. A library with no model installed takes
|
|
// this path, and the registry is simply shorter.
|
|
let loaded = models.as_ref().and_then(|paths| {
|
|
let detector = match dr_face::Detector::from_path(&paths.detector) {
|
|
Ok(d) => d,
|
|
Err(e) => {
|
|
log::warn!("repairs: cannot load the detector: {e}");
|
|
return None;
|
|
}
|
|
};
|
|
let embedder = match dr_face::Embedder::from_path(
|
|
&paths.embedder,
|
|
dr_face::ModelId::new(model_id.clone()),
|
|
) {
|
|
Ok(e) => e,
|
|
Err(e) => {
|
|
log::warn!("repairs: cannot load the embedder: {e}");
|
|
return None;
|
|
}
|
|
};
|
|
Some(FaceModels {
|
|
detector,
|
|
embedder,
|
|
eyes: paths.load_eyes(),
|
|
})
|
|
});
|
|
let caps = Capabilities {
|
|
gpu: gpu.is_some(),
|
|
face_models: loaded.is_some(),
|
|
eye_models: loaded.as_ref().is_some_and(|m| m.eyes.is_some()),
|
|
};
|
|
let mut tk = Toolkit {
|
|
gpu,
|
|
models: loaded,
|
|
model_id: model_id.clone(),
|
|
store,
|
|
options,
|
|
};
|
|
let repairs = registry(scope, &model_id, detector, caps);
|
|
|
|
let (planned, sets) = match plan(&catalog, &tk.store, &repairs) {
|
|
Ok(p) => p,
|
|
Err(e) => {
|
|
log::warn!("repairs: listing the work: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
// Over budget, decided on the byte count the catalog holds rather
|
|
// than by fetching the file to find out. Given up on where a
|
|
// repair says how, left alone otherwise, and counted as failed
|
|
// either way: not done, and said so.
|
|
let mut skipped = 0usize;
|
|
let planned: Vec<Planned> = planned
|
|
.into_iter()
|
|
.filter(|p| {
|
|
if p.input == Input::Header || p.target.size <= MAX_ORIGINAL_BYTES {
|
|
return true;
|
|
}
|
|
log::warn!(
|
|
"repairs: {} is {} MB, over the {} MB budget for a background fetch; skipped",
|
|
p.target.path,
|
|
p.target.size >> 20,
|
|
MAX_ORIGINAL_BYTES >> 20
|
|
);
|
|
give_up(&tk, &catalog, &repairs, &sets, &p.target);
|
|
skipped += 1;
|
|
false
|
|
})
|
|
.collect();
|
|
|
|
let total = planned.len();
|
|
if total == 0 {
|
|
log::info!("repairs: nothing is incomplete");
|
|
let _ = tx.send(FaceSweepMessage::Finished {
|
|
images: 0,
|
|
faces: 0,
|
|
failed: skipped,
|
|
});
|
|
return;
|
|
}
|
|
log::info!("repairs: {total} image(s) to visit");
|
|
if tx.send(FaceSweepMessage::Total(total)).is_err() {
|
|
return;
|
|
}
|
|
|
|
let rt = match crate::net_runtime::build() {
|
|
Ok(rt) => rt,
|
|
Err(e) => {
|
|
log::warn!("repairs: no runtime: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
rt.block_on(async {
|
|
let backend = match crate::remote::connect(&conn) {
|
|
Ok(b) => b,
|
|
Err(e) => {
|
|
log::warn!("repairs: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
let (mut images, mut found, mut failed) = (0usize, 0usize, skipped);
|
|
let mut offline = false;
|
|
|
|
// TRACES: FR-NC-6c | NFR-RES-2
|
|
// Fetch wide, render narrow: one original per lane in flight,
|
|
// and the render sequential, because there is one GPU and each
|
|
// render materialises a native frame -- 96 MB for a 24 MP
|
|
// photograph. Every file borrowed is given back (ARCH §9.0a).
|
|
let pool = dr_sync_folder::BorrowPool::new();
|
|
|
|
for chunk in planned.chunks(LANES) {
|
|
let fetched = crate::library::futures_join_all(chunk.iter().map(|p| {
|
|
let backend = &*backend;
|
|
let pool = &pool;
|
|
async move {
|
|
let id = RemoteId::Path(RemotePath::new(&p.target.path));
|
|
let got = if p.input == Input::Header {
|
|
backend.get(&id, Some(0..dr_decode::HEADER_BYTES)).await
|
|
} else {
|
|
let held = match pool
|
|
.borrow(backend, &RemotePath::new(&p.target.path))
|
|
.await
|
|
{
|
|
Ok(h) => h,
|
|
Err(e) => return (p, Err(e)),
|
|
};
|
|
let got = backend.get(&id, None).await;
|
|
drop(held);
|
|
got
|
|
};
|
|
(p, got)
|
|
}
|
|
}))
|
|
.await;
|
|
|
|
let mut lane_failed = 0usize;
|
|
for (p, got) in fetched {
|
|
let bytes = match got {
|
|
Ok(b) => b,
|
|
Err(e) if e.indicates_offline() => {
|
|
log::info!("repairs: server unreachable: {e}");
|
|
offline = true;
|
|
continue;
|
|
}
|
|
Err(e) => {
|
|
log::debug!("repairs: {}: {e}", p.target.path);
|
|
lane_failed += 1;
|
|
continue;
|
|
}
|
|
};
|
|
|
|
let mut fetched = Fetched::new(bytes, p.input);
|
|
let mut wrote = 0usize;
|
|
let mut broke = false;
|
|
for (repair, set) in repairs.iter().zip(sets.iter()) {
|
|
if !still_owed(&catalog, repair, set.as_ref(), p.target.image) {
|
|
continue;
|
|
}
|
|
match (repair.apply)(&mut tk, &catalog, &p.target, &mut fetched) {
|
|
Ok(n) => wrote += n,
|
|
Err(Failure::Unreadable(e)) => {
|
|
// Fails the same way on every pass, and every
|
|
// pass fetched it first: a 521 MB panorama the
|
|
// decoder refuses was downloaded once per
|
|
// sweep, on a tablet.
|
|
log::warn!("repairs: {}: {e}", p.target.path);
|
|
give_up(&tk, &catalog, &repairs, &sets, &p.target);
|
|
broke = true;
|
|
break;
|
|
}
|
|
Err(Failure::Other(e)) => {
|
|
log::warn!("repairs: {} on {}: {e}", repair.name, p.target.path);
|
|
broke = true;
|
|
}
|
|
}
|
|
}
|
|
if broke {
|
|
lane_failed += 1;
|
|
continue;
|
|
}
|
|
images += 1;
|
|
found += wrote;
|
|
if tx
|
|
.send(FaceSweepMessage::Indexed {
|
|
image: p.target.image,
|
|
faces: wrote,
|
|
})
|
|
.is_err()
|
|
{
|
|
// Receiver dropped: the screen closed, or Stop.
|
|
log::info!("repairs: cancelled after {images} image(s)");
|
|
pool.release_all(&*backend).await;
|
|
return;
|
|
}
|
|
}
|
|
|
|
failed += lane_failed;
|
|
if lane_failed > 0
|
|
&& tx
|
|
.send(FaceSweepMessage::Failed {
|
|
images: lane_failed,
|
|
})
|
|
.is_err()
|
|
{
|
|
pool.release_all(&*backend).await;
|
|
return;
|
|
}
|
|
if offline {
|
|
break;
|
|
}
|
|
}
|
|
|
|
let returned = pool.release_all(&*backend).await;
|
|
if returned.released > 0 {
|
|
log::info!("repairs: released {} borrowed file(s)", returned.released);
|
|
}
|
|
log::info!(
|
|
"repairs: {found} record(s) written across {images} image(s), {failed} failed{}",
|
|
if offline { ", server went away" } else { "" }
|
|
);
|
|
let _ = tx.send(FaceSweepMessage::Finished {
|
|
images,
|
|
faces: found,
|
|
failed,
|
|
});
|
|
});
|
|
});
|
|
|
|
rx
|
|
}
|
|
|
|
/// Record, for every repair that still claims an image and knows how, that
|
|
/// it can never be done.
|
|
fn give_up(
|
|
tk: &Toolkit,
|
|
catalog: &Catalog,
|
|
repairs: &[Repair],
|
|
sets: &[Option<HashSet<i64>>],
|
|
target: &Target,
|
|
) {
|
|
for (repair, set) in repairs.iter().zip(sets.iter()) {
|
|
let Some(f) = repair.give_up else { continue };
|
|
if !still_owed(catalog, repair, set.as_ref(), target.image) {
|
|
continue;
|
|
}
|
|
if let Err(e) = f(tk, catalog, target) {
|
|
log::warn!("repairs: giving up {} on {}: {e}", repair.name, target.path);
|
|
}
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use dr_catalog::faces::DetectedFace;
|
|
|
|
fn with_images(n: usize) -> Catalog {
|
|
crate::library::test_support::with_images(n)
|
|
}
|
|
|
|
fn image_ids(catalog: &Catalog) -> Vec<ImageId> {
|
|
crate::library::test_support::image_ids(catalog)
|
|
}
|
|
|
|
fn store() -> (ThumbStore, PathBuf) {
|
|
let dir = std::env::temp_dir().join(format!(
|
|
"dr-repairs-test-{}-{:?}",
|
|
std::process::id(),
|
|
std::thread::current().id()
|
|
));
|
|
let _ = std::fs::remove_dir_all(&dir);
|
|
std::fs::create_dir_all(&dir).unwrap();
|
|
(ThumbStore::open(&dir).unwrap(), dir)
|
|
}
|
|
|
|
fn face(model: &str, quality: Option<f32>, crop: Vec<u8>) -> DetectedFace {
|
|
DetectedFace {
|
|
x: 0.1,
|
|
y: 0.1,
|
|
w: 0.2,
|
|
h: 0.2,
|
|
landmarks: [(0.0, 0.0); 5],
|
|
confidence: 0.9,
|
|
embedding: vec![0u8; 1024],
|
|
crop_px: 120.0,
|
|
quality,
|
|
eyes: None,
|
|
landmarks_dense: Vec::new(),
|
|
crop,
|
|
model_id: model.into(),
|
|
}
|
|
}
|
|
|
|
/// A face with every field a native detection writes.
|
|
fn complete(model: &str) -> DetectedFace {
|
|
DetectedFace {
|
|
eyes: Some(dr_face::EyeReading {
|
|
right: dr_face::Eye {
|
|
open: 0.9,
|
|
px: 40.0,
|
|
sharpness: 0.2,
|
|
},
|
|
left: dr_face::Eye {
|
|
open: 0.9,
|
|
px: 40.0,
|
|
sharpness: 0.2,
|
|
},
|
|
sunglasses: 0.1,
|
|
}),
|
|
landmarks_dense: vec![7; 424],
|
|
..face(model, Some(20.0), vec![1])
|
|
}
|
|
}
|
|
|
|
const ALL: Capabilities = Capabilities {
|
|
gpu: true,
|
|
face_models: true,
|
|
eye_models: true,
|
|
};
|
|
|
|
fn by_name<'a>(repairs: &'a [Repair], name: &str) -> &'a Repair {
|
|
repairs.iter().find(|r| r.name == name).expect(name)
|
|
}
|
|
|
|
fn ids_of(targets: &[Target]) -> Vec<u64> {
|
|
targets.iter().map(|t| t.image.0).collect()
|
|
}
|
|
|
|
/// The registry is what the device can do, and no more: a device that
|
|
/// cannot read eyes lists no faces to read, or it would fetch every
|
|
/// original in the library to do nothing to it.
|
|
#[test]
|
|
fn the_registry_is_cut_to_what_the_device_can_do() {
|
|
let names = |caps: Capabilities| -> Vec<&'static str> {
|
|
registry(
|
|
Scope::Outstanding,
|
|
"scrfd_10g+w600k_mbf",
|
|
FaceDetector::Scrfd10g,
|
|
caps,
|
|
)
|
|
.iter()
|
|
.map(|r| r.name)
|
|
.collect()
|
|
};
|
|
assert_eq!(
|
|
names(ALL),
|
|
vec![
|
|
"face-proxy",
|
|
"face-quality",
|
|
"face-eyes",
|
|
"face-crop",
|
|
"face-detection",
|
|
"face-upgrade",
|
|
"metadata"
|
|
]
|
|
);
|
|
assert!(!names(Capabilities {
|
|
eye_models: false,
|
|
..ALL
|
|
})
|
|
.contains(&"face-eyes"));
|
|
assert_eq!(names(Capabilities::default()), vec!["metadata"]);
|
|
// The fast detector supersedes nothing, so there is nothing to
|
|
// upgrade.
|
|
let fast = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
assert!(!fast.iter().any(|r| r.name == "face-upgrade"));
|
|
// Under a re-index, detection leads and the proxy repair is
|
|
// subsumed by it.
|
|
let re: Vec<_> = registry(
|
|
Scope::Reindex,
|
|
"scrfd_10g+w600k_mbf",
|
|
FaceDetector::Scrfd10g,
|
|
ALL,
|
|
)
|
|
.iter()
|
|
.map(|r| r.name)
|
|
.collect();
|
|
assert_eq!(re[0], "face-detection");
|
|
assert!(!re.contains(&"face-proxy"));
|
|
assert!(!re.contains(&"face-upgrade"));
|
|
}
|
|
|
|
/// The regression the whole-library pass exists for: nothing here puts
|
|
/// a proxy on disk, and every image is still work.
|
|
#[test]
|
|
fn every_unindexed_image_is_work_even_with_no_proxy_anywhere() {
|
|
let catalog = with_images(10);
|
|
let (store, dir) = store();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let wanted = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert_eq!(wanted.len(), 10);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// `face_index` records that detection *ran*, so an image with no face
|
|
/// in it must not come back -- otherwise a personal library, which is
|
|
/// mostly landscapes and documents, never finishes.
|
|
#[test]
|
|
fn an_image_already_run_over_is_not_work_again() {
|
|
let catalog = with_images(3);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
faces::record_detections(catalog.connection(), ids[0], "w600k_mbf", 1024, &[]).unwrap();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let wanted = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert_eq!(ids_of(&wanted), vec![ids[1].0, ids[2].0]);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// A detector change keeps the embedder, so it is not a new library:
|
|
/// the images the old detector ran over are an *upgrade*, listed after
|
|
/// what nothing has examined and only when the chosen detector outranks
|
|
/// the one that indexed them.
|
|
#[test]
|
|
fn a_stronger_detector_upgrades_rather_than_re_indexes() {
|
|
let catalog = with_images(3);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
let conn = catalog.connection();
|
|
faces::record_detections(conn, ids[0], "w600k_mbf", 1024, &[]).unwrap();
|
|
faces::record_detections(conn, ids[1], "scrfd_10g+w600k_mbf", 1024, &[]).unwrap();
|
|
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"scrfd_10g+w600k_mbf",
|
|
FaceDetector::Scrfd10g,
|
|
ALL,
|
|
);
|
|
let fresh = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert_eq!(ids_of(&fresh), vec![ids[2].0]);
|
|
let up = listed(&catalog, &store, by_name(&repairs, "face-upgrade")).unwrap();
|
|
assert_eq!(ids_of(&up), vec![ids[0].0]);
|
|
|
|
// And the plan puts the never-examined image first.
|
|
let (planned, _) = plan(&catalog, &store, &repairs).unwrap();
|
|
let order: Vec<u64> = planned.iter().map(|p| p.target.image.0).collect();
|
|
assert_eq!(order[..2], [ids[2].0, ids[0].0]);
|
|
// The third is the metadata repair's, and last.
|
|
assert_eq!(order[2], ids[1].0);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// The state schema V14 leaves: a face with no quality and an image
|
|
/// with no marker. It is the quality repair's work, and *only* that
|
|
/// repair's -- a full re-detection of the same image would throw away
|
|
/// every suggestion on it for nothing.
|
|
#[test]
|
|
fn an_unmeasured_face_is_measured_rather_than_re_detected() {
|
|
let catalog = with_images(3);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
let conn = catalog.connection();
|
|
faces::record_detections(
|
|
conn,
|
|
ids[0],
|
|
"w600k_mbf",
|
|
4000,
|
|
&[face("w600k_mbf", None, vec![1])],
|
|
)
|
|
.unwrap();
|
|
faces::record_detections(
|
|
conn,
|
|
ids[1],
|
|
"w600k_mbf",
|
|
4000,
|
|
&[face("w600k_mbf", Some(18.0), vec![1])],
|
|
)
|
|
.unwrap();
|
|
faces::clear_index_marker(conn, ids[0], "w600k_mbf").unwrap();
|
|
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let quality = listed(&catalog, &store, by_name(&repairs, "face-quality")).unwrap();
|
|
assert_eq!(ids_of(&quality), vec![ids[0].0]);
|
|
let detect = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert_eq!(
|
|
ids_of(&detect),
|
|
vec![ids[2].0],
|
|
"the unmeasured image is not re-detected"
|
|
);
|
|
// Neither face has an eye reading: both are the eye repair's.
|
|
let eyes = listed(&catalog, &store, by_name(&repairs, "face-eyes")).unwrap();
|
|
assert_eq!(ids_of(&eyes), vec![ids[0].0, ids[1].0]);
|
|
|
|
// One image, one fetch, three claims: the plan asks for the render
|
|
// once.
|
|
let (planned, _) = plan(&catalog, &store, &repairs).unwrap();
|
|
assert_eq!(
|
|
planned.iter().filter(|p| p.target.image == ids[0]).count(),
|
|
1
|
|
);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// The re-index is keyed on the chosen detector, not the embedder: an
|
|
/// image the fast detector examined is work, one the chosen detector
|
|
/// examined in either of its forms is not -- and it converges, because
|
|
/// a re-detection under the chosen detector takes the image off.
|
|
#[test]
|
|
fn the_re_index_lists_what_the_chosen_detector_has_not_been_over() {
|
|
let catalog = with_images(6);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
let conn = catalog.connection();
|
|
let current = FaceDetector::Scrfd10g.model_ids();
|
|
// 0: the fast detector found a face on a proxy -- work.
|
|
faces::record_detections(
|
|
conn,
|
|
ids[0],
|
|
"w600k_mbf",
|
|
1024,
|
|
&[face("w600k_mbf", None, vec![])],
|
|
)
|
|
.unwrap();
|
|
// 1: the fast detector found nothing -- still work.
|
|
faces::record_detections(conn, ids[1], "w600k_mbf", 1024, &[]).unwrap();
|
|
// 2, 3: the chosen detector, native, in each of its forms -- done.
|
|
faces::record_detections(conn, ids[2], current[0], 4000, &[complete(current[0])]).unwrap();
|
|
faces::record_detections(conn, ids[3], current[1], 4000, &[complete(current[1])]).unwrap();
|
|
// 4: the chosen detector's marker, but a face a peer's shard brought
|
|
// without its crop -- the crop repair's, not detection's.
|
|
faces::record_detections(
|
|
conn,
|
|
ids[4],
|
|
current[0],
|
|
4000,
|
|
&[face(current[0], Some(20.0), vec![])],
|
|
)
|
|
.unwrap();
|
|
// 5: never examined -- work.
|
|
|
|
let repairs = registry(Scope::Reindex, current[0], FaceDetector::Scrfd10g, ALL);
|
|
let detect = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
// Images holding faces first, then the rest, each in id order.
|
|
assert_eq!(ids_of(&detect), vec![ids[0].0, ids[1].0, ids[5].0]);
|
|
let crop = listed(&catalog, &store, by_name(&repairs, "face-crop")).unwrap();
|
|
assert_eq!(ids_of(&crop), vec![ids[0].0, ids[4].0]);
|
|
|
|
// Re-detecting the first under the chosen detector takes it off
|
|
// every list, since a detection writes every field.
|
|
faces::record_detections(conn, ids[0], current[0], 4000, &[complete(current[0])]).unwrap();
|
|
for r in &repairs {
|
|
if r.name.starts_with("face-") {
|
|
assert!(
|
|
!still_owed(&catalog, r, None, ids[0]),
|
|
"{} still claims a re-detected image",
|
|
r.name
|
|
);
|
|
}
|
|
}
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// The proxy repair is the one list the catalog cannot compute alone,
|
|
/// and it leads the plan: the image the screen cannot draw is fetched
|
|
/// before the rest of the library.
|
|
#[test]
|
|
fn repairs_are_reached_before_the_rest_of_the_library() {
|
|
let catalog = with_images(50);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
let orphan = ids[40];
|
|
faces::record_detections(
|
|
catalog.connection(),
|
|
orphan,
|
|
"w600k_mbf",
|
|
1024,
|
|
&[face("w600k_mbf", Some(20.0), vec![1])],
|
|
)
|
|
.unwrap();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let (planned, sets) = plan(&catalog, &store, &repairs).unwrap();
|
|
assert_eq!(planned[0].target.image, orphan);
|
|
assert_eq!(
|
|
planned.len(),
|
|
50,
|
|
"49 un-indexed plus the one being repaired"
|
|
);
|
|
assert!(still_owed(
|
|
&catalog,
|
|
by_name(&repairs, "face-proxy"),
|
|
sets[0].as_ref(),
|
|
orphan
|
|
));
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// A face with no proxy left draws "no preview" and cannot repair
|
|
/// itself: the image has its `face_index` row, so it is not outstanding
|
|
/// work. The job has to pick it up by a second route.
|
|
#[test]
|
|
fn a_face_whose_proxy_is_gone_is_work_again() {
|
|
let catalog = with_images(3);
|
|
let ids = image_ids(&catalog);
|
|
// An empty store, which is the state the bug lives in: the face is
|
|
// recorded and there is nothing on disk to cut it out of.
|
|
let (store, dir) = store();
|
|
faces::record_detections(
|
|
catalog.connection(),
|
|
ids[0],
|
|
"w600k_mbf",
|
|
1024,
|
|
&[face("w600k_mbf", Some(20.0), vec![1])],
|
|
)
|
|
.unwrap();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let detect = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert!(
|
|
!detect.iter().any(|t| t.image == ids[0]),
|
|
"indexed, so not outstanding"
|
|
);
|
|
let proxy = listed(&catalog, &store, by_name(&repairs, "face-proxy")).unwrap();
|
|
assert_eq!(
|
|
ids_of(&proxy),
|
|
vec![ids[0].0],
|
|
"the orphaned face was not picked up"
|
|
);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// Metadata is a repair like any other, and a header is all it asks
|
|
/// for -- so an image both claim is fetched whole, and one only it
|
|
/// claims is not.
|
|
#[test]
|
|
fn an_image_needs_the_most_any_repair_asks_of_it() {
|
|
let catalog = with_images(2);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
faces::record_detections(catalog.connection(), ids[1], "w600k_mbf", 1024, &[]).unwrap();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let (planned, _) = plan(&catalog, &store, &repairs).unwrap();
|
|
let input_of = |id: ImageId| planned.iter().find(|p| p.target.image == id).unwrap().input;
|
|
assert_eq!(input_of(ids[0]), Input::NativeRender);
|
|
assert_eq!(input_of(ids[1]), Input::Header);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
}
|