dtourolle
6b1aac477d
docs/ had 26 developer documents flat beside the manual, and the two audiences are very differently sized: most readers want the manual and the gesture reference, a few want the register, the designs and the measurements. The manual and gestures.md stay at the top; everything for someone changing the code moves to docs/dev/, and the two documents that name their own successors — the v0.1 milestone and the UI-refinement plan — go to docs/dev/archive/ rather than being deleted, since both are still cited. docs/README.md is the index, users first. Every reference follows: code comments, Cargo manifests, the workflows, the pre-commit hook, the bench and traceability tools (which locate the repo root by docs/dev/requirements.md now), packaging, the Docker READMEs, CLAUDE.md, CONTRIBUTING.md and the README. The matrix links one level deeper and is regenerated. Links out of the moved documents into the tree gain a level; a link checker over every Markdown file finds none broken.
1776 lines
66 KiB
Rust
1776 lines
66 KiB
Rust
//! TRACES: FR-CULL-8 | FR-CULL-10 | FR-CAT-3 | NFR-ARCH-2 | NFR-RES-2
|
|
//! The completeness job: what a catalog record can lack, and how to fill it.
|
|
//!
|
|
//! A library's records are never all complete at once. A face found before
|
|
//! its quality was kept has no quality; one found before the eye models
|
|
//! existed has no reading; one adopted from a peer's shard has no crop; an
|
|
//! image the fast detector examined on a 1024 px proxy has boxes the current
|
|
//! detector would not have drawn; an image the scan stat'ed has no capture
|
|
//! date. Every one of those used to be its own pass with its own work list,
|
|
//! its own button and its own idea of "done", and adding a field to a record
|
|
//! meant adding a pass.
|
|
//!
|
|
//! This module is one job over a **registry**. A [`Repair`] names one thing
|
|
//! a record can lack: the predicate that says which images still owe it,
|
|
//! the input its handler needs (a header, the original, or a native
|
|
//! render), and the handler that fills it. The job unions the predicates
|
|
//! into one work list, fetches each image once and renders it once, and
|
|
//! runs every handler whose predicate that image still matches — checked
|
|
//! again before each handler, because one handler's write satisfies the
|
|
//! next's predicate (a re-detection writes every field a per-face handler
|
|
//! would have filled). Adding a per-face field, or a per-image one, is one
|
|
//! entry in [`registry`].
|
|
//!
|
|
//! # Convergence is the property, and the predicate is what carries it
|
|
//!
|
|
//! A repair's `needs` is the *only* definition of its work: the count the
|
|
//! settings page shows, the list the job fetches and the check before its
|
|
//! handler runs are one predicate, so a record the count reports is one the
|
|
//! job fetches, and a record the job fetches is one the handler fills — and
|
|
//! the job ends. A handler that cannot fill what its predicate lists is a
|
|
//! job that fetches the same originals on every press, which is why an eye
|
|
//! reading that cannot be cut is *not* a criterion (a face stays unread
|
|
//! however often it is detected) and why a degenerate face is dropped
|
|
//! rather than left.
|
|
//!
|
|
//! # Two scopes
|
|
//!
|
|
//! [`Scope::Outstanding`] is the converging pass behind "Index faces":
|
|
//! detection runs over what nothing has examined, and every other repair
|
|
//! over what it lists. [`Scope::Reindex`] is "Re-index every face": detection
|
|
//! runs over everything the *chosen detector* has not been over, whatever a
|
|
//! weaker one found there. Same job, one predicate differs.
|
|
|
|
use std::collections::HashSet;
|
|
use std::path::PathBuf;
|
|
use std::sync::mpsc::{Receiver, Sender};
|
|
|
|
use dr_catalog::faces::{self, FaceUpdate};
|
|
use dr_catalog::Catalog;
|
|
use dr_sync::{Connection, RemoteId, RemotePath};
|
|
use dr_thumbs::ThumbStore;
|
|
use dr_types::{FaceDetector, ImageId};
|
|
|
|
use crate::faces::FaceSweepMessage;
|
|
use crate::library::{FaceModelPaths, MetadataFound};
|
|
|
|
/// What a handler needs in hand for one image, cheapest first.
|
|
///
|
|
/// Ordered, because an image several repairs claim is fetched once at the
|
|
/// most a handler asks for: a header serves the metadata handler, and a
|
|
/// native render serves every face handler and the metadata one besides.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
|
|
pub enum Input {
|
|
/// The first `dr_decode::HEADER_BYTES` of the file: EXIF, and the
|
|
/// embedded preview's offset.
|
|
Header,
|
|
/// The whole original, undecoded.
|
|
Original,
|
|
/// The whole original, rendered at native resolution through the export
|
|
/// path (FR-CULL-8's first resolution).
|
|
NativeRender,
|
|
}
|
|
|
|
/// A set of image ids the catalog cannot compute alone -- one that asks the
|
|
/// thumbnail store.
|
|
pub type SetFn =
|
|
Box<dyn Fn(&Catalog, &ThumbStore) -> Result<HashSet<i64>, dr_catalog::CatalogError>>;
|
|
|
|
/// Which images still owe a repair.
|
|
pub enum Needs {
|
|
/// SQL over `images i JOIN remote r ON r.image_id = i.id`, true where the
|
|
/// image still owes it. Evaluated for the list, for the count, and again
|
|
/// per image before the handler runs.
|
|
Sql(String),
|
|
/// SQL over `faces f`, true where the face still owes it; the image owes
|
|
/// the repair if any of its faces does.
|
|
///
|
|
/// Kept as the per-face fragment rather than folded into an image
|
|
/// predicate, because the two questions asked of it want opposite
|
|
/// shapes. The list and the per-image check want `EXISTS (... WHERE
|
|
/// f.image_id = i.id AND fragment)`, one probe per image. The count
|
|
/// wants to start from the faces, where the partial indexes V19 keeps
|
|
/// for exactly these fragments make it a walk over the few thousand
|
|
/// still owing rather than a probe into eight-kilobyte rows for every
|
|
/// image in the library -- and the planner will not use those indexes
|
|
/// from inside the EXISTS.
|
|
Face(String),
|
|
/// Evaluated once, at the start of the job.
|
|
Set(SetFn),
|
|
}
|
|
|
|
/// [`Needs::Face`] as an image predicate: the image holds a face owing it.
|
|
fn any_face(fragment: &str) -> String {
|
|
format!("EXISTS (SELECT 1 FROM faces f WHERE f.image_id = i.id AND {fragment})")
|
|
}
|
|
|
|
/// A handler: fill one image, given what was fetched for it.
|
|
pub type ApplyFn = fn(&mut Toolkit, &Catalog, &Target, &mut Fetched) -> Result<usize, Failure>;
|
|
|
|
/// What to record for an image that can never be done.
|
|
pub type GiveUpFn = fn(&Toolkit, &Catalog, &Target) -> Result<(), String>;
|
|
|
|
/// Why a handler could not do its work on one image.
|
|
#[derive(Debug)]
|
|
pub enum Failure {
|
|
/// The decoder refused the file. It will refuse it on every pass, so
|
|
/// the repair's `give_up` is called and the image is not fetched again.
|
|
Unreadable(String),
|
|
/// Anything else -- a catalog error, a model error. Left for the next
|
|
/// pass.
|
|
Other(String),
|
|
}
|
|
|
|
impl From<String> for Failure {
|
|
fn from(s: String) -> Self {
|
|
Failure::Other(s)
|
|
}
|
|
}
|
|
|
|
/// One thing the catalog can be missing for an image, and how to fill it.
|
|
pub struct Repair {
|
|
/// For the log.
|
|
pub name: &'static str,
|
|
/// For the settings line, after a count of images: "images with faces to
|
|
/// read for quality".
|
|
pub label: &'static str,
|
|
pub needs: Needs,
|
|
pub input: Input,
|
|
/// Fill it for one image. Returns how many records it wrote -- faces,
|
|
/// for a face handler; one, for a per-image one.
|
|
pub apply: ApplyFn,
|
|
/// What to record for an image that can never be done -- over the
|
|
/// fetch budget, or refused by the decoder -- so the job does not offer
|
|
/// it again. `None` leaves the image exactly as it was, which is right
|
|
/// for a repair over records that already exist.
|
|
pub give_up: Option<GiveUpFn>,
|
|
}
|
|
|
|
/// One image on the work list.
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
pub struct Target {
|
|
pub image: ImageId,
|
|
pub path: String,
|
|
/// The thumbnail store's key (`oc:fileid`).
|
|
pub file_id: Option<u64>,
|
|
pub size: u64,
|
|
}
|
|
|
|
/// What was fetched for one image, and what has been made of it so far.
|
|
///
|
|
/// The render is lazy and cached: the first handler that asks pays for it,
|
|
/// the rest read it. That is the whole reason the job runs handlers per
|
|
/// image rather than per repair -- a native render is the expensive step,
|
|
/// and the reference library holds three per-face fields to fill from one.
|
|
pub struct Fetched {
|
|
bytes: Vec<u8>,
|
|
input: Input,
|
|
session: Option<crate::develop::DevelopSession>,
|
|
frame: Option<dr_export::Frame>,
|
|
}
|
|
|
|
impl Fetched {
|
|
fn new(bytes: Vec<u8>, input: Input) -> Self {
|
|
Self {
|
|
bytes,
|
|
input,
|
|
session: None,
|
|
frame: None,
|
|
}
|
|
}
|
|
|
|
/// The bytes fetched -- the header, or the whole file, per [`Input`].
|
|
pub fn bytes(&self) -> &[u8] {
|
|
&self.bytes
|
|
}
|
|
|
|
/// The open session, for a handler that wants a second render out of
|
|
/// it -- the proxy the People screen crops from.
|
|
pub fn session(
|
|
&mut self,
|
|
gpu: &dr_gpu::GpuContext,
|
|
) -> Result<&mut crate::develop::DevelopSession, Failure> {
|
|
if self.input < Input::Original {
|
|
return Err(Failure::Other("only the header was fetched".into()));
|
|
}
|
|
if self.session.is_none() {
|
|
self.session =
|
|
Some(crate::library::open_native(gpu, &self.bytes).map_err(Failure::Unreadable)?);
|
|
}
|
|
Ok(self.session.as_mut().expect("just opened"))
|
|
}
|
|
|
|
/// The native render, made once.
|
|
pub fn frame(&mut self, gpu: &dr_gpu::GpuContext) -> Result<&dr_export::Frame, Failure> {
|
|
if self.frame.is_none() {
|
|
let frame = self
|
|
.session(gpu)?
|
|
.render_for_export(dr_types::ColourSpace::Srgb)
|
|
.map_err(Failure::Unreadable)?;
|
|
self.frame = Some(frame);
|
|
}
|
|
Ok(self.frame.as_ref().expect("just rendered"))
|
|
}
|
|
}
|
|
|
|
/// The models this device has loaded, handed to every face handler.
|
|
pub struct FaceModels {
|
|
pub detector: dr_face::Detector,
|
|
pub embedder: dr_face::Embedder,
|
|
/// `None` on a device without them: it detects and embeds, and its
|
|
/// faces have no eye reading until a device that has them measures.
|
|
pub eyes: Option<dr_face::EyeModels>,
|
|
}
|
|
|
|
/// Everything a handler may need besides the catalog and the image.
|
|
pub struct Toolkit {
|
|
/// `None` on a build with no adapter. Every handler that renders needs
|
|
/// it, and says so by failing rather than by being left out of the
|
|
/// registry: the registry is built from [`Capabilities`], which is
|
|
/// where a missing GPU takes those repairs out.
|
|
pub gpu: Option<dr_gpu::GpuContext>,
|
|
pub models: Option<FaceModels>,
|
|
/// The pipeline id this device indexes under.
|
|
pub model_id: String,
|
|
pub store: ThumbStore,
|
|
pub options: dr_face::DetectOptions,
|
|
}
|
|
|
|
impl Toolkit {
|
|
fn gpu(&self) -> Result<&dr_gpu::GpuContext, Failure> {
|
|
self.gpu
|
|
.as_ref()
|
|
.ok_or_else(|| Failure::Other("no GPU to render with".into()))
|
|
}
|
|
|
|
fn models(&mut self) -> Result<&mut FaceModels, Failure> {
|
|
self.models
|
|
.as_mut()
|
|
.ok_or_else(|| Failure::Other("face models not loaded".into()))
|
|
}
|
|
}
|
|
|
|
/// What this device can do, which decides which repairs are registered.
|
|
///
|
|
/// A repair a device cannot perform is left out rather than listed and
|
|
/// skipped, and that is not tidiness: a repair in the registry is a count
|
|
/// on the settings page and a set of originals the job will fetch, and a
|
|
/// device without the eye models must not fetch every original in the
|
|
/// library to do nothing to it.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
|
pub struct Capabilities {
|
|
pub gpu: bool,
|
|
pub face_models: bool,
|
|
pub eye_models: bool,
|
|
}
|
|
|
|
/// Which images the job visits. See the module note.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub enum Scope {
|
|
Outstanding,
|
|
Reindex,
|
|
}
|
|
|
|
const VISIBLE: &str = "i.shadowed_by IS NULL AND i.trashed_at IS NULL";
|
|
|
|
/// The registry: every repair this device can do, in the order the job
|
|
/// runs them over one image -- and the order the work list is built in,
|
|
/// so an image only the last repair lists comes after one the first does.
|
|
///
|
|
/// The order is deliberate. Under [`Scope::Outstanding`]: the faces the
|
|
/// People screen is drawing now and cannot (no proxy) first, then the ones
|
|
/// it is grouping on vectors the gallery rule cannot act on, then what
|
|
/// nothing has examined, then what a weaker detector examined; metadata
|
|
/// last, since the timeline is the grid's own concern and the grid fills
|
|
/// it as it browses. Under [`Scope::Reindex`] detection leads, because
|
|
/// after it the per-face repairs have nothing left to do on that image.
|
|
///
|
|
/// Within the list, detection runs before the per-face repairs on any one
|
|
/// image, for the same reason: a detection writes every field they fill.
|
|
pub fn registry(
|
|
scope: Scope,
|
|
model_id: &str,
|
|
detector: FaceDetector,
|
|
caps: Capabilities,
|
|
) -> Vec<Repair> {
|
|
let mut out = Vec::new();
|
|
let embedder = faces::embedder_of(model_id).to_string();
|
|
let fi_embedder = faces::embedder_sql("fi.model_id");
|
|
let f_embedder = faces::embedder_sql("f.model_id");
|
|
let faces_here = format!(
|
|
"EXISTS (SELECT 1 FROM faces f WHERE f.image_id = i.id AND {f_embedder} = '{embedder}')"
|
|
);
|
|
let marker_under = |ids: &[&str]| {
|
|
let list = ids
|
|
.iter()
|
|
.map(|id| format!("'{id}'"))
|
|
.collect::<Vec<_>>()
|
|
.join(", ");
|
|
format!("EXISTS (SELECT 1 FROM face_index fi WHERE fi.image_id = i.id AND fi.model_id IN ({list}))")
|
|
};
|
|
// The per-face fragment `Needs::Face` carries: this embedder's face,
|
|
// still owing the pass. Spelled as the partial indexes' WHERE clauses
|
|
// are (V19), which is what lets the count be served from them.
|
|
let face_needing = |pred: &str| format!("{f_embedder} = '{embedder}' AND ({pred})");
|
|
|
|
let can_detect = caps.gpu && caps.face_models;
|
|
|
|
if can_detect && scope == Scope::Reindex {
|
|
// Keyed on the chosen detector, in both its forms, where everything
|
|
// else in this subsystem keys on the embedder. The sweep converges
|
|
// on coverage -- has anything looked -- and treats a face a weaker
|
|
// detector found on a proxy as found. The re-index converges on
|
|
// provenance: every box, landmark, crop and vector from the current
|
|
// detector over the native render, because those are what every
|
|
// later per-face pass reads (docs/dev/faces.md §17.4a).
|
|
out.push(Repair {
|
|
name: "face-detection",
|
|
label: "images to detect faces in with the chosen detector",
|
|
needs: Needs::Sql(format!("NOT {}", marker_under(&detector.model_ids()))),
|
|
input: Input::NativeRender,
|
|
apply: detect,
|
|
// Faces already there stay: a re-detection with nothing found
|
|
// would delete them, and "cannot fetch" is not "no faces".
|
|
give_up: None,
|
|
});
|
|
}
|
|
|
|
if can_detect && scope == Scope::Outstanding {
|
|
// Faces with nothing left to be cut out of: the proxy an indexing
|
|
// pass fetched and did not keep, or an ordinary eviction. The
|
|
// People screen draws "no preview" for every cell and cannot repair
|
|
// itself, because the image has its marker.
|
|
let embedder_for_set = embedder.clone();
|
|
out.push(Repair {
|
|
name: "face-proxy",
|
|
label: "images whose faces have no proxy to draw",
|
|
needs: Needs::Set(Box::new(move |catalog, store| {
|
|
faces_without_proxy(catalog, store, &embedder_for_set)
|
|
})),
|
|
input: Input::NativeRender,
|
|
apply: detect,
|
|
give_up: None,
|
|
});
|
|
}
|
|
|
|
if can_detect {
|
|
out.push(Repair {
|
|
name: "face-quality",
|
|
label: "images with faces to read for quality",
|
|
needs: Needs::Face(face_needing(NEEDS_QUALITY)),
|
|
input: Input::NativeRender,
|
|
apply: quality,
|
|
give_up: None,
|
|
});
|
|
}
|
|
if can_detect && caps.eye_models {
|
|
out.push(Repair {
|
|
name: "face-eyes",
|
|
label: "images with faces to read for eye state",
|
|
needs: Needs::Face(face_needing(NEEDS_EYES)),
|
|
input: Input::NativeRender,
|
|
apply: eyes,
|
|
give_up: None,
|
|
});
|
|
}
|
|
if caps.gpu {
|
|
out.push(Repair {
|
|
name: "face-crop",
|
|
label: "images with faces without a crop",
|
|
needs: Needs::Face(face_needing(NEEDS_CROP)),
|
|
input: Input::NativeRender,
|
|
apply: crop,
|
|
give_up: None,
|
|
});
|
|
}
|
|
|
|
if can_detect && scope == Scope::Outstanding {
|
|
// What nothing has examined. `face_index` records that detection
|
|
// *ran*, and an image with no face in it must not come back on the
|
|
// next pass -- otherwise a personal library, which is mostly
|
|
// landscapes and documents, never finishes. An image holding this
|
|
// embedder's faces but no marker is the state V14 left, and it is
|
|
// `face-quality`'s work, not this one's: a re-detection would carry
|
|
// the identities across by matching, where the update keeps them.
|
|
out.push(Repair {
|
|
name: "face-detection",
|
|
label: "images to index",
|
|
needs: Needs::Sql(format!(
|
|
"NOT EXISTS (SELECT 1 FROM face_index fi WHERE fi.image_id = i.id AND {fi_embedder} = '{embedder}')
|
|
AND NOT {faces_here}"
|
|
)),
|
|
input: Input::NativeRender,
|
|
apply: detect,
|
|
// An examination that found nothing is the honest record for an
|
|
// image that cannot be examined, and it is what stops the half
|
|
// gigabyte being spent once per sweep.
|
|
give_up: Some(mark_examined_empty),
|
|
});
|
|
let weaker = detector.supersedes();
|
|
if !weaker.is_empty() {
|
|
// Images a weaker detector indexed. Only ever upwards: a device
|
|
// set to the fast detector leaves a peer's thorough pass alone.
|
|
out.push(Repair {
|
|
name: "face-upgrade",
|
|
label: "images indexed by a weaker detector",
|
|
needs: Needs::Sql(marker_under(weaker)),
|
|
input: Input::NativeRender,
|
|
apply: detect,
|
|
give_up: None,
|
|
});
|
|
}
|
|
}
|
|
|
|
// Not a face at all, and here to say that this is not a face job. The
|
|
// grid dates images as it browses and the thumbnail sweep dates the
|
|
// rest; an image neither has reached is one this pass has the header
|
|
// of anyway, or can fetch for 256 KB.
|
|
out.push(Repair {
|
|
name: "metadata",
|
|
label: "images without capture metadata",
|
|
needs: Needs::Sql("i.metadata_state < 2".into()),
|
|
input: Input::Header,
|
|
apply: metadata,
|
|
give_up: None,
|
|
});
|
|
|
|
out
|
|
}
|
|
|
|
/// The per-face predicates, over `faces f`. Named once each because the
|
|
/// registry lists by them and the handler selects by them, and the two
|
|
/// agreeing is what makes a repair converge.
|
|
const NEEDS_QUALITY: &str = "f.quality IS NULL";
|
|
const NEEDS_EYES: &str = "(f.eye_right IS NULL OR f.landmarks_dense IS NULL)";
|
|
const NEEDS_CROP: &str = "f.crop IS NULL";
|
|
|
|
// ── the handlers ──────────────────────────────────────────────────────────
|
|
|
|
/// Detect and embed from scratch, replacing the image's faces and carrying
|
|
/// their identities across (`faces::record_detections`).
|
|
fn detect(
|
|
tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let gpu = tk.gpu()?.clone();
|
|
let options = tk.options;
|
|
let model_id = tk.model_id.clone();
|
|
let frame = fetched.frame(&gpu)?;
|
|
let edge = frame.width.max(frame.height);
|
|
let models = tk.models()?;
|
|
let found = crate::faces::index_native(
|
|
&mut models.detector,
|
|
&mut models.embedder,
|
|
models.eyes.as_mut(),
|
|
&frame.rgba,
|
|
frame.width as usize,
|
|
frame.height as usize,
|
|
&options,
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
|
|
// The proxy the People screen crops from, only where there is a face to
|
|
// cut out of it, and before the detections: a kill between the two
|
|
// leaves a proxy with no faces recorded -- which the next pass simply
|
|
// re-indexes -- rather than faces with no proxy, which is the state
|
|
// that draws an empty grid and cannot repair itself.
|
|
if !found.is_empty() {
|
|
if let Some(file_id) = target.file_id {
|
|
match fetched
|
|
.session(&gpu)?
|
|
.render_thumbnail(dr_thumbs::ThumbSize::Large.edge())
|
|
{
|
|
Ok((w, h, rgba)) => match dr_thumbs::encode_rgba(w, h, &rgba) {
|
|
Ok(bytes) => {
|
|
crate::library::store_thumbnail(
|
|
&mut tk.store,
|
|
file_id,
|
|
dr_thumbs::ThumbSize::Large,
|
|
&dr_thumbs::Thumbnail {
|
|
width: w,
|
|
height: h,
|
|
bytes,
|
|
},
|
|
);
|
|
}
|
|
Err(e) => log::debug!("encoding a face proxy: {e}"),
|
|
},
|
|
Err(e) => log::debug!("rendering a face proxy: {e}"),
|
|
}
|
|
}
|
|
}
|
|
|
|
faces::record_detections(catalog.connection(), target.image, &model_id, edge, &found)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
Ok(found.len())
|
|
}
|
|
|
|
/// Mark an image examined with nothing found, at a zero edge that says why.
|
|
fn mark_examined_empty(tk: &Toolkit, catalog: &Catalog, target: &Target) -> Result<(), String> {
|
|
faces::record_detections(catalog.connection(), target.image, &tk.model_id, 0, &[])
|
|
.map(|_| ())
|
|
.map_err(|e| e.to_string())
|
|
}
|
|
|
|
/// TRACES: FR-CULL-9
|
|
/// Embed a face again from the native render, for the raw vector and its
|
|
/// length (schema V14).
|
|
///
|
|
/// The size and sharpness gates are deliberately not re-applied. They
|
|
/// decide whether a face is worth *storing*, and these are stored; what is
|
|
/// being established now is how much the model can make of each, which is
|
|
/// the quality itself, and a face that would have failed a gate is precisely
|
|
/// one that should come out short and stop vouching for anyone.
|
|
fn quality(
|
|
tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let owed = faces::faces_needing(
|
|
catalog.connection(),
|
|
target.image,
|
|
&tk.model_id,
|
|
NEEDS_QUALITY,
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
if owed.is_empty() {
|
|
return Ok(0);
|
|
}
|
|
let gpu = tk.gpu()?.clone();
|
|
let frame = fetched.frame(&gpu)?;
|
|
let edge = frame.width.max(frame.height);
|
|
let models = tk.models()?;
|
|
// The eyes are read on the same warp where the device can, for the
|
|
// faces that have none: the pixels are in hand, and it spares the eye
|
|
// repair a second claim on this image.
|
|
let measured = crate::faces::measure_native(
|
|
&mut models.embedder,
|
|
models.eyes.as_mut(),
|
|
&frame.rgba,
|
|
frame.width as usize,
|
|
frame.height as usize,
|
|
&owed,
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
let n = measured.measured.len();
|
|
faces::record_updates(
|
|
catalog.connection(),
|
|
target.image,
|
|
&tk.model_id,
|
|
edge,
|
|
&measured.measured,
|
|
&measured.dropped,
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
Ok(n)
|
|
}
|
|
|
|
/// TRACES: FR-CULL-8a
|
|
/// Read a face's eyes and dense landmarks from the native render, with the
|
|
/// box and five landmarks it already has (schema V16, V18).
|
|
fn eyes(
|
|
tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let owed = faces::faces_needing(catalog.connection(), target.image, &tk.model_id, NEEDS_EYES)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
if owed.is_empty() {
|
|
return Ok(0);
|
|
}
|
|
let gpu = tk.gpu()?.clone();
|
|
let frame = fetched.frame(&gpu)?;
|
|
let edge = frame.width.max(frame.height);
|
|
let long_edge = edge as f32;
|
|
let Some(models) = tk.models()?.eyes.as_mut() else {
|
|
return Err(Failure::Other("eye models not loaded".into()));
|
|
};
|
|
let mut updates = Vec::new();
|
|
for f in &owed {
|
|
let mut landmarks = [(0.0_f32, 0.0_f32); 5];
|
|
for (o, &(x, y)) in landmarks.iter_mut().zip(f.landmarks.iter()) {
|
|
*o = (x * long_edge, y * long_edge);
|
|
}
|
|
let bbox = (
|
|
f.x * long_edge,
|
|
f.y * long_edge,
|
|
(f.x + f.w) * long_edge,
|
|
(f.y + f.h) * long_edge,
|
|
);
|
|
match models.read(
|
|
dr_face::Pixels::Rgba8(&frame.rgba),
|
|
frame.width as usize,
|
|
frame.height as usize,
|
|
bbox,
|
|
&landmarks,
|
|
) {
|
|
Ok(Some((reading, dense))) => updates.push(FaceUpdate {
|
|
eyes: Some((reading, dense.to_packed_bytes(long_edge))),
|
|
..FaceUpdate::for_face(f.id)
|
|
}),
|
|
// Nothing could be cut: the face stays unread, and is not
|
|
// listed again by anything -- see the module note.
|
|
Ok(None) => {}
|
|
Err(e) => log::debug!("eye reading failed: {e}"),
|
|
}
|
|
}
|
|
let n = updates.len();
|
|
if n > 0 {
|
|
faces::record_updates(
|
|
catalog.connection(),
|
|
target.image,
|
|
&tk.model_id,
|
|
edge,
|
|
&updates,
|
|
&[],
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
}
|
|
Ok(n)
|
|
}
|
|
|
|
/// Cut a face's crop out of the native render, for the People screen.
|
|
fn crop(
|
|
tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let owed = faces::faces_needing(catalog.connection(), target.image, &tk.model_id, NEEDS_CROP)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
if owed.is_empty() {
|
|
return Ok(0);
|
|
}
|
|
let gpu = tk.gpu()?.clone();
|
|
let frame = fetched.frame(&gpu)?;
|
|
let edge = frame.width.max(frame.height);
|
|
let long_edge = edge as f32;
|
|
let updates: Vec<FaceUpdate> = owed
|
|
.iter()
|
|
.filter_map(|f| {
|
|
let cut = crate::faces::cut_crop_native(
|
|
dr_face::Pixels::Rgba8(&frame.rgba),
|
|
frame.width as usize,
|
|
frame.height as usize,
|
|
(
|
|
f.x * long_edge,
|
|
f.y * long_edge,
|
|
f.w * long_edge,
|
|
f.h * long_edge,
|
|
),
|
|
)?;
|
|
Some(FaceUpdate {
|
|
crop: Some(cut),
|
|
..FaceUpdate::for_face(f.id)
|
|
})
|
|
})
|
|
.collect();
|
|
let n = updates.len();
|
|
if n > 0 {
|
|
faces::record_updates(
|
|
catalog.connection(),
|
|
target.image,
|
|
&tk.model_id,
|
|
edge,
|
|
&updates,
|
|
&[],
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
}
|
|
Ok(n)
|
|
}
|
|
|
|
/// TRACES: FR-CAT-3
|
|
/// Read the capture metadata out of the header and promote the image.
|
|
///
|
|
/// What the thumbnail sweep does per image, as a repair: a real date takes
|
|
/// the image to `metadata_state = 2`, and a file that genuinely has none is
|
|
/// marked examined too, so it is not fetched again -- the state the sweep
|
|
/// records for the dateless, for the same reason.
|
|
fn metadata(
|
|
_tk: &mut Toolkit,
|
|
catalog: &Catalog,
|
|
target: &Target,
|
|
fetched: &mut Fetched,
|
|
) -> Result<usize, Failure> {
|
|
let found = match dr_decode::metadata(fetched.bytes()) {
|
|
Ok(md) => MetadataFound {
|
|
image_id: target.image.0 as i64,
|
|
captured_at: md.captured_at,
|
|
captured_offset: md.captured_offset,
|
|
camera: crate::library::camera_label(md.make.as_deref(), md.model.as_deref()),
|
|
lens: md.lens.map(|l| l.trim().to_string()),
|
|
iso: md.iso,
|
|
},
|
|
Err(e) => {
|
|
log::debug!("metadata for {}: {e}", target.path);
|
|
MetadataFound {
|
|
image_id: target.image.0 as i64,
|
|
captured_at: None,
|
|
captured_offset: None,
|
|
camera: None,
|
|
lens: None,
|
|
iso: None,
|
|
}
|
|
}
|
|
};
|
|
let dated = found.captured_at.is_some();
|
|
crate::library::write_metadata(catalog, std::slice::from_ref(&found))
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
if !dated {
|
|
catalog
|
|
.connection()
|
|
.execute(
|
|
"UPDATE images SET metadata_state = 2 WHERE id = ?1",
|
|
[target.image.0 as i64],
|
|
)
|
|
.map_err(|e| Failure::Other(e.to_string()))?;
|
|
}
|
|
Ok(1)
|
|
}
|
|
|
|
// ── the scan ──────────────────────────────────────────────────────────────
|
|
|
|
/// Images holding this embedder's faces whose proxy is not in the store.
|
|
fn faces_without_proxy(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
embedder: &str,
|
|
) -> Result<HashSet<i64>, dr_catalog::CatalogError> {
|
|
let mut stmt = catalog.connection().prepare(&format!(
|
|
"SELECT DISTINCT i.id, r.file_id
|
|
FROM images i
|
|
JOIN remote r ON r.image_id = i.id
|
|
JOIN faces f ON f.image_id = i.id
|
|
WHERE r.file_id IS NOT NULL AND {VISIBLE} AND {} = ?1",
|
|
faces::embedder_sql("f.model_id"),
|
|
))?;
|
|
// The index once, not a probe per image with faces — see
|
|
// `ThumbStore::held`. An unreadable index reads as empty, as `contains`
|
|
// would have reported it.
|
|
let held = store.held(dr_thumbs::ThumbSize::Large).unwrap_or_else(|e| {
|
|
log::warn!("repairs: reading the thumbnail index: {e}");
|
|
Default::default()
|
|
});
|
|
let rows = stmt
|
|
.query_map([embedder], |r| {
|
|
Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?))
|
|
})?
|
|
.filter_map(Result::ok)
|
|
.filter(|(_, file_id)| !held.contains(&(*file_id as u64)))
|
|
.map(|(id, _)| id)
|
|
.collect();
|
|
Ok(rows)
|
|
}
|
|
|
|
/// The images one repair lists, in the order the job visits them: the ones
|
|
/// holding faces first -- they are what the People screen is drawing and
|
|
/// what a re-detection carries names across -- then the rest, by id.
|
|
fn listed(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
repair: &Repair,
|
|
) -> Result<Vec<Target>, dr_catalog::CatalogError> {
|
|
let (predicate, set) = match &repair.needs {
|
|
Needs::Sql(sql) => (sql.clone(), None),
|
|
Needs::Face(fragment) => (any_face(fragment), None),
|
|
Needs::Set(f) => ("1".to_string(), Some(f(catalog, store)?)),
|
|
};
|
|
let mut stmt = catalog.connection().prepare(&format!(
|
|
"SELECT i.id, i.source_ref, r.file_id, i.file_size
|
|
FROM images i
|
|
JOIN remote r ON r.image_id = i.id
|
|
WHERE r.file_id IS NOT NULL AND {VISIBLE} AND ({predicate})
|
|
ORDER BY EXISTS (SELECT 1 FROM faces f WHERE f.image_id = i.id) DESC, i.id"
|
|
))?;
|
|
let rows = stmt
|
|
.query_map([], |r| {
|
|
Ok(Target {
|
|
image: ImageId(r.get::<_, i64>(0)? as u64),
|
|
path: r.get(1)?,
|
|
file_id: r.get::<_, Option<i64>>(2)?.map(|v| v as u64),
|
|
size: r.get::<_, Option<i64>>(3)?.unwrap_or(0) as u64,
|
|
})
|
|
})?
|
|
.filter_map(Result::ok)
|
|
.filter(|t| set.as_ref().is_none_or(|s| s.contains(&(t.image.0 as i64))))
|
|
.collect();
|
|
Ok(rows)
|
|
}
|
|
|
|
/// Whether one image still owes a repair -- asked again before each
|
|
/// handler runs, because an earlier handler's write may have answered it.
|
|
fn still_owed(
|
|
catalog: &Catalog,
|
|
repair: &Repair,
|
|
set: Option<&HashSet<i64>>,
|
|
image: ImageId,
|
|
) -> bool {
|
|
let sql = match (&repair.needs, set) {
|
|
(Needs::Set(_), Some(s)) => return s.contains(&(image.0 as i64)),
|
|
(Needs::Set(_), None) => return false,
|
|
(Needs::Sql(sql), _) => sql.clone(),
|
|
(Needs::Face(fragment), _) => any_face(fragment),
|
|
};
|
|
catalog
|
|
.connection()
|
|
.query_row(
|
|
&format!(
|
|
"SELECT EXISTS (SELECT 1 FROM images i JOIN remote r ON r.image_id = i.id
|
|
WHERE i.id = ?1 AND ({sql}))"
|
|
),
|
|
[image.0 as i64],
|
|
|r| r.get::<_, bool>(0),
|
|
)
|
|
.unwrap_or(false)
|
|
}
|
|
|
|
/// How many images each repair still lists, for the settings line and the
|
|
/// coverage line.
|
|
///
|
|
/// Counted, not listed. [`listed`] builds a `Target` per image — its path,
|
|
/// its size — and sorts the faces-first order the job visits them in, none of
|
|
/// which a count reads; asked for six repairs on a 24,000-image library that
|
|
/// was 350 ms of `source_ref` strings built to be dropped. A `COUNT(*)` over
|
|
/// the same predicate is the same number in a tenth of the time.
|
|
pub fn counts(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
repairs: &[Repair],
|
|
) -> Result<Vec<(&'static str, u64)>, dr_catalog::CatalogError> {
|
|
repairs
|
|
.iter()
|
|
.map(|r| Ok((r.label, count(catalog, store, r)?)))
|
|
.collect()
|
|
}
|
|
|
|
/// How many images one repair lists — the size of [`listed`]'s answer,
|
|
/// without building it.
|
|
fn count(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
repair: &Repair,
|
|
) -> Result<u64, dr_catalog::CatalogError> {
|
|
match &repair.needs {
|
|
Needs::Sql(sql) => {
|
|
let n: i64 = catalog.connection().query_row(
|
|
&format!(
|
|
"SELECT COUNT(*)
|
|
FROM images i
|
|
JOIN remote r ON r.image_id = i.id
|
|
WHERE r.file_id IS NOT NULL AND {VISIBLE} AND ({sql})"
|
|
),
|
|
[],
|
|
|r| r.get(0),
|
|
)?;
|
|
Ok(n as u64)
|
|
}
|
|
// From the faces, not the images: see `Needs::Face`.
|
|
Needs::Face(fragment) => {
|
|
let n: i64 = catalog.connection().query_row(
|
|
&format!(
|
|
"SELECT COUNT(DISTINCT f.image_id)
|
|
FROM faces f
|
|
JOIN images i ON i.id = f.image_id
|
|
JOIN remote r ON r.image_id = i.id
|
|
WHERE {fragment} AND r.file_id IS NOT NULL AND {VISIBLE}"
|
|
),
|
|
[],
|
|
|r| r.get(0),
|
|
)?;
|
|
Ok(n as u64)
|
|
}
|
|
// The set is built from its own query and may name images `listed`
|
|
// would not visit, so it is intersected with the same base rather
|
|
// than trusted for its size.
|
|
Needs::Set(f) => {
|
|
let set = f(catalog, store)?;
|
|
let mut stmt = catalog.connection().prepare(&format!(
|
|
"SELECT i.id
|
|
FROM images i
|
|
JOIN remote r ON r.image_id = i.id
|
|
WHERE r.file_id IS NOT NULL AND {VISIBLE}"
|
|
))?;
|
|
let n = stmt
|
|
.query_map([], |r| r.get::<_, i64>(0))?
|
|
.filter_map(Result::ok)
|
|
.filter(|id| set.contains(id))
|
|
.count();
|
|
Ok(n as u64)
|
|
}
|
|
}
|
|
}
|
|
|
|
/// One image on the work list, with the most any repair claiming it asks
|
|
/// for.
|
|
struct Planned {
|
|
target: Target,
|
|
input: Input,
|
|
}
|
|
|
|
/// The work list, and each [`Needs::Set`] repair's set (`None` for a SQL
|
|
/// one), indexed like the registry.
|
|
type Plan = (Vec<Planned>, Vec<Option<HashSet<i64>>>);
|
|
|
|
/// The union of every repair's list, first claim first.
|
|
///
|
|
/// The order across repairs is the registry's, so the images the first
|
|
/// repair lists come first however many the last lists -- which is what
|
|
/// puts a few hundred proxy repairs ahead of twenty thousand un-indexed
|
|
/// images, where appended they would sit two hours down the queue.
|
|
fn plan(
|
|
catalog: &Catalog,
|
|
store: &ThumbStore,
|
|
repairs: &[Repair],
|
|
) -> Result<Plan, dr_catalog::CatalogError> {
|
|
let mut out: Vec<Planned> = Vec::new();
|
|
let mut at: std::collections::HashMap<u64, usize> = std::collections::HashMap::new();
|
|
let mut sets = Vec::with_capacity(repairs.len());
|
|
for repair in repairs {
|
|
let set = match &repair.needs {
|
|
Needs::Set(f) => Some(f(catalog, store)?),
|
|
Needs::Sql(_) | Needs::Face(_) => None,
|
|
};
|
|
let listed = listed(catalog, store, repair)?;
|
|
if !listed.is_empty() {
|
|
log::info!("repairs: {}: {} image(s)", repair.name, listed.len());
|
|
}
|
|
for target in listed {
|
|
match at.get(&target.image.0) {
|
|
Some(&i) => out[i].input = out[i].input.max(repair.input),
|
|
None => {
|
|
at.insert(target.image.0, out.len());
|
|
out.push(Planned {
|
|
target,
|
|
input: repair.input,
|
|
});
|
|
}
|
|
}
|
|
}
|
|
sets.push(set);
|
|
}
|
|
Ok((out, sets))
|
|
}
|
|
|
|
/// Originals over this are not fetched for a repair that wants the whole
|
|
/// file. See `library::SWEEP_MAX_ORIGINAL_BYTES`.
|
|
const MAX_ORIGINAL_BYTES: u64 = crate::library::SWEEP_MAX_ORIGINAL_BYTES;
|
|
|
|
/// How many originals are in flight at once. See `library::SWEEP_LANES`.
|
|
const LANES: usize = crate::library::SWEEP_LANES;
|
|
|
|
// ── the job ───────────────────────────────────────────────────────────────
|
|
|
|
/// Run the registry over the library, at native resolution, in the
|
|
/// background.
|
|
///
|
|
/// Every image on the union work list is fetched once -- the header, or
|
|
/// the whole original, per the most any repair claiming it asks for --
|
|
/// rendered at most once, and handed to each repair whose predicate it
|
|
/// still matches. The receiver is the cancellation handle: dropping it
|
|
/// stops the job at the next image, and everything written stays written.
|
|
///
|
|
/// Resumable by construction, because the work list is what the catalog
|
|
/// says is incomplete: a kill costs the images in flight and nothing else.
|
|
///
|
|
/// # Cost, stated plainly
|
|
///
|
|
/// One whole original per image any face repair claims, and one full
|
|
/// render. On the reference library that is 412 GB and roughly a hundred
|
|
/// minutes of decode for a whole-library pass -- which is why FR-CULL-8
|
|
/// makes it a transfer under FR-NC-6 that starts when the user says so.
|
|
/// Nothing is kept that was not already wanted: the original is borrowed
|
|
/// and given back (ARCH §9.0a), and the only thing written per image
|
|
/// besides the catalog is the proxy the People screen crops from.
|
|
#[allow(clippy::too_many_arguments)]
|
|
pub fn spawn(
|
|
conn: Connection,
|
|
catalog_path: PathBuf,
|
|
store_dir: PathBuf,
|
|
models: Option<FaceModelPaths>,
|
|
model_id: String,
|
|
detector: FaceDetector,
|
|
scope: Scope,
|
|
options: dr_face::DetectOptions,
|
|
gpu: Option<dr_gpu::GpuContext>,
|
|
) -> Receiver<FaceSweepMessage> {
|
|
let (tx, rx) = std::sync::mpsc::channel();
|
|
|
|
std::thread::spawn(move || {
|
|
let finish_empty = |tx: &Sender<FaceSweepMessage>| {
|
|
let _ = tx.send(FaceSweepMessage::Finished {
|
|
images: 0,
|
|
faces: 0,
|
|
failed: 0,
|
|
});
|
|
};
|
|
|
|
let catalog = match Catalog::open(&catalog_path) {
|
|
Ok(c) => c,
|
|
Err(e) => {
|
|
log::warn!("repairs: cannot open catalog: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
let store = match ThumbStore::open(&store_dir) {
|
|
Ok(s) => s,
|
|
Err(e) => {
|
|
log::warn!("repairs: cannot open the thumbnail store: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
// Models before the work list: they are the expensive failure, and
|
|
// listing twenty thousand images before discovering the weights are
|
|
// missing helps nobody. A library with no model installed takes
|
|
// this path, and the registry is simply shorter.
|
|
let loaded = models.as_ref().and_then(|paths| {
|
|
let detector = match dr_face::Detector::from_path(&paths.detector) {
|
|
Ok(d) => d,
|
|
Err(e) => {
|
|
log::warn!("repairs: cannot load the detector: {e}");
|
|
return None;
|
|
}
|
|
};
|
|
let embedder = match dr_face::Embedder::from_path(
|
|
&paths.embedder,
|
|
dr_face::ModelId::new(model_id.clone()),
|
|
) {
|
|
Ok(e) => e,
|
|
Err(e) => {
|
|
log::warn!("repairs: cannot load the embedder: {e}");
|
|
return None;
|
|
}
|
|
};
|
|
Some(FaceModels {
|
|
detector,
|
|
embedder,
|
|
eyes: paths.load_eyes(),
|
|
})
|
|
});
|
|
let caps = Capabilities {
|
|
gpu: gpu.is_some(),
|
|
face_models: loaded.is_some(),
|
|
eye_models: loaded.as_ref().is_some_and(|m| m.eyes.is_some()),
|
|
};
|
|
let mut tk = Toolkit {
|
|
gpu,
|
|
models: loaded,
|
|
model_id: model_id.clone(),
|
|
store,
|
|
options,
|
|
};
|
|
let repairs = registry(scope, &model_id, detector, caps);
|
|
|
|
let (planned, sets) = match plan(&catalog, &tk.store, &repairs) {
|
|
Ok(p) => p,
|
|
Err(e) => {
|
|
log::warn!("repairs: listing the work: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
// Over budget, decided on the byte count the catalog holds rather
|
|
// than by fetching the file to find out. Given up on where a
|
|
// repair says how, left alone otherwise, and counted as failed
|
|
// either way: not done, and said so.
|
|
let mut skipped = 0usize;
|
|
let planned: Vec<Planned> = planned
|
|
.into_iter()
|
|
.filter(|p| {
|
|
if p.input == Input::Header || p.target.size <= MAX_ORIGINAL_BYTES {
|
|
return true;
|
|
}
|
|
log::warn!(
|
|
"repairs: {} is {} MB, over the {} MB budget for a background fetch; skipped",
|
|
p.target.path,
|
|
p.target.size >> 20,
|
|
MAX_ORIGINAL_BYTES >> 20
|
|
);
|
|
give_up(&tk, &catalog, &repairs, &sets, &p.target);
|
|
skipped += 1;
|
|
false
|
|
})
|
|
.collect();
|
|
|
|
let total = planned.len();
|
|
if total == 0 {
|
|
log::info!("repairs: nothing is incomplete");
|
|
let _ = tx.send(FaceSweepMessage::Finished {
|
|
images: 0,
|
|
faces: 0,
|
|
failed: skipped,
|
|
});
|
|
return;
|
|
}
|
|
log::info!("repairs: {total} image(s) to visit");
|
|
if tx.send(FaceSweepMessage::Total(total)).is_err() {
|
|
return;
|
|
}
|
|
|
|
let rt = match crate::net_runtime::build() {
|
|
Ok(rt) => rt,
|
|
Err(e) => {
|
|
log::warn!("repairs: no runtime: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
rt.block_on(async {
|
|
let backend = match crate::remote::connect(&conn) {
|
|
Ok(b) => b,
|
|
Err(e) => {
|
|
log::warn!("repairs: {e}");
|
|
finish_empty(&tx);
|
|
return;
|
|
}
|
|
};
|
|
|
|
let (mut images, mut found, mut failed) = (0usize, 0usize, skipped);
|
|
let mut offline = false;
|
|
|
|
// TRACES: FR-NC-6c | NFR-RES-2
|
|
// Fetch wide, render narrow: one original per lane in flight,
|
|
// and the render sequential, because there is one GPU and each
|
|
// render materialises a native frame -- 96 MB for a 24 MP
|
|
// photograph. Every file borrowed is given back (ARCH §9.0a).
|
|
let pool = dr_sync_folder::BorrowPool::new();
|
|
|
|
for chunk in planned.chunks(LANES) {
|
|
let fetched = crate::library::futures_join_all(chunk.iter().map(|p| {
|
|
let backend = &*backend;
|
|
let pool = &pool;
|
|
async move {
|
|
let id = RemoteId::Path(RemotePath::new(&p.target.path));
|
|
let got = if p.input == Input::Header {
|
|
backend.get(&id, Some(0..dr_decode::HEADER_BYTES)).await
|
|
} else {
|
|
let held = match pool
|
|
.borrow(backend, &RemotePath::new(&p.target.path))
|
|
.await
|
|
{
|
|
Ok(h) => h,
|
|
Err(e) => return (p, Err(e)),
|
|
};
|
|
let got = backend.get(&id, None).await;
|
|
drop(held);
|
|
got
|
|
};
|
|
(p, got)
|
|
}
|
|
}))
|
|
.await;
|
|
|
|
let mut lane_failed = 0usize;
|
|
for (p, got) in fetched {
|
|
let bytes = match got {
|
|
Ok(b) => b,
|
|
Err(e) if e.indicates_offline() => {
|
|
log::info!("repairs: server unreachable: {e}");
|
|
offline = true;
|
|
continue;
|
|
}
|
|
Err(e) => {
|
|
log::debug!("repairs: {}: {e}", p.target.path);
|
|
lane_failed += 1;
|
|
continue;
|
|
}
|
|
};
|
|
|
|
let mut fetched = Fetched::new(bytes, p.input);
|
|
let mut wrote = 0usize;
|
|
let mut broke = false;
|
|
for (repair, set) in repairs.iter().zip(sets.iter()) {
|
|
if !still_owed(&catalog, repair, set.as_ref(), p.target.image) {
|
|
continue;
|
|
}
|
|
match (repair.apply)(&mut tk, &catalog, &p.target, &mut fetched) {
|
|
Ok(n) => wrote += n,
|
|
Err(Failure::Unreadable(e)) => {
|
|
// Fails the same way on every pass, and every
|
|
// pass fetched it first: a 521 MB panorama the
|
|
// decoder refuses was downloaded once per
|
|
// sweep, on a tablet.
|
|
log::warn!("repairs: {}: {e}", p.target.path);
|
|
give_up(&tk, &catalog, &repairs, &sets, &p.target);
|
|
broke = true;
|
|
break;
|
|
}
|
|
Err(Failure::Other(e)) => {
|
|
log::warn!("repairs: {} on {}: {e}", repair.name, p.target.path);
|
|
broke = true;
|
|
}
|
|
}
|
|
}
|
|
if broke {
|
|
lane_failed += 1;
|
|
continue;
|
|
}
|
|
images += 1;
|
|
found += wrote;
|
|
if tx
|
|
.send(FaceSweepMessage::Indexed {
|
|
image: p.target.image,
|
|
faces: wrote,
|
|
})
|
|
.is_err()
|
|
{
|
|
// Receiver dropped: the screen closed, or Stop.
|
|
log::info!("repairs: cancelled after {images} image(s)");
|
|
pool.release_all(&*backend).await;
|
|
return;
|
|
}
|
|
}
|
|
|
|
failed += lane_failed;
|
|
if lane_failed > 0
|
|
&& tx
|
|
.send(FaceSweepMessage::Failed {
|
|
images: lane_failed,
|
|
})
|
|
.is_err()
|
|
{
|
|
pool.release_all(&*backend).await;
|
|
return;
|
|
}
|
|
if offline {
|
|
break;
|
|
}
|
|
}
|
|
|
|
let returned = pool.release_all(&*backend).await;
|
|
if returned.released > 0 {
|
|
log::info!("repairs: released {} borrowed file(s)", returned.released);
|
|
}
|
|
log::info!(
|
|
"repairs: {found} record(s) written across {images} image(s), {failed} failed{}",
|
|
if offline { ", server went away" } else { "" }
|
|
);
|
|
let _ = tx.send(FaceSweepMessage::Finished {
|
|
images,
|
|
faces: found,
|
|
failed,
|
|
});
|
|
});
|
|
});
|
|
|
|
rx
|
|
}
|
|
|
|
/// Record, for every repair that still claims an image and knows how, that
|
|
/// it can never be done.
|
|
fn give_up(
|
|
tk: &Toolkit,
|
|
catalog: &Catalog,
|
|
repairs: &[Repair],
|
|
sets: &[Option<HashSet<i64>>],
|
|
target: &Target,
|
|
) {
|
|
for (repair, set) in repairs.iter().zip(sets.iter()) {
|
|
let Some(f) = repair.give_up else { continue };
|
|
if !still_owed(catalog, repair, set.as_ref(), target.image) {
|
|
continue;
|
|
}
|
|
if let Err(e) = f(tk, catalog, target) {
|
|
log::warn!("repairs: giving up {} on {}: {e}", repair.name, target.path);
|
|
}
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use dr_catalog::faces::DetectedFace;
|
|
|
|
fn with_images(n: usize) -> Catalog {
|
|
crate::library::test_support::with_images(n)
|
|
}
|
|
|
|
fn image_ids(catalog: &Catalog) -> Vec<ImageId> {
|
|
crate::library::test_support::image_ids(catalog)
|
|
}
|
|
|
|
fn store() -> (ThumbStore, PathBuf) {
|
|
let dir = std::env::temp_dir().join(format!(
|
|
"dr-repairs-test-{}-{:?}",
|
|
std::process::id(),
|
|
std::thread::current().id()
|
|
));
|
|
let _ = std::fs::remove_dir_all(&dir);
|
|
std::fs::create_dir_all(&dir).unwrap();
|
|
(ThumbStore::open(&dir).unwrap(), dir)
|
|
}
|
|
|
|
fn face(model: &str, quality: Option<f32>, crop: Vec<u8>) -> DetectedFace {
|
|
DetectedFace {
|
|
x: 0.1,
|
|
y: 0.1,
|
|
w: 0.2,
|
|
h: 0.2,
|
|
landmarks: [(0.0, 0.0); 5],
|
|
confidence: 0.9,
|
|
embedding: vec![0u8; 1024],
|
|
crop_px: 120.0,
|
|
quality,
|
|
eyes: None,
|
|
landmarks_dense: Vec::new(),
|
|
crop,
|
|
model_id: model.into(),
|
|
}
|
|
}
|
|
|
|
/// A face with every field a native detection writes.
|
|
fn complete(model: &str) -> DetectedFace {
|
|
DetectedFace {
|
|
eyes: Some(dr_face::EyeReading {
|
|
right: dr_face::Eye {
|
|
open: 0.9,
|
|
px: 40.0,
|
|
sharpness: 0.2,
|
|
},
|
|
left: dr_face::Eye {
|
|
open: 0.9,
|
|
px: 40.0,
|
|
sharpness: 0.2,
|
|
},
|
|
sunglasses: 0.1,
|
|
}),
|
|
landmarks_dense: vec![7; 424],
|
|
..face(model, Some(20.0), vec![1])
|
|
}
|
|
}
|
|
|
|
const ALL: Capabilities = Capabilities {
|
|
gpu: true,
|
|
face_models: true,
|
|
eye_models: true,
|
|
};
|
|
|
|
fn by_name<'a>(repairs: &'a [Repair], name: &str) -> &'a Repair {
|
|
repairs.iter().find(|r| r.name == name).expect(name)
|
|
}
|
|
|
|
fn ids_of(targets: &[Target]) -> Vec<u64> {
|
|
targets.iter().map(|t| t.image.0).collect()
|
|
}
|
|
|
|
/// The registry is what the device can do, and no more: a device that
|
|
/// cannot read eyes lists no faces to read, or it would fetch every
|
|
/// original in the library to do nothing to it.
|
|
#[test]
|
|
fn the_registry_is_cut_to_what_the_device_can_do() {
|
|
let names = |caps: Capabilities| -> Vec<&'static str> {
|
|
registry(
|
|
Scope::Outstanding,
|
|
"scrfd_10g+w600k_mbf",
|
|
FaceDetector::Scrfd10g,
|
|
caps,
|
|
)
|
|
.iter()
|
|
.map(|r| r.name)
|
|
.collect()
|
|
};
|
|
assert_eq!(
|
|
names(ALL),
|
|
vec![
|
|
"face-proxy",
|
|
"face-quality",
|
|
"face-eyes",
|
|
"face-crop",
|
|
"face-detection",
|
|
"face-upgrade",
|
|
"metadata"
|
|
]
|
|
);
|
|
assert!(!names(Capabilities {
|
|
eye_models: false,
|
|
..ALL
|
|
})
|
|
.contains(&"face-eyes"));
|
|
assert_eq!(names(Capabilities::default()), vec!["metadata"]);
|
|
// The fast detector supersedes nothing, so there is nothing to
|
|
// upgrade.
|
|
let fast = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
assert!(!fast.iter().any(|r| r.name == "face-upgrade"));
|
|
// Under a re-index, detection leads and the proxy repair is
|
|
// subsumed by it.
|
|
let re: Vec<_> = registry(
|
|
Scope::Reindex,
|
|
"scrfd_10g+w600k_mbf",
|
|
FaceDetector::Scrfd10g,
|
|
ALL,
|
|
)
|
|
.iter()
|
|
.map(|r| r.name)
|
|
.collect();
|
|
assert_eq!(re[0], "face-detection");
|
|
assert!(!re.contains(&"face-proxy"));
|
|
assert!(!re.contains(&"face-upgrade"));
|
|
}
|
|
|
|
/// The regression the whole-library pass exists for: nothing here puts
|
|
/// a proxy on disk, and every image is still work.
|
|
#[test]
|
|
fn every_unindexed_image_is_work_even_with_no_proxy_anywhere() {
|
|
let catalog = with_images(10);
|
|
let (store, dir) = store();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let wanted = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert_eq!(wanted.len(), 10);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// `face_index` records that detection *ran*, so an image with no face
|
|
/// in it must not come back -- otherwise a personal library, which is
|
|
/// mostly landscapes and documents, never finishes.
|
|
#[test]
|
|
fn an_image_already_run_over_is_not_work_again() {
|
|
let catalog = with_images(3);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
faces::record_detections(catalog.connection(), ids[0], "w600k_mbf", 1024, &[]).unwrap();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let wanted = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert_eq!(ids_of(&wanted), vec![ids[1].0, ids[2].0]);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// A detector change keeps the embedder, so it is not a new library:
|
|
/// the images the old detector ran over are an *upgrade*, listed after
|
|
/// what nothing has examined and only when the chosen detector outranks
|
|
/// the one that indexed them.
|
|
#[test]
|
|
fn a_stronger_detector_upgrades_rather_than_re_indexes() {
|
|
let catalog = with_images(3);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
let conn = catalog.connection();
|
|
faces::record_detections(conn, ids[0], "w600k_mbf", 1024, &[]).unwrap();
|
|
faces::record_detections(conn, ids[1], "scrfd_10g+w600k_mbf", 1024, &[]).unwrap();
|
|
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"scrfd_10g+w600k_mbf",
|
|
FaceDetector::Scrfd10g,
|
|
ALL,
|
|
);
|
|
let fresh = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert_eq!(ids_of(&fresh), vec![ids[2].0]);
|
|
let up = listed(&catalog, &store, by_name(&repairs, "face-upgrade")).unwrap();
|
|
assert_eq!(ids_of(&up), vec![ids[0].0]);
|
|
|
|
// And the plan puts the never-examined image first.
|
|
let (planned, _) = plan(&catalog, &store, &repairs).unwrap();
|
|
let order: Vec<u64> = planned.iter().map(|p| p.target.image.0).collect();
|
|
assert_eq!(order[..2], [ids[2].0, ids[0].0]);
|
|
// The third is the metadata repair's, and last.
|
|
assert_eq!(order[2], ids[1].0);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// `counts` answers from the faces, `listed` from the images (see
|
|
/// `Needs::Face`), and the two spellings of each predicate have to
|
|
/// agree -- for every repair, on a library where each has something to
|
|
/// do and something already done.
|
|
#[test]
|
|
fn counts_are_the_sizes_of_the_lists() {
|
|
let catalog = with_images(5);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
let conn = catalog.connection();
|
|
// 0: two faces, one measured, neither read for eyes, one without a
|
|
// crop -- the per-face repairs disagree about it face by face.
|
|
faces::record_detections(
|
|
conn,
|
|
ids[0],
|
|
"w600k_mbf",
|
|
4000,
|
|
&[
|
|
face("w600k_mbf", None, vec![1]),
|
|
face("w600k_mbf", Some(18.0), Vec::new()),
|
|
],
|
|
)
|
|
.unwrap();
|
|
// 1: done, under the chosen detector.
|
|
faces::record_detections(
|
|
conn,
|
|
ids[1],
|
|
"scrfd_10g+w600k_mbf",
|
|
4000,
|
|
&[complete("scrfd_10g+w600k_mbf")],
|
|
)
|
|
.unwrap();
|
|
// 2: examined by a weaker detector, nothing found.
|
|
faces::record_detections(conn, ids[2], "w600k_mbf", 4000, &[]).unwrap();
|
|
// 3, 4: never examined.
|
|
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"scrfd_10g+w600k_mbf",
|
|
FaceDetector::Scrfd10g,
|
|
ALL,
|
|
);
|
|
let counted = counts(&catalog, &store, &repairs).unwrap();
|
|
for (repair, (label, n)) in repairs.iter().zip(counted) {
|
|
assert_eq!(label, repair.label);
|
|
let list = listed(&catalog, &store, repair).unwrap();
|
|
assert_eq!(n as usize, list.len(), "{}", repair.name);
|
|
}
|
|
// And the fixture exercised what it claims to.
|
|
let names: Vec<&str> = repairs.iter().map(|r| r.name).collect();
|
|
for name in [
|
|
"face-quality",
|
|
"face-eyes",
|
|
"face-crop",
|
|
"face-detection",
|
|
"face-upgrade",
|
|
] {
|
|
assert!(names.contains(&name), "{name} missing from the registry");
|
|
assert!(
|
|
!listed(&catalog, &store, by_name(&repairs, name))
|
|
.unwrap()
|
|
.is_empty(),
|
|
"{name} has nothing to do"
|
|
);
|
|
}
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// The state schema V14 leaves: a face with no quality and an image
|
|
/// with no marker. It is the quality repair's work, and *only* that
|
|
/// repair's -- a full re-detection of the same image would throw away
|
|
/// every suggestion on it for nothing.
|
|
#[test]
|
|
fn an_unmeasured_face_is_measured_rather_than_re_detected() {
|
|
let catalog = with_images(3);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
let conn = catalog.connection();
|
|
faces::record_detections(
|
|
conn,
|
|
ids[0],
|
|
"w600k_mbf",
|
|
4000,
|
|
&[face("w600k_mbf", None, vec![1])],
|
|
)
|
|
.unwrap();
|
|
faces::record_detections(
|
|
conn,
|
|
ids[1],
|
|
"w600k_mbf",
|
|
4000,
|
|
&[face("w600k_mbf", Some(18.0), vec![1])],
|
|
)
|
|
.unwrap();
|
|
faces::clear_index_marker(conn, ids[0], "w600k_mbf").unwrap();
|
|
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let quality = listed(&catalog, &store, by_name(&repairs, "face-quality")).unwrap();
|
|
assert_eq!(ids_of(&quality), vec![ids[0].0]);
|
|
let detect = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert_eq!(
|
|
ids_of(&detect),
|
|
vec![ids[2].0],
|
|
"the unmeasured image is not re-detected"
|
|
);
|
|
// Neither face has an eye reading: both are the eye repair's.
|
|
let eyes = listed(&catalog, &store, by_name(&repairs, "face-eyes")).unwrap();
|
|
assert_eq!(ids_of(&eyes), vec![ids[0].0, ids[1].0]);
|
|
|
|
// One image, one fetch, three claims: the plan asks for the render
|
|
// once.
|
|
let (planned, _) = plan(&catalog, &store, &repairs).unwrap();
|
|
assert_eq!(
|
|
planned.iter().filter(|p| p.target.image == ids[0]).count(),
|
|
1
|
|
);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// The re-index is keyed on the chosen detector, not the embedder: an
|
|
/// image the fast detector examined is work, one the chosen detector
|
|
/// examined in either of its forms is not -- and it converges, because
|
|
/// a re-detection under the chosen detector takes the image off.
|
|
#[test]
|
|
fn the_re_index_lists_what_the_chosen_detector_has_not_been_over() {
|
|
let catalog = with_images(6);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
let conn = catalog.connection();
|
|
let current = FaceDetector::Scrfd10g.model_ids();
|
|
// 0: the fast detector found a face on a proxy -- work.
|
|
faces::record_detections(
|
|
conn,
|
|
ids[0],
|
|
"w600k_mbf",
|
|
1024,
|
|
&[face("w600k_mbf", None, vec![])],
|
|
)
|
|
.unwrap();
|
|
// 1: the fast detector found nothing -- still work.
|
|
faces::record_detections(conn, ids[1], "w600k_mbf", 1024, &[]).unwrap();
|
|
// 2, 3: the chosen detector, native, in each of its forms -- done.
|
|
faces::record_detections(conn, ids[2], current[0], 4000, &[complete(current[0])]).unwrap();
|
|
faces::record_detections(conn, ids[3], current[1], 4000, &[complete(current[1])]).unwrap();
|
|
// 4: the chosen detector's marker, but a face a peer's shard brought
|
|
// without its crop -- the crop repair's, not detection's.
|
|
faces::record_detections(
|
|
conn,
|
|
ids[4],
|
|
current[0],
|
|
4000,
|
|
&[face(current[0], Some(20.0), vec![])],
|
|
)
|
|
.unwrap();
|
|
// 5: never examined -- work.
|
|
|
|
let repairs = registry(Scope::Reindex, current[0], FaceDetector::Scrfd10g, ALL);
|
|
let detect = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
// Images holding faces first, then the rest, each in id order.
|
|
assert_eq!(ids_of(&detect), vec![ids[0].0, ids[1].0, ids[5].0]);
|
|
let crop = listed(&catalog, &store, by_name(&repairs, "face-crop")).unwrap();
|
|
assert_eq!(ids_of(&crop), vec![ids[0].0, ids[4].0]);
|
|
|
|
// Re-detecting the first under the chosen detector takes it off
|
|
// every list, since a detection writes every field.
|
|
faces::record_detections(conn, ids[0], current[0], 4000, &[complete(current[0])]).unwrap();
|
|
for r in &repairs {
|
|
if r.name.starts_with("face-") {
|
|
assert!(
|
|
!still_owed(&catalog, r, None, ids[0]),
|
|
"{} still claims a re-detected image",
|
|
r.name
|
|
);
|
|
}
|
|
}
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// The proxy repair is the one list the catalog cannot compute alone,
|
|
/// and it leads the plan: the image the screen cannot draw is fetched
|
|
/// before the rest of the library.
|
|
#[test]
|
|
fn repairs_are_reached_before_the_rest_of_the_library() {
|
|
let catalog = with_images(50);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
let orphan = ids[40];
|
|
faces::record_detections(
|
|
catalog.connection(),
|
|
orphan,
|
|
"w600k_mbf",
|
|
1024,
|
|
&[face("w600k_mbf", Some(20.0), vec![1])],
|
|
)
|
|
.unwrap();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let (planned, sets) = plan(&catalog, &store, &repairs).unwrap();
|
|
assert_eq!(planned[0].target.image, orphan);
|
|
assert_eq!(
|
|
planned.len(),
|
|
50,
|
|
"49 un-indexed plus the one being repaired"
|
|
);
|
|
assert!(still_owed(
|
|
&catalog,
|
|
by_name(&repairs, "face-proxy"),
|
|
sets[0].as_ref(),
|
|
orphan
|
|
));
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// A face with no proxy left draws "no preview" and cannot repair
|
|
/// itself: the image has its `face_index` row, so it is not outstanding
|
|
/// work. The job has to pick it up by a second route.
|
|
#[test]
|
|
fn a_face_whose_proxy_is_gone_is_work_again() {
|
|
let catalog = with_images(3);
|
|
let ids = image_ids(&catalog);
|
|
// An empty store, which is the state the bug lives in: the face is
|
|
// recorded and there is nothing on disk to cut it out of.
|
|
let (store, dir) = store();
|
|
faces::record_detections(
|
|
catalog.connection(),
|
|
ids[0],
|
|
"w600k_mbf",
|
|
1024,
|
|
&[face("w600k_mbf", Some(20.0), vec![1])],
|
|
)
|
|
.unwrap();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let detect = listed(&catalog, &store, by_name(&repairs, "face-detection")).unwrap();
|
|
assert!(
|
|
!detect.iter().any(|t| t.image == ids[0]),
|
|
"indexed, so not outstanding"
|
|
);
|
|
let proxy = listed(&catalog, &store, by_name(&repairs, "face-proxy")).unwrap();
|
|
assert_eq!(
|
|
ids_of(&proxy),
|
|
vec![ids[0].0],
|
|
"the orphaned face was not picked up"
|
|
);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
|
|
/// Metadata is a repair like any other, and a header is all it asks
|
|
/// for -- so an image both claim is fetched whole, and one only it
|
|
/// claims is not.
|
|
#[test]
|
|
fn an_image_needs_the_most_any_repair_asks_of_it() {
|
|
let catalog = with_images(2);
|
|
let ids = image_ids(&catalog);
|
|
let (store, dir) = store();
|
|
faces::record_detections(catalog.connection(), ids[1], "w600k_mbf", 1024, &[]).unwrap();
|
|
let repairs = registry(
|
|
Scope::Outstanding,
|
|
"w600k_mbf",
|
|
FaceDetector::Scrfd500m,
|
|
ALL,
|
|
);
|
|
let (planned, _) = plan(&catalog, &store, &repairs).unwrap();
|
|
let input_of = |id: ImageId| planned.iter().find(|p| p.target.image == id).unwrap().input;
|
|
assert_eq!(input_of(ids[0]), Input::NativeRender);
|
|
assert_eq!(input_of(ids[1]), Input::Header);
|
|
let _ = std::fs::remove_dir_all(dir);
|
|
}
|
|
}
|