Borrow the library to index it, and give it back
The passes that need every photograph's bytes — thumbnails, face indexing — now borrow each one and release it at the end. On a placeholder library that is the difference between peak disk being the working set and being the whole library. Including on cancellation, which was nearly missed: the face sweep returns mid-loop when the user presses Stop, and without releasing there the disk is spent and nothing is delivered for it. `materialise` now answers whether *it* fetched the content. The pool used to work that out by listing a file's parent directory — one listing per file across a library — when the backend already had to `stat` it to decide whether to ask. One syscall instead of a directory walk, and it removes the bug class the tests found earlier: a file at the library root has no `parent()`, so every one of them read as already-downloaded. **Pinning is the retention control**, and it drives the model the catalog already had rather than a second one. `tier_desired` is what the user asked to keep hydrated, `pending_pins` is the resumable work list, and a pinned collection is never dehydrated for the same reason it was never evicted. It was in fact *broken* here before: `get` on a stub failed, and the pin worker logged "one unreadable file must not abandon the whole pin" and silently did nothing. Pinned originals on such a library are recorded with `path = NULL` (`Cache::record_in_place`) rather than copied under `originals/`. Two reasons, and the second is the important one. A copy would hold every pinned photograph twice, with the budget able to evict the half that was not costing the disk. And `release` deletes the file a row names — so a row that names none cannot delete anything, which puts the one catastrophic operation out of reach by construction rather than by remembering not to call it. Deleting a materialised file inside a synced tree removes the photograph from the server and every other device. Handing disk back is `spawn_dehydrate`, which asks the client. Two gaps written down rather than papered over (docs/storage.md §7): a hydrating pass cannot yet quote its cost, because a stub reports no size; and the two sweeps hold separate pools, so a library indexed for both fetches twice.
This commit is contained in:
+205
-1
@@ -1565,6 +1565,48 @@ pub fn spawn_pin_fetch(
|
||||
};
|
||||
|
||||
let id = RemoteId::Path(RemotePath::new(&source_ref));
|
||||
|
||||
// TRACES: FR-NC-6c
|
||||
// On a placeholder library "pin" means *keep it downloaded*,
|
||||
// not "make a second copy". The original materialises in the
|
||||
// library folder itself, so copying it under `originals/`
|
||||
// would hold every pinned photograph twice — and the copy
|
||||
// would be the half the budget could evict while the real disk
|
||||
// cost stayed. Only the bookkeeping is recorded, with no path,
|
||||
// so nothing here can ever delete a file inside a synced tree
|
||||
// (see `Cache::record_in_place`).
|
||||
if backend.capabilities().materialisation.can_materialise() {
|
||||
match backend.materialise(&id).await {
|
||||
Ok(_) => {
|
||||
let bytes = size_of(&catalog, image).unwrap_or(0);
|
||||
if let Err(e) = store.record_in_place(
|
||||
catalog.connection(),
|
||||
image,
|
||||
bytes,
|
||||
true,
|
||||
now_secs(),
|
||||
) {
|
||||
log::warn!("recording pinned {source_ref}: {e}");
|
||||
continue;
|
||||
}
|
||||
stored += 1;
|
||||
bytes_total += bytes;
|
||||
if tx.send(PinMessage::Stored { done: stored }).is_err() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
Err(e) if e.indicates_offline() => {
|
||||
let _ = tx.send(PinMessage::Failed {
|
||||
message: e.to_string(),
|
||||
offline: true,
|
||||
});
|
||||
return;
|
||||
}
|
||||
Err(e) => log::warn!("pinning {source_ref}: {e}"),
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
match backend.get(&id, None).await {
|
||||
Ok(bytes) => {
|
||||
// `pinned: true` — this is the population the budget
|
||||
@@ -1614,6 +1656,82 @@ pub fn spawn_pin_fetch(
|
||||
rx
|
||||
}
|
||||
|
||||
/// TRACES: FR-NC-6c
|
||||
/// Hand a set of photographs back to the sync client, freeing their disk.
|
||||
///
|
||||
/// The other half of pinning on a placeholder library. `Cache::release` drops
|
||||
/// the bookkeeping and — correctly — deletes nothing, because the rows it
|
||||
/// holds for a library like this name no file of ours (`record_in_place`).
|
||||
/// The bytes are in the library folder, and only the client may take them
|
||||
/// back.
|
||||
///
|
||||
/// **This is a dehydration, not a deletion, and the distinction is the whole
|
||||
/// safety of the feature.** Removing a materialised file inside a synced tree
|
||||
/// propagates to the server and deletes the photograph everywhere.
|
||||
///
|
||||
/// Best effort per image: a file the client refuses to release simply stays,
|
||||
/// which costs disk and loses nothing.
|
||||
pub fn spawn_dehydrate(
|
||||
conn: Connection,
|
||||
catalog_path: PathBuf,
|
||||
images: Vec<dr_types::ImageId>,
|
||||
) -> Receiver<usize> {
|
||||
let (tx, rx) = std::sync::mpsc::channel();
|
||||
|
||||
std::thread::spawn(move || {
|
||||
let Ok(catalog) = Catalog::open(&catalog_path) else {
|
||||
return;
|
||||
};
|
||||
let Ok(rt) = crate::net_runtime::build() else {
|
||||
return;
|
||||
};
|
||||
rt.block_on(async {
|
||||
let Ok(backend) = crate::remote::connect(&conn) else {
|
||||
return;
|
||||
};
|
||||
// Nothing to do where content is not a thing that can be given
|
||||
// back — a server library, or a plain folder.
|
||||
if !backend.capabilities().materialisation.can_materialise() {
|
||||
return;
|
||||
}
|
||||
|
||||
let mut released = 0usize;
|
||||
for image in images {
|
||||
let Some(source_ref) = source_ref_of(&catalog, image) else {
|
||||
continue;
|
||||
};
|
||||
let id = RemoteId::Path(RemotePath::new(&source_ref));
|
||||
match backend.dematerialise(&id).await {
|
||||
Ok(()) => released += 1,
|
||||
Err(e) => log::debug!("releasing {source_ref}: {e}"),
|
||||
}
|
||||
}
|
||||
log::info!("released {released} photograph(s) back to the sync client");
|
||||
let _ = tx.send(released);
|
||||
});
|
||||
});
|
||||
|
||||
rx
|
||||
}
|
||||
|
||||
/// What an image occupies, as the catalog recorded it.
|
||||
///
|
||||
/// Zero where the scan could not tell — a placeholder reports no size, because
|
||||
/// a one-byte stub says nothing about what it stands for (ARCH §9.0a). A pin
|
||||
/// that cannot state its cost is better than one that states a wrong one.
|
||||
fn size_of(catalog: &Catalog, image: dr_types::ImageId) -> Option<u64> {
|
||||
catalog
|
||||
.connection()
|
||||
.query_row(
|
||||
"SELECT file_size FROM images WHERE id = ?1",
|
||||
rusqlite::params![image.0 as i64],
|
||||
|r| r.get::<_, Option<i64>>(0),
|
||||
)
|
||||
.ok()
|
||||
.flatten()
|
||||
.map(|v| v.max(0) as u64)
|
||||
}
|
||||
|
||||
/// The remote path for a catalogued image.
|
||||
fn source_ref_of(catalog: &Catalog, image: dr_types::ImageId) -> Option<String> {
|
||||
catalog
|
||||
@@ -3017,6 +3135,13 @@ pub fn spawn_face_sweep(
|
||||
let (mut done, mut images, mut found, mut failed) = (0usize, 0usize, 0usize, 0usize);
|
||||
let mut offline = false;
|
||||
|
||||
// TRACES: FR-NC-6c
|
||||
// The same borrow the thumbnail sweep makes, and deliberately its
|
||||
// own pool: the two passes run at different times, so sharing one
|
||||
// would keep every file the earlier pass touched hydrated until
|
||||
// the later one finished. Each gives its own back (ARCH §9.0a).
|
||||
let pool = dr_sync_folder::BorrowPool::new();
|
||||
|
||||
for chunk in wanted.chunks(SWEEP_CHUNK) {
|
||||
let lanes: Vec<Vec<&ThumbnailRequest>> = (0..SWEEP_LANES)
|
||||
.map(|lane| chunk.iter().skip(lane).step_by(SWEEP_LANES).collect())
|
||||
@@ -3026,6 +3151,7 @@ pub fn spawn_face_sweep(
|
||||
let backend = &*backend;
|
||||
let models = ⊧
|
||||
let options = &options;
|
||||
let pool = &pool;
|
||||
async move {
|
||||
let mut indexed: Vec<IndexedImage> = Vec::new();
|
||||
let mut discard = Vec::new();
|
||||
@@ -3034,6 +3160,23 @@ pub fn spawn_face_sweep(
|
||||
let mut offline = false;
|
||||
for req in lane {
|
||||
attempted += 1;
|
||||
|
||||
let _held =
|
||||
match pool.borrow(backend, &RemotePath::new(&req.path)).await {
|
||||
Ok(h) => h,
|
||||
Err(e) if e.indicates_offline() => {
|
||||
log::info!("face sweep: {e}");
|
||||
attempted -= 1;
|
||||
offline = true;
|
||||
break;
|
||||
}
|
||||
Err(e) => {
|
||||
log::debug!("face sweep: {}: {e}", req.path);
|
||||
failed += 1;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
||||
match fetch_preview(backend, req, &mut discard).await {
|
||||
PreviewOutcome::Ready(mut preview) => {
|
||||
// No await inside this borrow — see the
|
||||
@@ -3138,8 +3281,13 @@ pub fn spawn_face_sweep(
|
||||
{
|
||||
// Receiver dropped: the screen closed, or
|
||||
// the user pressed Stop. Everything written
|
||||
// so far stays written.
|
||||
// so far stays written — and everything
|
||||
// borrowed is given back. A cancelled pass
|
||||
// that kept the library hydrated would be
|
||||
// the worst of both: the disk spent and
|
||||
// the work abandoned.
|
||||
log::info!("face sweep: cancelled after {images} image(s)");
|
||||
pool.release_all(&*backend).await;
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -3157,6 +3305,14 @@ pub fn spawn_face_sweep(
|
||||
}
|
||||
}
|
||||
|
||||
let returned = pool.release_all(&*backend).await;
|
||||
if returned.released > 0 {
|
||||
log::info!(
|
||||
"face sweep: released {} borrowed file(s)",
|
||||
returned.released
|
||||
);
|
||||
}
|
||||
|
||||
log::info!(
|
||||
"face sweep: {found} face(s) across {images} image(s), {failed} failed{}",
|
||||
if offline { ", server went away" } else { "" }
|
||||
@@ -3317,6 +3473,15 @@ pub fn spawn_thumbnail_sweep(
|
||||
let mut offline = false;
|
||||
let mut found = Vec::new();
|
||||
|
||||
// TRACES: FR-NC-6c
|
||||
// On a placeholder library the bytes may not be here at all, and
|
||||
// this is a pass the user asked for — so it may fetch them, which
|
||||
// browsing may not (ARCH §9.0a). Every file is *borrowed*: what
|
||||
// this pass downloads it gives back, and what the user already had
|
||||
// it leaves alone. Against a server or a plain folder every borrow
|
||||
// is a no-op, so there is one code path rather than two.
|
||||
let pool = dr_sync_folder::BorrowPool::new();
|
||||
|
||||
for chunk in wanted.chunks(SWEEP_CHUNK) {
|
||||
// Each lane owns a disjoint slice and its own output, so
|
||||
// nothing is shared and no lock is needed. The store is not
|
||||
@@ -3327,6 +3492,7 @@ pub fn spawn_thumbnail_sweep(
|
||||
|
||||
let results = futures_join_all(lanes.into_iter().map(|lane| {
|
||||
let backend: &dyn RemoteBackend = &*backend;
|
||||
let pool = &pool;
|
||||
async move {
|
||||
let mut made: Vec<(u64, dr_thumbs::Thumbnail)> = Vec::new();
|
||||
let mut found = Vec::new();
|
||||
@@ -3339,6 +3505,27 @@ pub fn spawn_thumbnail_sweep(
|
||||
// store on and is not a candidate.
|
||||
let Some(file_id) = req.file_id else { continue };
|
||||
attempted += 1;
|
||||
|
||||
// Held for this image only. A failure to fetch is
|
||||
// this image's verdict, not the batch's: a client
|
||||
// that cannot reach the server reports it as
|
||||
// offline through the usual path below.
|
||||
let _held =
|
||||
match pool.borrow(backend, &RemotePath::new(&req.path)).await {
|
||||
Ok(h) => h,
|
||||
Err(e) if e.indicates_offline() => {
|
||||
log::info!("thumbnail sweep: {e}");
|
||||
attempted -= 1;
|
||||
offline = true;
|
||||
break;
|
||||
}
|
||||
Err(e) => {
|
||||
log::debug!("thumbnail sweep: {}: {e}", req.path);
|
||||
failed += 1;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
||||
match fetch_preview(backend, req, &mut found).await {
|
||||
PreviewOutcome::Ready(preview) => {
|
||||
match encode_preview(file_id, &preview) {
|
||||
@@ -3389,6 +3576,10 @@ pub fn spawn_thumbnail_sweep(
|
||||
.send(ThumbSweepMessage::Progress { done, stored })
|
||||
.is_err()
|
||||
{
|
||||
// Cancelled. Hand back what was borrowed before leaving,
|
||||
// or a stopped pass costs the disk of everything it had
|
||||
// reached and delivers nothing for it.
|
||||
pool.release_all(&*backend).await;
|
||||
return;
|
||||
}
|
||||
if offline {
|
||||
@@ -3397,6 +3588,19 @@ pub fn spawn_thumbnail_sweep(
|
||||
}
|
||||
|
||||
flush_sweep(&catalog, &mut found);
|
||||
|
||||
// Give back everything this pass fetched, before reporting done —
|
||||
// a user watching the disk should see it return, and a pass that
|
||||
// reported success while still holding the library would be
|
||||
// lying about what it cost.
|
||||
let returned = pool.release_all(&*backend).await;
|
||||
if returned.released > 0 {
|
||||
log::info!(
|
||||
"thumbnail sweep: released {} borrowed file(s)",
|
||||
returned.released
|
||||
);
|
||||
}
|
||||
|
||||
log::info!("thumbnail sweep: {stored} stored, {failed} without a usable preview");
|
||||
let _ = tx.send(ThumbSweepMessage::Finished {
|
||||
stored,
|
||||
|
||||
Reference in New Issue
Block a user