Borrow the library to index it, and give it back

The passes that need every photograph's bytes — thumbnails, face
indexing — now borrow each one and release it at the end. On a
placeholder library that is the difference between peak disk being the
working set and being the whole library.

Including on cancellation, which was nearly missed: the face sweep
returns mid-loop when the user presses Stop, and without releasing there
the disk is spent and nothing is delivered for it.

`materialise` now answers whether *it* fetched the content. The pool used
to work that out by listing a file's parent directory — one listing per
file across a library — when the backend already had to `stat` it to
decide whether to ask. One syscall instead of a directory walk, and it
removes the bug class the tests found earlier: a file at the library root
has no `parent()`, so every one of them read as already-downloaded.

**Pinning is the retention control**, and it drives the model the catalog
already had rather than a second one. `tier_desired` is what the user
asked to keep hydrated, `pending_pins` is the resumable work list, and a
pinned collection is never dehydrated for the same reason it was never
evicted. It was in fact *broken* here before: `get` on a stub failed, and
the pin worker logged "one unreadable file must not abandon the whole
pin" and silently did nothing.

Pinned originals on such a library are recorded with `path = NULL`
(`Cache::record_in_place`) rather than copied under `originals/`. Two
reasons, and the second is the important one. A copy would hold every
pinned photograph twice, with the budget able to evict the half that was
not costing the disk. And `release` deletes the file a row names — so a
row that names none cannot delete anything, which puts the one
catastrophic operation out of reach by construction rather than by
remembering not to call it. Deleting a materialised file inside a synced
tree removes the photograph from the server and every other device.

Handing disk back is `spawn_dehydrate`, which asks the client.

Two gaps written down rather than papered over (docs/storage.md §7): a
hydrating pass cannot yet quote its cost, because a stub reports no size;
and the two sweeps hold separate pools, so a library indexed for both
fetches twice.
This commit is contained in:
2026-08-29 09:57:53 +02:00
parent c102ba9df2
commit 5768100816
9 changed files with 406 additions and 114 deletions
+8 -65
View File
@@ -85,27 +85,6 @@ impl BorrowPool {
&'p self,
backend: &dyn RemoteBackend,
path: &RemotePath,
) -> Result<Borrowed<'p>, RemoteError> {
self.borrow_known(backend, path, None).await
}
/// [`borrow`](Self::borrow) where the caller already knows whether the
/// content is here.
///
/// A sweep reads availability out of the catalog on its way to building a
/// work list, so making the pool re-derive it costs a directory listing
/// per file for an answer already in hand.
///
/// **`None` means "find out", and being wrong is not symmetric.** Claiming
/// a file was already local leaves it downloaded — disk, and nothing else.
/// Claiming it was not releases a file the user may have pinned. So an
/// uncertain caller passes `Some(true)` or `None`, never a guess at
/// `false`.
pub async fn borrow_known<'p>(
&'p self,
backend: &dyn RemoteBackend,
path: &RemotePath,
already_local: Option<bool>,
) -> Result<Borrowed<'p>, RemoteError> {
if !backend.capabilities().materialisation.can_materialise() {
return Ok(Borrowed {
@@ -129,32 +108,19 @@ impl BorrowPool {
}
}
let id = RemoteId::Path(path.clone());
// The backend answers whether *it* fetched the content, because it had
// to look before deciding. Determining that here instead would cost a
// directory listing per file, and getting it wrong in the wrong
// direction releases a file the user pinned.
let ours = backend.materialise(&RemoteId::Path(path.clone())).await?;
// What was here *before* we asked. The whole contract rests on this
// being read first: after `materialise` there is no way to tell what
// we brought from what was already there.
let was_local = match already_local {
Some(known) => known,
None => is_materialised(backend, path).await,
};
if !was_local {
backend.materialise(&id).await?;
}
self.lock().insert(
path.clone(),
Held {
borrowers: 1,
ours: !was_local,
},
);
self.lock()
.insert(path.clone(), Held { borrowers: 1, ours });
Ok(Borrowed {
pool: Some(self),
path: path.clone(),
hydrated: !was_local,
hydrated: ours,
})
}
@@ -237,28 +203,5 @@ impl Drop for Borrowed<'_> {
}
}
/// Whether the backend currently holds this file's content.
///
/// Asked by listing its parent, because the trait has no "stat one object" —
/// and adding one for this would put a method on every backend to serve a case
/// only one of them has.
async fn is_materialised(backend: &dyn RemoteBackend, path: &RemotePath) -> bool {
// `parent()` is `None` for a file directly under the library root, whose
// parent *is* the root — not "no parent". Treating the two alike reported
// every top-level photograph as already downloaded, so nothing was ever
// hydrated and nothing was ever released.
let parent = path.parent().unwrap_or_else(RemotePath::root);
match backend.list(&parent, None).await {
Ok(entries) => entries
.iter()
.find(|e| &e.path == path)
.map(|e| e.materialised)
// Not listed at all: nothing to hydrate, and the caller's own read
// will report the miss properly.
.unwrap_or(true),
Err(_) => true,
}
}
#[cfg(test)]
mod tests;