Borrow the library to index it, and give it back
The passes that need every photograph's bytes — thumbnails, face indexing — now borrow each one and release it at the end. On a placeholder library that is the difference between peak disk being the working set and being the whole library. Including on cancellation, which was nearly missed: the face sweep returns mid-loop when the user presses Stop, and without releasing there the disk is spent and nothing is delivered for it. `materialise` now answers whether *it* fetched the content. The pool used to work that out by listing a file's parent directory — one listing per file across a library — when the backend already had to `stat` it to decide whether to ask. One syscall instead of a directory walk, and it removes the bug class the tests found earlier: a file at the library root has no `parent()`, so every one of them read as already-downloaded. **Pinning is the retention control**, and it drives the model the catalog already had rather than a second one. `tier_desired` is what the user asked to keep hydrated, `pending_pins` is the resumable work list, and a pinned collection is never dehydrated for the same reason it was never evicted. It was in fact *broken* here before: `get` on a stub failed, and the pin worker logged "one unreadable file must not abandon the whole pin" and silently did nothing. Pinned originals on such a library are recorded with `path = NULL` (`Cache::record_in_place`) rather than copied under `originals/`. Two reasons, and the second is the important one. A copy would hold every pinned photograph twice, with the budget able to evict the half that was not costing the disk. And `release` deletes the file a row names — so a row that names none cannot delete anything, which puts the one catastrophic operation out of reach by construction rather than by remembering not to call it. Deleting a materialised file inside a synced tree removes the photograph from the server and every other device. Handing disk back is `spawn_dehydrate`, which asks the client. Two gaps written down rather than papered over (docs/storage.md §7): a hydrating pass cannot yet quote its cost, because a stub reports no size; and the two sweeps hold separate pools, so a library indexed for both fetches twice.
This commit is contained in:
@@ -85,27 +85,6 @@ impl BorrowPool {
|
||||
&'p self,
|
||||
backend: &dyn RemoteBackend,
|
||||
path: &RemotePath,
|
||||
) -> Result<Borrowed<'p>, RemoteError> {
|
||||
self.borrow_known(backend, path, None).await
|
||||
}
|
||||
|
||||
/// [`borrow`](Self::borrow) where the caller already knows whether the
|
||||
/// content is here.
|
||||
///
|
||||
/// A sweep reads availability out of the catalog on its way to building a
|
||||
/// work list, so making the pool re-derive it costs a directory listing
|
||||
/// per file for an answer already in hand.
|
||||
///
|
||||
/// **`None` means "find out", and being wrong is not symmetric.** Claiming
|
||||
/// a file was already local leaves it downloaded — disk, and nothing else.
|
||||
/// Claiming it was not releases a file the user may have pinned. So an
|
||||
/// uncertain caller passes `Some(true)` or `None`, never a guess at
|
||||
/// `false`.
|
||||
pub async fn borrow_known<'p>(
|
||||
&'p self,
|
||||
backend: &dyn RemoteBackend,
|
||||
path: &RemotePath,
|
||||
already_local: Option<bool>,
|
||||
) -> Result<Borrowed<'p>, RemoteError> {
|
||||
if !backend.capabilities().materialisation.can_materialise() {
|
||||
return Ok(Borrowed {
|
||||
@@ -129,32 +108,19 @@ impl BorrowPool {
|
||||
}
|
||||
}
|
||||
|
||||
let id = RemoteId::Path(path.clone());
|
||||
// The backend answers whether *it* fetched the content, because it had
|
||||
// to look before deciding. Determining that here instead would cost a
|
||||
// directory listing per file, and getting it wrong in the wrong
|
||||
// direction releases a file the user pinned.
|
||||
let ours = backend.materialise(&RemoteId::Path(path.clone())).await?;
|
||||
|
||||
// What was here *before* we asked. The whole contract rests on this
|
||||
// being read first: after `materialise` there is no way to tell what
|
||||
// we brought from what was already there.
|
||||
let was_local = match already_local {
|
||||
Some(known) => known,
|
||||
None => is_materialised(backend, path).await,
|
||||
};
|
||||
|
||||
if !was_local {
|
||||
backend.materialise(&id).await?;
|
||||
}
|
||||
|
||||
self.lock().insert(
|
||||
path.clone(),
|
||||
Held {
|
||||
borrowers: 1,
|
||||
ours: !was_local,
|
||||
},
|
||||
);
|
||||
self.lock()
|
||||
.insert(path.clone(), Held { borrowers: 1, ours });
|
||||
|
||||
Ok(Borrowed {
|
||||
pool: Some(self),
|
||||
path: path.clone(),
|
||||
hydrated: !was_local,
|
||||
hydrated: ours,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -237,28 +203,5 @@ impl Drop for Borrowed<'_> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the backend currently holds this file's content.
|
||||
///
|
||||
/// Asked by listing its parent, because the trait has no "stat one object" —
|
||||
/// and adding one for this would put a method on every backend to serve a case
|
||||
/// only one of them has.
|
||||
async fn is_materialised(backend: &dyn RemoteBackend, path: &RemotePath) -> bool {
|
||||
// `parent()` is `None` for a file directly under the library root, whose
|
||||
// parent *is* the root — not "no parent". Treating the two alike reported
|
||||
// every top-level photograph as already downloaded, so nothing was ever
|
||||
// hydrated and nothing was ever released.
|
||||
let parent = path.parent().unwrap_or_else(RemotePath::root);
|
||||
match backend.list(&parent, None).await {
|
||||
Ok(entries) => entries
|
||||
.iter()
|
||||
.find(|e| &e.path == path)
|
||||
.map(|e| e.materialised)
|
||||
// Not listed at all: nothing to hydrate, and the caller's own read
|
||||
// will report the miss properly.
|
||||
.unwrap_or(true),
|
||||
Err(_) => true,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
||||
Reference in New Issue
Block a user