Borrow the library to index it, and give it back
The passes that need every photograph's bytes — thumbnails, face indexing — now borrow each one and release it at the end. On a placeholder library that is the difference between peak disk being the working set and being the whole library. Including on cancellation, which was nearly missed: the face sweep returns mid-loop when the user presses Stop, and without releasing there the disk is spent and nothing is delivered for it. `materialise` now answers whether *it* fetched the content. The pool used to work that out by listing a file's parent directory — one listing per file across a library — when the backend already had to `stat` it to decide whether to ask. One syscall instead of a directory walk, and it removes the bug class the tests found earlier: a file at the library root has no `parent()`, so every one of them read as already-downloaded. **Pinning is the retention control**, and it drives the model the catalog already had rather than a second one. `tier_desired` is what the user asked to keep hydrated, `pending_pins` is the resumable work list, and a pinned collection is never dehydrated for the same reason it was never evicted. It was in fact *broken* here before: `get` on a stub failed, and the pin worker logged "one unreadable file must not abandon the whole pin" and silently did nothing. Pinned originals on such a library are recorded with `path = NULL` (`Cache::record_in_place`) rather than copied under `originals/`. Two reasons, and the second is the important one. A copy would hold every pinned photograph twice, with the budget able to evict the half that was not costing the disk. And `release` deletes the file a row names — so a row that names none cannot delete anything, which puts the one catastrophic operation out of reach by construction rather than by remembering not to call it. Deleting a materialised file inside a synced tree removes the photograph from the server and every other device. Handing disk back is `spawn_dehydrate`, which asks the client. Two gaps written down rather than papered over (docs/storage.md §7): a hydrating pass cannot yet quote its cost, because a stub reports no size; and the two sweeps hold separate pools, so a library indexed for both fetches twice.
This commit is contained in:
@@ -221,22 +221,46 @@ async fn borrowing_against_a_plain_folder_does_nothing_at_all() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_uncertain_caller_keeps_the_file_rather_than_releasing_it() {
|
||||
// The asymmetry stated on `borrow_known`: claiming "already local" costs
|
||||
// disk, claiming "not local" can release a pin.
|
||||
let t = Tmp::new("uncertain");
|
||||
t.real("a.CR2");
|
||||
async fn the_backend_is_what_decides_whether_a_file_was_ours() {
|
||||
// Not the pool, and not the caller. The backend had to look before
|
||||
// deciding whether to ask, so it can answer for the cost of that same
|
||||
// `stat`; a borrower working it out separately would pay a directory
|
||||
// listing per file and could get it wrong in the direction that releases
|
||||
// a file the user pinned.
|
||||
let t = Tmp::new("who-decides");
|
||||
t.real("had.CR2").stub("wanted.CR2");
|
||||
let b = t.backend(FakeClient::new());
|
||||
|
||||
assert!(
|
||||
!b.materialise(&RemoteId::Path(RemotePath::new("had.CR2")))
|
||||
.await
|
||||
.unwrap(),
|
||||
"already here, so not ours to release"
|
||||
);
|
||||
assert!(
|
||||
b.materialise(&RemoteId::Path(RemotePath::new("wanted.CR2")))
|
||||
.await
|
||||
.unwrap(),
|
||||
"this call fetched it"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_file_borrowed_twice_in_one_pass_is_fetched_once_and_released_once() {
|
||||
// Thumbnailing and face indexing visit the same photograph. Fetching it
|
||||
// per stage doubles the transfer over the whole library.
|
||||
let t = Tmp::new("sequential");
|
||||
t.stub("a.CR2");
|
||||
let client = FakeClient::new();
|
||||
let b = t.backend(client.clone());
|
||||
let pool = BorrowPool::new();
|
||||
let path = RemotePath::new("a.CR2");
|
||||
|
||||
{
|
||||
let _held = pool
|
||||
.borrow_known(&b, &RemotePath::new("a.CR2"), Some(true))
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
pool.release_all(&b).await;
|
||||
assert!(t.has("a.CR2"), "kept");
|
||||
assert_eq!(client.dehydrations.load(Ordering::SeqCst), 0);
|
||||
// Sequential borrows, as two passes over one work list would make.
|
||||
drop(pool.borrow(&b, &path).await.unwrap());
|
||||
drop(pool.borrow(&b, &path).await.unwrap());
|
||||
|
||||
assert_eq!(client.hydrations.load(Ordering::SeqCst), 1, "paid once");
|
||||
let stats = pool.release_all(&b).await;
|
||||
assert_eq!(stats.released, 1, "given back once");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user