From 2fb3eb5d2dfb5699a63aaeba63de8af2280bf595 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Sat, 29 Aug 2026 08:32:01 +0200 Subject: [PATCH] Upload a shard that was sealed after the server saw it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tablet showed 442 of a person's 648 faces and could never catch up. Its shard ledger said why: it had adopted the laptop's shard 0 at 2,711,552 bytes, and that shard is 20,402,176 bytes on disk. The upload skipped it. A sealed shard the server already had under that name was taken to be "byte-identical by construction", so its mere presence was proof enough and the size was never consulted. That premise is false, and this library is the counter-example: an *open* shard is uploaded on every pass as it fills — that is how a growing shard reaches the other devices — so the server ordinarily holds a partial copy of a shard that is later completed and sealed. Sealing then froze that partial copy in place for ever. Nothing downstream could recover from it. The tablet's `has_adopted` check is keyed on name *and* size and would have re-downloaded a changed shard gladly; it was never offered one, because the sending side had stopped looking. So the comparison is on size, sealed or not. The client id is in the name, so nobody else can have written the file and size is a sound test. A sealed shard whose size already matches is still skipped on the first comparison, which is all the original cheapness was worth. The thumbnail upload had the identical test and the identical hole, and is fixed with it. Co-Authored-By: Claude Opus 5 (1M context) --- ui/dr-ui/src/derived_sync.rs | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/ui/dr-ui/src/derived_sync.rs b/ui/dr-ui/src/derived_sync.rs index 7eeff5a..7b16b89 100644 --- a/ui/dr-ui/src/derived_sync.rs +++ b/ui/dr-ui/src/derived_sync.rs @@ -235,13 +235,15 @@ async fn sync_shards( }; let name = shard_name(&client, shard.id); - // A sealed shard the server already has is byte-identical by - // construction, so its presence is proof enough — and with the client - // in the name, no one else can have written it. The open shard is - // re-uploaded whenever its size differs, which is the only way it - // changes. + // **Size, sealed or not.** This used to take the server merely *having* + // a sealed shard as proof it had the whole thing — "byte-identical by + // construction". It is not: an open shard is uploaded on every pass as + // it grows, so the server routinely holds a partial copy of a shard + // that is later filled and sealed. From the moment of sealing, the name + // matched and the size was never looked at again, and the completed + // shard could never be sent. The client id in the name still means + // nobody else can have written it, so size is a sound comparison. let skip = match remote.get(&name) { - Some(_) if shard.sealed => true, Some(size) => *size == bytes.len() as u64, None => false, }; @@ -404,14 +406,21 @@ async fn sync_face_shards( let name = shard_name(&client, shard.id); let path = store.shard_path(shard.id); - // **Decided before the file is read.** A sealed shard the server - // already has is byte-identical by construction, and the client is in - // the name so nobody else could have written it — the name alone - // settles it. Reading first meant every idle sync pulled ninety-four + // **Decided before the file is read**, and on size rather than mere + // presence. Reading first meant every idle sync pulled ninety-four // megabytes off disk to conclude it had nothing to send. + // + // The presence test was worse than wasteful. A sealed shard was taken + // to be byte-identical to whatever the server already had under that + // name — but an *open* shard is uploaded on every pass as it fills, so + // the server ordinarily holds a partial copy of a shard that is later + // completed and sealed. Sealing then froze that partial copy in place: + // the name matched, the size was never consulted, and the finished + // shard was skipped for ever. This library's shard 0 sat on the server + // at 2.7 MB against 20 MB on disk, and the tablet adopted the 2.7 MB — + // which is why it showed a fraction of the faces and never caught up. let on_disk = std::fs::metadata(&path).map(|m| m.len()).unwrap_or(0); let skip = match remote.get(&name) { - Some(_) if shard.sealed => true, Some(size) => *size == on_disk, None => false, };