Upload a shard that was sealed after the server saw it
Build and test / Desktop (Linux) (push) Successful in 2h7m52s
Build and test / Layer separation (push) Successful in 57s
🐳 Android image / Build and push (push) Successful in 12s
Build and test / android-image (push) Successful in 13s
Traceability / Requirement traces (push) Successful in 57s
Build and test / Android (aarch64) (push) Successful in 20m58s
Build and test / Desktop (Linux) (push) Successful in 2h7m52s
Build and test / Layer separation (push) Successful in 57s
🐳 Android image / Build and push (push) Successful in 12s
Build and test / android-image (push) Successful in 13s
Traceability / Requirement traces (push) Successful in 57s
Build and test / Android (aarch64) (push) Successful in 20m58s
The tablet showed 442 of a person's 648 faces and could never catch up. Its shard ledger said why: it had adopted the laptop's shard 0 at 2,711,552 bytes, and that shard is 20,402,176 bytes on disk. The upload skipped it. A sealed shard the server already had under that name was taken to be "byte-identical by construction", so its mere presence was proof enough and the size was never consulted. That premise is false, and this library is the counter-example: an *open* shard is uploaded on every pass as it fills — that is how a growing shard reaches the other devices — so the server ordinarily holds a partial copy of a shard that is later completed and sealed. Sealing then froze that partial copy in place for ever. Nothing downstream could recover from it. The tablet's `has_adopted` check is keyed on name *and* size and would have re-downloaded a changed shard gladly; it was never offered one, because the sending side had stopped looking. So the comparison is on size, sealed or not. The client id is in the name, so nobody else can have written the file and size is a sound test. A sealed shard whose size already matches is still skipped on the first comparison, which is all the original cheapness was worth. The thumbnail upload had the identical test and the identical hole, and is fixed with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -235,13 +235,15 @@ async fn sync_shards(
|
|||||||
};
|
};
|
||||||
let name = shard_name(&client, shard.id);
|
let name = shard_name(&client, shard.id);
|
||||||
|
|
||||||
// A sealed shard the server already has is byte-identical by
|
// **Size, sealed or not.** This used to take the server merely *having*
|
||||||
// construction, so its presence is proof enough — and with the client
|
// a sealed shard as proof it had the whole thing — "byte-identical by
|
||||||
// in the name, no one else can have written it. The open shard is
|
// construction". It is not: an open shard is uploaded on every pass as
|
||||||
// re-uploaded whenever its size differs, which is the only way it
|
// it grows, so the server routinely holds a partial copy of a shard
|
||||||
// changes.
|
// that is later filled and sealed. From the moment of sealing, the name
|
||||||
|
// matched and the size was never looked at again, and the completed
|
||||||
|
// shard could never be sent. The client id in the name still means
|
||||||
|
// nobody else can have written it, so size is a sound comparison.
|
||||||
let skip = match remote.get(&name) {
|
let skip = match remote.get(&name) {
|
||||||
Some(_) if shard.sealed => true,
|
|
||||||
Some(size) => *size == bytes.len() as u64,
|
Some(size) => *size == bytes.len() as u64,
|
||||||
None => false,
|
None => false,
|
||||||
};
|
};
|
||||||
@@ -404,14 +406,21 @@ async fn sync_face_shards(
|
|||||||
let name = shard_name(&client, shard.id);
|
let name = shard_name(&client, shard.id);
|
||||||
let path = store.shard_path(shard.id);
|
let path = store.shard_path(shard.id);
|
||||||
|
|
||||||
// **Decided before the file is read.** A sealed shard the server
|
// **Decided before the file is read**, and on size rather than mere
|
||||||
// already has is byte-identical by construction, and the client is in
|
// presence. Reading first meant every idle sync pulled ninety-four
|
||||||
// the name so nobody else could have written it — the name alone
|
|
||||||
// settles it. Reading first meant every idle sync pulled ninety-four
|
|
||||||
// megabytes off disk to conclude it had nothing to send.
|
// megabytes off disk to conclude it had nothing to send.
|
||||||
|
//
|
||||||
|
// The presence test was worse than wasteful. A sealed shard was taken
|
||||||
|
// to be byte-identical to whatever the server already had under that
|
||||||
|
// name — but an *open* shard is uploaded on every pass as it fills, so
|
||||||
|
// the server ordinarily holds a partial copy of a shard that is later
|
||||||
|
// completed and sealed. Sealing then froze that partial copy in place:
|
||||||
|
// the name matched, the size was never consulted, and the finished
|
||||||
|
// shard was skipped for ever. This library's shard 0 sat on the server
|
||||||
|
// at 2.7 MB against 20 MB on disk, and the tablet adopted the 2.7 MB —
|
||||||
|
// which is why it showed a fraction of the faces and never caught up.
|
||||||
let on_disk = std::fs::metadata(&path).map(|m| m.len()).unwrap_or(0);
|
let on_disk = std::fs::metadata(&path).map(|m| m.len()).unwrap_or(0);
|
||||||
let skip = match remote.get(&name) {
|
let skip = match remote.get(&name) {
|
||||||
Some(_) if shard.sealed => true,
|
|
||||||
Some(size) => *size == on_disk,
|
Some(size) => *size == on_disk,
|
||||||
None => false,
|
None => false,
|
||||||
};
|
};
|
||||||
|
|||||||
Reference in New Issue
Block a user