Give a shard's remote name the client that wrote it
Build and test / Desktop (Linux) (push) Failing after 50s
Build and test / Layer separation (push) Successful in 24s
🐳 Android image / Build and push (push) Successful in 3s
Build and test / android-image (push) Successful in 3s
Traceability / Requirement traces (push) Failing after 59s
Build and test / Android (aarch64) (push) Failing after 9m39s

Shard ids are per store: every client fills its own numbering from 0, so
"shard 3" names different thumbnails on every device. The derived sync
published them into a flat shard-NNNN.sqlite namespace anyway, which left
two clients writing one name.

Both failures that follow were live. On upload, a client's open shard
overwrote a peer's file of the same id — content the peer still believed
was published and would never restore, because its own copy was sealed and
the name existed. On download, the loop skipped any remote id it already
held locally, which is the only safe reading of a name that says nothing
about who wrote it, so a client holding shards 0..5 never fetched the
peer's 0..5 at all. Between them, two populated clients exchanged almost
nothing: only shards numbered above the other's highest. A fresh device
worked, having no local shards to collide with, which is why this went
unnoticed — it is exactly the case the feature was written for.

The name is now shard-<client>-NNNN.sqlite. The client id is minted per
store in index.sqlite, beside the numbering it qualifies rather than in
settings: a store deleted and rebuilt restarts at shard 0 and must not
claim the remote names its predecessor wrote. Since our own ids now say
nothing about what we have taken from others, index.sqlite also keeps a
ledger of adopted remote names and the size each had when merged. A size
rather than a flag, because a peer's sealed shard never returns but its
open one grows, and re-merging the grown copy is how the thumbnails it
gained since arrive.

Flat names already on servers still parse, reporting no owner, so each
client adopts them once, and nothing is written under that form again. One
whose id and byte size match a local shard is that client's own earlier
upload by the same identity argument the upload path already makes for
sealed shards, so the rename does not cost every client a re-download of
its whole store. Older builds ignore the new names and stop receiving
shards until updated; their own uploads are still adopted, so nothing is
lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-16 21:15:12 +02:00
co-authored by Claude Opus 5
parent f7e8cc99b1
commit 9a51cc88d6
3 changed files with 233 additions and 28 deletions
+27 -4
View File
@@ -497,7 +497,7 @@ sync to Nextcloud**, so a second device gets a full grid without re-fetching a b
```text
thumbs/
index.sqlite fileid → shard, plus size accounting
index.sqlite fileid → shard, size accounting, client id, adoption ledger
shard-0000.sqlite ≤ 25 MB, sealed
shard-0001.sqlite ≤ 25 MB, active
```
@@ -530,9 +530,32 @@ Three invariants, each tested:
| Re-storing an existing id updates in place, never migrates | Migrating would rewrite a sealed shard |
| Merging another client's shard is insert-only and idempotent | Both copies derive from the same bytes by the same code, so neither is better; preferring ours avoids dirtying a shard others have synced |
**Not yet built:** the transfer itself. `ThumbStore::shards()` reports which shards are sealed —
the input a sync pass needs — and `merge_shard` adopts a downloaded one, but nothing uploads or
downloads them yet. Until that lands the store is a local cache that happens to have the right shape.
**The transfer**, in `dr-ui`'s `derived_sync`, exchanges shards with `.darkroom-derived/` under the
library root. `ThumbStore::shards()` reports which are sealed, so an up-to-date client's whole pass
is one listing plus whichever shard is still open.
**Why a remote name carries a client id.** Corrected 2026-08-16. Shard ids are *per store* — every
client fills its own numbering from 0 — so the flat `shard-NNNN.sqlite` namespace the transfer first
used had two clients writing one name. Two failures followed from it, and both were live: the second
client's upload **overwrote** content the first still believed was published, and no client could
distinguish a peer's shard 3 from its own, so the only safe reading of "I already hold 3" was to skip
it. Between them, two populated clients exchanged almost nothing — only shards numbered above the
other's highest. A fresh device worked, which is why it went unnoticed: with no local shards there is
nothing to collide with.
The name is now `shard-<client>-NNNN.sqlite`, where `<client>` is minted per store in `index.sqlite`
beside the numbering it qualifies — a store deleted and rebuilt restarts at shard 0 and must not
claim its predecessor's names. Since a client's own ids no longer say anything about what it has
taken from others, `index.sqlite` also keeps an **adoption ledger** of merged remote names and the
size each had. Size, not a flag: a peer's sealed shard never returns, but its open one grows, and
re-merging the grown copy is how the thumbnails it gained arrive.
Flat names left on servers by earlier builds are still read — they report no owner, so each client
adopts them once — and nothing is written under that form again. A flat name whose id and byte size
match a local shard is that client's own earlier upload by the same identity argument used for
sealed shards, so the rename does not cost every client a re-download of its whole store. Older
builds ignore the new names, so they stop receiving shards until updated; nothing is lost, since
their own uploads are still adopted.
Two size classes remain planned — grid (256px) and filmstrip/loupe (1024px). Only the grid class is
implemented. The cache is LRU-capped per NFR-RES-4, and thumbnails evict before proxies and long