14 Commits

Include renames

Author	SHA1	Message	Date
dtourolle	21d599b420	Test the guard, since the bug was a branch nobody ran ... The catalog clobber existed because `if let Ok(bytes)` had a failure arm that was never exercised. Fixing it without covering that arm leaves the next person free to collapse it back. Three tests against a backend whose read fails in a chosen way, counting writes — because what went wrong was not a wrong value but a write that should not have happened at all: - a dehydrated snapshot uploads nothing - an unreadable one uploads nothing either, since "refused" is no more "absent" than "not downloaded" is - and a genuine first sync still uploads, which is the half that keeps `NotFound` distinct from `NotMaterialised` rather than merely cautious Checked against the original shape: the first two fail on it and the third passes. A guard test that cannot tell the bug from the fix is decoration. `async-trait` joins dev-dependencies to stand the double up behind `dyn RemoteBackend`.	2026-08-29 10:40:38 +02:00
dtourolle	4168d67cfa	Do not push a catalog over one we could not read ... `sync_catalog` is a read-modify-write over a file another device also writes: take theirs, merge, push the union. It was shaped if let Ok(bytes) = backend.get(&RemoteId::Path(target), None).await { which folds *every* failure into "there is no remote catalog" and carries straight on to the upload. On a placeholder library the snapshot in `.darkroom-derived/` is dehydrated like anything else, so the read failed every time and each sync pushed our catalog over theirs unmerged — taking the other device's collections and their members with it. The same shape as the sidecar bug, and the same fix: a read that fails for anything other than `NotFound` stops the upload and says why. An unreadable or unopenable snapshot stops it too — "will not parse" is not "is not there". This is what `NotFound` and `NotMaterialised` being separate errors is *for*: one means ours is the whole truth, the other means do not dare. Shard downloads go through the same fetch-on-demand read. They logged and skipped before, which on a library the client keeps dehydrated is every shard, every pass, and a peer's thumbnails and faces silently never arriving. And `put` over a placeholder now replaces it rather than refusing. Refusing was over-cautious of me: derived state lives inside the library folder, so a folder the client had dehydrated could never be written to again. An unconditional write replaces the whole file, so there is nothing in the stub to keep — content first, then the placeholder, since in a synced tree an absence is a deletion that propagates. `IfMatch` still refuses, because a stub's validator describes the stub; `IfAbsent` fails, because the file is there and only its content is not.	2026-08-29 10:36:26 +02:00
dtourolle	f12aece07e	Make storage pluggable, and prove it with a folder backend ... `RemoteBackend` existed from the first release and bought nothing it was designed for. Seven files in `dr-ui` constructed a `NextcloudBackend` directly, an account *was* a server URL beside a DAV user id, the local cache directory was named after a hostname, and the launch screen knew that signing in meant a browser handshake. The trait was real; the seam was documentation. A trait over operations is only a quarter of it. Pluggable storage needs four things, and this adds the other three: - **Capabilities** — already there, and the reason the engine can drive two backends at the speed each actually runs at. - **Configuration** — `dr_sync::Account`: where a library lives, in whatever form its connector addresses, with no server in it. Loads every existing config unchanged (`backend` defaults to `nextcloud`, `endpoint` is stored under its historical `server` key), and `Account::namespace()` reproduces the old catalog directory byte for byte, because changing it would abandon a catalog, its thumbnail shards, and the sidecars holding unsynced offline work. - **Registration** — `BackendProvider` and `BackendRegistry`. `ui/dr-ui/src/remote.rs` is now the only file above `dr-sync` that names a connector. `Connection` (an account plus an optional `Secret`) replaces the credentials-and-user-id pair that was threaded through fifteen signatures in an order that could be swapped. `Secret`'s inner string is reachable only through `expose()` and its `Debug` prints `Secret(***)`, so the indirect leak — a `{:?}` on anything holding one — no longer compiles into a leak. Nextcloud is unchanged and keeps every peculiarity: propagating ETags, chunked upload v2, `oc:fileid`, the `oc:permissions` probe on a refused PUT, the 423 retry classification, Login Flow v2. Those are what the capability model exists to serve, not something to hide. `dr-sync-folder` is the second connector: a local disk, a network mount, an external drive, or a folder a Nextcloud client already syncs. No account, no credential — the route that works where no secrets daemon does. It declares `LocalEtags` rather than claiming propagation a POSIX directory cannot provide, which costs nothing because 50k `stat` calls are not 50k PROPFINDs. Identity is a path hash, not an inode: an inode survives a rename but differs between devices and is reused after a delete, so two machines would disagree about which photograph a thumbnail belonged to. Re-deriving a thumbnail is a cost; showing the wrong one is a bug. docs/storage.md is the contract — the traits, the four steps to add a backend, and what each connector declares. ARCH §8.0 and §8.4a, and FR-NC-13, say why.	2026-08-29 09:57:52 +02:00
dtourolleandClaude Opus 5	2fb3eb5d2d	Upload a shard that was sealed after the server saw it ... The tablet showed 442 of a person's 648 faces and could never catch up. Its shard ledger said why: it had adopted the laptop's shard 0 at 2,711,552 bytes, and that shard is 20,402,176 bytes on disk. The upload skipped it. A sealed shard the server already had under that name was taken to be "byte-identical by construction", so its mere presence was proof enough and the size was never consulted. That premise is false, and this library is the counter-example: an *open* shard is uploaded on every pass as it fills — that is how a growing shard reaches the other devices — so the server ordinarily holds a partial copy of a shard that is later completed and sealed. Sealing then froze that partial copy in place for ever. Nothing downstream could recover from it. The tablet's `has_adopted` check is keyed on name *and* size and would have re-downloaded a changed shard gladly; it was never offered one, because the sending side had stopped looking. So the comparison is on size, sealed or not. The client id is in the name, so nobody else can have written the file and size is a sound test. A sealed shard whose size already matches is still skipped on the first comparison, which is all the original cheapness was worth. The thumbnail upload had the identical test and the identical hole, and is fixed with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-29 08:32:01 +02:00
dtourolleandClaude Opus 5	d70fe9da8d	Decide whether to send a shard before reading it ... The upload read every shard off disk and only then asked whether it needed sending. For a library with 94 MB of face shards already on the server, every idle sync read 94 MB to conclude it had nothing to do. The question does not need the bytes. A sealed shard the server already has is byte-identical by construction, and the client id is in the name, so nobody else could have written it — the name settles it. The open shard is compared on size, which `stat` answers. The progress line moves after the skip for the same reason: announced before it, an idle pass claimed to be sending five shards and sent none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 23:08:16 +02:00
dtourolleandClaude Opus 5	a1e8f494b9	Say what the face sync is doing while it does it ... The face pass set the status to "checking faces…" once and then said nothing until it was finished. On a library whose first export after a re-index is 9,849 photographs and 85 MB of shards, that is eight minutes of a progress bar sitting still — which is indistinguishable from a hang, and was reported as one twice. Nothing was wrong with the sync. The only fault was that it was silent. Three places now report, which are the three that take real time: - **Preparing**, per image with a count, since this is the long one and the only one whose length the user cannot guess from anything on screen. - **Sending**, per shard with its size, because a face shard carries crops and runs to tens of megabytes — one of them is a visible wait on any connection. Announced before the upload rather than after, since the wait *is* the upload. - **Taking in** a peer's shard, which is a download and then a row-by-row merge. The export reports every 25 images rather than every one, so the channel behind it stays lost in the write it accompanies. `export_to_shards` keeps its old signature and delegates, so the callers that do not want progress do not grow a parameter for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:16 +02:00
dtourolleandClaude Opus 5	242374fd0f	Let the interface hold a backend without knowing whose it is ... `dr-sync` defines `RemoteBackend` and a capability model the engine adapts to, so a second backend can be added without touching the code that uses one. That boundary was documentation. Seven files in `dr-ui` constructed a `NextcloudBackend` directly, ten functions took one by concrete type, and exactly two call sites in the tree — both inside `dr-sync` itself — ever held the trait object. A WebDAV or local-folder backend would have had a well-written trait to implement and nowhere to go afterwards. The change is smaller than the finding suggests, because the trait was already right. Every method the UI has ever called on a backend — `get`, `put`, `list`, `delete`, `create_dir`, `move_to` — was already on it, so nothing had to be added and no behaviour moved. Ten signatures widened to `&dyn RemoteBackend`, sixteen constructions became `remote::connect`, and `remote.rs` is now the only file in the interface that names a connector. `connect` returns `Result<Box<dyn RemoteBackend>, RemoteError>`. The error type is `dr-sync`'s rather than the connector's, which is why every call site kept its shape — the `match`, the `let Ok(..) else`, and `.map_err(ScanFailure::local)?` all still read as they did. One wrinkle worth recording: `&Box<dyn Trait>` does not reach `&dyn Trait` on its own. The compiler reaches for unsizing, which wants `Box<dyn RemoteBackend>: RemoteBackend`, and reports a confusing missing impl rather than suggesting a deref. Twelve call sites therefore say `&*backend`, and two say `let backend: &dyn RemoteBackend = &*backend` where a borrow is shared across lanes. What this does *not* do is abstract credentials. `AppCredentials` is an app password from Login Flow v2 — a Nextcloud protocol, not a general notion of authenticating to a remote — and seven files still name it. An OAuth token, a bucket key pair and an app password have no useful common shape, so deciding what an account is across backends before a second one exists would be a confident guess. code-health.md CH-2 now records that as the remaining half, and it should wait for the backend that forces it. Verified: fmt clean, clippy clean at -D warnings, 2041 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-27 19:51:23 +02:00
dtourolleandClaude Opus 5	96d07da15f	Sync face data as sealed shards, so a second device does not re-index ... Indexing 23,500 images is about two hours of CPU, and the result is byte-identical on every device: the same model over the same proxy produces the same embedding. Paying for it once per account rather than once per device is the point. Shards rather than the catalog snapshot, because the snapshot goes up whole on every sync and a fully indexed library carries roughly 30 MB of embeddings. That is exactly the cost the thumbnail store's 25 MB cap exists to bound, so face shards use the same cap -- imported from dr_thumbs rather than restated, since the number is a statement about sync cost and the two must not drift apart. The split follows the one already there: bulk immutable data in sealed shards, small mutable data in the catalog snapshot. Faces, landmarks, embeddings and run markers shard; people, names and assignments ride the catalog and merge by uuid. Keyed on oc:fileid throughout, never on image_id, because a row id means nothing on another device. The run marker travels with the faces it describes. Without it a receiving device cannot tell an image with no faces from one never examined, and would re-detect every landscape it had just adopted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 22:41:50 +02:00
dtourolleandClaude Opus 5	11a63fc7a8	Refresh the collection tree when a sync brings membership ... New-York gained 127 photographs from the tablet and the sidebar went on showing no number beside it. Two reasons, and the second is why the first was never noticed. The sync's completion handler reloads the grid and never rebuilds the tree — the scan already does, and the sync merges the same tables. And the condition it reloads under was `collections_gained > 0`, which counts collections, not members: a sync that files 127 photographs into a collection both devices already had gains no collection at all, so the count was zero and nothing refreshed. `SyncReport` now carries `members_gained` from the merge report, and either one rebuilds the tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:10:40 +02:00
dtourolleandClaude Opus 5	9e47133304	Run the formatter over the shard-naming change ... `cargo fmt --all -- --check` is a CI gate and 9a51cc8 landed two files past it, so the build has been red on master since regardless of what came after. Whitespace only — a wrapped `Ok(...)` and two `assert_eq!`s split across lines. No logic is touched and the tests are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 22:29:28 +02:00
dtourolleandClaude Opus 5	9a51cc88d6	Give a shard's remote name the client that wrote it ... Shard ids are per store: every client fills its own numbering from 0, so "shard 3" names different thumbnails on every device. The derived sync published them into a flat shard-NNNN.sqlite namespace anyway, which left two clients writing one name. Both failures that follow were live. On upload, a client's open shard overwrote a peer's file of the same id — content the peer still believed was published and would never restore, because its own copy was sealed and the name existed. On download, the loop skipped any remote id it already held locally, which is the only safe reading of a name that says nothing about who wrote it, so a client holding shards 0..5 never fetched the peer's 0..5 at all. Between them, two populated clients exchanged almost nothing: only shards numbered above the other's highest. A fresh device worked, having no local shards to collide with, which is why this went unnoticed — it is exactly the case the feature was written for. The name is now shard-<client>-NNNN.sqlite. The client id is minted per store in index.sqlite, beside the numbering it qualifies rather than in settings: a store deleted and rebuilt restarts at shard 0 and must not claim the remote names its predecessor wrote. Since our own ids now say nothing about what we have taken from others, index.sqlite also keeps a ledger of adopted remote names and the size each had when merged. A size rather than a flag, because a peer's sealed shard never returns but its open one grows, and re-merging the grown copy is how the thumbnails it gained since arrive. Flat names already on servers still parse, reporting no owner, so each client adopts them once, and nothing is written under that form again. One whose id and byte size match a local shard is that client's own earlier upload by the same identity argument the upload path already makes for sealed shards, so the rename does not cost every client a re-download of its whole store. Older builds ignore the new names and stop receiving shards until updated; their own uploads are still adopted, so nothing is lost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 21:15:12 +02:00
dtourolleandClaude Opus 5	03326242a1	Make the CI checks say what they mean, and format the workspace ... The Android job's "Verify minimum API level" step has never verified the minimum API level. It took the first `*.so` anywhere under the target directory, which is a host proc-macro from debug/deps — an x86-64 object built by the runner's gcc, whose .comment section cannot mention Android and so can never contradict the expected value. It now reads the artifact under the target triple, compares against MIN_API parsed from the Dockerfile rather than a second copy of the number, and fails on a mismatch. Both sides are checked non-empty first: two failed parses would otherwise compare equal and pass, which is the same silent success in a new costume. The Android image installs one SDK package per layer and keeps the output. sdkmanager is a JVM program that aborts when it cannot get memory, and the single `> /dev/null` step reported that as a bare "exit code 134" while a retry re-downloaded everything that had already succeeded. tools/ci-local.sh runs all four jobs — desktop, android, layering, traceability — against the host toolchain, which is pinned to the same 1.92.0 CI installs. Its matrix check compares regeneration against the working tree rather than against HEAD: CI starts from a clean checkout, so git's answer is the right one there and reports every local run stale here. The rest is rustfmt across the workspace, and the clippy findings that surfaced once it did: manual_contains in dr-thumbs and collections_ui, a map iterated as pairs for its keys, an index loop over a slice, and two runtime assertions on a constant now made at compile time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 12:02:51 +02:00
dtourolle	4d78041d1d	Many imorovments	2026-08-12 22:16:15 +02:00
dtourolle	8ad5c86ff9	Add the library, collections, and trash views; theme from style.yaml ... The UI gains the views the catalog work was building toward: a windowed library grid with ratings and flags, the collection tree with drag-to-add, and trash with restore. derived_sync pushes thumbnail shards and the catalog snapshot to the server's derived folder. Tokens now have one source of truth. build.rs reads style.yaml and generates theme.slint into OUT_DIR, which answers every existing `import { Theme } from "theme.slint"` unchanged, because Slint resolves imports against the importing file's directory first and the include paths after. Generating into OUT_DIR rather than beside the hand-written Slint is the point: a generated file sitting in ui/ looks exactly like the files around it that are meant to be edited, and an edit to it would survive until the next touch of style.yaml — a bug that hides for weeks. build.rs fails loudly if a stale ui/theme.slint exists, which would otherwise shadow the generated one silently and make every palette change vanish with no error. The palette moves to near-neutral dark with achromatic signalling, so the accent means "modified" or "active" rather than "heading". Shared components land in widgets.slint: a token that binds several values into one concept is a component, not a row in a YAML file. Adds an optional live-style feature that makes the tokens in-out so they can be written at startup — a feature rather than the default because it stops the properties being constant-folded. serde_norway is the YAML crate: serde_yaml and serde_yml are both deprecated, and its mappings preserve insertion order, which is what lets the generated Slint keep the token ordering the author chose. Assisted-by: LLM	2026-08-09 21:11:38 +02:00