936490880bd85b0c924446fab4c9864408ee357a
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1ad35e2b87 |
Read the library's sidecars, so a cull done elsewhere arrives
Judgements only ever travelled outward. A rating went to the catalog and to the photograph's sidecar, the sidecar reached the server, and there it stopped: the scan indexes files, `derived_sync` exchanges thumbnails, face shards and collections, `dr_catalog::merge` reconciles everything in a catalog except `versions.rating` and `versions.flag`, and the one sidecar reader that existed ran when a single photograph was opened in develop and handed its answer to the develop graph. `JobKind::ReadSidecar` was declared for exactly this when the job queue was written and was never enqueued or handled anywhere. The grid draws `versions.rating`. So a day of culling on the tablet could not reach the laptop by any path the application had, and the laptop's catalog says so plainly: 23,568 images, one of them judged. `pull_sidecars` closes it, off the back of work the scan already does. `dr_sync::scan` reports the `.drsc` files it meets in listings it was making anyway — no extra request, and a directory whose ETag is unchanged is still pruned before it is listed at all. A new `sidecars` table records the ETag of each one this device has taken in, so the fetch is one GET per sidecar that genuinely changed rather than one per photograph. A library nobody has edited costs nothing. The judgement is taken rather than maximised. The sidecar is the authoritative store and the fuse has already settled any contest between devices on `revision`, so lowering a rating from four to one on the tablet lowers it here — taking the larger would have refused every demotion the photographer ever made, which is most of what a second pass over a shoot is. A zero is the exception: it means *never judged*, not "judged zero", so a sidecar carrying none cannot erase a star this device holds. That is `merge_judgement`'s asymmetry and it carries the same known cost — clearing a rating does not propagate. A sidecar names a stem, so both halves of a RAW-and-JPEG pair are judged: they are one photograph (FR-CAT-11) sharing one document, and judging only one of them would leave the grid disagreeing with itself over which it drew. The `LIKE` that finds them is a filter, not the decision — `sidecar_path` is applied to every candidate, because a folder is entitled to contain a `%` and a rating landing on the wrong frame would be silent and permanent. Failing to read one is not a failure to scan: the ETag goes unrecorded, the ratings already here stay where they are, and the next scan tries again. The count is reported to the status line as well as the log, because a grid that silently gains three hundred stars is indistinguishable from one that has gone wrong — and because while this number was structurally zero there was nothing to tell the photographer their cull had not arrived. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
7418b040da |
Refuse a scan whose root has gone, instead of reporting it empty
FR-PLAT-AND-2, and a silent failure on both platforms. `dr_sync::scan` stepped over a NotFound or PermissionDenied the way it does for a child that vanished mid-walk -- correct for a child, wrong for the root, where it ended the walk, returned Ok with nothing in it, and reported a successful scan of a library that was no longer there. A lost root is now its own error. The images under it are marked Availability::Offline per FR-CAT-9 and no catalog row is deleted; `library::persist` clears the mark per file as each one is listed again, so a root that comes back needs no repair step. Partly satisfied rather than closed, and the gap is worth stating. The recovery half is real and reachable on Android today, because `map_status` turns Nextcloud's 403 and 404 into it and Nextcloud is how a phone actually gets a library in this build. The causes the requirement names -- revocation, reinstall, a removed card -- are properties of a persisted tree permission, and there is none: SAF does not exist here, `SourceRef::Document` is constructed only in test modules, and `LocalStorage` rejects the variant outright. When SAF lands it becomes a third producer of this error and nothing above it changes, which is why the discovery belongs in the connector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
5768100816 |
Borrow the library to index it, and give it back
The passes that need every photograph's bytes — thumbnails, face indexing — now borrow each one and release it at the end. On a placeholder library that is the difference between peak disk being the working set and being the whole library. Including on cancellation, which was nearly missed: the face sweep returns mid-loop when the user presses Stop, and without releasing there the disk is spent and nothing is delivered for it. `materialise` now answers whether *it* fetched the content. The pool used to work that out by listing a file's parent directory — one listing per file across a library — when the backend already had to `stat` it to decide whether to ask. One syscall instead of a directory walk, and it removes the bug class the tests found earlier: a file at the library root has no `parent()`, so every one of them read as already-downloaded. **Pinning is the retention control**, and it drives the model the catalog already had rather than a second one. `tier_desired` is what the user asked to keep hydrated, `pending_pins` is the resumable work list, and a pinned collection is never dehydrated for the same reason it was never evicted. It was in fact *broken* here before: `get` on a stub failed, and the pin worker logged "one unreadable file must not abandon the whole pin" and silently did nothing. Pinned originals on such a library are recorded with `path = NULL` (`Cache::record_in_place`) rather than copied under `originals/`. Two reasons, and the second is the important one. A copy would hold every pinned photograph twice, with the budget able to evict the half that was not costing the disk. And `release` deletes the file a row names — so a row that names none cannot delete anything, which puts the one catastrophic operation out of reach by construction rather than by remembering not to call it. Deleting a materialised file inside a synced tree removes the photograph from the server and every other device. Handing disk back is `spawn_dehydrate`, which asks the client. Two gaps written down rather than papered over (docs/storage.md §7): a hydrating pass cannot yet quote its cost, because a stub reports no size; and the two sweeps hold separate pools, so a library indexed for both fetches twice. |
||
|
|
c102ba9df2 |
Treat a placeholder as the photograph, not as a one-byte file
The folder connector was pointed at a Nextcloud VFS tree and got three things wrong, the first of which loses work. **A dehydrated sidecar read as absent.** `a.drsc` does not exist when the client has dehydrated it — only `a.drsc.nextcloud` does — so `get` missed, `.ok()` swallowed the `NotFound`, and the sidecar writer took that for "there is no sidecar yet" and wrote a fresh document over the existing one. Every edit another device had put there went with it. That function's own doc comment calls this the exact loss the format's unknown-key preservation exists to prevent. **A stub was catalogued as a 1-byte image**, and ARCH §9.0 measured this machine at 121,785 placeholders against 10,267 real files — so a folder library on a synced tree was ~92% broken rows. **Identity changed on hydration**, so downloading a photograph looked like a delete and an add, orphaning its thumbnail and its face rows. Entries now carry the photograph's own name and a `materialised` flag; `get` on a stub returns the new `RemoteError::NotMaterialised`, which is distinct from `NotFound` precisely because the sidecar writer must treat them differently — it fetches the sidecar and merges, or leaves the entry queued. Hydration is a **borrow**. `BorrowPool` records what was on disk before it asked, so `release_all` dehydrates only what a pass brought and leaves what the user already had. Reference counted: the thumbnail pass and the face pass meet on the same RAW, and without counting the first to finish dehydrates the file the second is reading. A borrow against a plain folder or a server does nothing, so a pass written for VFS runs everywhere. Releasing means asking the client to dehydrate and never deleting: a deletion inside a synced tree propagates to the server and removes the photograph from every device. Not a second backend — the capability is per *connection*, not per type, since the same folder hydrates only while the client runs. The convention arrives through a detector the registry supplies, so `dr-sync-folder` still knows nothing about any client's protocol. ARCH §9.0a records this as an amendment: finding 3 rejected hydration because it costs 100× a range read, and that comparison assumed a connector was available. A folder library has none. |
||
|
|
f12aece07e |
Make storage pluggable, and prove it with a folder backend
`RemoteBackend` existed from the first release and bought nothing it was
designed for. Seven files in `dr-ui` constructed a `NextcloudBackend`
directly, an account *was* a server URL beside a DAV user id, the local
cache directory was named after a hostname, and the launch screen knew
that signing in meant a browser handshake. The trait was real; the seam
was documentation.
A trait over operations is only a quarter of it. Pluggable storage needs
four things, and this adds the other three:
- **Capabilities** — already there, and the reason the engine can drive
two backends at the speed each actually runs at.
- **Configuration** — `dr_sync::Account`: where a library lives, in
whatever form its connector addresses, with no server in it. Loads
every existing config unchanged (`backend` defaults to `nextcloud`,
`endpoint` is stored under its historical `server` key), and
`Account::namespace()` reproduces the old catalog directory byte for
byte, because changing it would abandon a catalog, its thumbnail
shards, and the sidecars holding unsynced offline work.
- **Registration** — `BackendProvider` and `BackendRegistry`.
`ui/dr-ui/src/remote.rs` is now the only file above `dr-sync` that
names a connector.
`Connection` (an account plus an optional `Secret`) replaces the
credentials-and-user-id pair that was threaded through fifteen
signatures in an order that could be swapped. `Secret`'s inner string is
reachable only through `expose()` and its `Debug` prints `Secret(***)`,
so the indirect leak — a `{:?}` on anything holding one — no longer
compiles into a leak.
Nextcloud is unchanged and keeps every peculiarity: propagating ETags,
chunked upload v2, `oc:fileid`, the `oc:permissions` probe on a refused
PUT, the 423 retry classification, Login Flow v2. Those are what the
capability model exists to serve, not something to hide.
`dr-sync-folder` is the second connector: a local disk, a network mount,
an external drive, or a folder a Nextcloud client already syncs. No
account, no credential — the route that works where no secrets daemon
does. It declares `LocalEtags` rather than claiming propagation a POSIX
directory cannot provide, which costs nothing because 50k `stat` calls
are not 50k PROPFINDs. Identity is a path hash, not an inode: an inode
survives a rename but differs between devices and is reused after a
delete, so two machines would disagree about which photograph a
thumbnail belonged to. Re-deriving a thumbnail is a cost; showing the
wrong one is a bug.
docs/storage.md is the contract — the traits, the four steps to add a
backend, and what each connector declares. ARCH §8.0 and §8.4a, and
FR-NC-13, say why.
|
||
|
|
e3dd1526e0 |
Recognise a card whose photographs are already held
Re-inserting a card that has already been imported produced a folder full of `-1` copies. Both halves of the placement logic treated a taken name as a collision to rename around, which is right for the case they were written for — two cameras both writing IMG_0001.CR3 — and exactly wrong for the far more common one, where the taken name is the same photograph. Locally this cannot be answered from the catalog. On a library whose catalog describes a *server*, a file sitting in the local destination has no row to be found by, so the only way to know whether it has already been copied is to look. The destination folder is listed once per folder rather than probed per file: a card is two thousand frames landing in a handful of days. Remotely the same question is one PROPFIND that was already being made to resolve the name, so `upload_original` now returns `Placed::AlreadyThere` instead of inventing a second copy of work that is already safe. The count is reported apart from `uploaded`, because "12 already on the server" and "12 uploaded" are different answers to whether this run backed anything up — and apart from the local duplicate count, because these files *were* copied here. Name plus length decides it, not a digest: this runs before any transfer, and hashing to answer it would read the whole card to avoid reading the whole card. A camera reusing a filename after IMG_9999 writes a different number of bytes essentially always, which leaves the rename for the case it is really for. The digest tier still catches the same frame under a different name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
125fccbb46 |
Put an uploaded original in its dated folder
The transport was built and had nowhere to aim: put_chunked has existed since the connector landed, and the only things pushed through it are thumbnail shards and the catalog snapshot. The folder segments arrive already expanded, from dr_ingest::layout, rather than being re-derived here. That split is what makes FR-NC-7a's first consequence true — if this module worked out the folders from whatever the local library happened to look like, two machines with differently organised libraries would file the same photograph in two different places on one server. A name is resolved against one listing rather than a probe per candidate: a day folder is one PROPFIND and the answer covers every collision in it, where probing costs a round trip per attempt over a link that may be mobile data. Two cameras both produce IMG_0001.CR3, and the second must not overwrite the first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
6621c11ad6 |
Diagnose the refused sidecar: the library mount is create-only
Ratings and edits made on the tablet queue and are then refused on reconnect,
on a credential that pushes the catalog to the same library root in the same
sync pass. Chased on the device, since that is where the account lives.
A failed PUT now logs the server's own words, and on a 403 asks the parent
what rights it reports. The answer:
PUT .../PhotosRaw/2026/2026-08-03/_MG_9221.drsc -> 403
<s:exception>Sabre\DAV\Exception\Forbidden</s:exception>
parent permissions: MGNVCK
`M` mounted, `G` readable, `N` renameable, `V` moveable, `CK` create files and
folders. Absent: `W`, update an existing file, and `D`, delete. So `PhotosRaw`
is a mounted share that accepts a file once and refuses every change to it
afterwards.
That is the whole bug, and it is not one this side can retry its way out of. A
sidecar is rewritten on every rating and every edit, so the first judgement on
a photograph is written and every later one is refused — which reads as sync
being broken rather than as a share missing one permission. **The fix is to
grant update, and ideally delete, on that mount.**
`PermissionDenied` now says so, rather than "not allowed to write here", which
sent the reader to re-check a login that was working. Its test asserts the
intent — points at the folder, never at the credential — rather than a
phrase, so saying it better cannot read as a regression.
Also adds a `put_probe` example that makes the same request from a stored
session, for diagnosing this from a desktop when one is signed in.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
03326242a1 |
Make the CI checks say what they mean, and format the workspace
The Android job's "Verify minimum API level" step has never verified the minimum API level. It took the first `*.so` anywhere under the target directory, which is a host proc-macro from debug/deps — an x86-64 object built by the runner's gcc, whose .comment section cannot mention Android and so can never contradict the expected value. It now reads the artifact under the target triple, compares against MIN_API parsed from the Dockerfile rather than a second copy of the number, and fails on a mismatch. Both sides are checked non-empty first: two failed parses would otherwise compare equal and pass, which is the same silent success in a new costume. The Android image installs one SDK package per layer and keeps the output. sdkmanager is a JVM program that aborts when it cannot get memory, and the single `> /dev/null` step reported that as a bare "exit code 134" while a retry re-downloaded everything that had already succeeded. tools/ci-local.sh runs all four jobs — desktop, android, layering, traceability — against the host toolchain, which is pinned to the same 1.92.0 CI installs. Its matrix check compares regeneration against the working tree rather than against HEAD: CI starts from a clean checkout, so git's answer is the right one there and reports every local run stale here. The rest is rustfmt across the workspace, and the clippy findings that surfaced once it did: manual_contains in dr-thumbs and collections_ui, a map iterated as pairs for its keys, an index loop over a slice, and two runtime assertions on a constant now made at compile time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
4d78041d1d | Many imorovments | ||
|
|
cd75e5a4c6 |
Run the library from local data when the server is unreachable
Also carries in-flight work that shared these files: the zoom structure-key fix in the adjust pipeline, nearest-neighbour filtering past 1:1, the timeline scrub marker correction, the 423-Locked retry in the metadata sweep, and the thumbnail size-class migration. # Offline mode (FR-CAT-9) The app previously assumed the server was reachable and treated its absence as a series of unrelated per-operation failures. A launch without a connection produced an empty grid, even with a complete catalog on disk and every thumbnail already in the shards. Reachability is now inferred from traffic the app was already making, rather than probed for. `RemoteError::indicates_offline` draws the line that makes this possible: a dead connection is offline, a 403 or a 500 is not — the server answered, so blanking the library over one forbidden file would be a worse error than the one being reported. `Reachability` turns those outcomes into a state, so a library browsing happily never issues a probe at all. Going offline takes one failure, because the user is already experiencing it. Coming back requires evidence — a completed scan or a fetched thumbnail — with a capped exponential backoff behind the manual retry, so twelve sweep lanes failing together do not schedule twelve immediate probes. What keeps working: the catalog opens even when the scan that normally provides it failed, so the grid fills from the last successful scan. Thumbnails come from the shards. Rating, flagging and collecting are catalog writes that never touched the network. What stops is opening an original that was never stored locally, and it now says so in those words instead of reporting "network error: connection refused" over a photograph. Work that is pure network is refused rather than left to fail slowly: the metadata sweep, derived sync, and sidecar writes. The sweep would otherwise spend a timeout per image across the whole library while the progress bar implied something was happening. Deferring sidecars is a real gap rather than a hidden one — a rating made offline reaches its sidecar only when that image is judged again while connected — and it is recorded as such at the call site. # The "On this device" filter A chip beside the rating filters, narrowing the grid to images whose original is held locally. It composes with the rating terms rather than replacing them, so "five-star frames I can actually edit on this train" is one filter. The predicate is SQL, like the rating terms and for the same reason: the count in the header has to agree with the cells drawn. It reads `image_cache.tier_actual`, which nothing writes yet — the next commit fills it. Until then the chip honestly reports zero. `Tier` gains an explicit on-disk encoding. The variants are ordered by generosity and the derived `Ord` invites reordering them, which would silently reinterpret every cached row; the round-trip test is what holds the two in agreement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
b4c1645c7a |
Address trash moves and deletes by path, not fileid
Trashing or emptying the trash failed on every scanned image with "operation unsupported by this backend: fetch by fileid requires a path". RemoteId::Stable(oc:fileid) is an identity — it answers whether a file is the same one after a move, and keys the thumbnail shards. It is not an address: WebDAV exposes no fileid-addressable endpoint, so the Nextcloud backend serves get/delete/move_to by path and rejects a bare Stable. Both trash workers preferred the fileid whenever the catalog knew one, so the unreachable Path fallback was the only arm that would have worked, and the failure hit every properly-scanned image rather than some edge case. Address by path at both sites, and keep the fileid for what it is for: the identity MOVE preserves, and the key the thumbnail cleanup uses. Move::file_id was documented as "the id the MOVE addresses", which is the wrong claim that seeded this; corrected, along with a note on RemoteId itself so the distinction is stated where the type is defined. Only the backend rejection was covered by a test. Added the positive case, since that contract is what the call sites now depend on. The workers build their own NextcloudBackend, so no test can reach the call sites directly — closing that would mean injecting the backend, which is left alone here. Verified by build and test; not exercised against a live server. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
81e89de7ea |
Add remote move and mkdir; distinguish 403 from 401
Soft delete needs to move a photograph into the trash folder and back, and the stable id must survive the trip. WebDAV MOVE is one request and preserves oc:fileid; a copy-then-delete would allocate a new one, orphaning the thumbnail shard entry and the sidecar mapping and turning a restore into a full re-download. Overwrite: F, because a header that permits overwriting is one that eventually does. create_dir does MKCOL outermost-first and treats 405 — Nextcloud's answer for an existing collection — as the goal state rather than an error. Nothing else creates the trash folder, so without it the first trashed image of every library fails with a 409 that reads like a permission problem. PermissionDenied is now separate from AuthFailed. Folding 403 into 401 sent a user to re-check a credential that was working perfectly, with reads succeeding and only the write refused (observed against a real server). The usual cause is an app password created without "Allow filesystem access" — which signing in again will not fix. Assisted-by: LLM |
||
|
|
57f8d42a5c |
Record directory ETags only after actually listing them
The scan recorded a directory's ETag when it was *discovered* as a child of another, not when its own contents were read. A scan that stopped early therefore stored ETags for subtrees it had never listed; the next scan probed those ETags, found them unchanged, and pruned folders whose contents had never been seen. Their images never entered the catalog at all, and no later scan would ever look again. Each stack entry now carries the validator its parent reported, recorded only once the directory has been listed. An interrupted scan re-reads that folder next time — slower, and correct. Two related fixes fall out. The scan root is now probed and recorded even though no parent described it, without which the one-request no-op sync that ETag pruning exists for could never fire at the root. And a pruned directory keeps its recorded entry, rather than dropping out and forcing a full walk of that subtree on the following scan. Assisted-by: LLM |
||
|
|
c8bb08e661 |
Add folder scan with format selection; validate A3 on a real library
Library setup as the user described it: pick a folder, choose which RAW
types to look for, scan recursively.
dr-types::FormatFilter the tick-box selection, seeing through VFS
placeholder suffixes so a dehydrated CR2 still
matches as a CR2
dr-sync::scan recursive walk, Depth:1 per directory, pruning
unchanged subtrees where the backend propagates
directory ETags
Verified against nextcloud.tourolle.paris (34.0.2) on a real library:
browse root 32 entries, 98ms
scan PhotosRaw 17,185 RAW files in 334 directories, 34.1s
(7,836 CR2 + 9,349 DNG)
range read 262KB of a 21.5MB DNG in 119ms — 1.22% of the file,
and enough to read "Canon EOS 6D | ISO 100"
That last line is assumption A3 validated on real data. Cataloguing this
library by whole-file fetch would move roughly 370GB; the range path
moves a few MB.
Pruning is capability-gated rather than assumed: with per-entry ETags a
probe costs a request and proves nothing about children, so it is skipped
entirely. A test asserts zero probes in that case.
Still unresolved: /core/preview returns 400 for every parameter
combination tried, including on a JPEG the server reports as having a
preview. Not a request-shape bug — it fails identically bare. Recorded
rather than worked around; ARCH §6.7 already treats server previews as
opportunistic, so nothing depends on it.
|
||
|
|
f8a718f42e |
Add Nextcloud connector; reject VFS as a transfer mechanism
Investigated using the Nextcloud desktop client's Virtual Files as a
cache instead of talking to the server directly. Measured on this
machine (client 4.0.7): the configured folder holds 121,785 placeholders
against 10,267 materialised files, including 7,037 CR2 and 9,411 DNG.
Three findings, each independently disqualifying:
- Linux VFS is *suffix* mode. A dehydrated IMG.CR2 exists only as
IMG.CR2.nextcloud holding one byte; the real name is absent.
- Reading a placeholder does not hydrate it. dd of the first 256KB
returned 1 byte, the stub was unchanged, and the real name never
appeared. There is no FUSE layer — the stub is an inert marker.
- Even with hydration the granularity is wrong: VFS has two states,
1 byte or all bytes, and the preview tier needs a ~256KB prefix of
a 27MB file. That is ~100x what FR-NC-3 requires.
Recorded as ARCH §9.0. Coexistence is still supported: dr-types now
recognises *.nextcloud stubs, and the viewer lists them as "not
downloaded" rather than as corrupt files or not at all.
So the connector talks to the server directly, as D7 specified.
Implemented: Login Flow v2, PROPFIND with oc:fileid and nc:has-preview,
ETag pruning via a Depth:0 probe, range GET with local slicing when the
server ignores the header, conditional PUT, and /core/preview with
forceIcon=false. delta() returns Unsupported and says why.
Chunked upload v2 is not implemented yet — put() rejects bodies over
5MB explicitly rather than silently truncating.
Two bugs found by testing: my hand-computed epoch in a date test was a
day out (the parser was right), and quick-xml reaches EOF on truncated
input without erroring, so unbalanced elements needed an explicit check
— a half-parsed multistatus must not look like an empty directory.
83 tests passing.
|
||
|
|
0f202fd3f9 |
Add requirements traceability gate and Gitea pipelines
Ports JellyTau's traceability tooling to Rust, carrying across the bug it was repaired for. That gate divided a traced count by frozen literal denominators; the requirements file outgrew them and it reported 158% coverage, so it could never fail its own threshold. Two rules, both enforced by the extractor's own tests: - denominators parsed from docs/requirements.md at run time - coverage is |traced ∩ defined| / |defined|, never a raw traced count The gate additionally fails hard on a misconfigured run — zero requirements parsed or zero files scanned — rather than reporting a plausible 0%, and on any orphan tag naming a requirement that does not exist. Adapted for DarkRoom: IDs are FR-CAT-1 / NFR-P13 / FR-DEV-3a shapes rather than JellyTau's fixed three digits, and decisions (D), spikes (S), milestone items (M) and test ids remain taggable while being excluded from the denominator — counting them inflated it by 25. Also adds dr-sync: the RemoteBackend trait and capability model, so the Nextcloud connector is one implementation rather than the only shape the engine understands. No mature Nextcloud crate exists (reqwest_dav is too thin), so the connector will be hand-rolled over reqwest per D7. Gitea workflows follow the same style: containerised, commented with the reasoning, desktop and Android on every push, plus a CI check that no core/ crate depends on the UI toolkit (ARCH §6.5a). Coverage today: 13.3% (19/143). 50 tests passing. |