# Storage backends How DarkRoom talks to wherever a library lives, and what it takes to add somewhere new. This document is the contract. `docs/architecture.md` §8 says why sync is built on capability negotiation rather than a common denominator; this says what the seam actually is, where each piece lives, and what a third connector has to do. --- ## 1. What "pluggable" has to mean A trait alone does not make storage pluggable. `RemoteBackend` existed from the first release and every layer above it still knew it was talking to Nextcloud: seven files in `dr-ui` constructed a `NextcloudBackend` directly, ten functions took one by concrete type, the account model was a server URL beside a DAV user id, and the local cache directory was named after a hostname. The abstraction was real and bought nothing, because everything that *reached* a backend was still shaped like one product. Pluggable means all four of these, not just the first: 1. **Operations** — what a backend can do. `RemoteBackend`. 2. **Capabilities** — what it can do *cheaply*, so the engine adapts instead of assuming. `Capabilities`. 3. **Configuration** — what an account is, with no server in it. `Account`. 4. **Registration** — how the application discovers a connector at all, without naming it. `BackendProvider` + `BackendRegistry`. Two connectors ship. Nextcloud is unchanged and keeps every one of its peculiarities — those are the point of the capability model, not an embarrassment it has to hide. The folder connector serves a plain directory and exists partly because it is genuinely useful and partly because a second implementation is the only way to find out whether the first was an abstraction. --- ## 2. Where each piece lives ``` core/dr-sync/ the contract, and nothing that speaks a protocol ├─ types.rs RemotePath, RemoteId, RemoteEntry, Validator, … ├─ capability.rs Capabilities, ChangeDetection, ServerPreviews ├─ error.rs RemoteError — the one error every caller handles ├─ account.rs Account, AccountStore, Secret, Connection ├─ provider.rs BackendProvider, BackendRegistry, SignIn ├─ lib.rs RemoteBackend, SyncStrategy ├─ scan.rs the walk, driven by capabilities ├─ upload.rs where an original is placed └─ reachability.rs online/offline, inferred from observed results core/dr-sync-nextcloud/ WebDAV, oc:fileid, chunked upload v2, Login Flow v2 core/dr-sync-folder/ a directory on a filesystem ui/dr-ui/src/remote.rs the registry — the ONLY file above dr-sync that names a connector ``` `dr-sync` depends on no connector. That is deliberate and load-bearing: a build that only wants a folder library must not compile a TLS stack to get one, and the registry therefore lives in the crate that already depends on everything — the interface. --- ## 3. The four traits and types a connector meets ### 3.1 `RemoteBackend` — operations ```rust #[async_trait] pub trait RemoteBackend: Send + Sync { fn capabilities(&self) -> &Capabilities; fn name(&self) -> &str; // discovery async fn list(&self, dir: &RemotePath, since: Option<&Validator>) -> Result, RemoteError>; async fn dir_validator(&self, dir: &RemotePath) -> Result; async fn delta(&self, cursor: &Cursor) -> Result<(Vec, Cursor), RemoteError>; // transfer async fn get(&self, id: &RemoteId, range: Option>) -> Result, RemoteError>; async fn put(&self, path: &RemotePath, body: Vec, precond: Option) -> Result; async fn put_many(&self, items: Vec<(RemotePath, Vec)>) // defaulted -> Result>, RemoteError>; async fn delete(&self, id: &RemoteId, precond: Option) -> Result<(), RemoteError>; async fn move_to(&self, from: &RemoteId, to: &RemotePath) -> Result<(), RemoteError>; async fn create_dir(&self, path: &RemotePath) -> Result<(), RemoteError>; // optional async fn thumbnail(&self, id: &RemoteId, size: u32) // defaulted to None -> Result>, RemoteError>; } ``` Rules that are not obvious from the signatures: - **`dir_validator` and `delta` are capability-gated.** Return `RemoteError::Unsupported` unless your `ChangeDetection` is `PropagatingEtags` or `DeltaCursor` respectively. Answering `dir_validator` with something that does not actually propagate is worse than refusing: it lets a caller prune a subtree whose contents changed, and hides those changes for as long as the folder list holds still. - **`get` takes an optional range, and it is a hint.** A backend without cheap ranges may return the whole object; the caller slices. Correctness holds either way and `Capabilities::range_reads` says whether it was cheap. - **Chunked upload is not in the trait.** It is an implementation detail of `put`, chosen by body size. Exposing it would leak one server's protocol. - **`move_to` must preserve identity where the backend has stable ids.** This is what a soft delete uses (`FR-CAT-15`): a move implemented as copy + delete allocates a new id, orphaning the thumbnail shard and turning a restore into a full re-download. - **`create_dir` makes parents and succeeds if the directory exists.** Callers use it to guarantee a destination, not to claim they created one. ### 3.2 `Capabilities` — what is cheap The engine reads these once at connect time and picks a `SyncStrategy`. See ARCH §8.1–8.2 for the tiers. The two that change behaviour rather than speed: | Absent | Consequence the engine handles | |---|---| | `range_reads` | Embedded-preview extraction is impossible; browsing falls back to server previews or full download, and is refused on a metered connection | | `conditional_write` | Sidecar conflict detection falls back to revision counters inside the sidecar — narrows the race, does not close it. Reported as a reduced-safety mode | **Declare what is true, not what is flattering.** A backend claiming `PropagatingEtags` it does not have does not merely run slowly; it silently hides changes. ### 3.3 `Account` — configuration with no server in it ```rust pub struct Account { pub backend: String, // BackendProvider::id; defaults to "nextcloud" on load pub endpoint: String, // stored as "server" — a URL, a path, a bucket pub login: String, // empty where the connector has no notion of a user pub user_id: String, // connector-defined sub-address; Nextcloud's DAV segment pub root: String, // the folder chosen as the library root pub formats: Vec, pub last_scan: Option, } ``` Everything but `backend` is the connector's to interpret. Code above `dr-sync` reads these for display and for cache keys, never for meaning. Two properties are load-bearing: - **The on-disk form is backwards compatible.** `backend` defaults to `"nextcloud"` and `endpoint` is stored under its historical key `server`, so every account written before there was a choice loads unchanged. A config the app refuses to parse is an account the user has to set up again. - **`Account::namespace()` is frozen for Nextcloud.** It names the directory holding the catalog, the thumbnail shards, the sidecar spool and the export outbox. Changing it does not lose that data, it *abandons* it — silently, as an upgrade — and costs a full rescan on top. The Nextcloud form is reproduced byte for byte from what `catalog_path` computed before; every other backend is prefixed by its connector id, and long endpoints are truncated with a hash tail so two deep paths cannot collide inside one filesystem's 255-byte component limit. ### 3.4 `Connection` and `Secret` — the credential split ```rust pub struct Connection { pub account: Account, pub secret: Option } ``` Credentials go to platform secure storage (`FR-NC-2`, `NFR-SEC-2`). Never the catalog, never the config file, never a log line. `AccountStore` writes the account as plain JSON and the secret to the keyring, which is what lets the app show "signed in as duncan, watching /PhotosRaw" before it has touched the keyring at all. `Secret`'s inner string is reachable only through `expose()`, and its `Debug` prints `Secret(***)`. That closes the indirect leak — a `{:?}` on any struct that happens to hold a connection — by construction rather than by review. `Connection` is also what replaced a pair of arguments (credentials, user id) threaded together through fifteen signatures in an order that could be swapped. ### 3.5 `BackendProvider` — registration ```rust pub trait BackendProvider: Send + Sync { fn id(&self) -> &'static str; // written to Account::backend fn display_name(&self) -> &'static str; fn endpoint_label(&self) -> &'static str; // "Server" / "Folder" fn endpoint_placeholder(&self) -> &'static str; fn sign_in(&self) -> SignIn; fn normalise_endpoint(&self, input: &str) -> Result; fn account_for(&self, endpoint: &str) -> Result; // defaulted fn connect(&self, conn: &Connection) -> Result, RemoteError>; } pub enum SignIn { /// A handshake the user completes outside the app, yielding a credential. Browser, /// The endpoint is the whole account. No credential, no waiting state. EndpointOnly, } ``` - **`id` is on-disk configuration.** Changing it after anyone has an account orphans that account. Pick it once. - **`normalise_endpoint` is where a bad endpoint is *rejected*,** before an account is written for a library that does not exist. Its error string is shown to the user, so it says what to fix rather than naming a type. The Nextcloud provider upgrades `http://` to `https://` here (`NFR-SEC-3`); the folder provider canonicalises the path, so two spellings of one directory do not become two accounts indexing the same photographs. - **`connect` is synchronous and cheap.** It validates configuration and builds a client; it does not talk to the remote. Workers call it per task. - **`SignIn` is a shape, not a method.** It would be tidier to expose `async fn sign_in()`, and wrong: Login Flow v2 is a browser handshake the user completes elsewhere while the app polls, so it is not one call, it does not finish on our schedule, and the screen has to render a URL and a waiting state in the middle of it. `SignIn` tells the launch screen which of the two shapes to draw; the flow stays where its protocol is. **Credentials are deliberately not abstracted.** An app password, an OAuth token and a bucket key pair have no useful common shape, and inventing one before a third backend exists would produce a wrong answer confidently. The general form is `Connection` — an account plus an opaque secret — and each connector translates that into what its protocol needs (`NextcloudProvider::credentials`). --- ## 4. Adding a backend 1. **Implement `RemoteBackend`** over your protocol, in a new `core/dr-sync-` crate depending on `dr-sync` and nothing else of ours. 2. **Declare `Capabilities` honestly.** Start from `Capabilities::minimal()` and raise only what you can actually deliver. 3. **Implement `BackendProvider`** beside it. 4. **Register it** in `ui/dr-ui/src/remote.rs::registry()` and add the crate to `ui/dr-ui/Cargo.toml`. That is the whole list. Nothing else in `dr-ui` changes, because nothing else in `dr-ui` names a connector. **Two things to get right, because they are silent when wrong:** - **Identity.** `RemoteEntry::id` should be `RemoteId::Stable(u64)` wherever you can produce a `u64` that names the same photograph on every device looking at the same library. The catalog keys the thumbnail shards and the face index on it (`catalog.md` §10.1), and an entry without one gets neither. Set `Capabilities::stable_ids` only if that id also survives a rename — the two are different questions and only the second is a capability. - **Path safety.** A `RemotePath` is built from names on the remote and from a catalog another device wrote. If you resolve one against a real filesystem, reject `..` before you open anything. Register a test double the same way — `BackendRegistry::register` replaces an existing id rather than shadowing it — so an integration test can stand a fake server behind `"nextcloud"` without the registry knowing it happened. --- ## 5. The connectors that ship ### 5.1 Nextcloud (`dr-sync-nextcloud`, id `"nextcloud"`) Unchanged by the abstraction, peculiarities intact — see ARCH §8.4 for the full mapping. What matters here is that none of them had to be given up to make room for a second backend: | | | |---|---| | `change_detection` | `PropagatingEtags` — the one-request no-op sync | | `stable_ids` | yes, `oc:fileid`, survives server-side rename and move | | `range_reads` | yes, detected by `206` vs `200`, never `HEAD` | | `chunked_upload` | v2, 5 MB – 5 GB, `MKCOL` → `PUT` chunks → `MOVE .file` | | `bulk_upload` | yes, `POST /remote.php/dav/bulk` | | `conditional_write` | yes, `If-Match` | | `server_previews` | `CommonFormatsOnly` — stock Nextcloud renders no RAW | | sign-in | `SignIn::Browser`, Login Flow v2, system browser, app password | Also kept: the `oc:permissions` probe on a refused `PUT`, which is what distinguishes a create-only share from a bad credential; the `423 Locked` retry classification; and the bundled ISRG Root YE certificate. ### 5.2 Folder (`dr-sync-folder`, id `"folder"`) A local disk, an NFS or SMB mount, an external drive, or the directory a Nextcloud desktop client already syncs. No server, no account, no credential — which makes it the route that works on a machine with no secrets daemon at all. | | | |---|---| | `change_detection` | `LocalEtags` — see below | | `stable_ids` | **no** — the id is a path hash and does not survive a rename | | `range_reads` | yes, `seek` + `take` | | `chunked_upload` | none; a write is a write | | `bulk_upload` | no | | `conditional_write` | yes, with a documented residual race | | `server_previews` | `None` | | sign-in | `SignIn::EndpointOnly` | **Why `LocalEtags` and not `PropagatingEtags`.** A POSIX directory's mtime changes when its own entry list changes and at no other time — not when a child's contents are edited, and not for a grandchild. There is nothing to propagate, so `dir_validator` returns `Unsupported` and the engine walks the tree every scan. Which costs almost nothing, because the walk that was expensive was expensive for a reason this backend does not have: fifty thousand `stat` calls against a filesystem take well under a second, and fifty thousand `PROPFIND`s do not. The capability model is what lets both be driven by the same engine at the speed each actually runs at. **Identity is a hash of the path relative to the library root**, FNV-1a 64 (written out, because `DefaultHasher` is explicitly unstable between Rust releases and this value is written into the catalog). It gives the catalog a `u64` that names a photograph, is the same on every device looking at the same folder, and does not change when the file is edited. It does not survive a rename, and `stable_ids: false` says so: a moved photograph is seen as a delete and an add, and its thumbnail is derived again. The alternative — keying on the inode — is stable across a rename but *differs between devices* and is reused by the filesystem after a delete. Two machines would disagree about which photograph a thumbnail belonged to, and a recycled inode would silently attach an old thumbnail to a new image. Re-deriving a thumbnail is a cost; showing the wrong one is a bug. **Conditional writes.** `IfAbsent` is genuinely atomic (`O_CREAT | O_EXCL`). `IfMatch` is compare-then-swap: a `stat`, then a write to a temporary beside the destination and a `rename` over it. A POSIX filesystem has no compare-and-swap, so the race is narrowed to the microseconds between the two syscalls rather than closed — still far tighter than the fallback the engine uses for a backend that declares no conditional write at all, which spans a whole read-modify-write. The capability is declared, and the residual race is documented at the call site. **Two deliberate divergences from WebDAV semantics:** - **`delete` is not recursive.** A folder library is the user's own photographs on their own disk with no server-side trash behind it, so a caller that passed the wrong path would have no way back. Deleting a non-empty directory returns `RemoteError::Configuration`. Nothing in the engine deletes a directory — the soft delete is a `move_to` into the trash folder — so the guard is free. - **Every filesystem call runs on the blocking pool.** On a local disk that is overkill; on the NFS mount this backend is most useful over, a stalled server would otherwise wedge the async worker that made the call and every other request sharing it. **Failure classification** matters as much as the operations. A vanished mount (`ESTALE`, `ENOTCONN`, `EIO`) maps to `RemoteError::Network`, which is what puts the app into offline mode and leaves the catalog readable — exactly as a dead server does. A permissions problem maps to `PermissionDenied` and does *not*, because going offline over one forbidden file would hide a fixable problem behind a network banner. An endpoint that is not a directory at all maps to `RemoteError::Configuration`: nothing was unreachable and no credential was wrong, so neither of the other two would send the user anywhere useful. --- ## 6. What the abstraction does not yet cover Stated so the next person does not have to rediscover it. - **Multiple accounts at once.** `AccountStore` holds a list and the launch screen uses the most recent. Nothing in the model prevents two open libraries; the interface has no place to show them. - **Per-backend settings.** A connector has no way to contribute a settings page. Anything configurable is on the `Account` or is not configurable. - **Capability probing at runtime.** `Capabilities` is fixed at construction. Nextcloud's `server_previews` should really be probed per account — a server with `camerarawpreviews` installed can render RAW — and today it is assumed to be `CommonFormatsOnly`. - **A general notion of an account.** Credentials stay connector-specific on purpose (§3.5). A third connector with an OAuth flow will need a third `SignIn` variant, and that is the right place for it to appear.