DarkRoom/docs/storage.md

Storage backends

How DarkRoom talks to wherever a library lives, and what it takes to add somewhere new.

This document is the contract. docs/architecture.md §8 says why sync is built on capability negotiation rather than a common denominator; this says what the seam actually is, where each piece lives, and what a third connector has to do.

---

1. What "pluggable" has to mean

A trait alone does not make storage pluggable. RemoteBackend existed from the first release and every layer above it still knew it was talking to Nextcloud: seven files in dr-ui constructed a NextcloudBackend directly, ten functions took one by concrete type, the account model was a server URL beside a DAV user id, and the local cache directory was named after a hostname. The abstraction was real and bought nothing, because everything that reached a backend was still shaped like one product.

Pluggable means all four of these, not just the first:

1. Operations — what a backend can do. RemoteBackend.
2. Capabilities — what it can do cheaply, so the engine adapts instead of assuming. Capabilities.
3. Configuration — what an account is, with no server in it. Account.
4. Registration — how the application discovers a connector at all, without naming it. BackendProvider + BackendRegistry.

Two connectors ship. Nextcloud is unchanged and keeps every one of its peculiarities — those are the point of the capability model, not an embarrassment it has to hide. The folder connector serves a plain directory and exists partly because it is genuinely useful and partly because a second implementation is the only way to find out whether the first was an abstraction.

---

2. Where each piece lives

core/dr-sync/                 the contract, and nothing that speaks a protocol
  ├─ types.rs                 RemotePath, RemoteId, RemoteEntry, Validator, …
  ├─ capability.rs            Capabilities, ChangeDetection, ServerPreviews
  ├─ error.rs                 RemoteError — the one error every caller handles
  ├─ account.rs               Account, AccountStore, Secret, Connection
  ├─ provider.rs              BackendProvider, BackendRegistry, SignIn
  ├─ lib.rs                   RemoteBackend, SyncStrategy
  ├─ scan.rs                  the walk, driven by capabilities
  ├─ upload.rs                where an original is placed
  └─ reachability.rs          online/offline, inferred from observed results

core/dr-sync-nextcloud/       WebDAV, oc:fileid, chunked upload v2, Login Flow v2
core/dr-sync-folder/          a directory on a filesystem

ui/dr-ui/src/remote.rs        the registry — the ONLY file above dr-sync that
                              names a connector

dr-sync depends on no connector. That is deliberate and load-bearing: a build that only wants a folder library must not compile a TLS stack to get one, and the registry therefore lives in the crate that already depends on everything — the interface.

---

3. The four traits and types a connector meets

3.1 RemoteBackend — operations

#[async_trait]
pub trait RemoteBackend: Send + Sync {
    fn capabilities(&self) -> &Capabilities;
    fn name(&self) -> &str;

    // discovery
    async fn list(&self, dir: &RemotePath, since: Option<&Validator>)
        -> Result<Vec<RemoteEntry>, RemoteError>;
    async fn dir_validator(&self, dir: &RemotePath) -> Result<Validator, RemoteError>;
    async fn delta(&self, cursor: &Cursor)
        -> Result<(Vec<RemoteChange>, Cursor), RemoteError>;

    // transfer
    async fn get(&self, id: &RemoteId, range: Option<Range<u64>>)
        -> Result<Vec<u8>, RemoteError>;
    async fn put(&self, path: &RemotePath, body: Vec<u8>, precond: Option<Precondition>)
        -> Result<Validator, RemoteError>;
    async fn put_many(&self, items: Vec<(RemotePath, Vec<u8>)>)   // defaulted
        -> Result<Vec<Result<Validator, RemoteError>>, RemoteError>;
    async fn delete(&self, id: &RemoteId, precond: Option<Precondition>)
        -> Result<(), RemoteError>;
    async fn move_to(&self, from: &RemoteId, to: &RemotePath) -> Result<(), RemoteError>;
    async fn create_dir(&self, path: &RemotePath) -> Result<(), RemoteError>;

    // optional
    async fn thumbnail(&self, id: &RemoteId, size: u32)   // defaulted to None
        -> Result<Option<Vec<u8>>, RemoteError>;
}

Rules that are not obvious from the signatures:

• dir_validator and delta are capability-gated. Return RemoteError::Unsupported unless your ChangeDetection is PropagatingEtags or DeltaCursor respectively. Answering dir_validator with something that does not actually propagate is worse than refusing: it lets a caller prune a subtree whose contents changed, and hides those changes for as long as the folder list holds still.
• get takes an optional range, and it is a hint. A backend without cheap ranges may return the whole object; the caller slices. Correctness holds either way and Capabilities::range_reads says whether it was cheap.
• Chunked upload is not in the trait. It is an implementation detail of put, chosen by body size. Exposing it would leak one server's protocol.
• move_to must preserve identity where the backend has stable ids. This is what a soft delete uses (FR-CAT-15): a move implemented as copy + delete allocates a new id, orphaning the thumbnail shard and turning a restore into a full re-download.
• create_dir makes parents and succeeds if the directory exists. Callers use it to guarantee a destination, not to claim they created one.

3.2 Capabilities — what is cheap

The engine reads these once at connect time and picks a SyncStrategy. See ARCH §8.1–8.2 for the tiers. The two that change behaviour rather than speed:

Absent	Consequence the engine handles
range_reads	Embedded-preview extraction is impossible; browsing falls back to server previews or full download, and is refused on a metered connection
conditional_write	Sidecar conflict detection falls back to revision counters inside the sidecar — narrows the race, does not close it. Reported as a reduced-safety mode

Declare what is true, not what is flattering. A backend claiming PropagatingEtags it does not have does not merely run slowly; it silently hides changes.

3.3 Account — configuration with no server in it

pub struct Account {
    pub backend: String,     // BackendProvider::id; defaults to "nextcloud" on load
    pub endpoint: String,    // stored as "server" — a URL, a path, a bucket
    pub login: String,       // empty where the connector has no notion of a user
    pub user_id: String,     // connector-defined sub-address; Nextcloud's DAV segment
    pub root: String,        // the folder chosen as the library root
    pub formats: Vec<String>,
    pub last_scan: Option<i64>,
}

Everything but backend is the connector's to interpret. Code above dr-sync reads these for display and for cache keys, never for meaning.

Two properties are load-bearing:

• The on-disk form is backwards compatible. backend defaults to "nextcloud" and endpoint is stored under its historical key server, so every account written before there was a choice loads unchanged. A config the app refuses to parse is an account the user has to set up again.
• Account::namespace() is frozen for Nextcloud. It names the directory holding the catalog, the thumbnail shards, the sidecar spool and the export outbox. Changing it does not lose that data, it abandons it — silently, as an upgrade — and costs a full rescan on top. The Nextcloud form is reproduced byte for byte from what catalog_path computed before; every other backend is prefixed by its connector id, and long endpoints are truncated with a hash tail so two deep paths cannot collide inside one filesystem's 255-byte component limit.

3.4 Connection and Secret — the credential split

pub struct Connection { pub account: Account, pub secret: Option<Secret> }

Credentials go to platform secure storage (FR-NC-2, NFR-SEC-2). Never the catalog, never the config file, never a log line. AccountStore writes the account as plain JSON and the secret to the keyring, which is what lets the app show "signed in as duncan, watching /PhotosRaw" before it has touched the keyring at all.

Secret's inner string is reachable only through expose(), and its Debug prints Secret(***). That closes the indirect leak — a {:?} on any struct that happens to hold a connection — by construction rather than by review.

Connection is also what replaced a pair of arguments (credentials, user id) threaded together through fifteen signatures in an order that could be swapped.

3.5 BackendProvider — registration

pub trait BackendProvider: Send + Sync {
    fn id(&self) -> &'static str;               // written to Account::backend
    fn display_name(&self) -> &'static str;
    fn endpoint_label(&self) -> &'static str;   // "Server" / "Folder"
    fn endpoint_placeholder(&self) -> &'static str;
    fn sign_in(&self) -> SignIn;
    fn normalise_endpoint(&self, input: &str) -> Result<String, String>;
    fn account_for(&self, endpoint: &str) -> Result<Account, RemoteError>;  // defaulted
    fn connect(&self, conn: &Connection) -> Result<Box<dyn RemoteBackend>, RemoteError>;
}

pub enum SignIn {
    /// A handshake the user completes outside the app, yielding a credential.
    Browser,
    /// The endpoint is the whole account. No credential, no waiting state.
    EndpointOnly,
}

• id is on-disk configuration. Changing it after anyone has an account orphans that account. Pick it once.
• normalise_endpoint is where a bad endpoint is rejected, before an account is written for a library that does not exist. Its error string is shown to the user, so it says what to fix rather than naming a type. The Nextcloud provider upgrades http:// to https:// here (NFR-SEC-3); the folder provider canonicalises the path, so two spellings of one directory do not become two accounts indexing the same photographs.
• connect is synchronous and cheap. It validates configuration and builds a client; it does not talk to the remote. Workers call it per task.
• SignIn is a shape, not a method. It would be tidier to expose async fn sign_in(), and wrong: Login Flow v2 is a browser handshake the user completes elsewhere while the app polls, so it is not one call, it does not finish on our schedule, and the screen has to render a URL and a waiting state in the middle of it. SignIn tells the launch screen which of the two shapes to draw; the flow stays where its protocol is.

Credentials are deliberately not abstracted. An app password, an OAuth token and a bucket key pair have no useful common shape, and inventing one before a third backend exists would produce a wrong answer confidently. The general form is Connection — an account plus an opaque secret — and each connector translates that into what its protocol needs (NextcloudProvider::credentials).

---

4. Adding a backend

1. Implement RemoteBackend over your protocol, in a new core/dr-sync-<name> crate depending on dr-sync and nothing else of ours.
2. Declare Capabilities honestly. Start from Capabilities::minimal() and raise only what you can actually deliver.
3. Implement BackendProvider beside it.
4. Register it in ui/dr-ui/src/remote.rs::registry() and add the crate to ui/dr-ui/Cargo.toml.

That is the whole list. Nothing else in dr-ui changes, because nothing else in dr-ui names a connector.

Two things to get right, because they are silent when wrong:

• Identity. RemoteEntry::id should be RemoteId::Stable(u64) wherever you can produce a u64 that names the same photograph on every device looking at the same library. The catalog keys the thumbnail shards and the face index on it (catalog.md §10.1), and an entry without one gets neither. Set Capabilities::stable_ids only if that id also survives a rename — the two are different questions and only the second is a capability.
• Path safety. A RemotePath is built from names on the remote and from a catalog another device wrote. If you resolve one against a real filesystem, reject .. before you open anything.

Register a test double the same way — BackendRegistry::register replaces an existing id rather than shadowing it — so an integration test can stand a fake server behind "nextcloud" without the registry knowing it happened.

---

5. The connectors that ship

5.1 Nextcloud (dr-sync-nextcloud, id "nextcloud")

Unchanged by the abstraction, peculiarities intact — see ARCH §8.4 for the full mapping. What matters here is that none of them had to be given up to make room for a second backend:

change_detection	PropagatingEtags — the one-request no-op sync
stable_ids	yes, oc:fileid, survives server-side rename and move
range_reads	yes, detected by 206 vs 200, never HEAD
chunked_upload	v2, 5 MB – 5 GB, MKCOL → PUT chunks → MOVE .file
bulk_upload	yes, POST /remote.php/dav/bulk
conditional_write	yes, If-Match
server_previews	CommonFormatsOnly — stock Nextcloud renders no RAW
sign-in	SignIn::Browser, Login Flow v2, system browser, app password

Also kept: the oc:permissions probe on a refused PUT, which is what distinguishes a create-only share from a bad credential; the 423 Locked retry classification; and the bundled ISRG Root YE certificate.

5.2 Folder (dr-sync-folder, id "folder")

A local disk, an NFS or SMB mount, an external drive, or the directory a Nextcloud desktop client already syncs. No server, no account, no credential — which makes it the route that works on a machine with no secrets daemon at all.

change_detection	LocalEtags — see below
stable_ids	no — the id is a path hash and does not survive a rename
range_reads	yes, seek + take
chunked_upload	none; a write is a write
bulk_upload	no
conditional_write	yes, with a documented residual race
server_previews	None
sign-in	SignIn::EndpointOnly

Why LocalEtags and not PropagatingEtags. A POSIX directory's mtime changes when its own entry list changes and at no other time — not when a child's contents are edited, and not for a grandchild. There is nothing to propagate, so dir_validator returns Unsupported and the engine walks the tree every scan. Which costs almost nothing, because the walk that was expensive was expensive for a reason this backend does not have.

Measured 2026-08-28, cargo run -p dr-sync-folder --example scan: a full uncached walk of 2,299 images across 233 directories completed in 137 ms, and 380 images across 13 directories in 29 ms — the same engine, the same Depth: 1-per-directory walk, with no pruning at all. The Nextcloud connector's comparable figure is 34.1 s for 17,185 RAWs across 334 directories with pruning available (ARCH §8.4). The capability model is what lets one engine drive both at the speed each actually runs at, instead of forcing the fast one down to the slow one's interface.

Identity is a hash of the path relative to the library root, FNV-1a 64 (written out, because DefaultHasher is explicitly unstable between Rust releases and this value is written into the catalog). It gives the catalog a u64 that names a photograph, is the same on every device looking at the same folder, and does not change when the file is edited. It does not survive a rename, and stable_ids: false says so: a moved photograph is seen as a delete and an add, and its thumbnail is derived again.

The alternative — keying on the inode — is stable across a rename but differs between devices and is reused by the filesystem after a delete. Two machines would disagree about which photograph a thumbnail belonged to, and a recycled inode would silently attach an old thumbnail to a new image. Re-deriving a thumbnail is a cost; showing the wrong one is a bug.

Conditional writes. IfAbsent is genuinely atomic (O_CREAT | O_EXCL). IfMatch is compare-then-swap: a stat, then a write to a temporary beside the destination and a rename over it. A POSIX filesystem has no compare-and-swap, so the race is narrowed to the microseconds between the two syscalls rather than closed — still far tighter than the fallback the engine uses for a backend that declares no conditional write at all, which spans a whole read-modify-write. The capability is declared, and the residual race is documented at the call site.

Two deliberate divergences from WebDAV semantics:

• delete is not recursive. A folder library is the user's own photographs on their own disk with no server-side trash behind it, so a caller that passed the wrong path would have no way back. Deleting a non-empty directory returns RemoteError::Configuration. Nothing in the engine deletes a directory — the soft delete is a move_to into the trash folder — so the guard is free.
• Every filesystem call runs on the blocking pool. On a local disk that is overkill; on the NFS mount this backend is most useful over, a stalled server would otherwise wedge the async worker that made the call and every other request sharing it.

Failure classification matters as much as the operations. A vanished mount (ESTALE, ENOTCONN, EIO) maps to RemoteError::Network, which is what puts the app into offline mode and leaves the catalog readable — exactly as a dead server does. A permissions problem maps to PermissionDenied and does not, because going offline over one forbidden file would hide a fixable problem behind a network banner. An endpoint that is not a directory at all maps to RemoteError::Configuration: nothing was unreachable and no credential was wrong, so neither of the other two would send the user anywhere useful.

---

6. What the abstraction does not yet cover

Stated so the next person does not have to rediscover it.

• Multiple accounts at once. AccountStore holds a list and the launch screen uses the most recent. Nothing in the model prevents two open libraries; the interface has no place to show them.
• Per-backend settings. A connector has no way to contribute a settings page. Anything configurable is on the Account or is not configurable.
• Capability probing at runtime. Capabilities is fixed at construction. Nextcloud's server_previews should really be probed per account — a server with camerarawpreviews installed can render RAW — and today it is assumed to be CommonFormatsOnly.
• A general notion of an account. Credentials stay connector-specific on purpose (§3.5). A third connector with an OAuth flow will need a third SignIn variant, and that is the right place for it to appear.