DarkRoom/docs/storage.md

# Storage backends

How DarkRoom talks to wherever a library lives, and what it takes to add
somewhere new.

This document is the contract. `docs/architecture.md` §8 says why sync is built
on capability negotiation rather than a common denominator; this says what the
seam actually is, where each piece lives, and what a third connector has to do.

---

## 1. What "pluggable" has to mean

A trait alone does not make storage pluggable. `RemoteBackend` existed from the
first release and every layer above it still knew it was talking to Nextcloud:
seven files in `dr-ui` constructed a `NextcloudBackend` directly, ten functions
took one by concrete type, the account model was a server URL beside a DAV user
id, and the local cache directory was named after a hostname. The abstraction
was real and bought nothing, because everything that *reached* a backend was
still shaped like one product.

Pluggable means all four of these, not just the first:

1. **Operations** — what a backend can do. `RemoteBackend`.
2. **Capabilities** — what it can do *cheaply*, so the engine adapts instead of
   assuming. `Capabilities`.
3. **Configuration** — what an account is, with no server in it. `Account`.
4. **Registration** — how the application discovers a connector at all, without
   naming it. `BackendProvider` + `BackendRegistry`.

Two connectors ship. Nextcloud is unchanged and keeps every one of its
peculiarities — those are the point of the capability model, not an
embarrassment it has to hide. The folder connector serves a plain directory and
exists partly because it is genuinely useful and partly because a second
implementation is the only way to find out whether the first was an
abstraction.

---

## 2. Where each piece lives

```
core/dr-sync/                 the contract, and nothing that speaks a protocol
  ├─ types.rs                 RemotePath, RemoteId, RemoteEntry, Validator, …
  ├─ capability.rs            Capabilities, ChangeDetection, ServerPreviews
  ├─ error.rs                 RemoteError — the one error every caller handles
  ├─ account.rs               Account, AccountStore, Secret, Connection
  ├─ provider.rs              BackendProvider, BackendRegistry, SignIn
  ├─ lib.rs                   RemoteBackend, SyncStrategy
  ├─ scan.rs                  the walk, driven by capabilities
  ├─ upload.rs                where an original is placed
  └─ reachability.rs          online/offline, inferred from observed results

core/dr-sync-nextcloud/       WebDAV, oc:fileid, chunked upload v2, Login Flow v2
core/dr-sync-folder/          a directory on a filesystem

ui/dr-ui/src/remote.rs        the registry — the ONLY file above dr-sync that
                              names a connector
```

`dr-sync` depends on no connector. That is deliberate and load-bearing: a build
that only wants a folder library must not compile a TLS stack to get one, and
the registry therefore lives in the crate that already depends on everything —
the interface.

---

## 3. The four traits and types a connector meets

### 3.1 `RemoteBackend` — operations

```rust
#[async_trait]
pub trait RemoteBackend: Send + Sync {
    fn capabilities(&self) -> &Capabilities;
    fn name(&self) -> &str;

    // discovery
    async fn list(&self, dir: &RemotePath, since: Option<&Validator>)
        -> Result<Vec<RemoteEntry>, RemoteError>;
    async fn dir_validator(&self, dir: &RemotePath) -> Result<Validator, RemoteError>;
    async fn delta(&self, cursor: &Cursor)
        -> Result<(Vec<RemoteChange>, Cursor), RemoteError>;

    // transfer
    async fn get(&self, id: &RemoteId, range: Option<Range<u64>>)
        -> Result<Vec<u8>, RemoteError>;
    async fn put(&self, path: &RemotePath, body: Vec<u8>, precond: Option<Precondition>)
        -> Result<Validator, RemoteError>;
    async fn put_many(&self, items: Vec<(RemotePath, Vec<u8>)>)   // defaulted
        -> Result<Vec<Result<Validator, RemoteError>>, RemoteError>;
    async fn delete(&self, id: &RemoteId, precond: Option<Precondition>)
        -> Result<(), RemoteError>;
    async fn move_to(&self, from: &RemoteId, to: &RemotePath) -> Result<(), RemoteError>;
    async fn create_dir(&self, path: &RemotePath) -> Result<(), RemoteError>;

    // optional
    async fn thumbnail(&self, id: &RemoteId, size: u32)   // defaulted to None
        -> Result<Option<Vec<u8>>, RemoteError>;
}
```

Rules that are not obvious from the signatures:

- **`dir_validator` and `delta` are capability-gated.** Return
  `RemoteError::Unsupported` unless your `ChangeDetection` is
  `PropagatingEtags` or `DeltaCursor` respectively. Answering
  `dir_validator` with something that does not actually propagate is worse than
  refusing: it lets a caller prune a subtree whose contents changed, and hides
  those changes for as long as the folder list holds still.
- **`get` takes an optional range, and it is a hint.** A backend without cheap
  ranges may return the whole object; the caller slices. Correctness holds
  either way and `Capabilities::range_reads` says whether it was cheap.
- **Chunked upload is not in the trait.** It is an implementation detail of
  `put`, chosen by body size. Exposing it would leak one server's protocol.
- **`move_to` must preserve identity where the backend has stable ids.** This
  is what a soft delete uses (`FR-CAT-15`): a move implemented as copy + delete
  allocates a new id, orphaning the thumbnail shard and turning a restore into a
  full re-download.
- **`create_dir` makes parents and succeeds if the directory exists.** Callers
  use it to guarantee a destination, not to claim they created one.

### 3.2 `Capabilities` — what is cheap

The engine reads these once at connect time and picks a `SyncStrategy`. See
ARCH §8.1–8.2 for the tiers. The two that change behaviour rather than speed:

| Absent | Consequence the engine handles |
|---|---|
| `range_reads` | Embedded-preview extraction is impossible; browsing falls back to server previews or full download, and is refused on a metered connection |
| `conditional_write` | Sidecar conflict detection falls back to revision counters inside the sidecar — narrows the race, does not close it. Reported as a reduced-safety mode |

**Declare what is true, not what is flattering.** A backend claiming
`PropagatingEtags` it does not have does not merely run slowly; it silently
hides changes.

### 3.3 `Account` — configuration with no server in it

```rust
pub struct Account {
    pub backend: String,     // BackendProvider::id; defaults to "nextcloud" on load
    pub endpoint: String,    // stored as "server" — a URL, a path, a bucket
    pub login: String,       // empty where the connector has no notion of a user
    pub user_id: String,     // connector-defined sub-address; Nextcloud's DAV segment
    pub root: String,        // the folder chosen as the library root
    pub formats: Vec<String>,
    pub last_scan: Option<i64>,
}
```

Everything but `backend` is the connector's to interpret. Code above `dr-sync`
reads these for display and for cache keys, never for meaning.

Two properties are load-bearing:

- **The on-disk form is backwards compatible.** `backend` defaults to
  `"nextcloud"` and `endpoint` is stored under its historical key `server`, so
  every account written before there was a choice loads unchanged. A config the
  app refuses to parse is an account the user has to set up again.
- **`Account::namespace()` is frozen for Nextcloud.** It names the directory
  holding the catalog, the thumbnail shards, the sidecar spool and the export
  outbox. Changing it does not lose that data, it *abandons* it — silently, as
  an upgrade — and costs a full rescan on top. The Nextcloud form is reproduced
  byte for byte from what `catalog_path` computed before; every other backend is
  prefixed by its connector id, and long endpoints are truncated with a hash
  tail so two deep paths cannot collide inside one filesystem's 255-byte
  component limit.

### 3.4 `Connection` and `Secret` — the credential split

```rust
pub struct Connection { pub account: Account, pub secret: Option<Secret> }
```

Credentials go to platform secure storage (`FR-NC-2`, `NFR-SEC-2`). Never the
catalog, never the config file, never a log line. `AccountStore` writes the
account as plain JSON and the secret to the keyring, which is what lets the app
show "signed in as duncan, watching /PhotosRaw" before it has touched the
keyring at all.

`Secret`'s inner string is reachable only through `expose()`, and its `Debug`
prints `Secret(***)`. That closes the indirect leak — a `{:?}` on any struct
that happens to hold a connection — by construction rather than by review.

`Connection` is also what replaced a pair of arguments (credentials, user id)
threaded together through fifteen signatures in an order that could be swapped.

### 3.5 `BackendProvider` — registration

```rust
pub trait BackendProvider: Send + Sync {
    fn id(&self) -> &'static str;               // written to Account::backend
    fn display_name(&self) -> &'static str;
    fn endpoint_label(&self) -> &'static str;   // "Server" / "Folder"
    fn endpoint_placeholder(&self) -> &'static str;
    fn sign_in(&self) -> SignIn;
    fn normalise_endpoint(&self, input: &str) -> Result<String, String>;
    fn account_for(&self, endpoint: &str) -> Result<Account, RemoteError>;  // defaulted
    fn connect(&self, conn: &Connection) -> Result<Box<dyn RemoteBackend>, RemoteError>;
}

pub enum SignIn {
    /// A handshake the user completes outside the app, yielding a credential.
    Browser,
    /// The endpoint is the whole account. No credential, no waiting state.
    EndpointOnly,
}
```

- **`id` is on-disk configuration.** Changing it after anyone has an account
  orphans that account. Pick it once.
- **`normalise_endpoint` is where a bad endpoint is *rejected*,** before an
  account is written for a library that does not exist. Its error string is
  shown to the user, so it says what to fix rather than naming a type. The
  Nextcloud provider upgrades `http://` to `https://` here (`NFR-SEC-3`); the
  folder provider canonicalises the path, so two spellings of one directory do
  not become two accounts indexing the same photographs.
- **`connect` is synchronous and cheap.** It validates configuration and builds
  a client; it does not talk to the remote. Workers call it per task.
- **`SignIn` is a shape, not a method.** It would be tidier to expose
  `async fn sign_in()`, and wrong: Login Flow v2 is a browser handshake the user
  completes elsewhere while the app polls, so it is not one call, it does not
  finish on our schedule, and the screen has to render a URL and a waiting state
  in the middle of it. `SignIn` tells the launch screen which of the two shapes
  to draw; the flow stays where its protocol is.

**Credentials are deliberately not abstracted.** An app password, an OAuth
token and a bucket key pair have no useful common shape, and inventing one
before a third backend exists would produce a wrong answer confidently. The
general form is `Connection` — an account plus an opaque secret — and each
connector translates that into what its protocol needs
(`NextcloudProvider::credentials`).

---

## 4. Adding a backend

1. **Implement `RemoteBackend`** over your protocol, in a new
   `core/dr-sync-<name>` crate depending on `dr-sync` and nothing else of ours.
2. **Declare `Capabilities` honestly.** Start from `Capabilities::minimal()` and
   raise only what you can actually deliver.
3. **Implement `BackendProvider`** beside it.
4. **Register it** in `ui/dr-ui/src/remote.rs::registry()` and add the crate to
   `ui/dr-ui/Cargo.toml`.

That is the whole list. Nothing else in `dr-ui` changes, because nothing else in
`dr-ui` names a connector.

**Two things to get right, because they are silent when wrong:**

- **Identity.** `RemoteEntry::id` should be `RemoteId::Stable(u64)` wherever you
  can produce a `u64` that names the same photograph on every device looking at
  the same library. The catalog keys the thumbnail shards and the face index on
  it (`catalog.md` §10.1), and an entry without one gets neither. Set
  `Capabilities::stable_ids` only if that id also survives a rename — the two
  are different questions and only the second is a capability.
- **Path safety.** A `RemotePath` is built from names on the remote and from a
  catalog another device wrote. If you resolve one against a real filesystem,
  reject `..` before you open anything.

Register a test double the same way — `BackendRegistry::register` replaces an
existing id rather than shadowing it — so an integration test can stand a fake
server behind `"nextcloud"` without the registry knowing it happened.

---

## 5. The connectors that ship

### 5.1 Nextcloud (`dr-sync-nextcloud`, id `"nextcloud"`)

Unchanged by the abstraction, peculiarities intact — see ARCH §8.4 for the full
mapping. What matters here is that none of them had to be given up to make room
for a second backend:

| | |
|---|---|
| `change_detection` | `PropagatingEtags` — the one-request no-op sync |
| `stable_ids` | yes, `oc:fileid`, survives server-side rename and move |
| `range_reads` | yes, detected by `206` vs `200`, never `HEAD` |
| `chunked_upload` | v2, 5 MB – 5 GB, `MKCOL` → `PUT` chunks → `MOVE .file` |
| `bulk_upload` | yes, `POST /remote.php/dav/bulk` |
| `conditional_write` | yes, `If-Match` |
| `server_previews` | `CommonFormatsOnly` — stock Nextcloud renders no RAW |
| sign-in | `SignIn::Browser`, Login Flow v2, system browser, app password |

Also kept: the `oc:permissions` probe on a refused `PUT`, which is what
distinguishes a create-only share from a bad credential; the `423 Locked`
retry classification; and the bundled ISRG Root YE certificate.

### 5.2 Folder (`dr-sync-folder`, id `"folder"`)

A local disk, an NFS or SMB mount, an external drive, or the directory a
Nextcloud desktop client already syncs. No server, no account, no credential —
which makes it the route that works on a machine with no secrets daemon at all.

| | |
|---|---|
| `change_detection` | `LocalEtags` — see below |
| `stable_ids` | **no** — the id is a path hash and does not survive a rename |
| `range_reads` | yes, `seek` + `take` |
| `chunked_upload` | none; a write is a write |
| `bulk_upload` | no |
| `conditional_write` | yes, with a documented residual race |
| `server_previews` | `None` |
| sign-in | `SignIn::EndpointOnly` |

**Why `LocalEtags` and not `PropagatingEtags`.** A POSIX directory's mtime
changes when its own entry list changes and at no other time — not when a
child's contents are edited, and not for a grandchild. There is nothing to
propagate, so `dir_validator` returns `Unsupported` and the engine walks the
tree every scan. Which costs almost nothing, because the walk that was expensive
was expensive for a reason this backend does not have: fifty thousand `stat`
calls against a filesystem take well under a second, and fifty thousand
`PROPFIND`s do not. The capability model is what lets both be driven by the same
engine at the speed each actually runs at.

**Identity is a hash of the path relative to the library root**, FNV-1a 64
(written out, because `DefaultHasher` is explicitly unstable between Rust
releases and this value is written into the catalog). It gives the catalog a
`u64` that names a photograph, is the same on every device looking at the same
folder, and does not change when the file is edited. It does not survive a
rename, and `stable_ids: false` says so: a moved photograph is seen as a delete
and an add, and its thumbnail is derived again.

The alternative — keying on the inode — is stable across a rename but *differs
between devices* and is reused by the filesystem after a delete. Two machines
would disagree about which photograph a thumbnail belonged to, and a recycled
inode would silently attach an old thumbnail to a new image. Re-deriving a
thumbnail is a cost; showing the wrong one is a bug.

**Conditional writes.** `IfAbsent` is genuinely atomic (`O_CREAT | O_EXCL`).
`IfMatch` is compare-then-swap: a `stat`, then a write to a temporary beside the
destination and a `rename` over it. A POSIX filesystem has no compare-and-swap,
so the race is narrowed to the microseconds between the two syscalls rather than
closed — still far tighter than the fallback the engine uses for a backend that
declares no conditional write at all, which spans a whole read-modify-write.
The capability is declared, and the residual race is documented at the call
site.

**Two deliberate divergences from WebDAV semantics:**

- **`delete` is not recursive.** A folder library is the user's own photographs
  on their own disk with no server-side trash behind it, so a caller that passed
  the wrong path would have no way back. Deleting a non-empty directory returns
  `RemoteError::Configuration`. Nothing in the engine deletes a directory — the
  soft delete is a `move_to` into the trash folder — so the guard is free.
- **Every filesystem call runs on the blocking pool.** On a local disk that is
  overkill; on the NFS mount this backend is most useful over, a stalled server
  would otherwise wedge the async worker that made the call and every other
  request sharing it.

**Failure classification** matters as much as the operations. A vanished mount
(`ESTALE`, `ENOTCONN`, `EIO`) maps to `RemoteError::Network`, which is what puts
the app into offline mode and leaves the catalog readable — exactly as a dead
server does. A permissions problem maps to `PermissionDenied` and does *not*,
because going offline over one forbidden file would hide a fixable problem
behind a network banner. An endpoint that is not a directory at all maps to
`RemoteError::Configuration`: nothing was unreachable and no credential was
wrong, so neither of the other two would send the user anywhere useful.

---

## 6. What the abstraction does not yet cover

Stated so the next person does not have to rediscover it.

- **Multiple accounts at once.** `AccountStore` holds a list and the launch
  screen uses the most recent. Nothing in the model prevents two open libraries;
  the interface has no place to show them.
- **Per-backend settings.** A connector has no way to contribute a settings
  page. Anything configurable is on the `Account` or is not configurable.
- **Capability probing at runtime.** `Capabilities` is fixed at construction.
  Nextcloud's `server_previews` should really be probed per account — a server
  with `camerarawpreviews` installed can render RAW — and today it is assumed to
  be `CommonFormatsOnly`.
- **A general notion of an account.** Credentials stay connector-specific on
  purpose (§3.5). A third connector with an OAuth flow will need a third `SignIn`
  variant, and that is the right place for it to appear.