dtourolle
dee509c6ef
storage.md's trait listing stopped at get; it now has get_reporting, what the default and the Nextcloud override do, and where develop reads the figures. A new §5.3 says how folders are chosen — the portal or Windows dialogue, the server browser whose New folder is create_dir, SAF on Android — and where an album's files go: a server folder relative to the account root, with the outbox's third .dest line, or a device folder that never syncs. catalog.md §8.2 said only collections merge, which had not been true since keywords, people and capture metadata joined them, and is less true with albums; it now lists what merges and why album_folders does not. §2 records that the album tables, like dedup_probes, are made on first use rather than by a migration. outstanding.md said there was no SAF code on Android. There is now, for album folders only, and it carries TRACES: FR-PLAT-AND-1, which the entry says overstates a requirement about the library; FR-PLAT-AND-2 and S10's row follow from that.
673 lines
35 KiB
Markdown
673 lines
35 KiB
Markdown
# Storage backends
|
||
|
||
How DarkRoom talks to wherever a library lives, and what it takes to add
|
||
somewhere new.
|
||
|
||
This document is the contract. `docs/architecture.md` §8 says why sync is built
|
||
on capability negotiation rather than a common denominator; this says what the
|
||
seam actually is, where each piece lives, and what a third connector has to do.
|
||
|
||
---
|
||
|
||
## 1. What "pluggable" has to mean
|
||
|
||
A trait alone does not make storage pluggable. `RemoteBackend` existed from the
|
||
first release and every layer above it still knew it was talking to Nextcloud:
|
||
seven files in `dr-ui` constructed a `NextcloudBackend` directly, ten functions
|
||
took one by concrete type, the account model was a server URL beside a DAV user
|
||
id, and the local cache directory was named after a hostname. The abstraction
|
||
was real and bought nothing, because everything that *reached* a backend was
|
||
still shaped like one product.
|
||
|
||
Pluggable means all four of these, not just the first:
|
||
|
||
1. **Operations** — what a backend can do. `RemoteBackend`.
|
||
2. **Capabilities** — what it can do *cheaply*, so the engine adapts instead of
|
||
assuming. `Capabilities`.
|
||
3. **Configuration** — what an account is, with no server in it. `Account`.
|
||
4. **Registration** — how the application discovers a connector at all, without
|
||
naming it. `BackendProvider` + `BackendRegistry`.
|
||
|
||
Two connectors ship. Nextcloud is unchanged and keeps every one of its
|
||
peculiarities — those are the point of the capability model, not an
|
||
embarrassment it has to hide. The folder connector serves a plain directory and
|
||
exists partly because it is genuinely useful and partly because a second
|
||
implementation is the only way to find out whether the first was an
|
||
abstraction.
|
||
|
||
---
|
||
|
||
## 2. Where each piece lives
|
||
|
||
```
|
||
core/dr-sync/ the contract, and nothing that speaks a protocol
|
||
├─ types.rs RemotePath, RemoteId, RemoteEntry, Validator, …
|
||
├─ capability.rs Capabilities, ChangeDetection, ServerPreviews
|
||
├─ error.rs RemoteError — the one error every caller handles
|
||
├─ account.rs Account, AccountStore, Secret, Connection
|
||
├─ provider.rs BackendProvider, BackendRegistry, SignIn
|
||
├─ lib.rs RemoteBackend, SyncStrategy
|
||
├─ scan.rs the walk, driven by capabilities
|
||
├─ upload.rs where an original is placed
|
||
└─ reachability.rs online/offline, inferred from observed results
|
||
|
||
core/dr-sync-nextcloud/ WebDAV, oc:fileid, chunked upload v2, Login Flow v2
|
||
core/dr-sync-folder/ a directory on a filesystem
|
||
|
||
ui/dr-ui/src/remote.rs the registry — the ONLY file above dr-sync that
|
||
names a connector
|
||
```
|
||
|
||
`dr-sync` depends on no connector. That is deliberate and load-bearing: a build
|
||
that only wants a folder library must not compile a TLS stack to get one, and
|
||
the registry therefore lives in the crate that already depends on everything —
|
||
the interface.
|
||
|
||
---
|
||
|
||
## 3. The four traits and types a connector meets
|
||
|
||
### 3.1 `RemoteBackend` — operations
|
||
|
||
```rust
|
||
#[async_trait]
|
||
pub trait RemoteBackend: Send + Sync {
|
||
fn capabilities(&self) -> &Capabilities;
|
||
fn name(&self) -> &str;
|
||
|
||
// discovery
|
||
async fn list(&self, dir: &RemotePath, since: Option<&Validator>)
|
||
-> Result<Vec<RemoteEntry>, RemoteError>;
|
||
async fn dir_validator(&self, dir: &RemotePath) -> Result<Validator, RemoteError>;
|
||
async fn delta(&self, cursor: &Cursor)
|
||
-> Result<(Vec<RemoteChange>, Cursor), RemoteError>;
|
||
|
||
// transfer
|
||
async fn get(&self, id: &RemoteId, range: Option<Range<u64>>)
|
||
-> Result<Vec<u8>, RemoteError>;
|
||
async fn get_reporting(&self, id: &RemoteId, // defaulted
|
||
progress: &(dyn Fn(u64, Option<u64>) + Send + Sync))
|
||
-> Result<Vec<u8>, RemoteError>;
|
||
async fn put(&self, path: &RemotePath, body: Vec<u8>, precond: Option<Precondition>)
|
||
-> Result<Validator, RemoteError>;
|
||
async fn put_many(&self, items: Vec<(RemotePath, Vec<u8>)>) // defaulted
|
||
-> Result<Vec<Result<Validator, RemoteError>>, RemoteError>;
|
||
async fn delete(&self, id: &RemoteId, precond: Option<Precondition>)
|
||
-> Result<(), RemoteError>;
|
||
async fn move_to(&self, from: &RemoteId, to: &RemotePath) -> Result<(), RemoteError>;
|
||
async fn create_dir(&self, path: &RemotePath) -> Result<(), RemoteError>;
|
||
|
||
// optional
|
||
async fn thumbnail(&self, id: &RemoteId, size: u32) // defaulted to None
|
||
-> Result<Option<Vec<u8>>, RemoteError>;
|
||
}
|
||
```
|
||
|
||
Rules that are not obvious from the signatures:
|
||
|
||
- **`dir_validator` and `delta` are capability-gated.** Return
|
||
`RemoteError::Unsupported` unless your `ChangeDetection` is
|
||
`PropagatingEtags` or `DeltaCursor` respectively. Answering
|
||
`dir_validator` with something that does not actually propagate is worse than
|
||
refusing: it lets a caller prune a subtree whose contents changed, and hides
|
||
those changes for as long as the folder list holds still.
|
||
- **`get` takes an optional range, and it is a hint.** A backend without cheap
|
||
ranges may return the whole object; the caller slices. Correctness holds
|
||
either way and `Capabilities::range_reads` says whether it was cheap.
|
||
- **`get_reporting` is for the one transfer somebody is watching.** An original
|
||
opened in develop is tens of megabytes, and "downloading" alone for that long
|
||
reads as stuck. It reports the bytes received and the length the server
|
||
declared, if it declared one. The default is `get` whole and one report at the
|
||
end, which is right for a backend whose read is local; Nextcloud overrides it
|
||
to read the body chunk by chunk. The develop view reads the figures by path
|
||
from the in-flight registry in `ui/dr-ui/src/library/thumbnails_fetch.rs`,
|
||
because a step along the roll usually lands on a frame the prefetcher is
|
||
already fetching, and falls back on the catalog's file length when no length
|
||
was declared.
|
||
- **Chunked upload is not in the trait.** It is an implementation detail of
|
||
`put`, chosen by body size. Exposing it would leak one server's protocol.
|
||
- **`move_to` must preserve identity where the backend has stable ids.** This
|
||
is what a soft delete uses (`FR-CAT-15`): a move implemented as copy + delete
|
||
allocates a new id, orphaning the thumbnail shard and turning a restore into a
|
||
full re-download.
|
||
- **`create_dir` makes parents and succeeds if the directory exists.** Callers
|
||
use it to guarantee a destination, not to claim they created one. It is also
|
||
the server browser's `New folder` (§5.3), which then lists the parent again
|
||
rather than inserting the name it asked for: the server may have normalised it.
|
||
|
||
### 3.2 `Capabilities` — what is cheap
|
||
|
||
The engine reads these once at connect time and picks a `SyncStrategy`. See
|
||
ARCH §8.1–8.2 for the tiers. The two that change behaviour rather than speed:
|
||
|
||
| Absent | Consequence the engine handles |
|
||
|---|---|
|
||
| `range_reads` | Embedded-preview extraction is impossible; browsing falls back to server previews or full download, and is refused on a metered connection |
|
||
| `conditional_write` | Sidecar conflict detection falls back to revision counters inside the sidecar — narrows the race, does not close it. Reported as a reduced-safety mode |
|
||
|
||
**Declare what is true, not what is flattering.** A backend claiming
|
||
`PropagatingEtags` it does not have does not merely run slowly; it silently
|
||
hides changes.
|
||
|
||
### 3.3 `Account` — configuration with no server in it
|
||
|
||
```rust
|
||
pub struct Account {
|
||
pub backend: String, // BackendProvider::id; defaults to "nextcloud" on load
|
||
pub endpoint: String, // stored as "server" — a URL, a path, a bucket
|
||
pub login: String, // empty where the connector has no notion of a user
|
||
pub user_id: String, // connector-defined sub-address; Nextcloud's DAV segment
|
||
pub root: String, // the folder chosen as the library root
|
||
pub formats: Vec<String>,
|
||
pub last_scan: Option<i64>,
|
||
}
|
||
```
|
||
|
||
Everything but `backend` is the connector's to interpret. Code above `dr-sync`
|
||
reads these for display and for cache keys, never for meaning.
|
||
|
||
Two properties are load-bearing:
|
||
|
||
- **The on-disk form is backwards compatible.** `backend` defaults to
|
||
`"nextcloud"` and `endpoint` is stored under its historical key `server`, so
|
||
every account written before there was a choice loads unchanged. A config the
|
||
app refuses to parse is an account the user has to set up again.
|
||
- **`Account::namespace()` is frozen for Nextcloud.** It names the directory
|
||
holding the catalog, the thumbnail shards, the sidecar spool and the export
|
||
outbox. Changing it does not lose that data, it *abandons* it — silently, as
|
||
an upgrade — and costs a full rescan on top. The Nextcloud form is reproduced
|
||
byte for byte from what `catalog_path` computed before; every other backend is
|
||
prefixed by its connector id, and long endpoints are truncated with a hash
|
||
tail so two deep paths cannot collide inside one filesystem's 255-byte
|
||
component limit.
|
||
|
||
### 3.4 `Connection` and `Secret` — the credential split
|
||
|
||
```rust
|
||
pub struct Connection { pub account: Account, pub secret: Option<Secret> }
|
||
```
|
||
|
||
Credentials go to platform secure storage (`FR-NC-2`, `NFR-SEC-2`). Never the
|
||
catalog, never the config file, never a log line. `AccountStore` writes the
|
||
account as plain JSON and the secret to the keyring, which is what lets the app
|
||
show "signed in as duncan, watching /PhotosRaw" before it has touched the
|
||
keyring at all.
|
||
|
||
`Secret`'s inner string is reachable only through `expose()`, and its `Debug`
|
||
prints `Secret(***)`. That closes the indirect leak — a `{:?}` on any struct
|
||
that happens to hold a connection — by construction rather than by review.
|
||
|
||
`Connection` is also what replaced a pair of arguments (credentials, user id)
|
||
threaded together through fifteen signatures in an order that could be swapped.
|
||
|
||
### 3.5 `BackendProvider` — registration
|
||
|
||
```rust
|
||
pub trait BackendProvider: Send + Sync {
|
||
fn id(&self) -> &'static str; // written to Account::backend
|
||
fn display_name(&self) -> &'static str;
|
||
fn endpoint_label(&self) -> &'static str; // "Server" / "Folder"
|
||
fn endpoint_placeholder(&self) -> &'static str;
|
||
fn sign_in(&self) -> SignIn;
|
||
fn normalise_endpoint(&self, input: &str) -> Result<String, String>;
|
||
fn upgrade_endpoint(&self, stored: &str) -> Option<String>; // defaulted: None
|
||
fn account_for(&self, endpoint: &str) -> Result<Account, RemoteError>; // defaulted
|
||
fn connect(&self, conn: &Connection) -> Result<Box<dyn RemoteBackend>, RemoteError>;
|
||
}
|
||
|
||
pub enum SignIn {
|
||
/// A handshake the user completes outside the app, yielding a credential.
|
||
Browser,
|
||
/// The endpoint is the whole account. No credential, no waiting state.
|
||
EndpointOnly,
|
||
}
|
||
```
|
||
|
||
- **`id` is on-disk configuration.** Changing it after anyone has an account
|
||
orphans that account. Pick it once.
|
||
- **`normalise_endpoint` is where a bad endpoint is *rejected*,** before an
|
||
account is written for a library that does not exist. Its error string is
|
||
shown to the user, so it says what to fix rather than naming a type. The
|
||
Nextcloud provider upgrades `http://` to `https://` here (`NFR-SEC-3`); the
|
||
folder provider canonicalises the path, so two spellings of one directory do
|
||
not become two accounts indexing the same photographs.
|
||
- **`upgrade_endpoint` is for accounts already saved,** and is not a second
|
||
call to `normalise_endpoint`: the folder provider's needs the path to exist,
|
||
so running it on every launch would fail a library on an unplugged disk.
|
||
`AccountStore::upgrade_endpoints` runs it at launch; only Nextcloud answers,
|
||
rewriting an `http://` account an older build saved to `https://`
|
||
(#65). The keyring entry is filed under the endpoint, so the secret is
|
||
copied across before the record changes, and a move that would change
|
||
`namespace()` is refused rather than performed. Below both, the Nextcloud
|
||
client sets `https_only`, so no URL it sends — a typed one, the login flow's
|
||
poll endpoint, a redirect — can carry the app password in the clear, and
|
||
the login flow keeps the address the user typed rather than the server's
|
||
idea of its own URL.
|
||
- **`connect` is synchronous and cheap.** It validates configuration and builds
|
||
a client; it does not talk to the remote. Workers call it per task.
|
||
- **`SignIn` is a shape, not a method.** It would be tidier to expose
|
||
`async fn sign_in()`, and wrong: Login Flow v2 is a browser handshake the user
|
||
completes elsewhere while the app polls, so it is not one call, it does not
|
||
finish on our schedule, and the screen has to render a URL and a waiting state
|
||
in the middle of it. `SignIn` tells the launch screen which of the two shapes
|
||
to draw; the flow stays where its protocol is.
|
||
|
||
**Credentials are deliberately not abstracted.** An app password, an OAuth
|
||
token and a bucket key pair have no useful common shape, and inventing one
|
||
before a third backend exists would produce a wrong answer confidently. The
|
||
general form is `Connection` — an account plus an opaque secret — and each
|
||
connector translates that into what its protocol needs
|
||
(`NextcloudProvider::credentials`).
|
||
|
||
---
|
||
|
||
## 4. Adding a backend
|
||
|
||
1. **Implement `RemoteBackend`** over your protocol, in a new
|
||
`core/dr-sync-<name>` crate depending on `dr-sync` and nothing else of ours.
|
||
2. **Declare `Capabilities` honestly.** Start from `Capabilities::minimal()` and
|
||
raise only what you can actually deliver.
|
||
3. **Implement `BackendProvider`** beside it.
|
||
4. **Register it** in `ui/dr-ui/src/remote.rs::registry()` and add the crate to
|
||
`ui/dr-ui/Cargo.toml`.
|
||
|
||
That is the whole list. Nothing else in `dr-ui` changes, because nothing else in
|
||
`dr-ui` names a connector.
|
||
|
||
**Two things to get right, because they are silent when wrong:**
|
||
|
||
- **Identity.** `RemoteEntry::id` should be `RemoteId::Stable(u64)` wherever you
|
||
can produce a `u64` that names the same photograph on every device looking at
|
||
the same library. The catalog keys the thumbnail shards and the face index on
|
||
it (`catalog.md` §10.1), and an entry without one gets neither. Set
|
||
`Capabilities::stable_ids` only if that id also survives a rename — the two
|
||
are different questions and only the second is a capability.
|
||
- **Path safety.** A `RemotePath` is built from names on the remote and from a
|
||
catalog another device wrote. If you resolve one against a real filesystem,
|
||
reject `..` before you open anything.
|
||
|
||
Register a test double the same way — `BackendRegistry::register` replaces an
|
||
existing id rather than shadowing it — so an integration test can stand a fake
|
||
server behind `"nextcloud"` without the registry knowing it happened.
|
||
|
||
---
|
||
|
||
## 5. The connectors that ship
|
||
|
||
### 5.1 Nextcloud (`dr-sync-nextcloud`, id `"nextcloud"`)
|
||
|
||
Unchanged by the abstraction, peculiarities intact — see ARCH §8.4 for the full
|
||
mapping. What matters here is that none of them had to be given up to make room
|
||
for a second backend:
|
||
|
||
| | |
|
||
|---|---|
|
||
| `change_detection` | `PropagatingEtags` — the one-request no-op sync |
|
||
| `stable_ids` | yes, `oc:fileid`, survives server-side rename and move |
|
||
| `range_reads` | yes, detected by `206` vs `200`, never `HEAD` |
|
||
| `chunked_upload` | v2, 5 MB – 5 GB, `MKCOL` → `PUT` chunks → `MOVE .file` |
|
||
| `bulk_upload` | yes, `POST /remote.php/dav/bulk` |
|
||
| `conditional_write` | yes, `If-Match` |
|
||
| `server_previews` | `CommonFormatsOnly` — stock Nextcloud renders no RAW |
|
||
| `get_reporting` | overridden: the body is read chunk by chunk, against `Content-Length` |
|
||
| sign-in | `SignIn::Browser`, Login Flow v2, system browser, app password |
|
||
|
||
Also kept: the `oc:permissions` probe on a refused `PUT`, which is what
|
||
distinguishes a create-only share from a bad credential; the `423 Locked`
|
||
retry classification; and the bundled ISRG Root YE certificate.
|
||
|
||
### 5.2 Folder (`dr-sync-folder`, id `"folder"`)
|
||
|
||
A local disk, an NFS or SMB mount, an external drive, or the directory a
|
||
Nextcloud desktop client already syncs. No server, no account, no credential —
|
||
which makes it the route that works on a machine with no secrets daemon at all.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `change_detection` | `LocalEtags` — see below |
|
||
| `stable_ids` | **no** — the id is a path hash and does not survive a rename |
|
||
| `range_reads` | yes, `seek` + `take` |
|
||
| `chunked_upload` | none; a write is a write |
|
||
| `bulk_upload` | no |
|
||
| `conditional_write` | yes, with a documented residual race |
|
||
| `server_previews` | `None` |
|
||
| `get_reporting` | the default: a local read, reported once when it is done |
|
||
| sign-in | `SignIn::EndpointOnly`, the folder chosen in the platform's dialogue (§5.3) |
|
||
|
||
**Why `LocalEtags` and not `PropagatingEtags`.** A POSIX directory's mtime
|
||
changes when its own entry list changes and at no other time — not when a
|
||
child's contents are edited, and not for a grandchild. There is nothing to
|
||
propagate, so `dir_validator` returns `Unsupported` and the engine walks the
|
||
tree every scan. Which costs almost nothing, because the walk that was expensive
|
||
was expensive for a reason this backend does not have.
|
||
|
||
**Measured 2026-08-28**, `cargo run -p dr-sync-folder --example scan`: a full
|
||
uncached walk of 2,299 images across 233 directories completed in **137 ms**,
|
||
and 380 images across 13 directories in **29 ms** — the same engine, the same
|
||
`Depth: 1`-per-directory walk, with no pruning at all. The Nextcloud connector's
|
||
comparable figure is 34.1 s for 17,185 RAWs across 334 directories *with*
|
||
pruning available (ARCH §8.4). The capability model is what lets one engine
|
||
drive both at the speed each actually runs at, instead of forcing the fast one
|
||
down to the slow one's interface.
|
||
|
||
**Identity is a hash of the path relative to the library root**, FNV-1a 64
|
||
(written out, because `DefaultHasher` is explicitly unstable between Rust
|
||
releases and this value is written into the catalog). It gives the catalog a
|
||
`u64` that names a photograph, is the same on every device looking at the same
|
||
folder, and does not change when the file is edited. It does not survive a
|
||
rename, and `stable_ids: false` says so: a moved photograph is seen as a delete
|
||
and an add, and its thumbnail is derived again.
|
||
|
||
The alternative — keying on the inode — is stable across a rename but *differs
|
||
between devices* and is reused by the filesystem after a delete. Two machines
|
||
would disagree about which photograph a thumbnail belonged to, and a recycled
|
||
inode would silently attach an old thumbnail to a new image. Re-deriving a
|
||
thumbnail is a cost; showing the wrong one is a bug.
|
||
|
||
**Conditional writes.** `IfAbsent` is genuinely atomic (`O_CREAT | O_EXCL`).
|
||
`IfMatch` is compare-then-swap: a `stat`, then a write to a temporary beside the
|
||
destination and a `rename` over it. A POSIX filesystem has no compare-and-swap,
|
||
so the race is narrowed to the microseconds between the two syscalls rather than
|
||
closed — still far tighter than the fallback the engine uses for a backend that
|
||
declares no conditional write at all, which spans a whole read-modify-write.
|
||
The capability is declared, and the residual race is documented at the call
|
||
site.
|
||
|
||
**Two deliberate divergences from WebDAV semantics:**
|
||
|
||
- **`delete` is not recursive.** A folder library is the user's own photographs
|
||
on their own disk with no server-side trash behind it, so a caller that passed
|
||
the wrong path would have no way back. Deleting a non-empty directory returns
|
||
`RemoteError::Configuration`. Nothing in the engine deletes a directory — the
|
||
soft delete is a `move_to` into the trash folder — so the guard is free.
|
||
- **Every filesystem call runs on the blocking pool.** On a local disk that is
|
||
overkill; on the NFS mount this backend is most useful over, a stalled server
|
||
would otherwise wedge the async worker that made the call and every other
|
||
request sharing it.
|
||
|
||
**Failure classification** matters as much as the operations. A vanished mount
|
||
(`ESTALE`, `ENOTCONN`, `EIO`) maps to `RemoteError::Network`, which is what puts
|
||
the app into offline mode and leaves the catalog readable — exactly as a dead
|
||
server does. A permissions problem maps to `PermissionDenied` and does *not*,
|
||
because going offline over one forbidden file would hide a fixable problem
|
||
behind a network banner. An endpoint that is not a directory at all maps to
|
||
`RemoteError::Configuration`: nothing was unreachable and no credential was
|
||
wrong, so neither of the other two would send the user anywhere useful.
|
||
|
||
### 5.3 Choosing folders, and the folders exports go to
|
||
|
||
**Folders are pointed at, never typed** (FR-EXP-6). On the desktop,
|
||
`ui/dr-ui/src/folder_dialog.rs` asks the platform: the XDG desktop portal's
|
||
FileChooser on Linux, through `rfd`, and the common item dialogue on Windows.
|
||
That is how the folder connector's endpoint is chosen on the launch screen, and
|
||
the path it returns still goes through `normalise_endpoint` as a typed one did.
|
||
A folder on the server is chosen in the in-app browser, which lists with `list`
|
||
and makes a folder with `create_dir`; the launch screen and the album sheet
|
||
share it through `ui/dr-ui/src/remote_folders.rs`, which keeps both round trips
|
||
off the interface thread. Android has no filesystem dialogue, so its library
|
||
folder is still typed, and an album's folder there comes from the Storage
|
||
Access Framework's tree picker.
|
||
|
||
**An album is where exports go** (FR-EXP-10), and never inside the library: a
|
||
JPEG written into the tree a scan catalogues comes back as a photograph beside
|
||
the RAW it was made from. Its folder is one of two kinds
|
||
(`dr_catalog::albums::Place`):
|
||
|
||
- **On the server,** relative to the *account* root, not the library root. The
|
||
browser refuses a folder inside the library and says why; on a server whose
|
||
whole account is the library, every folder is inside it, and it says that
|
||
instead. Exports reach it through the outbox like any upload, and a queued
|
||
file's `.dest` record gains a third line saying its folder is relative to
|
||
the account — a third line rather than a leading slash, because a record
|
||
written before albums may carry a stray slash and has to keep the meaning it
|
||
was written with.
|
||
- **On this device,** a path, or on Android a SAF tree URI with a persisted
|
||
grant, written through `DocumentsContract` (`ui/dr-ui/src/saf.rs`). A
|
||
provider renames on a collision by itself, so the album records the name it
|
||
was given rather than the one asked for.
|
||
|
||
A server folder lives on the album row and syncs with it. A device folder lives
|
||
in `album_folders`, which the merge never reads and the upload snapshot drops,
|
||
because a path or a grant on one device means nothing on another: an album made
|
||
on the desktop arrives on the tablet with no folder until one is chosen there.
|
||
|
||
---
|
||
|
||
## 6. Virtual filesystems
|
||
|
||
A sync client in virtual-files mode leaves a **placeholder** where a file is
|
||
catalogued but not downloaded. On Linux — the only mode it supports — that
|
||
means `IMG.CR2` does not exist at all and `IMG.CR2.nextcloud` does, holding one
|
||
byte. ARCH §9.0 measured a real machine: 121,785 placeholders against 10,267
|
||
materialised files.
|
||
|
||
A folder library that ignores this is not merely degraded, it is dangerous.
|
||
Before the handling below existed, the folder connector catalogued every stub
|
||
as a 1-byte image, gave it an identity that changed the moment it was
|
||
downloaded, and — worst — reported a dehydrated *sidecar* as absent, which made
|
||
the sidecar writer create a fresh document over an existing one and discard
|
||
every edit another device had put there.
|
||
|
||
### 6.1 Three questions, one trait
|
||
|
||
Everything else about a synced folder is an ordinary directory, so this is not
|
||
a second connector. `dr_sync_folder::Vfs` asks only what differs:
|
||
|
||
```rust
|
||
pub trait Vfs: Send + Sync {
|
||
fn name(&self) -> &'static str;
|
||
fn is_placeholder(&self, on_disk: &str) -> bool;
|
||
fn real_name<'a>(&self, on_disk: &'a str) -> &'a str;
|
||
fn placeholder_name(&self, name: &str) -> Cow<'_, str>;
|
||
fn can_materialise(&self) -> bool; // defaulted false
|
||
fn materialise(&self, local: &Path) -> Result<(), RemoteError>; // defaulted
|
||
fn dematerialise(&self, local: &Path) -> Result<(), RemoteError>; // defaulted
|
||
}
|
||
```
|
||
|
||
`NoVfs` for a plain directory; `dr_sync_nextcloud::NextcloudVfs` for a synced
|
||
one, wrapping the `DesktopClient` socket. A third convention is a third impl.
|
||
|
||
**Why not a `folder-vfs` provider.** The interesting capability is not a
|
||
property of the backend: the same directory can materialise on demand while the
|
||
client is running and cannot when it is down, so it must be computed per
|
||
connection either way. Registering two providers would ask the user to choose
|
||
between two things that differ by whether a background process is up. The
|
||
convention is detected instead, per connection, by a hook the registry supplies
|
||
(`FolderProvider::with_vfs_detector`) — which is what keeps `dr-sync-folder`
|
||
free of any client's protocol.
|
||
|
||
### 6.2 What the backend reports
|
||
|
||
| | |
|
||
|---|---|
|
||
| `RemoteEntry::path` | the photograph's name, never the stub's — so identity survives a download |
|
||
| `RemoteEntry::materialised` | `false` on a stub; the catalog maps it to `Availability::Offline` |
|
||
| `RemoteEntry::size` | `0` on a stub, meaning *unknown* — see below |
|
||
| `get` on a stub | `RemoteError::NotMaterialised`, **never** `NotFound` and never the stub's one byte |
|
||
| `put` over a stub, unconditional | **replaces it** — the whole file is being written, so there is nothing in the stub to keep, and the placeholder is removed after the content lands |
|
||
| `put` over a stub, `IfMatch` | `NotMaterialised` — a stub's validator describes the placeholder, so nothing here can satisfy the guard; the caller fetches and retries |
|
||
| `put` over a stub, `IfAbsent` | `PreconditionFailed` — the file *is* there, only its content is elsewhere |
|
||
| `move_to` a stub | moves the stub and keeps it a stub — culling without downloading is ordinary |
|
||
| `delete` a stub | deletes it; a photograph is deleted whether or not its bytes are here |
|
||
| `capabilities().materialisation` | `OnDemand` with a client, `Placeholders` without, `Always` on a plain folder |
|
||
|
||
**Size is genuinely unknown.** A Linux suffix-mode stub is one byte and carries
|
||
no record of what it stands for. The client's `._sync_*.db` has the real size,
|
||
but that is a private schema and reading it would couple us to their migrations.
|
||
FR-NC-6c wants a transfer size quoted before an operation starts; for a stub the
|
||
honest answer is that it cannot be, and the interface should say so rather than
|
||
report one byte or invent an estimate silently.
|
||
|
||
### 6.3 Hydration is a borrow
|
||
|
||
The rule: **a file is returned to the state it was found in.** What a pass
|
||
downloaded is released; what the user already had is left alone. `BorrowPool`
|
||
enforces it.
|
||
|
||
```rust
|
||
let pool = BorrowPool::new();
|
||
{
|
||
let held = pool.borrow(&backend, &path).await?; // downloads only if absent
|
||
// ... read it, thumbnail it, index its faces ...
|
||
} // borrow ends
|
||
let stats = pool.release_all(&backend).await; // dehydrates only what it hydrated
|
||
```
|
||
|
||
Three properties that are not obvious:
|
||
|
||
- **Reference counted.** The thumbnail pass and the face pass meet on the same
|
||
RAW. Without counting, the first to finish dehydrates the file the second is
|
||
reading; with it, the transfer is paid once and released when the last
|
||
borrower is done.
|
||
- **Prior state is read before asking.** After `materialise` there is no way to
|
||
tell what the pass brought from what was already there, so it is recorded
|
||
first. Getting this wrong silently undoes a pin, and "my pinned trip
|
||
evaporated after an indexing run" is the failure that would make people stop
|
||
trusting the feature.
|
||
- **Being unsure is not symmetric.** `borrow_known(.., Some(true))` keeps a file
|
||
that might have been ours — costing disk. `Some(false)` releases one that
|
||
might have been the user's. An uncertain caller passes `true` or `None`,
|
||
never a guess at `false`.
|
||
|
||
A borrow against a plain folder or a server backend short-circuits and does
|
||
nothing, so a pass written for a VFS library runs unchanged everywhere rather
|
||
than growing two code paths.
|
||
|
||
**Measured 2026-08-29**, `cargo run -p dr-sync-folder --example vfs_cycle`: a
|
||
library of 100 photographs at 25 MB each, 90 of them dehydrated and 10 the user
|
||
keeps. A pass over all 100, borrowing and releasing as it goes:
|
||
|
||
| | |
|
||
|---|---|
|
||
| on disk at rest | 250 MB |
|
||
| **peak during the pass** | **275 MB** — the resting set plus one photograph |
|
||
| without borrowing | 2,500 MB |
|
||
| on disk afterwards | 250 MB |
|
||
| of the 10 the user already had | 10 still there |
|
||
|
||
The peak is the working set, not the library, and the release is selective.
|
||
|
||
|
||
### 6.4 Derived state is dehydrated too
|
||
|
||
Shards, the catalog snapshot and the place record live in `.darkroom-derived/`
|
||
**inside the library folder**, so a sync client dehydrates them exactly as it
|
||
dehydrates a photograph. Unlike a photograph, none of them can be skipped: a
|
||
shard that will not open is a peer's thumbnails never merging, and a catalog
|
||
snapshot that will not open is their collections.
|
||
|
||
The folder holds three kinds of thing:
|
||
|
||
| File | What it is | How two devices reconcile it |
|
||
| --- | --- | --- |
|
||
| `shard-<client>-NNNN.sqlite` | Thumbnail and face shards | Sealed and immutable; a name match is a content match, so existence is the whole protocol |
|
||
| `catalog.sqlite` | The catalog snapshot, for its collections | Read-modify-write: **merge theirs, then push the union** |
|
||
| `place.json` | Where the photographer was (FR-UI-8) | Replace: the newer timestamp wins outright |
|
||
|
||
`place.json` is the odd one out, and deliberately. Everything else there is
|
||
*derived* — a faster way to learn something the device could have worked out for
|
||
itself from the originals and the sidecars — so losing it costs time. A place is
|
||
a fact only the other device knew, and losing it costs a scroll. That is why it
|
||
is exchanged last in the pass, why its failures are logged rather than reported,
|
||
and why it is the one file here that is replaced rather than merged: two devices
|
||
cannot both be where the photographer is, so there is nothing of theirs inside
|
||
ours to preserve.
|
||
|
||
It still refuses to upload over a copy it could not read, for a smaller version
|
||
of the reason below: a record we have not compared against might be the newer
|
||
one, and overwriting it would move the other device's photographer without ever
|
||
having seen where they were.
|
||
|
||
`derived_sync::read_derived` fetches on demand rather than giving up. More
|
||
important is what happens when it *cannot*:
|
||
|
||
The catalog sync is a read-modify-write over a file another device also writes.
|
||
It was shaped `if let Ok(bytes) = backend.get(..)`, which folded every failure
|
||
into "there is no remote catalog" and carried straight on to the upload — so a
|
||
dehydrated snapshot meant pushing ours over theirs unmerged, taking their
|
||
collections and members with it. The same shape as the sidecar bug in §6, and
|
||
the same fix: a read that fails for any reason other than `NotFound` **stops the
|
||
upload**.
|
||
|
||
That is why `NotFound` and `NotMaterialised` had to be separate errors. One
|
||
means "yours is the whole truth, write it"; the other means "do not dare".
|
||
|
||
### 6.5 Release means dehydrate, never delete
|
||
|
||
The single most dangerous thing in this feature. A synced folder is not a
|
||
cache: deleting a materialised file inside it propagates the deletion to the
|
||
server and removes the photograph from every device the user owns. `Vfs` and
|
||
`RemoteBackend::dematerialise` both say so, and an implementation that cannot
|
||
dehydrate returns `Unsupported` rather than approximating it.
|
||
|
||
This is also why the originals cache (`dr_catalog::cache`) cannot simply be
|
||
pointed at a VFS library: `Cache::release` deletes bytes, which is right for a
|
||
copy under `originals/` and catastrophic in place.
|
||
|
||
### 6.6 Which photographs stay downloaded
|
||
|
||
The user's half of the bargain: a pass borrows for a moment, but *some* of the
|
||
library should stay local — the trip you are about to take, the shoot you are
|
||
working on.
|
||
|
||
That is a **pin**, and it is the pin the originals cache already had
|
||
(`dr_catalog::cache`, FR-NC-6a). Nothing parallel was built, because the model
|
||
was already the right one:
|
||
|
||
| Cache concept | On a placeholder library |
|
||
|---|---|
|
||
| `tier_desired` | what the user asked to keep hydrated |
|
||
| `tier_actual` | what is actually materialised |
|
||
| `pending_pins()` | the work list — what to hydrate next, resumable |
|
||
| pinned rows are never evicted | a pinned collection is never dehydrated |
|
||
| passive rows, LRU under a budget | what a pass borrowed, released when it finishes |
|
||
|
||
So "keep this collection hydrated" is `Cache::pin`, and the existing pin worker
|
||
drives it — except that on a placeholder library it calls `materialise` instead
|
||
of downloading a copy.
|
||
|
||
**Why not a copy.** The original materialises *in the library folder*. Copying
|
||
it under `originals/` as well would hold every pinned photograph twice, and the
|
||
copy would be the half the budget could evict while the real disk cost stayed.
|
||
`Cache::record_in_place` records the bookkeeping with **`path = NULL`**, and
|
||
that null is load-bearing: `release` deletes the file a row names, and a row
|
||
that names none deletes nothing. The safety property is structural rather than
|
||
remembered.
|
||
|
||
Unpinning therefore frees nothing by itself — the bytes are not ours to delete.
|
||
`spawn_dehydrate` asks the client to take them back, which is what actually
|
||
returns the disk.
|
||
|
||
---
|
||
|
||
## 7. What the abstraction does not yet cover
|
||
|
||
Stated so the next person does not have to rediscover it.
|
||
|
||
- **Multiple accounts at once.** `AccountStore` holds a list and the launch
|
||
screen uses the most recent. Nothing in the model prevents two open libraries;
|
||
the interface has no place to show them.
|
||
- **Per-backend settings.** A connector has no way to contribute a settings
|
||
page. Anything configurable is on the `Account` or is not configurable.
|
||
- **Capability probing at runtime.** `Capabilities` is fixed at construction.
|
||
Nextcloud's `server_previews` should really be probed per account — a server
|
||
with `camerarawpreviews` installed can render RAW — and today it is assumed to
|
||
be `CommonFormatsOnly`.
|
||
- **A general notion of an account.** Credentials stay connector-specific on
|
||
purpose (§3.5). A third connector with an OAuth flow will need a third `SignIn`
|
||
variant, and that is the right place for it to appear.
|
||
- **A quoted cost before a hydrating pass.** FR-NC-6c wants the transfer size
|
||
stated before an operation that needs absent data. A placeholder reports no
|
||
size (§6.2), so the honest figure for "index this library" is a count and not
|
||
a byte total. The interface should say *n photographs, size unknown until
|
||
fetched* rather than estimate one silently — and it does not say anything yet.
|
||
- **Metadata-only placeholders.** Windows and macOS express these in filesystem
|
||
metadata rather than in the name, and carry the real size there. `Vfs` asks
|
||
its questions about a *name*, which is all the one convention this project has
|
||
met needs. Supporting them means widening the trait to take a `Metadata`, and
|
||
doing that before anyone has run this on those platforms would be guessing.
|
||
- **Hydration during browsing, deliberately.** It stays forbidden (ARCH §9.0
|
||
finding 3). A grid cell whose content is absent shows as not-downloaded; only
|
||
a pass the user asked for may fetch.
|