The claim that hydration-as-a-borrow makes peak disk the working set rather than the library was so far an argument. `--example vfs_cycle` runs it: 100 photographs of 25 MB, 90 dehydrated, a pass over all of them through the real engine. Peak 275 MB — the resting set plus one photograph — against 2,500 MB had the pass simply fetched everything. Back to 250 MB afterwards, and all ten files the user already kept still there, which is the half of the contract that matters more. Recorded in docs/storage.md §6.3 and ARCH §9.0a, because a bound argued from a number nobody measured is one that gets quietly lost.
560 lines
28 KiB
Markdown
560 lines
28 KiB
Markdown
# Storage backends
|
||
|
||
How DarkRoom talks to wherever a library lives, and what it takes to add
|
||
somewhere new.
|
||
|
||
This document is the contract. `docs/architecture.md` §8 says why sync is built
|
||
on capability negotiation rather than a common denominator; this says what the
|
||
seam actually is, where each piece lives, and what a third connector has to do.
|
||
|
||
---
|
||
|
||
## 1. What "pluggable" has to mean
|
||
|
||
A trait alone does not make storage pluggable. `RemoteBackend` existed from the
|
||
first release and every layer above it still knew it was talking to Nextcloud:
|
||
seven files in `dr-ui` constructed a `NextcloudBackend` directly, ten functions
|
||
took one by concrete type, the account model was a server URL beside a DAV user
|
||
id, and the local cache directory was named after a hostname. The abstraction
|
||
was real and bought nothing, because everything that *reached* a backend was
|
||
still shaped like one product.
|
||
|
||
Pluggable means all four of these, not just the first:
|
||
|
||
1. **Operations** — what a backend can do. `RemoteBackend`.
|
||
2. **Capabilities** — what it can do *cheaply*, so the engine adapts instead of
|
||
assuming. `Capabilities`.
|
||
3. **Configuration** — what an account is, with no server in it. `Account`.
|
||
4. **Registration** — how the application discovers a connector at all, without
|
||
naming it. `BackendProvider` + `BackendRegistry`.
|
||
|
||
Two connectors ship. Nextcloud is unchanged and keeps every one of its
|
||
peculiarities — those are the point of the capability model, not an
|
||
embarrassment it has to hide. The folder connector serves a plain directory and
|
||
exists partly because it is genuinely useful and partly because a second
|
||
implementation is the only way to find out whether the first was an
|
||
abstraction.
|
||
|
||
---
|
||
|
||
## 2. Where each piece lives
|
||
|
||
```
|
||
core/dr-sync/ the contract, and nothing that speaks a protocol
|
||
├─ types.rs RemotePath, RemoteId, RemoteEntry, Validator, …
|
||
├─ capability.rs Capabilities, ChangeDetection, ServerPreviews
|
||
├─ error.rs RemoteError — the one error every caller handles
|
||
├─ account.rs Account, AccountStore, Secret, Connection
|
||
├─ provider.rs BackendProvider, BackendRegistry, SignIn
|
||
├─ lib.rs RemoteBackend, SyncStrategy
|
||
├─ scan.rs the walk, driven by capabilities
|
||
├─ upload.rs where an original is placed
|
||
└─ reachability.rs online/offline, inferred from observed results
|
||
|
||
core/dr-sync-nextcloud/ WebDAV, oc:fileid, chunked upload v2, Login Flow v2
|
||
core/dr-sync-folder/ a directory on a filesystem
|
||
|
||
ui/dr-ui/src/remote.rs the registry — the ONLY file above dr-sync that
|
||
names a connector
|
||
```
|
||
|
||
`dr-sync` depends on no connector. That is deliberate and load-bearing: a build
|
||
that only wants a folder library must not compile a TLS stack to get one, and
|
||
the registry therefore lives in the crate that already depends on everything —
|
||
the interface.
|
||
|
||
---
|
||
|
||
## 3. The four traits and types a connector meets
|
||
|
||
### 3.1 `RemoteBackend` — operations
|
||
|
||
```rust
|
||
#[async_trait]
|
||
pub trait RemoteBackend: Send + Sync {
|
||
fn capabilities(&self) -> &Capabilities;
|
||
fn name(&self) -> &str;
|
||
|
||
// discovery
|
||
async fn list(&self, dir: &RemotePath, since: Option<&Validator>)
|
||
-> Result<Vec<RemoteEntry>, RemoteError>;
|
||
async fn dir_validator(&self, dir: &RemotePath) -> Result<Validator, RemoteError>;
|
||
async fn delta(&self, cursor: &Cursor)
|
||
-> Result<(Vec<RemoteChange>, Cursor), RemoteError>;
|
||
|
||
// transfer
|
||
async fn get(&self, id: &RemoteId, range: Option<Range<u64>>)
|
||
-> Result<Vec<u8>, RemoteError>;
|
||
async fn put(&self, path: &RemotePath, body: Vec<u8>, precond: Option<Precondition>)
|
||
-> Result<Validator, RemoteError>;
|
||
async fn put_many(&self, items: Vec<(RemotePath, Vec<u8>)>) // defaulted
|
||
-> Result<Vec<Result<Validator, RemoteError>>, RemoteError>;
|
||
async fn delete(&self, id: &RemoteId, precond: Option<Precondition>)
|
||
-> Result<(), RemoteError>;
|
||
async fn move_to(&self, from: &RemoteId, to: &RemotePath) -> Result<(), RemoteError>;
|
||
async fn create_dir(&self, path: &RemotePath) -> Result<(), RemoteError>;
|
||
|
||
// optional
|
||
async fn thumbnail(&self, id: &RemoteId, size: u32) // defaulted to None
|
||
-> Result<Option<Vec<u8>>, RemoteError>;
|
||
}
|
||
```
|
||
|
||
Rules that are not obvious from the signatures:
|
||
|
||
- **`dir_validator` and `delta` are capability-gated.** Return
|
||
`RemoteError::Unsupported` unless your `ChangeDetection` is
|
||
`PropagatingEtags` or `DeltaCursor` respectively. Answering
|
||
`dir_validator` with something that does not actually propagate is worse than
|
||
refusing: it lets a caller prune a subtree whose contents changed, and hides
|
||
those changes for as long as the folder list holds still.
|
||
- **`get` takes an optional range, and it is a hint.** A backend without cheap
|
||
ranges may return the whole object; the caller slices. Correctness holds
|
||
either way and `Capabilities::range_reads` says whether it was cheap.
|
||
- **Chunked upload is not in the trait.** It is an implementation detail of
|
||
`put`, chosen by body size. Exposing it would leak one server's protocol.
|
||
- **`move_to` must preserve identity where the backend has stable ids.** This
|
||
is what a soft delete uses (`FR-CAT-15`): a move implemented as copy + delete
|
||
allocates a new id, orphaning the thumbnail shard and turning a restore into a
|
||
full re-download.
|
||
- **`create_dir` makes parents and succeeds if the directory exists.** Callers
|
||
use it to guarantee a destination, not to claim they created one.
|
||
|
||
### 3.2 `Capabilities` — what is cheap
|
||
|
||
The engine reads these once at connect time and picks a `SyncStrategy`. See
|
||
ARCH §8.1–8.2 for the tiers. The two that change behaviour rather than speed:
|
||
|
||
| Absent | Consequence the engine handles |
|
||
|---|---|
|
||
| `range_reads` | Embedded-preview extraction is impossible; browsing falls back to server previews or full download, and is refused on a metered connection |
|
||
| `conditional_write` | Sidecar conflict detection falls back to revision counters inside the sidecar — narrows the race, does not close it. Reported as a reduced-safety mode |
|
||
|
||
**Declare what is true, not what is flattering.** A backend claiming
|
||
`PropagatingEtags` it does not have does not merely run slowly; it silently
|
||
hides changes.
|
||
|
||
### 3.3 `Account` — configuration with no server in it
|
||
|
||
```rust
|
||
pub struct Account {
|
||
pub backend: String, // BackendProvider::id; defaults to "nextcloud" on load
|
||
pub endpoint: String, // stored as "server" — a URL, a path, a bucket
|
||
pub login: String, // empty where the connector has no notion of a user
|
||
pub user_id: String, // connector-defined sub-address; Nextcloud's DAV segment
|
||
pub root: String, // the folder chosen as the library root
|
||
pub formats: Vec<String>,
|
||
pub last_scan: Option<i64>,
|
||
}
|
||
```
|
||
|
||
Everything but `backend` is the connector's to interpret. Code above `dr-sync`
|
||
reads these for display and for cache keys, never for meaning.
|
||
|
||
Two properties are load-bearing:
|
||
|
||
- **The on-disk form is backwards compatible.** `backend` defaults to
|
||
`"nextcloud"` and `endpoint` is stored under its historical key `server`, so
|
||
every account written before there was a choice loads unchanged. A config the
|
||
app refuses to parse is an account the user has to set up again.
|
||
- **`Account::namespace()` is frozen for Nextcloud.** It names the directory
|
||
holding the catalog, the thumbnail shards, the sidecar spool and the export
|
||
outbox. Changing it does not lose that data, it *abandons* it — silently, as
|
||
an upgrade — and costs a full rescan on top. The Nextcloud form is reproduced
|
||
byte for byte from what `catalog_path` computed before; every other backend is
|
||
prefixed by its connector id, and long endpoints are truncated with a hash
|
||
tail so two deep paths cannot collide inside one filesystem's 255-byte
|
||
component limit.
|
||
|
||
### 3.4 `Connection` and `Secret` — the credential split
|
||
|
||
```rust
|
||
pub struct Connection { pub account: Account, pub secret: Option<Secret> }
|
||
```
|
||
|
||
Credentials go to platform secure storage (`FR-NC-2`, `NFR-SEC-2`). Never the
|
||
catalog, never the config file, never a log line. `AccountStore` writes the
|
||
account as plain JSON and the secret to the keyring, which is what lets the app
|
||
show "signed in as duncan, watching /PhotosRaw" before it has touched the
|
||
keyring at all.
|
||
|
||
`Secret`'s inner string is reachable only through `expose()`, and its `Debug`
|
||
prints `Secret(***)`. That closes the indirect leak — a `{:?}` on any struct
|
||
that happens to hold a connection — by construction rather than by review.
|
||
|
||
`Connection` is also what replaced a pair of arguments (credentials, user id)
|
||
threaded together through fifteen signatures in an order that could be swapped.
|
||
|
||
### 3.5 `BackendProvider` — registration
|
||
|
||
```rust
|
||
pub trait BackendProvider: Send + Sync {
|
||
fn id(&self) -> &'static str; // written to Account::backend
|
||
fn display_name(&self) -> &'static str;
|
||
fn endpoint_label(&self) -> &'static str; // "Server" / "Folder"
|
||
fn endpoint_placeholder(&self) -> &'static str;
|
||
fn sign_in(&self) -> SignIn;
|
||
fn normalise_endpoint(&self, input: &str) -> Result<String, String>;
|
||
fn account_for(&self, endpoint: &str) -> Result<Account, RemoteError>; // defaulted
|
||
fn connect(&self, conn: &Connection) -> Result<Box<dyn RemoteBackend>, RemoteError>;
|
||
}
|
||
|
||
pub enum SignIn {
|
||
/// A handshake the user completes outside the app, yielding a credential.
|
||
Browser,
|
||
/// The endpoint is the whole account. No credential, no waiting state.
|
||
EndpointOnly,
|
||
}
|
||
```
|
||
|
||
- **`id` is on-disk configuration.** Changing it after anyone has an account
|
||
orphans that account. Pick it once.
|
||
- **`normalise_endpoint` is where a bad endpoint is *rejected*,** before an
|
||
account is written for a library that does not exist. Its error string is
|
||
shown to the user, so it says what to fix rather than naming a type. The
|
||
Nextcloud provider upgrades `http://` to `https://` here (`NFR-SEC-3`); the
|
||
folder provider canonicalises the path, so two spellings of one directory do
|
||
not become two accounts indexing the same photographs.
|
||
- **`connect` is synchronous and cheap.** It validates configuration and builds
|
||
a client; it does not talk to the remote. Workers call it per task.
|
||
- **`SignIn` is a shape, not a method.** It would be tidier to expose
|
||
`async fn sign_in()`, and wrong: Login Flow v2 is a browser handshake the user
|
||
completes elsewhere while the app polls, so it is not one call, it does not
|
||
finish on our schedule, and the screen has to render a URL and a waiting state
|
||
in the middle of it. `SignIn` tells the launch screen which of the two shapes
|
||
to draw; the flow stays where its protocol is.
|
||
|
||
**Credentials are deliberately not abstracted.** An app password, an OAuth
|
||
token and a bucket key pair have no useful common shape, and inventing one
|
||
before a third backend exists would produce a wrong answer confidently. The
|
||
general form is `Connection` — an account plus an opaque secret — and each
|
||
connector translates that into what its protocol needs
|
||
(`NextcloudProvider::credentials`).
|
||
|
||
---
|
||
|
||
## 4. Adding a backend
|
||
|
||
1. **Implement `RemoteBackend`** over your protocol, in a new
|
||
`core/dr-sync-<name>` crate depending on `dr-sync` and nothing else of ours.
|
||
2. **Declare `Capabilities` honestly.** Start from `Capabilities::minimal()` and
|
||
raise only what you can actually deliver.
|
||
3. **Implement `BackendProvider`** beside it.
|
||
4. **Register it** in `ui/dr-ui/src/remote.rs::registry()` and add the crate to
|
||
`ui/dr-ui/Cargo.toml`.
|
||
|
||
That is the whole list. Nothing else in `dr-ui` changes, because nothing else in
|
||
`dr-ui` names a connector.
|
||
|
||
**Two things to get right, because they are silent when wrong:**
|
||
|
||
- **Identity.** `RemoteEntry::id` should be `RemoteId::Stable(u64)` wherever you
|
||
can produce a `u64` that names the same photograph on every device looking at
|
||
the same library. The catalog keys the thumbnail shards and the face index on
|
||
it (`catalog.md` §10.1), and an entry without one gets neither. Set
|
||
`Capabilities::stable_ids` only if that id also survives a rename — the two
|
||
are different questions and only the second is a capability.
|
||
- **Path safety.** A `RemotePath` is built from names on the remote and from a
|
||
catalog another device wrote. If you resolve one against a real filesystem,
|
||
reject `..` before you open anything.
|
||
|
||
Register a test double the same way — `BackendRegistry::register` replaces an
|
||
existing id rather than shadowing it — so an integration test can stand a fake
|
||
server behind `"nextcloud"` without the registry knowing it happened.
|
||
|
||
---
|
||
|
||
## 5. The connectors that ship
|
||
|
||
### 5.1 Nextcloud (`dr-sync-nextcloud`, id `"nextcloud"`)
|
||
|
||
Unchanged by the abstraction, peculiarities intact — see ARCH §8.4 for the full
|
||
mapping. What matters here is that none of them had to be given up to make room
|
||
for a second backend:
|
||
|
||
| | |
|
||
|---|---|
|
||
| `change_detection` | `PropagatingEtags` — the one-request no-op sync |
|
||
| `stable_ids` | yes, `oc:fileid`, survives server-side rename and move |
|
||
| `range_reads` | yes, detected by `206` vs `200`, never `HEAD` |
|
||
| `chunked_upload` | v2, 5 MB – 5 GB, `MKCOL` → `PUT` chunks → `MOVE .file` |
|
||
| `bulk_upload` | yes, `POST /remote.php/dav/bulk` |
|
||
| `conditional_write` | yes, `If-Match` |
|
||
| `server_previews` | `CommonFormatsOnly` — stock Nextcloud renders no RAW |
|
||
| sign-in | `SignIn::Browser`, Login Flow v2, system browser, app password |
|
||
|
||
Also kept: the `oc:permissions` probe on a refused `PUT`, which is what
|
||
distinguishes a create-only share from a bad credential; the `423 Locked`
|
||
retry classification; and the bundled ISRG Root YE certificate.
|
||
|
||
### 5.2 Folder (`dr-sync-folder`, id `"folder"`)
|
||
|
||
A local disk, an NFS or SMB mount, an external drive, or the directory a
|
||
Nextcloud desktop client already syncs. No server, no account, no credential —
|
||
which makes it the route that works on a machine with no secrets daemon at all.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `change_detection` | `LocalEtags` — see below |
|
||
| `stable_ids` | **no** — the id is a path hash and does not survive a rename |
|
||
| `range_reads` | yes, `seek` + `take` |
|
||
| `chunked_upload` | none; a write is a write |
|
||
| `bulk_upload` | no |
|
||
| `conditional_write` | yes, with a documented residual race |
|
||
| `server_previews` | `None` |
|
||
| sign-in | `SignIn::EndpointOnly` |
|
||
|
||
**Why `LocalEtags` and not `PropagatingEtags`.** A POSIX directory's mtime
|
||
changes when its own entry list changes and at no other time — not when a
|
||
child's contents are edited, and not for a grandchild. There is nothing to
|
||
propagate, so `dir_validator` returns `Unsupported` and the engine walks the
|
||
tree every scan. Which costs almost nothing, because the walk that was expensive
|
||
was expensive for a reason this backend does not have.
|
||
|
||
**Measured 2026-08-28**, `cargo run -p dr-sync-folder --example scan`: a full
|
||
uncached walk of 2,299 images across 233 directories completed in **137 ms**,
|
||
and 380 images across 13 directories in **29 ms** — the same engine, the same
|
||
`Depth: 1`-per-directory walk, with no pruning at all. The Nextcloud connector's
|
||
comparable figure is 34.1 s for 17,185 RAWs across 334 directories *with*
|
||
pruning available (ARCH §8.4). The capability model is what lets one engine
|
||
drive both at the speed each actually runs at, instead of forcing the fast one
|
||
down to the slow one's interface.
|
||
|
||
**Identity is a hash of the path relative to the library root**, FNV-1a 64
|
||
(written out, because `DefaultHasher` is explicitly unstable between Rust
|
||
releases and this value is written into the catalog). It gives the catalog a
|
||
`u64` that names a photograph, is the same on every device looking at the same
|
||
folder, and does not change when the file is edited. It does not survive a
|
||
rename, and `stable_ids: false` says so: a moved photograph is seen as a delete
|
||
and an add, and its thumbnail is derived again.
|
||
|
||
The alternative — keying on the inode — is stable across a rename but *differs
|
||
between devices* and is reused by the filesystem after a delete. Two machines
|
||
would disagree about which photograph a thumbnail belonged to, and a recycled
|
||
inode would silently attach an old thumbnail to a new image. Re-deriving a
|
||
thumbnail is a cost; showing the wrong one is a bug.
|
||
|
||
**Conditional writes.** `IfAbsent` is genuinely atomic (`O_CREAT | O_EXCL`).
|
||
`IfMatch` is compare-then-swap: a `stat`, then a write to a temporary beside the
|
||
destination and a `rename` over it. A POSIX filesystem has no compare-and-swap,
|
||
so the race is narrowed to the microseconds between the two syscalls rather than
|
||
closed — still far tighter than the fallback the engine uses for a backend that
|
||
declares no conditional write at all, which spans a whole read-modify-write.
|
||
The capability is declared, and the residual race is documented at the call
|
||
site.
|
||
|
||
**Two deliberate divergences from WebDAV semantics:**
|
||
|
||
- **`delete` is not recursive.** A folder library is the user's own photographs
|
||
on their own disk with no server-side trash behind it, so a caller that passed
|
||
the wrong path would have no way back. Deleting a non-empty directory returns
|
||
`RemoteError::Configuration`. Nothing in the engine deletes a directory — the
|
||
soft delete is a `move_to` into the trash folder — so the guard is free.
|
||
- **Every filesystem call runs on the blocking pool.** On a local disk that is
|
||
overkill; on the NFS mount this backend is most useful over, a stalled server
|
||
would otherwise wedge the async worker that made the call and every other
|
||
request sharing it.
|
||
|
||
**Failure classification** matters as much as the operations. A vanished mount
|
||
(`ESTALE`, `ENOTCONN`, `EIO`) maps to `RemoteError::Network`, which is what puts
|
||
the app into offline mode and leaves the catalog readable — exactly as a dead
|
||
server does. A permissions problem maps to `PermissionDenied` and does *not*,
|
||
because going offline over one forbidden file would hide a fixable problem
|
||
behind a network banner. An endpoint that is not a directory at all maps to
|
||
`RemoteError::Configuration`: nothing was unreachable and no credential was
|
||
wrong, so neither of the other two would send the user anywhere useful.
|
||
|
||
---
|
||
|
||
## 6. Virtual filesystems
|
||
|
||
A sync client in virtual-files mode leaves a **placeholder** where a file is
|
||
catalogued but not downloaded. On Linux — the only mode it supports — that
|
||
means `IMG.CR2` does not exist at all and `IMG.CR2.nextcloud` does, holding one
|
||
byte. ARCH §9.0 measured a real machine: 121,785 placeholders against 10,267
|
||
materialised files.
|
||
|
||
A folder library that ignores this is not merely degraded, it is dangerous.
|
||
Before the handling below existed, the folder connector catalogued every stub
|
||
as a 1-byte image, gave it an identity that changed the moment it was
|
||
downloaded, and — worst — reported a dehydrated *sidecar* as absent, which made
|
||
the sidecar writer create a fresh document over an existing one and discard
|
||
every edit another device had put there.
|
||
|
||
### 6.1 Three questions, one trait
|
||
|
||
Everything else about a synced folder is an ordinary directory, so this is not
|
||
a second connector. `dr_sync_folder::Vfs` asks only what differs:
|
||
|
||
```rust
|
||
pub trait Vfs: Send + Sync {
|
||
fn name(&self) -> &'static str;
|
||
fn is_placeholder(&self, on_disk: &str) -> bool;
|
||
fn real_name<'a>(&self, on_disk: &'a str) -> &'a str;
|
||
fn placeholder_name(&self, name: &str) -> Cow<'_, str>;
|
||
fn can_materialise(&self) -> bool; // defaulted false
|
||
fn materialise(&self, local: &Path) -> Result<(), RemoteError>; // defaulted
|
||
fn dematerialise(&self, local: &Path) -> Result<(), RemoteError>; // defaulted
|
||
}
|
||
```
|
||
|
||
`NoVfs` for a plain directory; `dr_sync_nextcloud::NextcloudVfs` for a synced
|
||
one, wrapping the `DesktopClient` socket. A third convention is a third impl.
|
||
|
||
**Why not a `folder-vfs` provider.** The interesting capability is not a
|
||
property of the backend: the same directory can materialise on demand while the
|
||
client is running and cannot when it is down, so it must be computed per
|
||
connection either way. Registering two providers would ask the user to choose
|
||
between two things that differ by whether a background process is up. The
|
||
convention is detected instead, per connection, by a hook the registry supplies
|
||
(`FolderProvider::with_vfs_detector`) — which is what keeps `dr-sync-folder`
|
||
free of any client's protocol.
|
||
|
||
### 6.2 What the backend reports
|
||
|
||
| | |
|
||
|---|---|
|
||
| `RemoteEntry::path` | the photograph's name, never the stub's — so identity survives a download |
|
||
| `RemoteEntry::materialised` | `false` on a stub; the catalog maps it to `Availability::Offline` |
|
||
| `RemoteEntry::size` | `0` on a stub, meaning *unknown* — see below |
|
||
| `get` on a stub | `RemoteError::NotMaterialised`, **never** `NotFound` and never the stub's one byte |
|
||
| `put` over a stub | refused; writing a rival file hands the client a conflict to resolve arbitrarily |
|
||
| `move_to` a stub | moves the stub and keeps it a stub — culling without downloading is ordinary |
|
||
| `delete` a stub | deletes it; a photograph is deleted whether or not its bytes are here |
|
||
| `capabilities().materialisation` | `OnDemand` with a client, `Placeholders` without, `Always` on a plain folder |
|
||
|
||
**Size is genuinely unknown.** A Linux suffix-mode stub is one byte and carries
|
||
no record of what it stands for. The client's `._sync_*.db` has the real size,
|
||
but that is a private schema and reading it would couple us to their migrations.
|
||
FR-NC-6c wants a transfer size quoted before an operation starts; for a stub the
|
||
honest answer is that it cannot be, and the interface should say so rather than
|
||
report one byte or invent an estimate silently.
|
||
|
||
### 6.3 Hydration is a borrow
|
||
|
||
The rule: **a file is returned to the state it was found in.** What a pass
|
||
downloaded is released; what the user already had is left alone. `BorrowPool`
|
||
enforces it.
|
||
|
||
```rust
|
||
let pool = BorrowPool::new();
|
||
{
|
||
let held = pool.borrow(&backend, &path).await?; // downloads only if absent
|
||
// ... read it, thumbnail it, index its faces ...
|
||
} // borrow ends
|
||
let stats = pool.release_all(&backend).await; // dehydrates only what it hydrated
|
||
```
|
||
|
||
Three properties that are not obvious:
|
||
|
||
- **Reference counted.** The thumbnail pass and the face pass meet on the same
|
||
RAW. Without counting, the first to finish dehydrates the file the second is
|
||
reading; with it, the transfer is paid once and released when the last
|
||
borrower is done.
|
||
- **Prior state is read before asking.** After `materialise` there is no way to
|
||
tell what the pass brought from what was already there, so it is recorded
|
||
first. Getting this wrong silently undoes a pin, and "my pinned trip
|
||
evaporated after an indexing run" is the failure that would make people stop
|
||
trusting the feature.
|
||
- **Being unsure is not symmetric.** `borrow_known(.., Some(true))` keeps a file
|
||
that might have been ours — costing disk. `Some(false)` releases one that
|
||
might have been the user's. An uncertain caller passes `true` or `None`,
|
||
never a guess at `false`.
|
||
|
||
A borrow against a plain folder or a server backend short-circuits and does
|
||
nothing, so a pass written for a VFS library runs unchanged everywhere rather
|
||
than growing two code paths.
|
||
|
||
**Measured 2026-08-29**, `cargo run -p dr-sync-folder --example vfs_cycle`: a
|
||
library of 100 photographs at 25 MB each, 90 of them dehydrated and 10 the user
|
||
keeps. A pass over all 100, borrowing and releasing as it goes:
|
||
|
||
| | |
|
||
|---|---|
|
||
| on disk at rest | 250 MB |
|
||
| **peak during the pass** | **275 MB** — the resting set plus one photograph |
|
||
| without borrowing | 2,500 MB |
|
||
| on disk afterwards | 250 MB |
|
||
| of the 10 the user already had | 10 still there |
|
||
|
||
The peak is the working set, not the library, and the release is selective.
|
||
|
||
|
||
### 6.4 Release means dehydrate, never delete
|
||
|
||
The single most dangerous thing in this feature. A synced folder is not a
|
||
cache: deleting a materialised file inside it propagates the deletion to the
|
||
server and removes the photograph from every device the user owns. `Vfs` and
|
||
`RemoteBackend::dematerialise` both say so, and an implementation that cannot
|
||
dehydrate returns `Unsupported` rather than approximating it.
|
||
|
||
This is also why the originals cache (`dr_catalog::cache`) cannot simply be
|
||
pointed at a VFS library: `Cache::release` deletes bytes, which is right for a
|
||
copy under `originals/` and catastrophic in place.
|
||
|
||
### 6.5 Which photographs stay downloaded
|
||
|
||
The user's half of the bargain: a pass borrows for a moment, but *some* of the
|
||
library should stay local — the trip you are about to take, the shoot you are
|
||
working on.
|
||
|
||
That is a **pin**, and it is the pin the originals cache already had
|
||
(`dr_catalog::cache`, FR-NC-6a). Nothing parallel was built, because the model
|
||
was already the right one:
|
||
|
||
| Cache concept | On a placeholder library |
|
||
|---|---|
|
||
| `tier_desired` | what the user asked to keep hydrated |
|
||
| `tier_actual` | what is actually materialised |
|
||
| `pending_pins()` | the work list — what to hydrate next, resumable |
|
||
| pinned rows are never evicted | a pinned collection is never dehydrated |
|
||
| passive rows, LRU under a budget | what a pass borrowed, released when it finishes |
|
||
|
||
So "keep this collection hydrated" is `Cache::pin`, and the existing pin worker
|
||
drives it — except that on a placeholder library it calls `materialise` instead
|
||
of downloading a copy.
|
||
|
||
**Why not a copy.** The original materialises *in the library folder*. Copying
|
||
it under `originals/` as well would hold every pinned photograph twice, and the
|
||
copy would be the half the budget could evict while the real disk cost stayed.
|
||
`Cache::record_in_place` records the bookkeeping with **`path = NULL`**, and
|
||
that null is load-bearing: `release` deletes the file a row names, and a row
|
||
that names none deletes nothing. The safety property is structural rather than
|
||
remembered.
|
||
|
||
Unpinning therefore frees nothing by itself — the bytes are not ours to delete.
|
||
`spawn_dehydrate` asks the client to take them back, which is what actually
|
||
returns the disk.
|
||
|
||
---
|
||
|
||
## 7. What the abstraction does not yet cover
|
||
|
||
Stated so the next person does not have to rediscover it.
|
||
|
||
- **Multiple accounts at once.** `AccountStore` holds a list and the launch
|
||
screen uses the most recent. Nothing in the model prevents two open libraries;
|
||
the interface has no place to show them.
|
||
- **Per-backend settings.** A connector has no way to contribute a settings
|
||
page. Anything configurable is on the `Account` or is not configurable.
|
||
- **Capability probing at runtime.** `Capabilities` is fixed at construction.
|
||
Nextcloud's `server_previews` should really be probed per account — a server
|
||
with `camerarawpreviews` installed can render RAW — and today it is assumed to
|
||
be `CommonFormatsOnly`.
|
||
- **A general notion of an account.** Credentials stay connector-specific on
|
||
purpose (§3.5). A third connector with an OAuth flow will need a third `SignIn`
|
||
variant, and that is the right place for it to appear.
|
||
- **A quoted cost before a hydrating pass.** FR-NC-6c wants the transfer size
|
||
stated before an operation that needs absent data. A placeholder reports no
|
||
size (§6.2), so the honest figure for "index this library" is a count and not
|
||
a byte total. The interface should say *n photographs, size unknown until
|
||
fetched* rather than estimate one silently — and it does not say anything yet.
|
||
- **Metadata-only placeholders.** Windows and macOS express these in filesystem
|
||
metadata rather than in the name, and carry the real size there. `Vfs` asks
|
||
its questions about a *name*, which is all the one convention this project has
|
||
met needs. Supporting them means widening the trait to take a `Metadata`, and
|
||
doing that before anyone has run this on those platforms would be guessing.
|
||
- **Hydration during browsing, deliberately.** It stays forbidden (ARCH §9.0
|
||
finding 3). A grid cell whose content is absent shows as not-downloaded; only
|
||
a pass the user asked for may fetch.
|