Treat a placeholder as the photograph, not as a one-byte file

The folder connector was pointed at a Nextcloud VFS tree and got three
things wrong, the first of which loses work.

**A dehydrated sidecar read as absent.** `a.drsc` does not exist when the
client has dehydrated it — only `a.drsc.nextcloud` does — so `get` missed,
`.ok()` swallowed the `NotFound`, and the sidecar writer took that for
"there is no sidecar yet" and wrote a fresh document over the existing
one. Every edit another device had put there went with it. That function's
own doc comment calls this the exact loss the format's unknown-key
preservation exists to prevent.

**A stub was catalogued as a 1-byte image**, and ARCH §9.0 measured this
machine at 121,785 placeholders against 10,267 real files — so a folder
library on a synced tree was ~92% broken rows.

**Identity changed on hydration**, so downloading a photograph looked like
a delete and an add, orphaning its thumbnail and its face rows.

Entries now carry the photograph's own name and a `materialised` flag;
`get` on a stub returns the new `RemoteError::NotMaterialised`, which is
distinct from `NotFound` precisely because the sidecar writer must treat
them differently — it fetches the sidecar and merges, or leaves the entry
queued.

Hydration is a **borrow**. `BorrowPool` records what was on disk before it
asked, so `release_all` dehydrates only what a pass brought and leaves
what the user already had. Reference counted: the thumbnail pass and the
face pass meet on the same RAW, and without counting the first to finish
dehydrates the file the second is reading. A borrow against a plain folder
or a server does nothing, so a pass written for VFS runs everywhere.

Releasing means asking the client to dehydrate and never deleting: a
deletion inside a synced tree propagates to the server and removes the
photograph from every device.

Not a second backend — the capability is per *connection*, not per type,
since the same folder hydrates only while the client runs. The convention
arrives through a detector the registry supplies, so `dr-sync-folder`
still knows nothing about any client's protocol.

ARCH §9.0a records this as an amendment: finding 3 rejected hydration
because it costs 100× a range read, and that comparison assumed a
connector was available. A folder library has none.
This commit is contained in:
2026-08-29 09:57:52 +02:00
parent 6c363cee97
commit c102ba9df2
22 changed files with 1555 additions and 53 deletions
+114 -1
View File
@@ -366,7 +366,120 @@ wrong, so neither of the other two would send the user anywhere useful.
---
## 6. What the abstraction does not yet cover
## 6. Virtual filesystems
A sync client in virtual-files mode leaves a **placeholder** where a file is
catalogued but not downloaded. On Linux — the only mode it supports — that
means `IMG.CR2` does not exist at all and `IMG.CR2.nextcloud` does, holding one
byte. ARCH §9.0 measured a real machine: 121,785 placeholders against 10,267
materialised files.
A folder library that ignores this is not merely degraded, it is dangerous.
Before the handling below existed, the folder connector catalogued every stub
as a 1-byte image, gave it an identity that changed the moment it was
downloaded, and — worst — reported a dehydrated *sidecar* as absent, which made
the sidecar writer create a fresh document over an existing one and discard
every edit another device had put there.
### 6.1 Three questions, one trait
Everything else about a synced folder is an ordinary directory, so this is not
a second connector. `dr_sync_folder::Vfs` asks only what differs:
```rust
pub trait Vfs: Send + Sync {
fn name(&self) -> &'static str;
fn is_placeholder(&self, on_disk: &str) -> bool;
fn real_name<'a>(&self, on_disk: &'a str) -> &'a str;
fn placeholder_name(&self, name: &str) -> Cow<'_, str>;
fn can_materialise(&self) -> bool; // defaulted false
fn materialise(&self, local: &Path) -> Result<(), RemoteError>; // defaulted
fn dematerialise(&self, local: &Path) -> Result<(), RemoteError>; // defaulted
}
```
`NoVfs` for a plain directory; `dr_sync_nextcloud::NextcloudVfs` for a synced
one, wrapping the `DesktopClient` socket. A third convention is a third impl.
**Why not a `folder-vfs` provider.** The interesting capability is not a
property of the backend: the same directory can materialise on demand while the
client is running and cannot when it is down, so it must be computed per
connection either way. Registering two providers would ask the user to choose
between two things that differ by whether a background process is up. The
convention is detected instead, per connection, by a hook the registry supplies
(`FolderProvider::with_vfs_detector`) — which is what keeps `dr-sync-folder`
free of any client's protocol.
### 6.2 What the backend reports
| | |
|---|---|
| `RemoteEntry::path` | the photograph's name, never the stub's — so identity survives a download |
| `RemoteEntry::materialised` | `false` on a stub; the catalog maps it to `Availability::Offline` |
| `RemoteEntry::size` | `0` on a stub, meaning *unknown* — see below |
| `get` on a stub | `RemoteError::NotMaterialised`, **never** `NotFound` and never the stub's one byte |
| `put` over a stub | refused; writing a rival file hands the client a conflict to resolve arbitrarily |
| `move_to` a stub | moves the stub and keeps it a stub — culling without downloading is ordinary |
| `delete` a stub | deletes it; a photograph is deleted whether or not its bytes are here |
| `capabilities().materialisation` | `OnDemand` with a client, `Placeholders` without, `Always` on a plain folder |
**Size is genuinely unknown.** A Linux suffix-mode stub is one byte and carries
no record of what it stands for. The client's `._sync_*.db` has the real size,
but that is a private schema and reading it would couple us to their migrations.
FR-NC-6c wants a transfer size quoted before an operation starts; for a stub the
honest answer is that it cannot be, and the interface should say so rather than
report one byte or invent an estimate silently.
### 6.3 Hydration is a borrow
The rule: **a file is returned to the state it was found in.** What a pass
downloaded is released; what the user already had is left alone. `BorrowPool`
enforces it.
```rust
let pool = BorrowPool::new();
{
let held = pool.borrow(&backend, &path).await?; // downloads only if absent
// ... read it, thumbnail it, index its faces ...
} // borrow ends
let stats = pool.release_all(&backend).await; // dehydrates only what it hydrated
```
Three properties that are not obvious:
- **Reference counted.** The thumbnail pass and the face pass meet on the same
RAW. Without counting, the first to finish dehydrates the file the second is
reading; with it, the transfer is paid once and released when the last
borrower is done.
- **Prior state is read before asking.** After `materialise` there is no way to
tell what the pass brought from what was already there, so it is recorded
first. Getting this wrong silently undoes a pin, and "my pinned trip
evaporated after an indexing run" is the failure that would make people stop
trusting the feature.
- **Being unsure is not symmetric.** `borrow_known(.., Some(true))` keeps a file
that might have been ours — costing disk. `Some(false)` releases one that
might have been the user's. An uncertain caller passes `true` or `None`,
never a guess at `false`.
A borrow against a plain folder or a server backend short-circuits and does
nothing, so a pass written for a VFS library runs unchanged everywhere rather
than growing two code paths.
### 6.4 Release means dehydrate, never delete
The single most dangerous thing in this feature. A synced folder is not a
cache: deleting a materialised file inside it propagates the deletion to the
server and removes the photograph from every device the user owns. `Vfs` and
`RemoteBackend::dematerialise` both say so, and an implementation that cannot
dehydrate returns `Unsupported` rather than approximating it.
This is also why the originals cache (`dr_catalog::cache`) cannot simply be
pointed at a VFS library: `Cache::release` deletes bytes, which is right for a
copy under `originals/` and catastrophic in place.
---
## 7. What the abstraction does not yet cover
Stated so the next person does not have to rediscover it.