Treat a placeholder as the photograph, not as a one-byte file
The folder connector was pointed at a Nextcloud VFS tree and got three things wrong, the first of which loses work. **A dehydrated sidecar read as absent.** `a.drsc` does not exist when the client has dehydrated it — only `a.drsc.nextcloud` does — so `get` missed, `.ok()` swallowed the `NotFound`, and the sidecar writer took that for "there is no sidecar yet" and wrote a fresh document over the existing one. Every edit another device had put there went with it. That function's own doc comment calls this the exact loss the format's unknown-key preservation exists to prevent. **A stub was catalogued as a 1-byte image**, and ARCH §9.0 measured this machine at 121,785 placeholders against 10,267 real files — so a folder library on a synced tree was ~92% broken rows. **Identity changed on hydration**, so downloading a photograph looked like a delete and an add, orphaning its thumbnail and its face rows. Entries now carry the photograph's own name and a `materialised` flag; `get` on a stub returns the new `RemoteError::NotMaterialised`, which is distinct from `NotFound` precisely because the sidecar writer must treat them differently — it fetches the sidecar and merges, or leaves the entry queued. Hydration is a **borrow**. `BorrowPool` records what was on disk before it asked, so `release_all` dehydrates only what a pass brought and leaves what the user already had. Reference counted: the thumbnail pass and the face pass meet on the same RAW, and without counting the first to finish dehydrates the file the second is reading. A borrow against a plain folder or a server does nothing, so a pass written for VFS runs everywhere. Releasing means asking the client to dehydrate and never deleting: a deletion inside a synced tree propagates to the server and removes the photograph from every device. Not a second backend — the capability is per *connection*, not per type, since the same folder hydrates only while the client runs. The convention arrives through a detector the registry supplies, so `dr-sync-folder` still knows nothing about any client's protocol. ARCH §9.0a records this as an amendment: finding 3 rejected hydration because it costs 100× a range read, and that comparison assumed a connector was available. A folder library has none.
This commit is contained in:
+114
-1
@@ -366,7 +366,120 @@ wrong, so neither of the other two would send the user anywhere useful.
|
||||
|
||||
---
|
||||
|
||||
## 6. What the abstraction does not yet cover
|
||||
## 6. Virtual filesystems
|
||||
|
||||
A sync client in virtual-files mode leaves a **placeholder** where a file is
|
||||
catalogued but not downloaded. On Linux — the only mode it supports — that
|
||||
means `IMG.CR2` does not exist at all and `IMG.CR2.nextcloud` does, holding one
|
||||
byte. ARCH §9.0 measured a real machine: 121,785 placeholders against 10,267
|
||||
materialised files.
|
||||
|
||||
A folder library that ignores this is not merely degraded, it is dangerous.
|
||||
Before the handling below existed, the folder connector catalogued every stub
|
||||
as a 1-byte image, gave it an identity that changed the moment it was
|
||||
downloaded, and — worst — reported a dehydrated *sidecar* as absent, which made
|
||||
the sidecar writer create a fresh document over an existing one and discard
|
||||
every edit another device had put there.
|
||||
|
||||
### 6.1 Three questions, one trait
|
||||
|
||||
Everything else about a synced folder is an ordinary directory, so this is not
|
||||
a second connector. `dr_sync_folder::Vfs` asks only what differs:
|
||||
|
||||
```rust
|
||||
pub trait Vfs: Send + Sync {
|
||||
fn name(&self) -> &'static str;
|
||||
fn is_placeholder(&self, on_disk: &str) -> bool;
|
||||
fn real_name<'a>(&self, on_disk: &'a str) -> &'a str;
|
||||
fn placeholder_name(&self, name: &str) -> Cow<'_, str>;
|
||||
fn can_materialise(&self) -> bool; // defaulted false
|
||||
fn materialise(&self, local: &Path) -> Result<(), RemoteError>; // defaulted
|
||||
fn dematerialise(&self, local: &Path) -> Result<(), RemoteError>; // defaulted
|
||||
}
|
||||
```
|
||||
|
||||
`NoVfs` for a plain directory; `dr_sync_nextcloud::NextcloudVfs` for a synced
|
||||
one, wrapping the `DesktopClient` socket. A third convention is a third impl.
|
||||
|
||||
**Why not a `folder-vfs` provider.** The interesting capability is not a
|
||||
property of the backend: the same directory can materialise on demand while the
|
||||
client is running and cannot when it is down, so it must be computed per
|
||||
connection either way. Registering two providers would ask the user to choose
|
||||
between two things that differ by whether a background process is up. The
|
||||
convention is detected instead, per connection, by a hook the registry supplies
|
||||
(`FolderProvider::with_vfs_detector`) — which is what keeps `dr-sync-folder`
|
||||
free of any client's protocol.
|
||||
|
||||
### 6.2 What the backend reports
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `RemoteEntry::path` | the photograph's name, never the stub's — so identity survives a download |
|
||||
| `RemoteEntry::materialised` | `false` on a stub; the catalog maps it to `Availability::Offline` |
|
||||
| `RemoteEntry::size` | `0` on a stub, meaning *unknown* — see below |
|
||||
| `get` on a stub | `RemoteError::NotMaterialised`, **never** `NotFound` and never the stub's one byte |
|
||||
| `put` over a stub | refused; writing a rival file hands the client a conflict to resolve arbitrarily |
|
||||
| `move_to` a stub | moves the stub and keeps it a stub — culling without downloading is ordinary |
|
||||
| `delete` a stub | deletes it; a photograph is deleted whether or not its bytes are here |
|
||||
| `capabilities().materialisation` | `OnDemand` with a client, `Placeholders` without, `Always` on a plain folder |
|
||||
|
||||
**Size is genuinely unknown.** A Linux suffix-mode stub is one byte and carries
|
||||
no record of what it stands for. The client's `._sync_*.db` has the real size,
|
||||
but that is a private schema and reading it would couple us to their migrations.
|
||||
FR-NC-6c wants a transfer size quoted before an operation starts; for a stub the
|
||||
honest answer is that it cannot be, and the interface should say so rather than
|
||||
report one byte or invent an estimate silently.
|
||||
|
||||
### 6.3 Hydration is a borrow
|
||||
|
||||
The rule: **a file is returned to the state it was found in.** What a pass
|
||||
downloaded is released; what the user already had is left alone. `BorrowPool`
|
||||
enforces it.
|
||||
|
||||
```rust
|
||||
let pool = BorrowPool::new();
|
||||
{
|
||||
let held = pool.borrow(&backend, &path).await?; // downloads only if absent
|
||||
// ... read it, thumbnail it, index its faces ...
|
||||
} // borrow ends
|
||||
let stats = pool.release_all(&backend).await; // dehydrates only what it hydrated
|
||||
```
|
||||
|
||||
Three properties that are not obvious:
|
||||
|
||||
- **Reference counted.** The thumbnail pass and the face pass meet on the same
|
||||
RAW. Without counting, the first to finish dehydrates the file the second is
|
||||
reading; with it, the transfer is paid once and released when the last
|
||||
borrower is done.
|
||||
- **Prior state is read before asking.** After `materialise` there is no way to
|
||||
tell what the pass brought from what was already there, so it is recorded
|
||||
first. Getting this wrong silently undoes a pin, and "my pinned trip
|
||||
evaporated after an indexing run" is the failure that would make people stop
|
||||
trusting the feature.
|
||||
- **Being unsure is not symmetric.** `borrow_known(.., Some(true))` keeps a file
|
||||
that might have been ours — costing disk. `Some(false)` releases one that
|
||||
might have been the user's. An uncertain caller passes `true` or `None`,
|
||||
never a guess at `false`.
|
||||
|
||||
A borrow against a plain folder or a server backend short-circuits and does
|
||||
nothing, so a pass written for a VFS library runs unchanged everywhere rather
|
||||
than growing two code paths.
|
||||
|
||||
### 6.4 Release means dehydrate, never delete
|
||||
|
||||
The single most dangerous thing in this feature. A synced folder is not a
|
||||
cache: deleting a materialised file inside it propagates the deletion to the
|
||||
server and removes the photograph from every device the user owns. `Vfs` and
|
||||
`RemoteBackend::dematerialise` both say so, and an implementation that cannot
|
||||
dehydrate returns `Unsupported` rather than approximating it.
|
||||
|
||||
This is also why the originals cache (`dr_catalog::cache`) cannot simply be
|
||||
pointed at a VFS library: `Cache::release` deletes bytes, which is right for a
|
||||
copy under `originals/` and catastrophic in place.
|
||||
|
||||
---
|
||||
|
||||
## 7. What the abstraction does not yet cover
|
||||
|
||||
Stated so the next person does not have to rediscover it.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user