dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Make storage pluggable, and prove it with a folder backend

Browse Source 
`RemoteBackend` existed from the first release and bought nothing it was
designed for. Seven files in `dr-ui` constructed a `NextcloudBackend`
directly, an account *was* a server URL beside a DAV user id, the local
cache directory was named after a hostname, and the launch screen knew
that signing in meant a browser handshake. The trait was real; the seam
was documentation.

A trait over operations is only a quarter of it. Pluggable storage needs
four things, and this adds the other three:

- **Capabilities** — already there, and the reason the engine can drive
  two backends at the speed each actually runs at.
- **Configuration** — `dr_sync::Account`: where a library lives, in
  whatever form its connector addresses, with no server in it. Loads
  every existing config unchanged (`backend` defaults to `nextcloud`,
  `endpoint` is stored under its historical `server` key), and
  `Account::namespace()` reproduces the old catalog directory byte for
  byte, because changing it would abandon a catalog, its thumbnail
  shards, and the sidecars holding unsynced offline work.
- **Registration** — `BackendProvider` and `BackendRegistry`.
  `ui/dr-ui/src/remote.rs` is now the only file above `dr-sync` that
  names a connector.

`Connection` (an account plus an optional `Secret`) replaces the
credentials-and-user-id pair that was threaded through fifteen
signatures in an order that could be swapped. `Secret`'s inner string is
reachable only through `expose()` and its `Debug` prints `Secret(***)`,
so the indirect leak — a `{:?}` on anything holding one — no longer
compiles into a leak.

Nextcloud is unchanged and keeps every peculiarity: propagating ETags,
chunked upload v2, `oc:fileid`, the `oc:permissions` probe on a refused
PUT, the 423 retry classification, Login Flow v2. Those are what the
capability model exists to serve, not something to hide.

`dr-sync-folder` is the second connector: a local disk, a network mount,
an external drive, or a folder a Nextcloud client already syncs. No
account, no credential — the route that works where no secrets daemon
does. It declares `LocalEtags` rather than claiming propagation a POSIX
directory cannot provide, which costs nothing because 50k `stat` calls
are not 50k PROPFINDs. Identity is a path hash, not an inode: an inode
survives a rename but differs between devices and is reused after a
delete, so two machines would disagree about which photograph a
thumbnail belonged to. Re-deriving a thumbnail is a cost; showing the
wrong one is a bug.

docs/storage.md is the contract — the traits, the four steps to add a
backend, and what each connector declares. ARCH §8.0 and §8.4a, and
FR-NC-13, say why.
This commit is contained in:
dtourolle
2026-08-29 09:57:52 +02:00
parent 1b8b7998a2
commit f12aece07e
40 changed files with 3617 additions and 960 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/dr-sync-folder/src/lib.rs
+612
View File
@@ -0,0 +1,612 @@
// TRACES: FR-NC-13 | FR-NC-12
//! A library that is just a directory.
//!
//! The second [`RemoteBackend`], and the one that exists to prove the first
//! was an abstraction rather than a description. It serves a plain folder: a
//! local disk, an NFS or SMB mount, a Nextcloud desktop client's synced copy,
//! an external drive. No server, no account, no credential.
//!
//! # What it is honestly worse at, and why that is fine
//!
//! Nextcloud's fast path rests on directory ETags propagating up the tree, so
//! one request against the root proves a 50k-image library unchanged. A POSIX
//! directory's mtime says only that its own entry list changed — not that a
//! grandchild's *contents* did — so there is nothing here to propagate and
//! [`ChangeDetection::LocalEtags`] is the truthful answer. The engine reads
//! that and walks the tree every scan instead of pruning it.
//!
//! Which costs almost nothing, because the walk that was expensive was
//! expensive for a reason this backend does not have. Fifty thousand
//! `stat` calls against a local filesystem take well under a second; fifty
//! thousand `PROPFIND`s do not. The capability model is what lets both be
//! driven by the same engine at the speed each one actually runs at.
//!
//! # Identity
//!
//! [`RemoteId::Stable`] here is a hash of the path relative to the library
//! root. That gives the catalog what it needs — a `u64` that names a
//! photograph, is the same on every device looking at the same folder, and
//! does not change when the file is edited — which is what keys the thumbnail
//! shards and the face index (`catalog.md` §10.1).
//!
//! It does **not** survive a rename, and [`Capabilities::stable_ids`] says so.
//! A moved photograph is seen as a delete and an add, and its thumbnail is
//! derived again. That is the documented degradation for a backend without
//! server-assigned ids, and it is the right trade here: the alternative,
//! keying on the inode, is stable across a rename but *differs between
//! devices* and is reused by the filesystem after a delete — so two machines
//! would disagree about which photograph a thumbnail belonged to, and a
//! recycled inode would silently attach an old thumbnail to a new image.
//! Re-deriving a thumbnail is a cost; showing the wrong one is a bug.
//!
//! # Blocking
//!
//! Every filesystem call goes through the blocking pool. On a local disk that
//! is overkill; on the NFS mount this backend is most useful over, a stalled
//! server would otherwise wedge the async worker that made the call and every
//! other request sharing it.
use std::io::{Read, Seek, SeekFrom, Write};
use std::ops::Range;
use std::path::{Component, Path, PathBuf};
use async_trait::async_trait;
use dr_sync::{
Account, BackendProvider, Capabilities, ChangeDetection, Connection, Cursor, EntryKind,
Precondition, RemoteBackend, RemoteChange, RemoteEntry, RemoteError, RemoteId, RemotePath,
ServerPreviews, SignIn, Validator,
};
/// The id written to [`Account::backend`] for a folder library.
///
/// On-disk configuration: changing it orphans every folder account.
pub const BACKEND_ID: &str = "folder";
/// TRACES: FR-NC-13
/// Registers the folder connector.
///
/// See [`dr_sync::provider`] for what each method is for.
pub struct FolderProvider;
impl BackendProvider for FolderProvider {
fn id(&self) -> &'static str {
BACKEND_ID
}
fn display_name(&self) -> &'static str {
"Folder"
}
fn endpoint_label(&self) -> &'static str {
"Folder"
}
fn endpoint_placeholder(&self) -> &'static str {
"/home/you/Pictures"
}
fn sign_in(&self) -> SignIn {
SignIn::EndpointOnly
}
/// Check the directory before an account is written for it.
///
/// A typo here would otherwise be stored, skip the launch screen on the
/// next start, and surface as a scan that finds nothing — which reads as
/// a broken library rather than a wrong path. The messages say what to fix.
fn normalise_endpoint(&self, input: &str) -> Result<String, String> {
let trimmed = input.trim();
if trimmed.is_empty() {
return Err("Choose the folder your photographs are in.".into());
}
// `~` is what a person types and what a shell would have expanded;
// nothing expands it here, so a stored `~/Pictures` becomes a
// directory literally named `~`.
let expanded = match trimmed.strip_prefix("~/") {
Some(rest) => match std::env::var_os("HOME") {
Some(home) => PathBuf::from(home).join(rest),
None => return Err("No home directory to expand ~ against.".into()),
},
None => PathBuf::from(trimmed),
};
if !expanded.is_absolute() {
return Err("Give the full path to the folder, starting at /.".into());
}
if !expanded.exists() {
return Err(format!("No folder at {}.", expanded.display()));
}
if !expanded.is_dir() {
return Err(format!("{} is a file, not a folder.", expanded.display()));
}
// Resolved so a library reached through a symlink or a `..` is stored
// under one name. Two spellings of one folder would otherwise be two
// accounts with two catalogs indexing the same photographs.
let canonical = expanded
.canonicalize()
.map_err(|e| format!("Cannot read {}: {e}", expanded.display()))?;
Ok(canonical.to_string_lossy().into_owned())
}
fn account_for(&self, endpoint: &str) -> Result<Account, RemoteError> {
Ok(Account::new(BACKEND_ID, endpoint))
}
fn connect(&self, conn: &Connection) -> Result<Box<dyn RemoteBackend>, RemoteError> {
Ok(Box::new(FolderBackend::new(&conn.account.endpoint)?))
}
}
/// TRACES: FR-NC-13 | FR-NC-4
/// A library rooted at a directory.
#[derive(Debug, Clone)]
pub struct FolderBackend {
root: PathBuf,
caps: Capabilities,
}
impl FolderBackend {
/// Open the folder at `root`.
///
/// The directory must exist now. It may stop existing later — a drive
/// unplugged, a mount dropped — and that surfaces per-operation as
/// [`RemoteError::Network`], which is what puts the app into offline mode
/// and leaves the catalog readable, exactly as a dead server does.
pub fn new(root: impl Into<PathBuf>) -> Result<Self, RemoteError> {
let root = root.into();
if !root.is_dir() {
return Err(RemoteError::Configuration(format!(
"{} is not a folder",
root.display()
)));
}
Ok(Self {
root,
caps: Capabilities {
// A directory's mtime describes its own entry list and nothing
// below it, so there is no propagation to exploit; the engine
// walks and compares per entry.
change_detection: ChangeDetection::LocalEtags,
// A path hash does not survive a rename. See the module docs
// for why the inode is not used instead.
stable_ids: false,
range_reads: true,
// Not a protocol with a message size limit; a write is a write.
chunked_upload: None,
bulk_upload: false,
conditional_write: true,
server_previews: ServerPreviews::None,
},
})
}
pub fn root(&self) -> &Path {
&self.root
}
/// The local path for a remote path, refusing anything that escapes.
///
/// The guard is not theoretical. A `RemotePath` is built from strings that
/// reach us from a catalog written by another device and from filenames on
/// the remote itself, and this backend resolves them against a real
/// filesystem with the user's own permissions. `../../.ssh/id_ed25519` is
/// a legal path segment; without this it would be a legal *read*.
fn resolve(&self, path: &RemotePath) -> Result<PathBuf, RemoteError> {
let rel = Path::new(path.as_str());
for component in rel.components() {
match component {
Component::Normal(_) => {}
Component::CurDir => {}
Component::ParentDir | Component::RootDir | Component::Prefix(_) => {
return Err(RemoteError::Configuration(format!(
"{path} leaves the library folder"
)));
}
}
}
Ok(self.root.join(rel))
}
/// The local path a [`RemoteId`] names.
///
/// A stable id here is a hash and nothing can be resolved from it, exactly
/// as a Nextcloud `oc:fileid` names no WebDAV endpoint. Callers hold the
/// path alongside it in the catalog and pass that.
fn resolve_id(&self, id: &RemoteId) -> Result<PathBuf, RemoteError> {
match id {
RemoteId::Path(p) => self.resolve(p),
RemoteId::Stable(_) => Err(RemoteError::Unsupported(
"a folder cannot be addressed by id; use RemoteId::Path",
)),
}
}
}
/// Run a filesystem operation off the async worker that asked for it.
///
/// See the module docs: a stalled network mount must not take the caller's
/// runtime with it.
async fn blocking<T, F>(f: F) -> Result<T, RemoteError>
where
F: FnOnce() -> Result<T, RemoteError> + Send + 'static,
T: Send + 'static,
{
match tokio::task::spawn_blocking(f).await {
Ok(r) => r,
// The only way a blocking task fails to produce a result is a panic
// inside it, which is a bug here rather than a condition the caller
// can act on — but crashing the worker over it would lose a whole
// scan, so it is reported like any other failure.
Err(e) => Err(RemoteError::Protocol(format!("folder task failed: {e}"))),
}
}
/// Map an IO failure to the error the engine already knows how to handle.
///
/// The classification is the point. [`RemoteError::indicates_offline`] drives
/// offline mode, so a vanished mount must reach it as `Network` — that is
/// precisely the "the library is unreachable, keep working from the catalog"
/// case — while a permissions problem must not, because going offline over one
/// forbidden file would hide a fixable problem behind a network banner.
fn map_io(e: std::io::Error, what: &str) -> RemoteError {
use std::io::ErrorKind as K;
match e.kind() {
K::NotFound => RemoteError::NotFound(what.to_string()),
K::PermissionDenied => RemoteError::PermissionDenied,
K::AlreadyExists => RemoteError::PreconditionFailed,
// ENOSPC and friends. Quota is what the engine calls "no room".
K::StorageFull | K::QuotaExceeded | K::FileTooLarge => RemoteError::QuotaExceeded,
// A dropped mount answers ESTALE/EIO/ENOTCONN, and the honest reading
// is the same as a dead server: the library cannot be reached now, and
// may be again shortly.
K::HostUnreachable
| K::NetworkUnreachable
| K::NetworkDown
| K::ConnectionAborted
| K::ConnectionReset
| K::NotConnected
| K::BrokenPipe
| K::TimedOut => RemoteError::Network(format!("{what}: {e}")),
_ => RemoteError::Protocol(format!("{what}: {e}")),
}
}
/// The identity of a file, from its path relative to the library root.
///
/// FNV-1a rather than `DefaultHasher`, whose output is explicitly unstable
/// between Rust releases: this value is written into the catalog and into the
/// thumbnail index, and must mean the same thing after a toolchain upgrade as
/// it did before one.
fn identity(path: &RemotePath) -> u64 {
let mut h: u64 = 0xcbf2_9ce4_8422_2325;
for b in path.as_str().as_bytes() {
h ^= *b as u64;
h = h.wrapping_mul(0x0000_0100_0000_01b3);
}
h
}
/// A file's validator: its size and modification time.
///
/// The pair, not either alone. An mtime with one-second granularity — which is
/// what some filesystems and most network mounts report — cannot distinguish
/// two writes in the same second, and a size alone cannot see an edit that
/// preserved it. Together they miss only a same-second write of identical
/// length, which for a photograph is a rewrite of the same frame.
fn validator_of(meta: &std::fs::Metadata) -> Validator {
let (secs, nanos) = meta
.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| (d.as_secs(), d.subsec_nanos()))
.unwrap_or((0, 0));
Validator::new(format!("{:x}-{:x}.{:x}", meta.len(), secs, nanos))
}
fn modified_secs(meta: &std::fs::Metadata) -> Option<i64> {
meta.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| d.as_secs() as i64)
}
#[async_trait]
impl RemoteBackend for FolderBackend {
fn capabilities(&self) -> &Capabilities {
&self.caps
}
fn name(&self) -> &str {
"Folder"
}
async fn list(
&self,
dir: &RemotePath,
_since: Option<&Validator>,
) -> Result<Vec<RemoteEntry>, RemoteError> {
let local = self.resolve(dir)?;
let dir = dir.clone();
blocking(move || {
let read =
std::fs::read_dir(&local).map_err(|e| map_io(e, &local.display().to_string()))?;
let mut out = Vec::new();
for entry in read {
let entry = match entry {
Ok(e) => e,
// One unreadable entry must not fail the listing: a
// scan of a real library meets a broken symlink or a
// file being written, and abandoning the whole
// directory over it loses every photograph beside it.
Err(e) => {
log::debug!("skipping an entry in {}: {e}", local.display());
continue;
}
};
let name = entry.file_name();
let Some(name) = name.to_str() else {
// A name that is not UTF-8 cannot round-trip through a
// `RemotePath`, and quietly mangling it would produce a
// path that addresses a different file — or none.
log::warn!("skipping a non-UTF-8 name in {}", local.display());
continue;
};
// `metadata`, not `symlink_metadata`: a symlinked shoot
// folder is a normal way to assemble a library, and the
// scan's depth limit is what stops a loop.
let meta = match entry.metadata() {
Ok(m) => m,
Err(e) => {
log::debug!("skipping {name}: {e}");
continue;
}
};
let path = dir.join(name);
out.push(RemoteEntry {
id: RemoteId::Stable(identity(&path)),
kind: if meta.is_dir() {
EntryKind::Directory
} else {
EntryKind::File
},
validator: validator_of(&meta),
size: meta.len(),
modified: modified_secs(&meta),
// No renderer behind a folder; previews are extracted
// locally from the file itself.
has_preview: false,
path,
});
}
Ok(out)
})
.await
}
/// Not offered.
///
/// A directory's mtime changes when its own entries are added or removed
/// and at no other time, so it cannot answer the question this method
/// exists for — "did anything below here change?". Returning it anyway
/// would let a future caller prune a subtree whose contents had been
/// edited, and hide those edits for as long as the folder list held still.
async fn dir_validator(&self, _dir: &RemotePath) -> Result<Validator, RemoteError> {
Err(RemoteError::Unsupported(
"a folder's mtime does not propagate; use per-entry validators",
))
}
async fn delta(&self, _cursor: &Cursor) -> Result<(Vec<RemoteChange>, Cursor), RemoteError> {
Err(RemoteError::Unsupported("a folder keeps no change feed"))
}
async fn get(&self, id: &RemoteId, range: Option<Range<u64>>) -> Result<Vec<u8>, RemoteError> {
let local = self.resolve_id(id)?;
blocking(move || {
let what = local.display().to_string();
let mut file = std::fs::File::open(&local).map_err(|e| map_io(e, &what))?;
let Some(r) = range else {
let mut buf = Vec::new();
file.read_to_end(&mut buf).map_err(|e| map_io(e, &what))?;
return Ok(buf);
};
// A short read at the end of the file is not an error: the header
// extractor asks for a fixed window and the file may be smaller
// than it, which is the ordinary case for a small JPEG.
file.seek(SeekFrom::Start(r.start))
.map_err(|e| map_io(e, &what))?;
let want = r.end.saturating_sub(r.start);
let mut buf = Vec::new();
file.take(want)
.read_to_end(&mut buf)
.map_err(|e| map_io(e, &what))?;
Ok(buf)
})
.await
}
async fn put(
&self,
path: &RemotePath,
body: Vec<u8>,
precond: Option<Precondition>,
) -> Result<Validator, RemoteError> {
let local = self.resolve(path)?;
blocking(move || {
let what = local.display().to_string();
if let Some(parent) = local.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| map_io(e, &parent.display().to_string()))?;
}
match &precond {
// Genuinely atomic: `O_CREAT | O_EXCL` is one syscall, so two
// devices racing to create a sidecar cannot both win.
Some(Precondition::IfAbsent) => {
let mut f = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(&local)
.map_err(|e| map_io(e, &what))?;
f.write_all(&body).map_err(|e| map_io(e, &what))?;
f.sync_all().map_err(|e| map_io(e, &what))?;
let meta = f.metadata().map_err(|e| map_io(e, &what))?;
return Ok(validator_of(&meta));
}
// Compare, then swap. A POSIX filesystem has no compare-and-
// swap, so this narrows the window to the microseconds between
// the `stat` and the `rename` rather than closing it. That is
// still far tighter than the fallback the engine uses when a
// backend declares no conditional write at all — comparing
// revision counters *inside* the sidecar, which spans a whole
// read-modify-write — which is why the capability is declared
// rather than refused.
Some(Precondition::IfMatch(expected)) => {
let meta = std::fs::metadata(&local).map_err(|e| map_io(e, &what))?;
if &validator_of(&meta) != expected {
return Err(RemoteError::PreconditionFailed);
}
}
None => {}
}
// Write beside the destination and rename over it, so a reader
// never sees a half-written sidecar and an interrupted write
// cannot destroy the file it was replacing. Beside, not in
// `/tmp`: a rename across filesystems is not atomic, and on
// Android `/tmp` is a different one.
let tmp = local.with_extension(format!(
"{}.darkroom-tmp",
local.extension().and_then(|e| e.to_str()).unwrap_or("")
));
let write = (|| -> Result<(), RemoteError> {
let mut f = std::fs::File::create(&tmp).map_err(|e| map_io(e, &what))?;
f.write_all(&body).map_err(|e| map_io(e, &what))?;
f.sync_all().map_err(|e| map_io(e, &what))
})();
if let Err(e) = write {
let _ = std::fs::remove_file(&tmp);
return Err(e);
}
if let Err(e) = std::fs::rename(&tmp, &local) {
let _ = std::fs::remove_file(&tmp);
return Err(map_io(e, &what));
}
let meta = std::fs::metadata(&local).map_err(|e| map_io(e, &what))?;
Ok(validator_of(&meta))
})
.await
}
/// Delete a file, or an empty directory.
///
/// **Not recursive, unlike WebDAV's `DELETE` on a collection.** The
/// divergence is deliberate: a folder library is the user's own
/// photographs on their own disk, with no server-side trash behind it, so
/// a caller that passed the wrong path would have no way back. Nothing in
/// the engine deletes a directory — the soft delete is a
/// [`move_to`](RemoteBackend::move_to) into the trash folder — so refusing
/// costs nothing and the guard is free.
async fn delete(
&self,
id: &RemoteId,
precond: Option<Precondition>,
) -> Result<(), RemoteError> {
let local = self.resolve_id(id)?;
blocking(move || {
let what = local.display().to_string();
let meta = std::fs::symlink_metadata(&local).map_err(|e| map_io(e, &what))?;
match &precond {
Some(Precondition::IfMatch(expected)) => {
if &validator_of(&meta) != expected {
return Err(RemoteError::PreconditionFailed);
}
}
// "Delete only if nothing is there" is not a thing to ask of a
// delete; something is there or the `stat` above already
// failed.
Some(Precondition::IfAbsent) => {
return Err(RemoteError::Unsupported(
"IfAbsent is not meaningful on a delete",
))
}
None => {}
}
if meta.is_dir() {
std::fs::remove_dir(&local).map_err(|e| {
if e.kind() == std::io::ErrorKind::DirectoryNotEmpty {
RemoteError::Configuration(format!(
"{what} is not empty; a folder library will not delete a tree"
))
} else {
map_io(e, &what)
}
})
} else {
std::fs::remove_file(&local).map_err(|e| map_io(e, &what))
}
})
.await
}
async fn move_to(&self, from: &RemoteId, to: &RemotePath) -> Result<(), RemoteError> {
let src = self.resolve_id(from)?;
let dst = self.resolve(to)?;
blocking(move || {
let what = dst.display().to_string();
// Parents first: the trash folder does not exist until the first
// photograph is trashed, and the trait promises this creates it.
if let Some(parent) = dst.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| map_io(e, &parent.display().to_string()))?;
}
match std::fs::rename(&src, &dst) {
Ok(()) => Ok(()),
// EXDEV. Both paths are inside one library root, so this
// needs a root that spans a mount point — a shoot folder
// that is its own mount, which is an ordinary way to attach
// an archive drive. Copy and unlink rather than refusing:
// the identity a rename would have preserved is a path hash
// here, and it changes either way.
Err(e) if e.raw_os_error() == Some(18) => {
std::fs::copy(&src, &dst).map_err(|e| map_io(e, &what))?;
std::fs::remove_file(&src).map_err(|e| {
// The copy landed. Leaving the original is a
// duplicate, which the next scan will show; losing
// the copy would be worse.
let _ = std::fs::remove_file(&dst);
map_io(e, &src.display().to_string())
})
}
Err(e) => Err(map_io(e, &what)),
}
})
.await
}
async fn create_dir(&self, path: &RemotePath) -> Result<(), RemoteError> {
let local = self.resolve(path)?;
blocking(move || {
// `create_dir_all` makes parents and succeeds on one that already
// exists, which is exactly the contract.
std::fs::create_dir_all(&local).map_err(|e| map_io(e, &local.display().to_string()))
})
.await
}
}
#[cfg(test)]
mod tests;