dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Make storage pluggable, and prove it with a folder backend

Browse Source 
`RemoteBackend` existed from the first release and bought nothing it was
designed for. Seven files in `dr-ui` constructed a `NextcloudBackend`
directly, an account *was* a server URL beside a DAV user id, the local
cache directory was named after a hostname, and the launch screen knew
that signing in meant a browser handshake. The trait was real; the seam
was documentation.

A trait over operations is only a quarter of it. Pluggable storage needs
four things, and this adds the other three:

- **Capabilities** — already there, and the reason the engine can drive
  two backends at the speed each actually runs at.
- **Configuration** — `dr_sync::Account`: where a library lives, in
  whatever form its connector addresses, with no server in it. Loads
  every existing config unchanged (`backend` defaults to `nextcloud`,
  `endpoint` is stored under its historical `server` key), and
  `Account::namespace()` reproduces the old catalog directory byte for
  byte, because changing it would abandon a catalog, its thumbnail
  shards, and the sidecars holding unsynced offline work.
- **Registration** — `BackendProvider` and `BackendRegistry`.
  `ui/dr-ui/src/remote.rs` is now the only file above `dr-sync` that
  names a connector.

`Connection` (an account plus an optional `Secret`) replaces the
credentials-and-user-id pair that was threaded through fifteen
signatures in an order that could be swapped. `Secret`'s inner string is
reachable only through `expose()` and its `Debug` prints `Secret(***)`,
so the indirect leak — a `{:?}` on anything holding one — no longer
compiles into a leak.

Nextcloud is unchanged and keeps every peculiarity: propagating ETags,
chunked upload v2, `oc:fileid`, the `oc:permissions` probe on a refused
PUT, the 423 retry classification, Login Flow v2. Those are what the
capability model exists to serve, not something to hide.

`dr-sync-folder` is the second connector: a local disk, a network mount,
an external drive, or a folder a Nextcloud client already syncs. No
account, no credential — the route that works where no secrets daemon
does. It declares `LocalEtags` rather than claiming propagation a POSIX
directory cannot provide, which costs nothing because 50k `stat` calls
are not 50k PROPFINDs. Identity is a path hash, not an inode: an inode
survives a rename but differs between devices and is reused after a
delete, so two machines would disagree about which photograph a
thumbnail belonged to. Re-deriving a thumbnail is a cost; showing the
wrong one is a bug.

docs/storage.md is the contract — the traits, the four steps to add a
backend, and what each connector declares. ARCH §8.0 and §8.4a, and
FR-NC-13, say why.
This commit is contained in:
dtourolle
2026-08-29 09:57:52 +02:00
parent 1b8b7998a2
commit f12aece07e
40 changed files with 3617 additions and 960 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/dr-sync-folder/src/lib.rs
+612
View File
@@ -0,0 +1,612 @@
// TRACES: FR-NC-13 | FR-NC-12
//! A library that is just a directory.
//!
//! The second [`RemoteBackend`], and the one that exists to prove the first
//! was an abstraction rather than a description. It serves a plain folder: a
//! local disk, an NFS or SMB mount, a Nextcloud desktop client's synced copy,
//! an external drive. No server, no account, no credential.
//!
//! # What it is honestly worse at, and why that is fine
//!
//! Nextcloud's fast path rests on directory ETags propagating up the tree, so
//! one request against the root proves a 50k-image library unchanged. A POSIX
//! directory's mtime says only that its own entry list changed — not that a
//! grandchild's *contents* did — so there is nothing here to propagate and
//! [`ChangeDetection::LocalEtags`] is the truthful answer. The engine reads
//! that and walks the tree every scan instead of pruning it.
//!
//! Which costs almost nothing, because the walk that was expensive was
//! expensive for a reason this backend does not have. Fifty thousand
//! `stat` calls against a local filesystem take well under a second; fifty
//! thousand `PROPFIND`s do not. The capability model is what lets both be
//! driven by the same engine at the speed each one actually runs at.
//!
//! # Identity
//!
//! [`RemoteId::Stable`] here is a hash of the path relative to the library
//! root. That gives the catalog what it needs — a `u64` that names a
//! photograph, is the same on every device looking at the same folder, and
//! does not change when the file is edited — which is what keys the thumbnail
//! shards and the face index (`catalog.md` §10.1).
//!
//! It does **not** survive a rename, and [`Capabilities::stable_ids`] says so.
//! A moved photograph is seen as a delete and an add, and its thumbnail is
//! derived again. That is the documented degradation for a backend without
//! server-assigned ids, and it is the right trade here: the alternative,
//! keying on the inode, is stable across a rename but *differs between
//! devices* and is reused by the filesystem after a delete — so two machines
//! would disagree about which photograph a thumbnail belonged to, and a
//! recycled inode would silently attach an old thumbnail to a new image.
//! Re-deriving a thumbnail is a cost; showing the wrong one is a bug.
//!
//! # Blocking
//!
//! Every filesystem call goes through the blocking pool. On a local disk that
//! is overkill; on the NFS mount this backend is most useful over, a stalled
//! server would otherwise wedge the async worker that made the call and every
//! other request sharing it.
use std::io::{Read, Seek, SeekFrom, Write};
use std::ops::Range;
use std::path::{Component, Path, PathBuf};
use async_trait::async_trait;
use dr_sync::{
Account, BackendProvider, Capabilities, ChangeDetection, Connection, Cursor, EntryKind,
Precondition, RemoteBackend, RemoteChange, RemoteEntry, RemoteError, RemoteId, RemotePath,
ServerPreviews, SignIn, Validator,
};
/// The id written to [`Account::backend`] for a folder library.
///
/// On-disk configuration: changing it orphans every folder account.
pub const BACKEND_ID: &str = "folder";
/// TRACES: FR-NC-13
/// Registers the folder connector.
///
/// See [`dr_sync::provider`] for what each method is for.
pub struct FolderProvider;
impl BackendProvider for FolderProvider {
fn id(&self) -> &'static str {
BACKEND_ID
}
fn display_name(&self) -> &'static str {
"Folder"
}
fn endpoint_label(&self) -> &'static str {
"Folder"
}
fn endpoint_placeholder(&self) -> &'static str {
"/home/you/Pictures"
}
fn sign_in(&self) -> SignIn {
SignIn::EndpointOnly
}
/// Check the directory before an account is written for it.
///
/// A typo here would otherwise be stored, skip the launch screen on the
/// next start, and surface as a scan that finds nothing — which reads as
/// a broken library rather than a wrong path. The messages say what to fix.
fn normalise_endpoint(&self, input: &str) -> Result<String, String> {
let trimmed = input.trim();
if trimmed.is_empty() {
return Err("Choose the folder your photographs are in.".into());
}
// `~` is what a person types and what a shell would have expanded;
// nothing expands it here, so a stored `~/Pictures` becomes a
// directory literally named `~`.
let expanded = match trimmed.strip_prefix("~/") {
Some(rest) => match std::env::var_os("HOME") {
Some(home) => PathBuf::from(home).join(rest),
None => return Err("No home directory to expand ~ against.".into()),
},
None => PathBuf::from(trimmed),
};
if !expanded.is_absolute() {
return Err("Give the full path to the folder, starting at /.".into());
}
if !expanded.exists() {
return Err(format!("No folder at {}.", expanded.display()));
}
if !expanded.is_dir() {
return Err(format!("{} is a file, not a folder.", expanded.display()));
}
// Resolved so a library reached through a symlink or a `..` is stored
// under one name. Two spellings of one folder would otherwise be two
// accounts with two catalogs indexing the same photographs.
let canonical = expanded
.canonicalize()
.map_err(|e| format!("Cannot read {}: {e}", expanded.display()))?;
Ok(canonical.to_string_lossy().into_owned())
}
fn account_for(&self, endpoint: &str) -> Result<Account, RemoteError> {
Ok(Account::new(BACKEND_ID, endpoint))
}
fn connect(&self, conn: &Connection) -> Result<Box<dyn RemoteBackend>, RemoteError> {
Ok(Box::new(FolderBackend::new(&conn.account.endpoint)?))
}
}
/// TRACES: FR-NC-13 | FR-NC-4
/// A library rooted at a directory.
#[derive(Debug, Clone)]
pub struct FolderBackend {
root: PathBuf,
caps: Capabilities,
}
impl FolderBackend {
/// Open the folder at `root`.
///
/// The directory must exist now. It may stop existing later — a drive
/// unplugged, a mount dropped — and that surfaces per-operation as
/// [`RemoteError::Network`], which is what puts the app into offline mode
/// and leaves the catalog readable, exactly as a dead server does.
pub fn new(root: impl Into<PathBuf>) -> Result<Self, RemoteError> {
let root = root.into();
if !root.is_dir() {
return Err(RemoteError::Configuration(format!(
"{} is not a folder",
root.display()
)));
}
Ok(Self {
root,
caps: Capabilities {
// A directory's mtime describes its own entry list and nothing
// below it, so there is no propagation to exploit; the engine
// walks and compares per entry.
change_detection: ChangeDetection::LocalEtags,
// A path hash does not survive a rename. See the module docs
// for why the inode is not used instead.
stable_ids: false,
range_reads: true,
// Not a protocol with a message size limit; a write is a write.
chunked_upload: None,
bulk_upload: false,
conditional_write: true,
server_previews: ServerPreviews::None,
},
})
}
pub fn root(&self) -> &Path {
&self.root
}
/// The local path for a remote path, refusing anything that escapes.
///
/// The guard is not theoretical. A `RemotePath` is built from strings that
/// reach us from a catalog written by another device and from filenames on
/// the remote itself, and this backend resolves them against a real
/// filesystem with the user's own permissions. `../../.ssh/id_ed25519` is
/// a legal path segment; without this it would be a legal *read*.
fn resolve(&self, path: &RemotePath) -> Result<PathBuf, RemoteError> {
let rel = Path::new(path.as_str());
for component in rel.components() {
match component {
Component::Normal(_) => {}
Component::CurDir => {}
Component::ParentDir | Component::RootDir | Component::Prefix(_) => {
return Err(RemoteError::Configuration(format!(
"{path} leaves the library folder"
)));
}
}
}
Ok(self.root.join(rel))
}
/// The local path a [`RemoteId`] names.
///
/// A stable id here is a hash and nothing can be resolved from it, exactly
/// as a Nextcloud `oc:fileid` names no WebDAV endpoint. Callers hold the
/// path alongside it in the catalog and pass that.
fn resolve_id(&self, id: &RemoteId) -> Result<PathBuf, RemoteError> {
match id {
RemoteId::Path(p) => self.resolve(p),
RemoteId::Stable(_) => Err(RemoteError::Unsupported(
"a folder cannot be addressed by id; use RemoteId::Path",
)),
}
}
}
/// Run a filesystem operation off the async worker that asked for it.
///
/// See the module docs: a stalled network mount must not take the caller's
/// runtime with it.
async fn blocking<T, F>(f: F) -> Result<T, RemoteError>
where
F: FnOnce() -> Result<T, RemoteError> + Send + 'static,
T: Send + 'static,
{
match tokio::task::spawn_blocking(f).await {
Ok(r) => r,
// The only way a blocking task fails to produce a result is a panic
// inside it, which is a bug here rather than a condition the caller
// can act on — but crashing the worker over it would lose a whole
// scan, so it is reported like any other failure.
Err(e) => Err(RemoteError::Protocol(format!("folder task failed: {e}"))),
}
}
/// Map an IO failure to the error the engine already knows how to handle.
///
/// The classification is the point. [`RemoteError::indicates_offline`] drives
/// offline mode, so a vanished mount must reach it as `Network` — that is
/// precisely the "the library is unreachable, keep working from the catalog"
/// case — while a permissions problem must not, because going offline over one
/// forbidden file would hide a fixable problem behind a network banner.
fn map_io(e: std::io::Error, what: &str) -> RemoteError {
use std::io::ErrorKind as K;
match e.kind() {
K::NotFound => RemoteError::NotFound(what.to_string()),
K::PermissionDenied => RemoteError::PermissionDenied,
K::AlreadyExists => RemoteError::PreconditionFailed,
// ENOSPC and friends. Quota is what the engine calls "no room".
K::StorageFull | K::QuotaExceeded | K::FileTooLarge => RemoteError::QuotaExceeded,
// A dropped mount answers ESTALE/EIO/ENOTCONN, and the honest reading
// is the same as a dead server: the library cannot be reached now, and
// may be again shortly.
K::HostUnreachable
| K::NetworkUnreachable
| K::NetworkDown
| K::ConnectionAborted
| K::ConnectionReset
| K::NotConnected
| K::BrokenPipe
| K::TimedOut => RemoteError::Network(format!("{what}: {e}")),
_ => RemoteError::Protocol(format!("{what}: {e}")),
}
}
/// The identity of a file, from its path relative to the library root.
///
/// FNV-1a rather than `DefaultHasher`, whose output is explicitly unstable
/// between Rust releases: this value is written into the catalog and into the
/// thumbnail index, and must mean the same thing after a toolchain upgrade as
/// it did before one.
fn identity(path: &RemotePath) -> u64 {
let mut h: u64 = 0xcbf2_9ce4_8422_2325;
for b in path.as_str().as_bytes() {
h ^= *b as u64;
h = h.wrapping_mul(0x0000_0100_0000_01b3);
}
h
}
/// A file's validator: its size and modification time.
///
/// The pair, not either alone. An mtime with one-second granularity — which is
/// what some filesystems and most network mounts report — cannot distinguish
/// two writes in the same second, and a size alone cannot see an edit that
/// preserved it. Together they miss only a same-second write of identical
/// length, which for a photograph is a rewrite of the same frame.
fn validator_of(meta: &std::fs::Metadata) -> Validator {
let (secs, nanos) = meta
.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| (d.as_secs(), d.subsec_nanos()))
.unwrap_or((0, 0));
Validator::new(format!("{:x}-{:x}.{:x}", meta.len(), secs, nanos))
}
fn modified_secs(meta: &std::fs::Metadata) -> Option<i64> {
meta.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| d.as_secs() as i64)
}
#[async_trait]
impl RemoteBackend for FolderBackend {
fn capabilities(&self) -> &Capabilities {
&self.caps
}
fn name(&self) -> &str {
"Folder"
}
async fn list(
&self,
dir: &RemotePath,
_since: Option<&Validator>,
) -> Result<Vec<RemoteEntry>, RemoteError> {
let local = self.resolve(dir)?;
let dir = dir.clone();
blocking(move || {
let read =
std::fs::read_dir(&local).map_err(|e| map_io(e, &local.display().to_string()))?;
let mut out = Vec::new();
for entry in read {
let entry = match entry {
Ok(e) => e,
// One unreadable entry must not fail the listing: a
// scan of a real library meets a broken symlink or a
// file being written, and abandoning the whole
// directory over it loses every photograph beside it.
Err(e) => {
log::debug!("skipping an entry in {}: {e}", local.display());
continue;
}
};
let name = entry.file_name();
let Some(name) = name.to_str() else {
// A name that is not UTF-8 cannot round-trip through a
// `RemotePath`, and quietly mangling it would produce a
// path that addresses a different file — or none.
log::warn!("skipping a non-UTF-8 name in {}", local.display());
continue;
};
// `metadata`, not `symlink_metadata`: a symlinked shoot
// folder is a normal way to assemble a library, and the
// scan's depth limit is what stops a loop.
let meta = match entry.metadata() {
Ok(m) => m,
Err(e) => {
log::debug!("skipping {name}: {e}");
continue;
}
};
let path = dir.join(name);
out.push(RemoteEntry {
id: RemoteId::Stable(identity(&path)),
kind: if meta.is_dir() {
EntryKind::Directory
} else {
EntryKind::File
},
validator: validator_of(&meta),
size: meta.len(),
modified: modified_secs(&meta),
// No renderer behind a folder; previews are extracted
// locally from the file itself.
has_preview: false,
path,
});
}
Ok(out)
})
.await
}
/// Not offered.
///
/// A directory's mtime changes when its own entries are added or removed
/// and at no other time, so it cannot answer the question this method
/// exists for — "did anything below here change?". Returning it anyway
/// would let a future caller prune a subtree whose contents had been
/// edited, and hide those edits for as long as the folder list held still.
async fn dir_validator(&self, _dir: &RemotePath) -> Result<Validator, RemoteError> {
Err(RemoteError::Unsupported(
"a folder's mtime does not propagate; use per-entry validators",
))
}
async fn delta(&self, _cursor: &Cursor) -> Result<(Vec<RemoteChange>, Cursor), RemoteError> {
Err(RemoteError::Unsupported("a folder keeps no change feed"))
}
async fn get(&self, id: &RemoteId, range: Option<Range<u64>>) -> Result<Vec<u8>, RemoteError> {
let local = self.resolve_id(id)?;
blocking(move || {
let what = local.display().to_string();
let mut file = std::fs::File::open(&local).map_err(|e| map_io(e, &what))?;
let Some(r) = range else {
let mut buf = Vec::new();
file.read_to_end(&mut buf).map_err(|e| map_io(e, &what))?;
return Ok(buf);
};
// A short read at the end of the file is not an error: the header
// extractor asks for a fixed window and the file may be smaller
// than it, which is the ordinary case for a small JPEG.
file.seek(SeekFrom::Start(r.start))
.map_err(|e| map_io(e, &what))?;
let want = r.end.saturating_sub(r.start);
let mut buf = Vec::new();
file.take(want)
.read_to_end(&mut buf)
.map_err(|e| map_io(e, &what))?;
Ok(buf)
})
.await
}
async fn put(
&self,
path: &RemotePath,
body: Vec<u8>,
precond: Option<Precondition>,
) -> Result<Validator, RemoteError> {
let local = self.resolve(path)?;
blocking(move || {
let what = local.display().to_string();
if let Some(parent) = local.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| map_io(e, &parent.display().to_string()))?;
}
match &precond {
// Genuinely atomic: `O_CREAT | O_EXCL` is one syscall, so two
// devices racing to create a sidecar cannot both win.
Some(Precondition::IfAbsent) => {
let mut f = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(&local)
.map_err(|e| map_io(e, &what))?;
f.write_all(&body).map_err(|e| map_io(e, &what))?;
f.sync_all().map_err(|e| map_io(e, &what))?;
let meta = f.metadata().map_err(|e| map_io(e, &what))?;
return Ok(validator_of(&meta));
}
// Compare, then swap. A POSIX filesystem has no compare-and-
// swap, so this narrows the window to the microseconds between
// the `stat` and the `rename` rather than closing it. That is
// still far tighter than the fallback the engine uses when a
// backend declares no conditional write at all — comparing
// revision counters *inside* the sidecar, which spans a whole
// read-modify-write — which is why the capability is declared
// rather than refused.
Some(Precondition::IfMatch(expected)) => {
let meta = std::fs::metadata(&local).map_err(|e| map_io(e, &what))?;
if &validator_of(&meta) != expected {
return Err(RemoteError::PreconditionFailed);
}
}
None => {}
}
// Write beside the destination and rename over it, so a reader
// never sees a half-written sidecar and an interrupted write
// cannot destroy the file it was replacing. Beside, not in
// `/tmp`: a rename across filesystems is not atomic, and on
// Android `/tmp` is a different one.
let tmp = local.with_extension(format!(
"{}.darkroom-tmp",
local.extension().and_then(|e| e.to_str()).unwrap_or("")
));
let write = (|| -> Result<(), RemoteError> {
let mut f = std::fs::File::create(&tmp).map_err(|e| map_io(e, &what))?;
f.write_all(&body).map_err(|e| map_io(e, &what))?;
f.sync_all().map_err(|e| map_io(e, &what))
})();
if let Err(e) = write {
let _ = std::fs::remove_file(&tmp);
return Err(e);
}
if let Err(e) = std::fs::rename(&tmp, &local) {
let _ = std::fs::remove_file(&tmp);
return Err(map_io(e, &what));
}
let meta = std::fs::metadata(&local).map_err(|e| map_io(e, &what))?;
Ok(validator_of(&meta))
})
.await
}
/// Delete a file, or an empty directory.
///
/// **Not recursive, unlike WebDAV's `DELETE` on a collection.** The
/// divergence is deliberate: a folder library is the user's own
/// photographs on their own disk, with no server-side trash behind it, so
/// a caller that passed the wrong path would have no way back. Nothing in
/// the engine deletes a directory — the soft delete is a
/// [`move_to`](RemoteBackend::move_to) into the trash folder — so refusing
/// costs nothing and the guard is free.
async fn delete(
&self,
id: &RemoteId,
precond: Option<Precondition>,
) -> Result<(), RemoteError> {
let local = self.resolve_id(id)?;
blocking(move || {
let what = local.display().to_string();
let meta = std::fs::symlink_metadata(&local).map_err(|e| map_io(e, &what))?;
match &precond {
Some(Precondition::IfMatch(expected)) => {
if &validator_of(&meta) != expected {
return Err(RemoteError::PreconditionFailed);
}
}
// "Delete only if nothing is there" is not a thing to ask of a
// delete; something is there or the `stat` above already
// failed.
Some(Precondition::IfAbsent) => {
return Err(RemoteError::Unsupported(
"IfAbsent is not meaningful on a delete",
))
}
None => {}
}
if meta.is_dir() {
std::fs::remove_dir(&local).map_err(|e| {
if e.kind() == std::io::ErrorKind::DirectoryNotEmpty {
RemoteError::Configuration(format!(
"{what} is not empty; a folder library will not delete a tree"
))
} else {
map_io(e, &what)
}
})
} else {
std::fs::remove_file(&local).map_err(|e| map_io(e, &what))
}
})
.await
}
async fn move_to(&self, from: &RemoteId, to: &RemotePath) -> Result<(), RemoteError> {
let src = self.resolve_id(from)?;
let dst = self.resolve(to)?;
blocking(move || {
let what = dst.display().to_string();
// Parents first: the trash folder does not exist until the first
// photograph is trashed, and the trait promises this creates it.
if let Some(parent) = dst.parent() {
std::fs::create_dir_all(parent)
.map_err(|e| map_io(e, &parent.display().to_string()))?;
}
match std::fs::rename(&src, &dst) {
Ok(()) => Ok(()),
// EXDEV. Both paths are inside one library root, so this
// needs a root that spans a mount point — a shoot folder
// that is its own mount, which is an ordinary way to attach
// an archive drive. Copy and unlink rather than refusing:
// the identity a rename would have preserved is a path hash
// here, and it changes either way.
Err(e) if e.raw_os_error() == Some(18) => {
std::fs::copy(&src, &dst).map_err(|e| map_io(e, &what))?;
std::fs::remove_file(&src).map_err(|e| {
// The copy landed. Leaving the original is a
// duplicate, which the next scan will show; losing
// the copy would be worse.
let _ = std::fs::remove_file(&dst);
map_io(e, &src.display().to_string())
})
}
Err(e) => Err(map_io(e, &what)),
}
})
.await
}
async fn create_dir(&self, path: &RemotePath) -> Result<(), RemoteError> {
let local = self.resolve(path)?;
blocking(move || {
// `create_dir_all` makes parents and succeeds on one that already
// exists, which is exactly the contract.
std::fs::create_dir_all(&local).map_err(|e| map_io(e, &local.display().to_string()))
})
.await
}
}
#[cfg(test)]
mod tests;
core/dr-sync-folder/src/tests.rs
+540
View File
@@ -0,0 +1,540 @@
//! Behaviour of the folder connector, against real directories.
//!
//! No mocks: the whole point of this backend is what a filesystem actually
//! does, and a double would only assert what this file assumes.
use super::*;
use dr_sync::{scan, RemoteBackend};
use dr_types::FormatFilter;
use std::collections::HashMap;
/// A throwaway library root.
///
/// Under the system temp directory, named for the test, and cleared first so a
/// crashed run cannot leave state that makes the next one pass.
struct Tmp(PathBuf);
impl Tmp {
fn new(name: &str) -> Self {
let d = std::env::temp_dir().join(format!("dr-folder-test-{name}"));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
Tmp(d)
}
fn file(&self, rel: &str, body: &[u8]) -> &Self {
let p = self.0.join(rel);
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(p, body).unwrap();
self
}
fn backend(&self) -> FolderBackend {
FolderBackend::new(&self.0).unwrap()
}
}
impl Drop for Tmp {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
fn names(entries: &[RemoteEntry]) -> Vec<String> {
let mut v: Vec<String> = entries.iter().map(|e| e.path.name().to_string()).collect();
v.sort();
v
}
// --- opening --------------------------------------------------------------
#[test]
fn a_missing_folder_is_a_configuration_error_not_a_network_one() {
// It must not put the app into offline mode: nothing was unreachable, the
// account names somewhere that is not a folder.
let err = FolderBackend::new("/definitely/not/here").unwrap_err();
assert!(matches!(err, RemoteError::Configuration(_)), "{err:?}");
assert!(!err.indicates_offline());
}
// --- listing --------------------------------------------------------------
#[tokio::test]
async fn listing_reports_files_and_directories() {
let t = Tmp::new("list");
t.file("a.CR2", b"raw").file("sub/b.CR2", b"raw");
let b = t.backend();
let root = b.list(&RemotePath::root(), None).await.unwrap();
assert_eq!(names(&root), vec!["a.CR2", "sub"]);
let kinds: HashMap<_, _> = root
.iter()
.map(|e| (e.path.name().to_string(), e.kind))
.collect();
assert_eq!(kinds["a.CR2"], EntryKind::File);
assert_eq!(kinds["sub"], EntryKind::Directory);
let sub = b.list(&RemotePath::new("sub"), None).await.unwrap();
assert_eq!(names(&sub), vec!["b.CR2"]);
// Paths are rooted at the library, not at the filesystem.
assert_eq!(sub[0].path.as_str(), "sub/b.CR2");
}
#[tokio::test]
async fn a_listing_carries_the_size_a_scan_needs() {
let t = Tmp::new("size");
t.file("a.CR2", &[7u8; 1234]);
let e = &t.backend().list(&RemotePath::root(), None).await.unwrap()[0];
assert_eq!(e.size, 1234);
assert!(e.modified.is_some());
// Nothing behind a folder renders anything.
assert!(!e.has_preview);
}
#[tokio::test]
async fn listing_a_missing_directory_is_not_found() {
let t = Tmp::new("missing");
let e = t
.backend()
.list(&RemotePath::new("nope"), None)
.await
.unwrap_err();
assert!(matches!(e, RemoteError::NotFound(_)), "{e:?}");
}
// --- identity and validators ---------------------------------------------
#[tokio::test]
async fn identity_is_stable_across_an_edit_but_not_across_a_rename() {
// The catalog keys thumbnails and faces on this id, so editing a file must
// not orphan its thumbnail. A rename is a different photograph as far as
// this backend can tell, which `Capabilities::stable_ids` reports.
let t = Tmp::new("identity");
t.file("a.CR2", b"one");
let b = t.backend();
let before = b.list(&RemotePath::root(), None).await.unwrap()[0]
.id
.clone();
t.file("a.CR2", b"two-different-length");
let after = b.list(&RemotePath::root(), None).await.unwrap()[0]
.id
.clone();
assert_eq!(before, after, "an edit is not a new photograph");
std::fs::rename(t.0.join("a.CR2"), t.0.join("b.CR2")).unwrap();
let renamed = b.list(&RemotePath::root(), None).await.unwrap()[0]
.id
.clone();
assert_ne!(before, renamed);
assert!(!b.capabilities().stable_ids, "and the capability says so");
}
#[tokio::test]
async fn two_libraries_agree_on_the_identity_of_the_same_photograph() {
// Two devices mounting one share must key the thumbnail index the same
// way, or each re-derives what the other already stored. This is why the
// id is a path hash and not an inode.
let a = Tmp::new("id-a");
let b = Tmp::new("id-b");
a.file("2026/x.CR2", b"one");
b.file("2026/x.CR2", b"quite different bytes");
let ida = a
.backend()
.list(&RemotePath::new("2026"), None)
.await
.unwrap()[0]
.id
.clone();
let idb = b
.backend()
.list(&RemotePath::new("2026"), None)
.await
.unwrap()[0]
.id
.clone();
assert_eq!(ida, idb);
}
#[tokio::test]
async fn a_validator_changes_when_the_content_does() {
let t = Tmp::new("validator");
t.file("a.CR2", b"one");
let b = t.backend();
let before = b.list(&RemotePath::root(), None).await.unwrap()[0]
.validator
.clone();
// A different length, so this holds on a filesystem with one-second mtime
// granularity as well as on one with nanoseconds.
t.file("a.CR2", b"a rather longer body");
let after = b.list(&RemotePath::root(), None).await.unwrap()[0]
.validator
.clone();
assert_ne!(before, after);
}
#[tokio::test]
async fn a_folder_does_not_pretend_to_prune() {
// Answering with the directory's own mtime would let a caller skip a
// subtree whose files had been edited, hiding those edits indefinitely.
let t = Tmp::new("prune");
let b = t.backend();
assert!(matches!(
b.dir_validator(&RemotePath::root()).await,
Err(RemoteError::Unsupported(_))
));
assert_eq!(
b.capabilities().change_detection,
ChangeDetection::LocalEtags
);
}
// --- reading --------------------------------------------------------------
#[tokio::test]
async fn a_whole_file_and_a_range_both_read() {
let t = Tmp::new("get");
t.file("a.CR2", b"0123456789");
let b = t.backend();
let id = RemoteId::Path(RemotePath::new("a.CR2"));
assert_eq!(b.get(&id, None).await.unwrap(), b"0123456789");
assert_eq!(b.get(&id, Some(2..5)).await.unwrap(), b"234");
}
#[tokio::test]
async fn a_range_past_the_end_returns_what_is_there() {
// The header extractor asks for a fixed window; a small JPEG is shorter
// than it, and failing would make every small file undatable.
let t = Tmp::new("shortrange");
t.file("a.JPG", b"abc");
let got = t
.backend()
.get(&RemoteId::Path(RemotePath::new("a.JPG")), Some(0..65536))
.await
.unwrap();
assert_eq!(got, b"abc");
}
#[tokio::test]
async fn a_bare_identity_cannot_address_a_file() {
// Same contract as the Nextcloud connector: the id says *which*
// photograph, the path says *where*. Callers hold both.
let t = Tmp::new("byid");
t.file("a.CR2", b"x");
let e = t
.backend()
.get(&RemoteId::Stable(1), None)
.await
.unwrap_err();
assert!(matches!(e, RemoteError::Unsupported(_)), "{e:?}");
}
#[tokio::test]
async fn nothing_reachable_from_a_remote_path_escapes_the_library() {
// A `RemotePath` is built from names on the remote and from a catalog
// another device wrote. Resolving one against a real filesystem with the
// user's own permissions makes `..` a read of anything they own.
let t = Tmp::new("escape");
let b = t.backend();
for attempt in ["../../../etc/passwd", "sub/../../outside"] {
let e = b
.get(&RemoteId::Path(RemotePath::new(attempt)), None)
.await
.unwrap_err();
assert!(
matches!(e, RemoteError::Configuration(_)),
"{attempt} was not refused: {e:?}"
);
}
}
// --- writing --------------------------------------------------------------
#[tokio::test]
async fn a_write_creates_the_folders_it_needs() {
let t = Tmp::new("put");
let b = t.backend();
b.put(&RemotePath::new("2026/03/a.xmp"), b"<x/>".to_vec(), None)
.await
.unwrap();
assert_eq!(std::fs::read(t.0.join("2026/03/a.xmp")).unwrap(), b"<x/>");
}
#[tokio::test]
async fn a_write_leaves_no_temporary_behind() {
// The rename-into-place is invisible from outside, and must stay that way:
// a stray `.darkroom-tmp` in a shoot folder would be listed by the scan.
let t = Tmp::new("puttmp");
let b = t.backend();
b.put(&RemotePath::new("a.xmp"), b"x".to_vec(), None)
.await
.unwrap();
assert_eq!(
names(&b.list(&RemotePath::root(), None).await.unwrap()),
vec!["a.xmp"]
);
}
#[tokio::test]
async fn an_overwrite_replaces_rather_than_appends() {
let t = Tmp::new("overwrite");
t.file("a.xmp", b"the older and much longer body");
let b = t.backend();
b.put(&RemotePath::new("a.xmp"), b"new".to_vec(), None)
.await
.unwrap();
assert_eq!(std::fs::read(t.0.join("a.xmp")).unwrap(), b"new");
}
#[tokio::test]
async fn if_absent_creates_once_and_refuses_after() {
let t = Tmp::new("ifabsent");
let b = t.backend();
let p = RemotePath::new("a.xmp");
b.put(&p, b"first".to_vec(), Some(Precondition::IfAbsent))
.await
.unwrap();
let e = b
.put(&p, b"second".to_vec(), Some(Precondition::IfAbsent))
.await
.unwrap_err();
assert!(matches!(e, RemoteError::PreconditionFailed), "{e:?}");
assert_eq!(std::fs::read(t.0.join("a.xmp")).unwrap(), b"first");
}
#[tokio::test]
async fn if_match_writes_on_the_expected_version_and_refuses_a_stale_one() {
// The sidecar conflict path (ARCH §8.5): a failure here means another
// device wrote first, and triggers a merge rather than an overwrite.
let t = Tmp::new("ifmatch");
t.file("a.xmp", b"one");
let b = t.backend();
let p = RemotePath::new("a.xmp");
let current = b.list(&RemotePath::root(), None).await.unwrap()[0]
.validator
.clone();
let after = b
.put(
&p,
b"two".to_vec(),
Some(Precondition::IfMatch(current.clone())),
)
.await
.unwrap();
assert_ne!(after, current);
let e = b
.put(&p, b"three".to_vec(), Some(Precondition::IfMatch(current)))
.await
.unwrap_err();
assert!(matches!(e, RemoteError::PreconditionFailed), "{e:?}");
assert_eq!(std::fs::read(t.0.join("a.xmp")).unwrap(), b"two");
}
#[tokio::test]
async fn the_validator_a_write_returns_is_the_one_a_listing_reports() {
// Otherwise the next conditional write fails against a file nobody else
// touched, and every sidecar update becomes a spurious conflict.
let t = Tmp::new("putvalidator");
let b = t.backend();
let p = RemotePath::new("a.xmp");
let written = b.put(&p, b"body".to_vec(), None).await.unwrap();
let listed = b.list(&RemotePath::root(), None).await.unwrap()[0]
.validator
.clone();
assert_eq!(written, listed);
}
// --- moving and deleting --------------------------------------------------
#[tokio::test]
async fn a_move_creates_the_trash_folder_it_needs() {
// The soft delete (FR-CAT-15): the trash does not exist until the first
// photograph goes into it, and the trait promises the move makes it.
let t = Tmp::new("move");
t.file("a.CR2", b"raw");
let b = t.backend();
b.move_to(
&RemoteId::Path(RemotePath::new("a.CR2")),
&RemotePath::new(".darkroom-trash/a.CR2"),
)
.await
.unwrap();
assert!(!t.0.join("a.CR2").exists());
assert_eq!(
std::fs::read(t.0.join(".darkroom-trash/a.CR2")).unwrap(),
b"raw"
);
}
#[tokio::test]
async fn deleting_a_file_removes_it() {
let t = Tmp::new("delete");
t.file("a.CR2", b"raw");
let b = t.backend();
b.delete(&RemoteId::Path(RemotePath::new("a.CR2")), None)
.await
.unwrap();
assert!(!t.0.join("a.CR2").exists());
}
#[tokio::test]
async fn deleting_refuses_to_take_a_tree_with_it() {
// Deliberately unlike WebDAV. There is no server-side trash behind a local
// folder, so a caller with a wrong path would have no way back.
let t = Tmp::new("deletetree");
t.file("shoot/a.CR2", b"raw");
let e = t
.backend()
.delete(&RemoteId::Path(RemotePath::new("shoot")), None)
.await
.unwrap_err();
assert!(matches!(e, RemoteError::Configuration(_)), "{e:?}");
assert!(t.0.join("shoot/a.CR2").exists());
}
#[tokio::test]
async fn a_conditional_delete_refuses_a_file_that_changed() {
let t = Tmp::new("deletecond");
t.file("a.CR2", b"raw");
let b = t.backend();
let stale = Validator::new("0-0.0");
let e = b
.delete(
&RemoteId::Path(RemotePath::new("a.CR2")),
Some(Precondition::IfMatch(stale)),
)
.await
.unwrap_err();
assert!(matches!(e, RemoteError::PreconditionFailed), "{e:?}");
assert!(t.0.join("a.CR2").exists());
}
#[tokio::test]
async fn creating_a_directory_twice_succeeds() {
// Callers use this to guarantee a destination, not to claim they made it.
let t = Tmp::new("mkdir");
let b = t.backend();
let p = RemotePath::new("2026/03");
b.create_dir(&p).await.unwrap();
b.create_dir(&p).await.unwrap();
assert!(t.0.join("2026/03").is_dir());
}
// --- driven by the engine -------------------------------------------------
#[tokio::test]
async fn the_scan_engine_walks_a_folder_library() {
// The claim this whole crate makes: the engine written for one backend
// drives another with no change. Nothing below is folder-specific.
let t = Tmp::new("scan");
t.file("2026/03/a.CR2", b"raw")
.file("2026/03/b.JPG", b"jpeg")
.file("2026/04/c.CR2", b"raw")
.file("2026/notes.txt", b"text")
.file(".darkroom-trash/deleted.CR2", b"raw");
let result = scan(
&t.backend(),
&RemotePath::root(),
&FormatFilter::from_formats([dr_types::Format::Cr2]),
&HashMap::new(),
|_| {},
)
.await
.unwrap();
let found: Vec<&str> = result.images.iter().map(|e| e.path.as_str()).collect();
// The filter picked the RAWs; the trash was skipped, or the soft delete
// would undo itself on the next scan.
assert_eq!(found, vec!["2026/03/a.CR2", "2026/04/c.CR2"]);
assert_eq!(result.progress.directories_pruned, 0, "nothing to prune");
}
#[tokio::test]
async fn an_upload_lands_where_the_engine_places_it() {
let t = Tmp::new("upload");
let b = t.backend();
let placed = dr_sync::upload_original(
&b,
&RemotePath::root(),
&["2026".to_string(), "03".to_string()],
"a.CR2",
b"raw".to_vec(),
)
.await
.unwrap();
assert_eq!(placed.path().as_str(), "2026/03/a.CR2");
assert_eq!(std::fs::read(t.0.join("2026/03/a.CR2")).unwrap(), b"raw");
}
// --- the provider ---------------------------------------------------------
#[test]
fn an_endpoint_is_checked_before_an_account_is_written_for_it() {
let t = Tmp::new("provider");
let p = FolderProvider;
assert!(p.normalise_endpoint(" ").is_err(), "empty");
assert!(p.normalise_endpoint("Pictures").is_err(), "relative");
assert!(p.normalise_endpoint("/no/such/place").is_err(), "missing");
t.file("a.CR2", b"x");
assert!(
p.normalise_endpoint(&t.0.join("a.CR2").to_string_lossy())
.is_err(),
"a file is not a library"
);
let ok = p.normalise_endpoint(&t.0.to_string_lossy()).unwrap();
assert_eq!(PathBuf::from(&ok), t.0.canonicalize().unwrap());
}
#[test]
fn two_spellings_of_one_folder_become_one_account() {
// Otherwise the same photographs are indexed twice, into two catalogs.
let t = Tmp::new("canonical");
t.file("sub/a.CR2", b"x");
let p = FolderProvider;
let direct = p
.normalise_endpoint(&t.0.join("sub").to_string_lossy())
.unwrap();
let roundabout = p
.normalise_endpoint(&t.0.join("sub/../sub").to_string_lossy())
.unwrap();
assert_eq!(direct, roundabout);
}
#[test]
fn a_folder_account_needs_no_credential() {
let p = FolderProvider;
assert_eq!(p.sign_in(), SignIn::EndpointOnly);
assert!(!p.sign_in().needs_secret());
let account = p.account_for("/mnt/photos").unwrap();
assert_eq!(account.backend, BACKEND_ID);
assert_eq!(account.endpoint, "/mnt/photos");
assert!(account.login.is_empty());
}
#[test]
fn the_registry_opens_a_folder_account() {
// End to end through the abstraction: an account, a registry, a backend —
// with nothing in between naming this crate.
let t = Tmp::new("registry");
let mut registry = dr_sync::BackendRegistry::new();
registry.register(std::sync::Arc::new(FolderProvider));
let account = Account::new(BACKEND_ID, t.0.to_string_lossy());
let backend = registry.connect(&Connection::new(account, None)).unwrap();
assert_eq!(backend.name(), "Folder");
}