Make storage pluggable, and prove it with a folder backend

`RemoteBackend` existed from the first release and bought nothing it was designed for. Seven files in `dr-ui` constructed a `NextcloudBackend` directly, an account *was* a server URL beside a DAV user id, the local cache directory was named after a hostname, and the launch screen knew that signing in meant a browser handshake. The trait was real; the seam was documentation. A trait over operations is only a quarter of it. Pluggable storage needs four things, and this adds the other three: - **Capabilities** — already there, and the reason the engine can drive two backends at the speed each actually runs at. - **Configuration** — `dr_sync::Account`: where a library lives, in whatever form its connector addresses, with no server in it. Loads every existing config unchanged (`backend` defaults to `nextcloud`, `endpoint` is stored under its historical `server` key), and `Account::namespace()` reproduces the old catalog directory byte for byte, because changing it would abandon a catalog, its thumbnail shards, and the sidecars holding unsynced offline work. - **Registration** — `BackendProvider` and `BackendRegistry`. `ui/dr-ui/src/remote.rs` is now the only file above `dr-sync` that names a connector. `Connection` (an account plus an optional `Secret`) replaces the credentials-and-user-id pair that was threaded through fifteen signatures in an order that could be swapped. `Secret`'s inner string is reachable only through `expose()` and its `Debug` prints `Secret(***)`, so the indirect leak — a `{:?}` on anything holding one — no longer compiles into a leak. Nextcloud is unchanged and keeps every peculiarity: propagating ETags, chunked upload v2, `oc:fileid`, the `oc:permissions` probe on a refused PUT, the 423 retry classification, Login Flow v2. Those are what the capability model exists to serve, not something to hide. `dr-sync-folder` is the second connector: a local disk, a network mount, an external drive, or a folder a Nextcloud client already syncs. No account, no credential — the route that works where no secrets daemon does. It declares `LocalEtags` rather than claiming propagation a POSIX directory cannot provide, which costs nothing because 50k `stat` calls are not 50k PROPFINDs. Identity is a path hash, not an inode: an inode survives a rename but differs between devices and is reused after a delete, so two machines would disagree about which photograph a thumbnail belonged to. Re-deriving a thumbnail is a cost; showing the wrong one is a bug. docs/storage.md is the contract — the traits, the four steps to add a backend, and what each connector declares. ARCH §8.0 and §8.4a, and FR-NC-13, say why.
2026-08-29 09:57:52 +02:00
parent 1b8b7998a2
commit f12aece07e
40 changed files with 3617 additions and 960 deletions
@@ -0,0 +1,540 @@
+//! Behaviour of the folder connector, against real directories.
+//!
+//! No mocks: the whole point of this backend is what a filesystem actually
+//! does, and a double would only assert what this file assumes.
+
+use super::*;
+use dr_sync::{scan, RemoteBackend};
+use dr_types::FormatFilter;
+use std::collections::HashMap;
+
+/// A throwaway library root.
+///
+/// Under the system temp directory, named for the test, and cleared first so a
+/// crashed run cannot leave state that makes the next one pass.
+struct Tmp(PathBuf);
+
+impl Tmp {
+    fn new(name: &str) -> Self {
+        let d = std::env::temp_dir().join(format!("dr-folder-test-{name}"));
+        let _ = std::fs::remove_dir_all(&d);
+        std::fs::create_dir_all(&d).unwrap();
+        Tmp(d)
+    }
+
+    fn file(&self, rel: &str, body: &[u8]) -> &Self {
+        let p = self.0.join(rel);
+        std::fs::create_dir_all(p.parent().unwrap()).unwrap();
+        std::fs::write(p, body).unwrap();
+        self
+    }
+
+    fn backend(&self) -> FolderBackend {
+        FolderBackend::new(&self.0).unwrap()
+    }
+}
+
+impl Drop for Tmp {
+    fn drop(&mut self) {
+        let _ = std::fs::remove_dir_all(&self.0);
+    }
+}
+
+fn names(entries: &[RemoteEntry]) -> Vec<String> {
+    let mut v: Vec<String> = entries.iter().map(|e| e.path.name().to_string()).collect();
+    v.sort();
+    v
+}
+
+// --- opening --------------------------------------------------------------
+
+#[test]
+fn a_missing_folder_is_a_configuration_error_not_a_network_one() {
+    // It must not put the app into offline mode: nothing was unreachable, the
+    // account names somewhere that is not a folder.
+    let err = FolderBackend::new("/definitely/not/here").unwrap_err();
+    assert!(matches!(err, RemoteError::Configuration(_)), "{err:?}");
+    assert!(!err.indicates_offline());
+}
+
+// --- listing --------------------------------------------------------------
+
+#[tokio::test]
+async fn listing_reports_files_and_directories() {
+    let t = Tmp::new("list");
+    t.file("a.CR2", b"raw").file("sub/b.CR2", b"raw");
+    let b = t.backend();
+
+    let root = b.list(&RemotePath::root(), None).await.unwrap();
+    assert_eq!(names(&root), vec!["a.CR2", "sub"]);
+
+    let kinds: HashMap<_, _> = root
+        .iter()
+        .map(|e| (e.path.name().to_string(), e.kind))
+        .collect();
+    assert_eq!(kinds["a.CR2"], EntryKind::File);
+    assert_eq!(kinds["sub"], EntryKind::Directory);
+
+    let sub = b.list(&RemotePath::new("sub"), None).await.unwrap();
+    assert_eq!(names(&sub), vec!["b.CR2"]);
+    // Paths are rooted at the library, not at the filesystem.
+    assert_eq!(sub[0].path.as_str(), "sub/b.CR2");
+}
+
+#[tokio::test]
+async fn a_listing_carries_the_size_a_scan_needs() {
+    let t = Tmp::new("size");
+    t.file("a.CR2", &[7u8; 1234]);
+    let e = &t.backend().list(&RemotePath::root(), None).await.unwrap()[0];
+    assert_eq!(e.size, 1234);
+    assert!(e.modified.is_some());
+    // Nothing behind a folder renders anything.
+    assert!(!e.has_preview);
+}
+
+#[tokio::test]
+async fn listing_a_missing_directory_is_not_found() {
+    let t = Tmp::new("missing");
+    let e = t
+        .backend()
+        .list(&RemotePath::new("nope"), None)
+        .await
+        .unwrap_err();
+    assert!(matches!(e, RemoteError::NotFound(_)), "{e:?}");
+}
+
+// --- identity and validators ---------------------------------------------
+
+#[tokio::test]
+async fn identity_is_stable_across_an_edit_but_not_across_a_rename() {
+    // The catalog keys thumbnails and faces on this id, so editing a file must
+    // not orphan its thumbnail. A rename is a different photograph as far as
+    // this backend can tell, which `Capabilities::stable_ids` reports.
+    let t = Tmp::new("identity");
+    t.file("a.CR2", b"one");
+    let b = t.backend();
+
+    let before = b.list(&RemotePath::root(), None).await.unwrap()[0]
+        .id
+        .clone();
+    t.file("a.CR2", b"two-different-length");
+    let after = b.list(&RemotePath::root(), None).await.unwrap()[0]
+        .id
+        .clone();
+    assert_eq!(before, after, "an edit is not a new photograph");
+
+    std::fs::rename(t.0.join("a.CR2"), t.0.join("b.CR2")).unwrap();
+    let renamed = b.list(&RemotePath::root(), None).await.unwrap()[0]
+        .id
+        .clone();
+    assert_ne!(before, renamed);
+    assert!(!b.capabilities().stable_ids, "and the capability says so");
+}
+
+#[tokio::test]
+async fn two_libraries_agree_on_the_identity_of_the_same_photograph() {
+    // Two devices mounting one share must key the thumbnail index the same
+    // way, or each re-derives what the other already stored. This is why the
+    // id is a path hash and not an inode.
+    let a = Tmp::new("id-a");
+    let b = Tmp::new("id-b");
+    a.file("2026/x.CR2", b"one");
+    b.file("2026/x.CR2", b"quite different bytes");
+
+    let ida = a
+        .backend()
+        .list(&RemotePath::new("2026"), None)
+        .await
+        .unwrap()[0]
+        .id
+        .clone();
+    let idb = b
+        .backend()
+        .list(&RemotePath::new("2026"), None)
+        .await
+        .unwrap()[0]
+        .id
+        .clone();
+    assert_eq!(ida, idb);
+}
+
+#[tokio::test]
+async fn a_validator_changes_when_the_content_does() {
+    let t = Tmp::new("validator");
+    t.file("a.CR2", b"one");
+    let b = t.backend();
+    let before = b.list(&RemotePath::root(), None).await.unwrap()[0]
+        .validator
+        .clone();
+
+    // A different length, so this holds on a filesystem with one-second mtime
+    // granularity as well as on one with nanoseconds.
+    t.file("a.CR2", b"a rather longer body");
+    let after = b.list(&RemotePath::root(), None).await.unwrap()[0]
+        .validator
+        .clone();
+    assert_ne!(before, after);
+}
+
+#[tokio::test]
+async fn a_folder_does_not_pretend_to_prune() {
+    // Answering with the directory's own mtime would let a caller skip a
+    // subtree whose files had been edited, hiding those edits indefinitely.
+    let t = Tmp::new("prune");
+    let b = t.backend();
+    assert!(matches!(
+        b.dir_validator(&RemotePath::root()).await,
+        Err(RemoteError::Unsupported(_))
+    ));
+    assert_eq!(
+        b.capabilities().change_detection,
+        ChangeDetection::LocalEtags
+    );
+}
+
+// --- reading --------------------------------------------------------------
+
+#[tokio::test]
+async fn a_whole_file_and_a_range_both_read() {
+    let t = Tmp::new("get");
+    t.file("a.CR2", b"0123456789");
+    let b = t.backend();
+    let id = RemoteId::Path(RemotePath::new("a.CR2"));
+
+    assert_eq!(b.get(&id, None).await.unwrap(), b"0123456789");
+    assert_eq!(b.get(&id, Some(2..5)).await.unwrap(), b"234");
+}
+
+#[tokio::test]
+async fn a_range_past_the_end_returns_what_is_there() {
+    // The header extractor asks for a fixed window; a small JPEG is shorter
+    // than it, and failing would make every small file undatable.
+    let t = Tmp::new("shortrange");
+    t.file("a.JPG", b"abc");
+    let got = t
+        .backend()
+        .get(&RemoteId::Path(RemotePath::new("a.JPG")), Some(0..65536))
+        .await
+        .unwrap();
+    assert_eq!(got, b"abc");
+}
+
+#[tokio::test]
+async fn a_bare_identity_cannot_address_a_file() {
+    // Same contract as the Nextcloud connector: the id says *which*
+    // photograph, the path says *where*. Callers hold both.
+    let t = Tmp::new("byid");
+    t.file("a.CR2", b"x");
+    let e = t
+        .backend()
+        .get(&RemoteId::Stable(1), None)
+        .await
+        .unwrap_err();
+    assert!(matches!(e, RemoteError::Unsupported(_)), "{e:?}");
+}
+
+#[tokio::test]
+async fn nothing_reachable_from_a_remote_path_escapes_the_library() {
+    // A `RemotePath` is built from names on the remote and from a catalog
+    // another device wrote. Resolving one against a real filesystem with the
+    // user's own permissions makes `..` a read of anything they own.
+    let t = Tmp::new("escape");
+    let b = t.backend();
+    for attempt in ["../../../etc/passwd", "sub/../../outside"] {
+        let e = b
+            .get(&RemoteId::Path(RemotePath::new(attempt)), None)
+            .await
+            .unwrap_err();
+        assert!(
+            matches!(e, RemoteError::Configuration(_)),
+            "{attempt} was not refused: {e:?}"
+        );
+    }
+}
+
+// --- writing --------------------------------------------------------------
+
+#[tokio::test]
+async fn a_write_creates_the_folders_it_needs() {
+    let t = Tmp::new("put");
+    let b = t.backend();
+    b.put(&RemotePath::new("2026/03/a.xmp"), b"<x/>".to_vec(), None)
+        .await
+        .unwrap();
+    assert_eq!(std::fs::read(t.0.join("2026/03/a.xmp")).unwrap(), b"<x/>");
+}
+
+#[tokio::test]
+async fn a_write_leaves_no_temporary_behind() {
+    // The rename-into-place is invisible from outside, and must stay that way:
+    // a stray `.darkroom-tmp` in a shoot folder would be listed by the scan.
+    let t = Tmp::new("puttmp");
+    let b = t.backend();
+    b.put(&RemotePath::new("a.xmp"), b"x".to_vec(), None)
+        .await
+        .unwrap();
+    assert_eq!(
+        names(&b.list(&RemotePath::root(), None).await.unwrap()),
+        vec!["a.xmp"]
+    );
+}
+
+#[tokio::test]
+async fn an_overwrite_replaces_rather_than_appends() {
+    let t = Tmp::new("overwrite");
+    t.file("a.xmp", b"the older and much longer body");
+    let b = t.backend();
+    b.put(&RemotePath::new("a.xmp"), b"new".to_vec(), None)
+        .await
+        .unwrap();
+    assert_eq!(std::fs::read(t.0.join("a.xmp")).unwrap(), b"new");
+}
+
+#[tokio::test]
+async fn if_absent_creates_once_and_refuses_after() {
+    let t = Tmp::new("ifabsent");
+    let b = t.backend();
+    let p = RemotePath::new("a.xmp");
+
+    b.put(&p, b"first".to_vec(), Some(Precondition::IfAbsent))
+        .await
+        .unwrap();
+    let e = b
+        .put(&p, b"second".to_vec(), Some(Precondition::IfAbsent))
+        .await
+        .unwrap_err();
+
+    assert!(matches!(e, RemoteError::PreconditionFailed), "{e:?}");
+    assert_eq!(std::fs::read(t.0.join("a.xmp")).unwrap(), b"first");
+}
+
+#[tokio::test]
+async fn if_match_writes_on_the_expected_version_and_refuses_a_stale_one() {
+    // The sidecar conflict path (ARCH §8.5): a failure here means another
+    // device wrote first, and triggers a merge rather than an overwrite.
+    let t = Tmp::new("ifmatch");
+    t.file("a.xmp", b"one");
+    let b = t.backend();
+    let p = RemotePath::new("a.xmp");
+
+    let current = b.list(&RemotePath::root(), None).await.unwrap()[0]
+        .validator
+        .clone();
+    let after = b
+        .put(
+            &p,
+            b"two".to_vec(),
+            Some(Precondition::IfMatch(current.clone())),
+        )
+        .await
+        .unwrap();
+    assert_ne!(after, current);
+
+    let e = b
+        .put(&p, b"three".to_vec(), Some(Precondition::IfMatch(current)))
+        .await
+        .unwrap_err();
+    assert!(matches!(e, RemoteError::PreconditionFailed), "{e:?}");
+    assert_eq!(std::fs::read(t.0.join("a.xmp")).unwrap(), b"two");
+}
+
+#[tokio::test]
+async fn the_validator_a_write_returns_is_the_one_a_listing_reports() {
+    // Otherwise the next conditional write fails against a file nobody else
+    // touched, and every sidecar update becomes a spurious conflict.
+    let t = Tmp::new("putvalidator");
+    let b = t.backend();
+    let p = RemotePath::new("a.xmp");
+    let written = b.put(&p, b"body".to_vec(), None).await.unwrap();
+    let listed = b.list(&RemotePath::root(), None).await.unwrap()[0]
+        .validator
+        .clone();
+    assert_eq!(written, listed);
+}
+
+// --- moving and deleting --------------------------------------------------
+
+#[tokio::test]
+async fn a_move_creates_the_trash_folder_it_needs() {
+    // The soft delete (FR-CAT-15): the trash does not exist until the first
+    // photograph goes into it, and the trait promises the move makes it.
+    let t = Tmp::new("move");
+    t.file("a.CR2", b"raw");
+    let b = t.backend();
+
+    b.move_to(
+        &RemoteId::Path(RemotePath::new("a.CR2")),
+        &RemotePath::new(".darkroom-trash/a.CR2"),
+    )
+    .await
+    .unwrap();
+
+    assert!(!t.0.join("a.CR2").exists());
+    assert_eq!(
+        std::fs::read(t.0.join(".darkroom-trash/a.CR2")).unwrap(),
+        b"raw"
+    );
+}
+
+#[tokio::test]
+async fn deleting_a_file_removes_it() {
+    let t = Tmp::new("delete");
+    t.file("a.CR2", b"raw");
+    let b = t.backend();
+    b.delete(&RemoteId::Path(RemotePath::new("a.CR2")), None)
+        .await
+        .unwrap();
+    assert!(!t.0.join("a.CR2").exists());
+}
+
+#[tokio::test]
+async fn deleting_refuses_to_take_a_tree_with_it() {
+    // Deliberately unlike WebDAV. There is no server-side trash behind a local
+    // folder, so a caller with a wrong path would have no way back.
+    let t = Tmp::new("deletetree");
+    t.file("shoot/a.CR2", b"raw");
+    let e = t
+        .backend()
+        .delete(&RemoteId::Path(RemotePath::new("shoot")), None)
+        .await
+        .unwrap_err();
+    assert!(matches!(e, RemoteError::Configuration(_)), "{e:?}");
+    assert!(t.0.join("shoot/a.CR2").exists());
+}
+
+#[tokio::test]
+async fn a_conditional_delete_refuses_a_file_that_changed() {
+    let t = Tmp::new("deletecond");
+    t.file("a.CR2", b"raw");
+    let b = t.backend();
+    let stale = Validator::new("0-0.0");
+    let e = b
+        .delete(
+            &RemoteId::Path(RemotePath::new("a.CR2")),
+            Some(Precondition::IfMatch(stale)),
+        )
+        .await
+        .unwrap_err();
+    assert!(matches!(e, RemoteError::PreconditionFailed), "{e:?}");
+    assert!(t.0.join("a.CR2").exists());
+}
+
+#[tokio::test]
+async fn creating_a_directory_twice_succeeds() {
+    // Callers use this to guarantee a destination, not to claim they made it.
+    let t = Tmp::new("mkdir");
+    let b = t.backend();
+    let p = RemotePath::new("2026/03");
+    b.create_dir(&p).await.unwrap();
+    b.create_dir(&p).await.unwrap();
+    assert!(t.0.join("2026/03").is_dir());
+}
+
+// --- driven by the engine -------------------------------------------------
+
+#[tokio::test]
+async fn the_scan_engine_walks_a_folder_library() {
+    // The claim this whole crate makes: the engine written for one backend
+    // drives another with no change. Nothing below is folder-specific.
+    let t = Tmp::new("scan");
+    t.file("2026/03/a.CR2", b"raw")
+        .file("2026/03/b.JPG", b"jpeg")
+        .file("2026/04/c.CR2", b"raw")
+        .file("2026/notes.txt", b"text")
+        .file(".darkroom-trash/deleted.CR2", b"raw");
+
+    let result = scan(
+        &t.backend(),
+        &RemotePath::root(),
+        &FormatFilter::from_formats([dr_types::Format::Cr2]),
+        &HashMap::new(),
+        |_| {},
+    )
+    .await
+    .unwrap();
+
+    let found: Vec<&str> = result.images.iter().map(|e| e.path.as_str()).collect();
+    // The filter picked the RAWs; the trash was skipped, or the soft delete
+    // would undo itself on the next scan.
+    assert_eq!(found, vec!["2026/03/a.CR2", "2026/04/c.CR2"]);
+    assert_eq!(result.progress.directories_pruned, 0, "nothing to prune");
+}
+
+#[tokio::test]
+async fn an_upload_lands_where_the_engine_places_it() {
+    let t = Tmp::new("upload");
+    let b = t.backend();
+    let placed = dr_sync::upload_original(
+        &b,
+        &RemotePath::root(),
+        &["2026".to_string(), "03".to_string()],
+        "a.CR2",
+        b"raw".to_vec(),
+    )
+    .await
+    .unwrap();
+    assert_eq!(placed.path().as_str(), "2026/03/a.CR2");
+    assert_eq!(std::fs::read(t.0.join("2026/03/a.CR2")).unwrap(), b"raw");
+}
+
+// --- the provider ---------------------------------------------------------
+
+#[test]
+fn an_endpoint_is_checked_before_an_account_is_written_for_it() {
+    let t = Tmp::new("provider");
+    let p = FolderProvider;
+
+    assert!(p.normalise_endpoint("   ").is_err(), "empty");
+    assert!(p.normalise_endpoint("Pictures").is_err(), "relative");
+    assert!(p.normalise_endpoint("/no/such/place").is_err(), "missing");
+
+    t.file("a.CR2", b"x");
+    assert!(
+        p.normalise_endpoint(&t.0.join("a.CR2").to_string_lossy())
+            .is_err(),
+        "a file is not a library"
+    );
+
+    let ok = p.normalise_endpoint(&t.0.to_string_lossy()).unwrap();
+    assert_eq!(PathBuf::from(&ok), t.0.canonicalize().unwrap());
+}
+
+#[test]
+fn two_spellings_of_one_folder_become_one_account() {
+    // Otherwise the same photographs are indexed twice, into two catalogs.
+    let t = Tmp::new("canonical");
+    t.file("sub/a.CR2", b"x");
+    let p = FolderProvider;
+    let direct = p
+        .normalise_endpoint(&t.0.join("sub").to_string_lossy())
+        .unwrap();
+    let roundabout = p
+        .normalise_endpoint(&t.0.join("sub/../sub").to_string_lossy())
+        .unwrap();
+    assert_eq!(direct, roundabout);
+}
+
+#[test]
+fn a_folder_account_needs_no_credential() {
+    let p = FolderProvider;
+    assert_eq!(p.sign_in(), SignIn::EndpointOnly);
+    assert!(!p.sign_in().needs_secret());
+
+    let account = p.account_for("/mnt/photos").unwrap();
+    assert_eq!(account.backend, BACKEND_ID);
+    assert_eq!(account.endpoint, "/mnt/photos");
+    assert!(account.login.is_empty());
+}
+
+#[test]
+fn the_registry_opens_a_folder_account() {
+    // End to end through the abstraction: an account, a registry, a backend —
+    // with nothing in between naming this crate.
+    let t = Tmp::new("registry");
+    let mut registry = dr_sync::BackendRegistry::new();
+    registry.register(std::sync::Arc::new(FolderProvider));
+
+    let account = Account::new(BACKEND_ID, t.0.to_string_lossy());
+    let backend = registry.connect(&Connection::new(account, None)).unwrap();
+    assert_eq!(backend.name(), "Folder");
+}