dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Make storage pluggable, and prove it with a folder backend

Browse Source 
`RemoteBackend` existed from the first release and bought nothing it was
designed for. Seven files in `dr-ui` constructed a `NextcloudBackend`
directly, an account *was* a server URL beside a DAV user id, the local
cache directory was named after a hostname, and the launch screen knew
that signing in meant a browser handshake. The trait was real; the seam
was documentation.

A trait over operations is only a quarter of it. Pluggable storage needs
four things, and this adds the other three:

- **Capabilities** — already there, and the reason the engine can drive
  two backends at the speed each actually runs at.
- **Configuration** — `dr_sync::Account`: where a library lives, in
  whatever form its connector addresses, with no server in it. Loads
  every existing config unchanged (`backend` defaults to `nextcloud`,
  `endpoint` is stored under its historical `server` key), and
  `Account::namespace()` reproduces the old catalog directory byte for
  byte, because changing it would abandon a catalog, its thumbnail
  shards, and the sidecars holding unsynced offline work.
- **Registration** — `BackendProvider` and `BackendRegistry`.
  `ui/dr-ui/src/remote.rs` is now the only file above `dr-sync` that
  names a connector.

`Connection` (an account plus an optional `Secret`) replaces the
credentials-and-user-id pair that was threaded through fifteen
signatures in an order that could be swapped. `Secret`'s inner string is
reachable only through `expose()` and its `Debug` prints `Secret(***)`,
so the indirect leak — a `{:?}` on anything holding one — no longer
compiles into a leak.

Nextcloud is unchanged and keeps every peculiarity: propagating ETags,
chunked upload v2, `oc:fileid`, the `oc:permissions` probe on a refused
PUT, the 423 retry classification, Login Flow v2. Those are what the
capability model exists to serve, not something to hide.

`dr-sync-folder` is the second connector: a local disk, a network mount,
an external drive, or a folder a Nextcloud client already syncs. No
account, no credential — the route that works where no secrets daemon
does. It declares `LocalEtags` rather than claiming propagation a POSIX
directory cannot provide, which costs nothing because 50k `stat` calls
are not 50k PROPFINDs. Identity is a path hash, not an inode: an inode
survives a rename but differs between devices and is reused after a
delete, so two machines would disagree about which photograph a
thumbnail belonged to. Re-deriving a thumbnail is a cost; showing the
wrong one is a bug.

docs/storage.md is the contract — the traits, the four steps to add a
backend, and what each connector declares. ARCH §8.0 and §8.4a, and
FR-NC-13, say why.
This commit is contained in:
dtourolle
2026-08-29 09:57:52 +02:00
parent 1b8b7998a2
commit f12aece07e
40 changed files with 3617 additions and 960 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ui/dr-ui/src/library_ui.rs
+70 -88
View File
@@ -17,7 +17,7 @@ use std::rc::Rc;
use std::sync::mpsc::Receiver;
use dr_catalog::Catalog;
use dr_sync_nextcloud::{AppCredentials, Session, SessionStore};
use dr_sync::{Account, AccountStore, Connection};
use dr_types::FormatFilter;
use slint::{ComponentHandle, Model as _};
@@ -177,7 +177,12 @@ pub struct LibraryController {
/// Pushing shards and the catalog to the server.
sync_timer: RefCell<Option<slint::Timer>>,
/// Kept so a rescan can run without going back through the launch screen.
session: RefCell<Option<(AppCredentials, Session, FormatFilter)>>,
///
/// A [`Connection`] rather than credentials beside an account: it is what
/// every worker needs, it is what `remote::connect` takes, and holding the
/// two halves separately is how they came to be threaded through fifteen
/// signatures in the wrong order.
session: RefCell<Option<(Connection, FormatFilter)>>,
/// Which collection narrows the grid, owned by [`crate::collections_ui`]
/// and read here. Shared rather than passed per call because a rescan, a
/// scrub and a drop all reload the window and must all honour it.
@@ -506,8 +511,8 @@ impl LibraryController {
/// (FR-NC-6a), and a queued edit must never be.
pub fn sidecar_cache_dir(&self) -> Option<PathBuf> {
let borrow = self.session.borrow();
let (_, session, _) = borrow.as_ref()?;
library::catalog_path(&session.server, &session.user_id)
let (conn, _) = borrow.as_ref()?;
library::catalog_path(&conn.account)
.parent()
.map(|p| p.join("sidecars"))
}
@@ -520,8 +525,8 @@ impl LibraryController {
/// catalog row is gone is unreachable anyway.
pub fn cache_dir(&self) -> Option<PathBuf> {
let borrow = self.session.borrow();
let (_, session, _) = borrow.as_ref()?;
library::catalog_path(&session.server, &session.user_id)
let (conn, _) = borrow.as_ref()?;
library::catalog_path(&conn.account)
.parent()
.map(|p| p.join("originals"))
}
@@ -565,8 +570,8 @@ impl LibraryController {
/// three hundred would re-download every one of them.
pub fn cache_context_for(&self, image: dr_types::ImageId) -> Option<library::CacheContext> {
let borrow = self.session.borrow();
let (_, session, _) = borrow.as_ref()?;
let catalog_path = library::catalog_path(&session.server, &session.user_id);
let (conn, _) = borrow.as_ref()?;
let catalog_path = library::catalog_path(&conn.account);
let dir = catalog_path.parent()?.join("originals");
drop(borrow);
@@ -602,15 +607,12 @@ impl LibraryController {
self.catalog.clone()
}
/// Credentials and session for the open library.
/// The open library's connection.
///
/// Needed by the trash, whose `MOVE` and `DELETE` go to the same account the
/// Needed by the trash, whose move and delete go to the same account the
/// scan and thumbnail workers use. `None` before a library is opened.
pub fn session(&self) -> Option<(AppCredentials, Session)> {
self.session
.borrow()
.as_ref()
.map(|(c, s, _)| (c.clone(), s.clone()))
pub fn session(&self) -> Option<Connection> {
self.session.borrow().as_ref().map(|(c, _)| c.clone())
}
/// Catalog ids of the rows currently in the model, in model order.
@@ -730,16 +732,12 @@ impl LibraryController {
.collect()
}
/// Credentials and account for the open library, if one is open.
/// The open library's connection, for a full-file fetch.
///
/// What a full-file fetch needs: the grid's paths are remote, so opening
/// an image means downloading it, and that needs the same session the
/// thumbnail workers use.
pub fn credentials(&self) -> Option<(AppCredentials, String)> {
self.session
.borrow()
.as_ref()
.map(|(creds, session, _)| (creds.clone(), session.user_id.clone()))
/// The grid's paths are remote, so opening an image means fetching it, and
/// that goes through the same account the thumbnail workers use.
pub fn credentials(&self) -> Option<Connection> {
self.session.borrow().as_ref().map(|(c, _)| c.clone())
}
/// Narrow the grid to a collection, or to the whole library with `None`.
@@ -786,10 +784,13 @@ pub fn open(
window: &AppWindow,
ctl: Rc<LibraryController>,
coll_ctl: Rc<crate::collections_ui::CollectionsController>,
store: &SessionStore,
session: Session,
store: &AccountStore,
account: Account,
) {
let creds = match store.credentials(&session) {
// The credential is fetched only where the connector wants one; a folder
// library has none, and asking the keyring for it would fail the one
// backend that needs nothing.
let conn = match store.connection(&account, crate::remote::needs_secret(&account)) {
Ok(c) => c,
Err(e) => {
window.set_library_error(format!("credentials: {e}").into());
@@ -798,8 +799,8 @@ pub fn open(
}
};
let filter = session.format_filter();
*ctl.session.borrow_mut() = Some((creds.clone(), session.clone(), filter.clone()));
let filter = account.format_filter();
*ctl.session.borrow_mut() = Some((conn.clone(), filter.clone()));
window.set_show_library(true);
window.set_library_open(true);
@@ -809,10 +810,10 @@ pub fn open(
// Always visible: two folders one letter apart are easy to confuse, and a
// scan of the wrong one is indistinguishable from a broken scan.
window.set_library_root_label(
if session.root.is_empty() {
format!("{} · whole account", session.user_id)
if conn.account.root.is_empty() {
format!("{} · whole account", conn.account.user_id)
} else {
format!("{}/{}", session.user_id, session.root)
format!("{}/{}", conn.account.user_id, conn.account.root)
}
.into(),
);
@@ -825,13 +826,13 @@ pub fn open(
return;
}
let path = library::catalog_path(&session.server, &session.user_id);
let path = library::catalog_path(&conn.account);
log::info!(
"scanning {} for {} format(s) → {}",
if session.root.is_empty() {
if conn.account.root.is_empty() {
"<account root>"
} else {
&session.root
&conn.account.root
},
filter.iter().count(),
path.display()
@@ -842,9 +843,8 @@ pub fn open(
show_catalog_now(window, &ctl, &path, &coll_ctl);
let rx = library::spawn_scan(
creds,
session.user_id.clone(),
session.root.clone(),
conn.clone(),
conn.account.root.clone(),
filter,
path.clone(),
);
@@ -870,8 +870,8 @@ fn drain_scan(
// Named after the folder, because two accounts or two roots produce rows
// that are otherwise identical.
let title = match ctl.session.borrow().as_ref() {
Some((_, session, _)) if !session.root.is_empty() => {
format!("Scanning {}", session.root)
Some((c, _)) if !c.account.root.is_empty() => {
format!("Scanning {}", c.account.root)
}
_ => "Scanning the library".to_string(),
};
@@ -1044,7 +1044,7 @@ fn start_rescan(
ctl: &Rc<LibraryController>,
coll_ctl: &Rc<crate::collections_ui::CollectionsController>,
) {
let Some((creds, session, filter)) = ctl.session.borrow().clone() else {
let Some((conn, filter)) = ctl.session.borrow().clone() else {
return;
};
@@ -1052,11 +1052,10 @@ fn start_rescan(
window.set_library_error(slint::SharedString::new());
window.set_library_status("Rescanning…".into());
let path = library::catalog_path(&session.server, &session.user_id);
let path = library::catalog_path(&conn.account);
let rx = library::spawn_scan(
creds,
session.user_id.clone(),
session.root.clone(),
conn.clone(),
conn.account.root.clone(),
filter,
path.clone(),
);
@@ -1413,7 +1412,7 @@ fn collection_images(catalog: &Catalog, ids: &[dr_types::CollectionId]) -> Vec<d
/// TRACES: FR-NC-6a
/// Download whatever the pins still want, reporting progress.
fn start_pin_fetch(window: &AppWindow, ctl: &Rc<LibraryController>) {
let Some((creds, session, _)) = ctl.session.borrow().clone() else {
let Some((conn, _)) = ctl.session.borrow().clone() else {
return;
};
let Some(cache_dir) = ctl.cache_dir() else {
@@ -1428,9 +1427,8 @@ fn start_pin_fetch(window: &AppWindow, ctl: &Rc<LibraryController>) {
}
let rx = library::spawn_pin_fetch(
creds,
session.user_id.clone(),
library::catalog_path(&session.server, &session.user_id),
conn.clone(),
library::catalog_path(&conn.account),
cache_dir,
// Pinned originals are exempt from the budget, but a pin fetch also
// stores passively when it finds an image already cached, so the worker
@@ -1629,11 +1627,11 @@ fn start_outbox_drain(window: &AppWindow, ctl: &Rc<LibraryController>) {
ctl.outbox_maybe_dirty.set(false);
return;
}
let Some((creds, session, _)) = ctl.session.borrow().clone() else {
let Some((conn, _)) = ctl.session.borrow().clone() else {
return;
};
let rx = library::spawn_outbox_drain(creds, session.user_id.clone(), cache_dir);
let rx = library::spawn_outbox_drain(conn.clone(), cache_dir);
let job = ctl
.activity
.begin(crate::activity::Kind::Upload, "Uploading queued edits");
@@ -2764,7 +2762,7 @@ pub(crate) fn start_sidecar_writes(
// write being conditional on it.
let offline = ctl.is_offline();
let Some((creds, session, _)) = ctl.session.borrow().clone() else {
let Some((conn, _)) = ctl.session.borrow().clone() else {
return;
};
let Some(cache_dir) = ctl.sidecar_cache_dir() else {
@@ -2772,8 +2770,7 @@ pub(crate) fn start_sidecar_writes(
};
let count = writes.len();
let rx =
library::spawn_sidecar_writes(creds, session.user_id.clone(), writes, cache_dir, offline);
let rx = library::spawn_sidecar_writes(conn.clone(), writes, cache_dir, offline);
let timer = slint::Timer::default();
let weak = window.as_weak();
@@ -2907,7 +2904,7 @@ fn fetch_rank(row: usize, first_on_screen: usize, on_screen: usize) -> (u8, usiz
/// Fetch thumbnails for rows in the model that do not have one yet.
fn request_thumbnails(window: &AppWindow, ctl: &Rc<LibraryController>) {
let Some((creds, session, _)) = ctl.session.borrow().clone() else {
let Some((conn, _)) = ctl.session.borrow().clone() else {
return;
};
@@ -2967,11 +2964,10 @@ fn request_thumbnails(window: &AppWindow, ctl: &Rc<LibraryController>) {
let requested = wanted.len();
let rx = library::spawn_thumbnails(
creds,
session.user_id.clone(),
conn.clone(),
wanted,
library::thumbs_dir(&session.server, &session.user_id),
library::catalog_path(&session.server, &session.user_id),
library::thumbs_dir(&conn.account),
library::catalog_path(&conn.account),
);
drain_thumbnails(window.as_weak(), ctl.clone(), rx, requested, class);
}
@@ -3022,14 +3018,11 @@ pub fn refresh_thumbnail(
// nothing here to correct.
return;
};
let Some((_, session, _)) = ctl.session.borrow().clone() else {
let Some((conn, _)) = ctl.session.borrow().clone() else {
return;
};
let mut store = match dr_thumbs::ThumbStore::open(&library::thumbs_dir(
&session.server,
&session.user_id,
)) {
let mut store = match dr_thumbs::ThumbStore::open(&library::thumbs_dir(&conn.account)) {
Ok(s) => s,
Err(e) => {
log::warn!("re-thumbnailing {remote_path}: opening the store: {e}");
@@ -3336,7 +3329,7 @@ fn start_derived_sync(window: &AppWindow, ctl: &Rc<LibraryController>) {
return;
}
let Some((creds, session, _)) = ctl.session.borrow().clone() else {
let Some((conn, _)) = ctl.session.borrow().clone() else {
return;
};
// Already running: a second pass would race the first over the same
@@ -3355,7 +3348,7 @@ fn start_derived_sync(window: &AppWindow, ctl: &Rc<LibraryController>) {
return;
}
let catalog_path = library::catalog_path(&session.server, &session.user_id);
let catalog_path = library::catalog_path(&conn.account);
let scratch = catalog_path
.parent()
.map(|p| p.join("scratch"))
@@ -3373,14 +3366,9 @@ fn start_derived_sync(window: &AppWindow, ctl: &Rc<LibraryController>) {
// for next time and nothing is lost by not watching it. It reports
// through the log until an export has a place in the activity list.
{
let outbox = crate::export::outbox_dir(&session.server, &session.user_id);
let outbox = crate::export::outbox_dir(&conn.account);
if crate::export::pending_count(&outbox) > 0 {
let rx = crate::export::spawn_upload(
creds.clone(),
session.user_id.clone(),
session.root.clone(),
outbox,
);
let rx = crate::export::spawn_upload(conn.clone(), conn.account.root.clone(), outbox);
std::thread::spawn(move || {
while let Ok(msg) = rx.recv() {
match msg {
@@ -3403,10 +3391,9 @@ fn start_derived_sync(window: &AppWindow, ctl: &Rc<LibraryController>) {
window.set_library_syncing(true);
let rx = crate::derived_sync::spawn_sync(
creds,
session.user_id.clone(),
session.root.clone(),
library::thumbs_dir(&session.server, &session.user_id),
conn.clone(),
conn.account.root.clone(),
library::thumbs_dir(&conn.account),
catalog_path,
scratch,
);
@@ -3529,7 +3516,7 @@ fn start_derived_sync(window: &AppWindow, ctl: &Rc<LibraryController>) {
/// describes the fraction of the library that happened to be scrolled past.
/// This covers the rest.
fn start_sweep(window: &AppWindow, ctl: &Rc<LibraryController>) {
let Some((creds, session, _)) = ctl.session.borrow().clone() else {
let Some((conn, _)) = ctl.session.borrow().clone() else {
return;
};
@@ -3544,11 +3531,7 @@ fn start_sweep(window: &AppWindow, ctl: &Rc<LibraryController>) {
return;
}
let rx = library::spawn_sweep(
creds,
session.user_id.clone(),
library::catalog_path(&session.server, &session.user_id),
);
let rx = library::spawn_sweep(conn.clone(), library::catalog_path(&conn.account));
let timer = slint::Timer::default();
let weak = window.as_weak();
@@ -3640,7 +3623,7 @@ fn start_sweep(window: &AppWindow, ctl: &Rc<LibraryController>) {
/// The sync at the end is not a separate courtesy: a filled store that never
/// leaves this device is most of the cost for none of the point.
fn start_thumbnail_sweep(window: &AppWindow, ctl: &Rc<LibraryController>) {
let Some((creds, session, _)) = ctl.session.borrow().clone() else {
let Some((conn, _)) = ctl.session.borrow().clone() else {
return;
};
@@ -3664,10 +3647,9 @@ fn start_thumbnail_sweep(window: &AppWindow, ctl: &Rc<LibraryController>) {
window.set_library_thumbnailing(true);
let rx = library::spawn_thumbnail_sweep(
creds,
session.user_id.clone(),
library::catalog_path(&session.server, &session.user_id),
library::thumbs_dir(&session.server, &session.user_id),
conn.clone(),
library::catalog_path(&conn.account),
library::thumbs_dir(&conn.account),
);
let timer = slint::Timer::default();