dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Make storage pluggable, and prove it with a folder backend

Browse Source 
`RemoteBackend` existed from the first release and bought nothing it was
designed for. Seven files in `dr-ui` constructed a `NextcloudBackend`
directly, an account *was* a server URL beside a DAV user id, the local
cache directory was named after a hostname, and the launch screen knew
that signing in meant a browser handshake. The trait was real; the seam
was documentation.

A trait over operations is only a quarter of it. Pluggable storage needs
four things, and this adds the other three:

- **Capabilities** — already there, and the reason the engine can drive
  two backends at the speed each actually runs at.
- **Configuration** — `dr_sync::Account`: where a library lives, in
  whatever form its connector addresses, with no server in it. Loads
  every existing config unchanged (`backend` defaults to `nextcloud`,
  `endpoint` is stored under its historical `server` key), and
  `Account::namespace()` reproduces the old catalog directory byte for
  byte, because changing it would abandon a catalog, its thumbnail
  shards, and the sidecars holding unsynced offline work.
- **Registration** — `BackendProvider` and `BackendRegistry`.
  `ui/dr-ui/src/remote.rs` is now the only file above `dr-sync` that
  names a connector.

`Connection` (an account plus an optional `Secret`) replaces the
credentials-and-user-id pair that was threaded through fifteen
signatures in an order that could be swapped. `Secret`'s inner string is
reachable only through `expose()` and its `Debug` prints `Secret(***)`,
so the indirect leak — a `{:?}` on anything holding one — no longer
compiles into a leak.

Nextcloud is unchanged and keeps every peculiarity: propagating ETags,
chunked upload v2, `oc:fileid`, the `oc:permissions` probe on a refused
PUT, the 423 retry classification, Login Flow v2. Those are what the
capability model exists to serve, not something to hide.

`dr-sync-folder` is the second connector: a local disk, a network mount,
an external drive, or a folder a Nextcloud client already syncs. No
account, no credential — the route that works where no secrets daemon
does. It declares `LocalEtags` rather than claiming propagation a POSIX
directory cannot provide, which costs nothing because 50k `stat` calls
are not 50k PROPFINDs. Identity is a path hash, not an inode: an inode
survives a rename but differs between devices and is reused after a
delete, so two machines would disagree about which photograph a
thumbnail belonged to. Re-deriving a thumbnail is a cost; showing the
wrong one is a bug.

docs/storage.md is the contract — the traits, the four steps to add a
backend, and what each connector declares. ARCH §8.0 and §8.4a, and
FR-NC-13, say why.
This commit is contained in:
dtourolle
2026-08-29 09:57:52 +02:00
parent 1b8b7998a2
commit f12aece07e
40 changed files with 3617 additions and 960 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/dr-sync-nextcloud/src/lib.rs
+8 -3
View File
@@ -1,4 +1,9 @@
//! Nextcloud connector — the only [`RemoteBackend`] implementation.
//! Nextcloud connector.
//!
//! One of two [`RemoteBackend`] implementations, registered through
//! [`NextcloudProvider`]. What an *account* is no longer lives here — that is
//! [`dr_sync::Account`], which has no server in it — so this crate is the
//! protocol and nothing else.
//!
//! Hand-rolled over `reqwest` rather than built on a WebDAV crate (D7). No
//! mature Nextcloud crate exists, and the operations that matter here are
@@ -16,11 +21,11 @@ use dr_sync::{
pub mod auth;
pub mod desktop_client;
mod propfind;
pub mod session;
pub mod provider;
pub use auth::{AppCredentials, LoginFlow};
pub use desktop_client::DesktopClient;
pub use session::{Session, SessionError, SessionStore};
pub use provider::NextcloudProvider;
/// Chunk sizes Nextcloud's chunked upload v2 accepts.
const CHUNKS: ChunkConstraints = ChunkConstraints {
core/dr-sync-nextcloud/src/provider.rs
+174
View File
@@ -0,0 +1,174 @@
// TRACES: FR-NC-12 | FR-NC-1
//! Registering Nextcloud as a storage backend.
//!
//! The account model this connector used to own now lives in
//! [`dr_sync::account`], where it has no server in it. What is left here is
//! the part that genuinely is Nextcloud: an endpoint is an HTTPS URL, an
//! account is established through Login Flow v2, and the credential is an app
//! password.
//!
//! Nothing above `dr_ui::remote` refers to this type.
use dr_sync::{
Account, BackendProvider, Connection, RemoteBackend, RemoteError, SignIn, LEGACY_BACKEND,
};
use crate::{AppCredentials, NextcloudBackend};
/// The id written to [`Account::backend`] for a Nextcloud account.
///
/// The same string [`dr_sync::LEGACY_BACKEND`] freezes, because every account
/// configured before there was a choice is one of these and must keep the
/// catalog directory it already has.
pub const BACKEND_ID: &str = LEGACY_BACKEND;
/// Registers the Nextcloud connector.
pub struct NextcloudProvider;
impl NextcloudProvider {
/// The account a completed login flow describes.
///
/// `user_id` is the DAV path segment, which is not always the login name:
/// a login can be an email address while the user id is something else,
/// and building `/remote.php/dav/files/<login>/` from the wrong one 404s
/// every request.
pub fn account_from(creds: &AppCredentials, user_id: impl Into<String>) -> Account {
Account::new(BACKEND_ID, creds.server.trim_end_matches('/'))
.with_login(creds.login_name.clone(), user_id)
}
/// The credentials a stored account plus its secret amount to.
///
/// [`AppCredentials`] stays the connector's own type rather than becoming
/// something general: an app password, an OAuth token and a bucket key
/// pair have no useful common shape, and inventing one would produce a
/// wrong answer confidently. The general form is [`Connection`]; this is
/// the translation into what one protocol needs.
pub fn credentials(conn: &Connection) -> Result<AppCredentials, RemoteError> {
Ok(AppCredentials {
server: conn.account.endpoint.clone(),
login_name: conn.account.login.clone(),
app_password: conn.require_secret()?.expose().to_string(),
})
}
}
impl BackendProvider for NextcloudProvider {
fn id(&self) -> &'static str {
BACKEND_ID
}
fn display_name(&self) -> &'static str {
"Nextcloud"
}
fn endpoint_label(&self) -> &'static str {
"Server"
}
fn endpoint_placeholder(&self) -> &'static str {
"https://cloud.example.com"
}
fn sign_in(&self) -> SignIn {
SignIn::Browser
}
/// Normalise a server address typed by hand.
///
/// Users type `cloud.example.com`, not a URL. Assume HTTPS rather than
/// failing, and never silently accept plain HTTP — NFR-SEC-3 requires TLS,
/// and an unencrypted default would be a security decision made on the
/// user's behalf without telling them.
fn normalise_endpoint(&self, input: &str) -> Result<String, String> {
let s = input.trim().trim_end_matches('/');
if s.is_empty() {
return Err("Enter the address of your Nextcloud server.".into());
}
if s.starts_with("https://") {
Ok(s.to_string())
} else if let Some(rest) = s.strip_prefix("http://") {
// Upgrade rather than accept. If the server genuinely has no TLS
// the connection fails loudly, which is the correct outcome.
Ok(format!("https://{rest}"))
} else {
Ok(format!("https://{s}"))
}
}
fn connect(&self, conn: &Connection) -> Result<Box<dyn RemoteBackend>, RemoteError> {
let creds = Self::credentials(conn)?;
Ok(Box::new(NextcloudBackend::new(
&creds,
&conn.account.user_id,
)?))
}
}
#[cfg(test)]
mod tests {
use super::*;
fn creds() -> AppCredentials {
AppCredentials {
server: "https://cloud.example/".into(),
login_name: "duncan@example.com".into(),
app_password: "token".into(),
}
}
#[test]
fn an_address_typed_by_hand_becomes_an_https_url() {
let p = NextcloudProvider;
assert_eq!(
p.normalise_endpoint("cloud.example.com/").unwrap(),
"https://cloud.example.com"
);
// Upgraded, never accepted: NFR-SEC-3.
assert_eq!(
p.normalise_endpoint("http://cloud.example.com").unwrap(),
"https://cloud.example.com"
);
assert!(p.normalise_endpoint(" ").is_err());
}
#[test]
fn the_account_keeps_the_dav_user_id_apart_from_the_login() {
// A login can be an email address while the user id is something
// else; building the DAV path from the wrong one 404s everything.
let a = NextcloudProvider::account_from(&creds(), "duncan");
assert_eq!(a.login, "duncan@example.com");
assert_eq!(a.user_id, "duncan");
assert_eq!(a.endpoint, "https://cloud.example");
}
#[test]
fn a_nextcloud_account_keeps_its_historical_catalog_directory() {
// Frozen: this names the directory holding the catalog, the thumbnail
// shards and un-uploaded sidecars.
let a = NextcloudProvider::account_from(&creds(), "duncan");
assert_eq!(a.namespace(), "cloud-example-duncan");
}
#[test]
fn connecting_without_a_credential_is_unauthenticated_not_a_crash() {
// A cleared keyring or a revoked app password arrives here as an
// account with no secret. The caller re-runs the login flow.
let account = NextcloudProvider::account_from(&creds(), "duncan");
match NextcloudProvider.connect(&Connection::new(account, None)) {
Err(RemoteError::Unauthenticated) => {}
Err(e) => panic!("wrong error: {e:?}"),
Ok(b) => panic!("connected without a credential as {}", b.name()),
}
}
#[test]
fn a_stored_account_and_its_secret_rebuild_the_credentials() {
let account = NextcloudProvider::account_from(&creds(), "duncan");
let conn = Connection::new(account, Some(dr_sync::Secret::new("token")));
let rebuilt = NextcloudProvider::credentials(&conn).unwrap();
assert_eq!(rebuilt.server, "https://cloud.example");
assert_eq!(rebuilt.login_name, "duncan@example.com");
assert_eq!(rebuilt.app_password, "token");
}
}
core/dr-sync-nextcloud/src/session.rs
-451
View File
@@ -1,451 +0,0 @@
//! Account sessions — logging in once and staying logged in.
//!
//! Splits deliberately in two:
//!
//! - **Credentials** go to platform secure storage (FR-NC-2). Never the
//! catalog, never a file, never a log line.
//! - **Everything else** — server, login, chosen root, format filter — is
//! ordinary configuration, safe to write as plain JSON.
//!
//! That split is what lets the app show "signed in as duncan, watching
//! /PhotosRaw" before it has touched the keyring, and re-authenticate cleanly
//! if the credential has been revoked server-side.
use std::path::{Path, PathBuf};
use dr_plat::{SecretError, SecretRef, SecretStore};
use dr_sync::RemoteError;
use dr_types::{Format, FormatFilter};
use serde::{Deserialize, Serialize};
use crate::AppCredentials;
/// Where configuration is written, when the platform has told us.
///
/// Android has no `$HOME` and no XDG directories, so the guess below resolves
/// to a path the app cannot write. Nothing failed loudly: the session list went
/// to a doomed path, so credentials survived only as long as the process did and
/// backgrounding the app lost the account (ARCH §6.9 — no core API may assume a
/// filesystem path on Android).
///
/// The platform layer sets this once at startup, before any store is opened.
static DATA_DIR: std::sync::OnceLock<PathBuf> = std::sync::OnceLock::new();
/// TRACES: FR-NC-2
/// Declare the per-app directory configuration belongs in.
///
/// Call before opening any store; later calls are ignored rather than racing.
/// On Android this is `AndroidApp::internal_data_path`, which is private to the
/// app and survives being backgrounded. Desktop needs no call — the XDG
/// fallback is correct there.
pub fn set_data_dir(dir: PathBuf) {
let _ = DATA_DIR.set(dir);
}
/// The directory configuration lives in.
fn config_dir() -> PathBuf {
if let Some(d) = DATA_DIR.get() {
return d.clone();
}
std::env::var_os("XDG_CONFIG_HOME")
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from(std::env::var("HOME").unwrap_or_default()).join(".config"))
.join("darkroom")
}
/// A configured account, minus its credential.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Session {
pub server: String,
pub login: String,
/// The DAV path segment, which may differ from `login` — a login can be
/// an email address while the user id is something else.
pub user_id: String,
/// The folder chosen as the library root. Empty means the account root.
#[serde(default)]
pub root: String,
/// Which formats the scan looks for (the tick-boxes).
#[serde(default)]
pub formats: Vec<String>,
/// Unix seconds of the last completed scan, for display.
#[serde(default)]
pub last_scan: Option<i64>,
}
impl Session {
pub fn new(creds: &AppCredentials, user_id: impl Into<String>) -> Self {
Self {
server: creds.server.trim_end_matches('/').to_string(),
login: creds.login_name.clone(),
user_id: user_id.into(),
root: String::new(),
formats: Vec::new(),
last_scan: None,
}
}
/// The stored format selection, defaulting to every supported format.
///
/// An unconfigured session must find everything rather than nothing.
pub fn format_filter(&self) -> FormatFilter {
if self.formats.is_empty() {
FormatFilter::all()
} else {
FormatFilter::from_formats(
self.formats
.iter()
.filter_map(|s| Format::from_extension(&s.to_ascii_lowercase())),
)
}
}
pub fn set_format_filter(&mut self, filter: &FormatFilter) {
self.formats = filter
.iter()
.map(|f| format!("{f:?}").to_lowercase())
.collect();
}
/// Where this session's credential lives.
pub fn secret_ref(&self) -> SecretRef {
SecretRef::app_password(&self.server, &self.login)
}
/// A short description for the UI.
pub fn describe(&self) -> String {
let host = self
.server
.trim_start_matches("https://")
.trim_start_matches("http://");
if self.root.is_empty() {
format!("{} on {host}", self.login)
} else {
format!("{} on {host}/{}", self.login, self.root)
}
}
}
/// TRACES: FR-NC-1 | FR-NC-2 | M-1 | M-2
/// Loads and saves sessions, keeping credentials in secure storage.
pub struct SessionStore {
config_path: PathBuf,
secrets: Box<dyn SecretStore>,
}
/// What is written to disk. Versioned so a format change is a migration
/// rather than a parse failure.
#[derive(Debug, Default, Serialize, Deserialize)]
struct ConfigFile {
#[serde(default = "one")]
version: u32,
#[serde(default)]
sessions: Vec<Session>,
}
fn one() -> u32 {
1
}
impl SessionStore {
/// Open the store at the platform config location.
///
/// Linux: `$XDG_CONFIG_HOME/darkroom/sessions.json`, falling back to
/// `~/.config` (FR-PLAT-LIN-1).
pub fn open(secrets: Box<dyn SecretStore>) -> Self {
Self::open_at(config_dir().join("sessions.json"), secrets)
}
/// Open at an explicit path — used by tests, and by anything wanting a
/// non-default config location.
/// Where configuration lives, for callers that need to sit files beside it.
pub fn data_dir() -> PathBuf {
config_dir()
}
pub fn open_at(config_path: PathBuf, secrets: Box<dyn SecretStore>) -> Self {
Self {
config_path,
secrets,
}
}
pub fn config_path(&self) -> &Path {
&self.config_path
}
/// Whether credentials can be remembered at all.
///
/// Where false the UI should say sign-in will not persist, rather than
/// letting the user discover it next launch.
pub fn can_remember(&self) -> bool {
self.secrets.is_available()
}
/// Every configured session. Missing or unreadable config yields an empty
/// list rather than an error — a first run is not a failure.
pub fn list(&self) -> Vec<Session> {
self.read_config().sessions
}
/// The most recently configured session, if any.
pub fn current(&self) -> Option<Session> {
self.read_config().sessions.into_iter().next_back()
}
/// Persist a session and its credential.
///
/// The credential goes to secure storage first: if that fails there is no
/// point recording a session that cannot authenticate.
pub fn save(&self, session: &Session, creds: &AppCredentials) -> Result<(), SessionError> {
self.secrets
.store(&session.secret_ref(), &creds.app_password)?;
let mut config = self.read_config();
config
.sessions
.retain(|s| !(s.server == session.server && s.login == session.login));
config.sessions.push(session.clone());
self.write_config(&config)
}
/// Update a session's settings, leaving its credential untouched.
pub fn update(&self, session: &Session) -> Result<(), SessionError> {
let mut config = self.read_config();
match config
.sessions
.iter_mut()
.find(|s| s.server == session.server && s.login == session.login)
{
Some(existing) => *existing = session.clone(),
None => config.sessions.push(session.clone()),
}
self.write_config(&config)
}
/// Rebuild credentials for a session from secure storage.
///
/// [`SecretError::NotFound`] means the credential was revoked or the
/// keyring was cleared — the caller re-runs the login flow.
pub fn credentials(&self, session: &Session) -> Result<AppCredentials, SessionError> {
let password = self.secrets.retrieve(&session.secret_ref())?;
Ok(AppCredentials {
server: session.server.clone(),
login_name: session.login.clone(),
app_password: password,
})
}
/// Forget a session and delete its credential.
///
/// The credential is removed even if the config write fails, so a logout
/// never leaves a usable secret behind.
pub fn forget(&self, session: &Session) -> Result<(), SessionError> {
let deleted = self.secrets.delete(&session.secret_ref());
let mut config = self.read_config();
config
.sessions
.retain(|s| !(s.server == session.server && s.login == session.login));
let written = self.write_config(&config);
deleted?;
written
}
fn read_config(&self) -> ConfigFile {
std::fs::read_to_string(&self.config_path)
.ok()
.and_then(|t| serde_json::from_str(&t).ok())
.unwrap_or_default()
}
fn write_config(&self, config: &ConfigFile) -> Result<(), SessionError> {
if let Some(parent) = self.config_path.parent() {
std::fs::create_dir_all(parent)?;
}
let json = serde_json::to_string_pretty(config)?;
// Write and rename, so an interrupted save cannot truncate an
// existing config.
let tmp = self.config_path.with_extension("tmp");
std::fs::write(&tmp, json)?;
std::fs::rename(&tmp, &self.config_path)?;
Ok(())
}
}
#[derive(Debug, thiserror::Error)]
pub enum SessionError {
#[error("secure storage: {0}")]
Secret(#[from] SecretError),
#[error("config io: {0}")]
Io(#[from] std::io::Error),
#[error("config format: {0}")]
Serde(#[from] serde_json::Error),
#[error(transparent)]
Remote(#[from] RemoteError),
}
#[cfg(test)]
mod tests {
use super::*;
use dr_plat::EphemeralSecretStore;
fn creds() -> AppCredentials {
AppCredentials {
server: "https://cloud.example/".into(),
login_name: "duncan".into(),
app_password: "secret-token".into(),
}
}
fn store_in(dir: &Path) -> SessionStore {
SessionStore::open_at(
dir.join("sessions.json"),
Box::new(EphemeralSecretStore::new()),
)
}
fn tmpdir(name: &str) -> PathBuf {
let d = std::env::temp_dir().join(format!("darkroom-test-{name}"));
let _ = std::fs::remove_dir_all(&d);
std::fs::create_dir_all(&d).unwrap();
d
}
#[test]
fn a_saved_session_survives_reopening() {
let dir = tmpdir("survives");
let secrets = Box::new(EphemeralSecretStore::new());
// Same secret store instance, as a real process would have.
let store = SessionStore::open_at(dir.join("sessions.json"), secrets);
let mut s = Session::new(&creds(), "duncan");
s.root = "PhotosRaw".into();
store.save(&s, &creds()).unwrap();
let reloaded = store.current().expect("session persisted");
assert_eq!(reloaded.login, "duncan");
assert_eq!(reloaded.root, "PhotosRaw");
// Trailing slash normalised, so URLs built from it are consistent.
assert_eq!(reloaded.server, "https://cloud.example");
}
#[test]
fn the_credential_never_reaches_the_config_file() {
// NFR-SEC-2: the whole point of the split.
let dir = tmpdir("nocreds");
let store = store_in(&dir);
let s = Session::new(&creds(), "duncan");
store.save(&s, &creds()).unwrap();
let text = std::fs::read_to_string(dir.join("sessions.json")).unwrap();
assert!(!text.contains("secret-token"), "credential leaked to disk");
assert!(text.contains("duncan"), "session metadata should be there");
}
#[test]
fn credentials_round_trip_through_secure_storage() {
let dir = tmpdir("roundtrip");
let store = store_in(&dir);
let s = Session::new(&creds(), "duncan");
store.save(&s, &creds()).unwrap();
let got = store.credentials(&s).unwrap();
assert_eq!(got.app_password, "secret-token");
assert_eq!(got.login_name, "duncan");
}
#[test]
fn forgetting_removes_both_halves() {
let dir = tmpdir("forget");
let store = store_in(&dir);
let s = Session::new(&creds(), "duncan");
store.save(&s, &creds()).unwrap();
store.forget(&s).unwrap();
assert!(store.current().is_none());
assert!(matches!(
store.credentials(&s),
Err(SessionError::Secret(SecretError::NotFound))
));
}
#[test]
fn saving_the_same_account_twice_does_not_duplicate_it() {
let dir = tmpdir("dedupe");
let store = store_in(&dir);
let mut s = Session::new(&creds(), "duncan");
store.save(&s, &creds()).unwrap();
s.root = "Photos".into();
store.save(&s, &creds()).unwrap();
assert_eq!(store.list().len(), 1);
assert_eq!(store.current().unwrap().root, "Photos");
}
#[test]
fn a_missing_config_is_a_first_run_not_an_error() {
let dir = tmpdir("firstrun");
let store = store_in(&dir);
assert!(store.list().is_empty());
assert!(store.current().is_none());
}
#[test]
fn a_corrupt_config_does_not_prevent_starting() {
// Better to present a first-run state than to refuse to launch.
let dir = tmpdir("corrupt");
std::fs::write(dir.join("sessions.json"), "{ not json").unwrap();
let store = store_in(&dir);
assert!(store.list().is_empty());
}
#[test]
fn format_selection_round_trips() {
let dir = tmpdir("formats");
let store = store_in(&dir);
let mut s = Session::new(&creds(), "duncan");
s.set_format_filter(&FormatFilter::from_formats([Format::Cr2, Format::Dng]));
store.save(&s, &creds()).unwrap();
let f = store.current().unwrap().format_filter();
assert!(f.allows(Format::Cr2));
assert!(f.allows(Format::Dng));
assert!(!f.allows(Format::Nef));
}
#[test]
fn an_unset_filter_means_every_format() {
// Never "no formats", which would silently find nothing.
let s = Session::new(&creds(), "duncan");
let f = s.format_filter();
assert!(f.allows(Format::Cr2));
assert!(f.allows(Format::Jpeg));
}
#[test]
fn describe_is_readable_and_hides_the_scheme() {
let mut s = Session::new(&creds(), "duncan");
assert_eq!(s.describe(), "duncan on cloud.example");
s.root = "PhotosRaw".into();
assert_eq!(s.describe(), "duncan on cloud.example/PhotosRaw");
}
#[test]
fn updating_settings_leaves_the_credential_alone() {
let dir = tmpdir("update");
let store = store_in(&dir);
let mut s = Session::new(&creds(), "duncan");
store.save(&s, &creds()).unwrap();
s.root = "Elsewhere".into();
store.update(&s).unwrap();
assert_eq!(store.current().unwrap().root, "Elsewhere");
assert_eq!(store.credentials(&s).unwrap().app_password, "secret-token");
}
}