Library setup as the user described it: pick a folder, choose which RAW
types to look for, scan recursively.
dr-types::FormatFilter the tick-box selection, seeing through VFS
placeholder suffixes so a dehydrated CR2 still
matches as a CR2
dr-sync::scan recursive walk, Depth:1 per directory, pruning
unchanged subtrees where the backend propagates
directory ETags
Verified against nextcloud.tourolle.paris (34.0.2) on a real library:
browse root 32 entries, 98ms
scan PhotosRaw 17,185 RAW files in 334 directories, 34.1s
(7,836 CR2 + 9,349 DNG)
range read 262KB of a 21.5MB DNG in 119ms — 1.22% of the file,
and enough to read "Canon EOS 6D | ISO 100"
That last line is assumption A3 validated on real data. Cataloguing this
library by whole-file fetch would move roughly 370GB; the range path
moves a few MB.
Pruning is capability-gated rather than assumed: with per-entry ETags a
probe costs a request and proves nothing about children, so it is skipped
entirely. A test asserts zero probes in that case.
Still unresolved: /core/preview returns 400 for every parameter
combination tried, including on a JPEG the server reports as having a
preview. Not a request-shape bug — it fails identically bare. Recorded
rather than worked around; ARCH §6.7 already treats server previews as
opportunistic, so nothing depends on it.
206 lines
7.5 KiB
Rust
206 lines
7.5 KiB
Rust
//! Pluggable remote storage for DarkRoom.
|
|
//!
|
|
//! Defines the [`RemoteBackend`] trait and the capability model the sync
|
|
//! engine adapts to. Only the Nextcloud connector is implemented
|
|
//! (`dr-sync-nextcloud`), but the boundary is designed so other backends can
|
|
//! be added without touching the engine.
|
|
//!
|
|
//! # Why capabilities rather than a common denominator
|
|
//!
|
|
//! Nextcloud's fast path relies on a behaviour that is *not* a WebDAV
|
|
//! guarantee: directory ETags propagate up the tree, so an unchanged root
|
|
//! ETag proves nothing anywhere in the library changed. That single property
|
|
//! turns a no-op sync over 50k images into one HTTP request.
|
|
//!
|
|
//! A trait built to what every backend can do would force full enumeration
|
|
//! every time — the exact cost the design exists to avoid. So backends
|
|
//! declare what they support and the engine picks a strategy (ARCH §8.1).
|
|
|
|
use std::ops::Range;
|
|
|
|
use async_trait::async_trait;
|
|
|
|
pub mod capability;
|
|
pub mod error;
|
|
pub mod scan;
|
|
pub mod types;
|
|
|
|
pub use capability::{Capabilities, ChangeDetection, ChunkConstraints, ServerPreviews};
|
|
pub use error::RemoteError;
|
|
pub use scan::{scan, ScanProgress, ScanResult};
|
|
pub use types::{
|
|
Cursor, EntryKind, Identity, Precondition, RemoteChange, RemoteEntry, RemoteId, RemotePath,
|
|
Validator,
|
|
};
|
|
|
|
/// TRACES: FR-NC-12
|
|
/// A remote storage backend.
|
|
///
|
|
/// Implementations are expected to be cheap to clone or to be used behind an
|
|
/// `Arc`; the engine may call them concurrently.
|
|
#[async_trait]
|
|
pub trait RemoteBackend: Send + Sync {
|
|
/// What this backend supports. Read once at connect time and used to pick
|
|
/// a sync strategy.
|
|
fn capabilities(&self) -> &Capabilities;
|
|
|
|
/// Human-readable backend name, for logs and the UI.
|
|
fn name(&self) -> &str;
|
|
|
|
// ---- discovery --------------------------------------------------------
|
|
|
|
/// List one directory level.
|
|
///
|
|
/// `since` carries the validator the caller last saw, so backends able to
|
|
/// skip unchanged entries may do so. Backends that cannot simply ignore
|
|
/// it.
|
|
async fn list(
|
|
&self,
|
|
dir: &RemotePath,
|
|
since: Option<&Validator>,
|
|
) -> Result<Vec<RemoteEntry>, RemoteError>;
|
|
|
|
/// Fetch a directory's validator without listing its contents.
|
|
///
|
|
/// The cheap probe that makes ETag pruning work: one request against the
|
|
/// root answers "did anything change?". Backends without
|
|
/// [`ChangeDetection::PropagatingEtags`] return
|
|
/// [`RemoteError::Unsupported`].
|
|
async fn dir_validator(&self, dir: &RemotePath) -> Result<Validator, RemoteError>;
|
|
|
|
/// Ask what changed since a cursor.
|
|
///
|
|
/// Only meaningful for [`ChangeDetection::DeltaCursor`] backends; others
|
|
/// return [`RemoteError::Unsupported`].
|
|
async fn delta(&self, cursor: &Cursor) -> Result<(Vec<RemoteChange>, Cursor), RemoteError>;
|
|
|
|
// ---- transfer ---------------------------------------------------------
|
|
|
|
/// Fetch an object, optionally a byte range.
|
|
///
|
|
/// The range is a hint, not a guarantee: backends without range support
|
|
/// may return the whole object, and the caller slices. Correctness holds
|
|
/// either way; [`Capabilities::range_reads`] says whether it was cheap.
|
|
async fn get(&self, id: &RemoteId, range: Option<Range<u64>>) -> Result<Vec<u8>, RemoteError>;
|
|
|
|
/// Upload, optionally guarded by a precondition.
|
|
///
|
|
/// Backends handle chunking internally based on body size — chunked
|
|
/// upload is an implementation detail, not part of this interface, since
|
|
/// exposing it would leak one server's protocol into the abstraction.
|
|
async fn put(
|
|
&self,
|
|
path: &RemotePath,
|
|
body: Vec<u8>,
|
|
precond: Option<Precondition>,
|
|
) -> Result<Validator, RemoteError>;
|
|
|
|
/// Upload many small objects.
|
|
///
|
|
/// Defaults to sequential [`put`](Self::put) calls; backends with a bulk
|
|
/// endpoint override it. Sidecars are the motivating case — hundreds of
|
|
/// a few KB each.
|
|
async fn put_many(
|
|
&self,
|
|
items: Vec<(RemotePath, Vec<u8>)>,
|
|
) -> Result<Vec<Result<Validator, RemoteError>>, RemoteError> {
|
|
let mut out = Vec::with_capacity(items.len());
|
|
for (path, body) in items {
|
|
out.push(self.put(&path, body, None).await);
|
|
}
|
|
Ok(out)
|
|
}
|
|
|
|
/// Delete an object.
|
|
async fn delete(&self, id: &RemoteId, precond: Option<Precondition>)
|
|
-> Result<(), RemoteError>;
|
|
|
|
// ---- optional ---------------------------------------------------------
|
|
|
|
/// Server-rendered thumbnail, where available.
|
|
///
|
|
/// `Ok(None)` means the server has no preview for this object — which is
|
|
/// common for RAW, since stock Nextcloud ships no RAW preview provider
|
|
/// (ARCH §6.7). Callers must have a local fallback.
|
|
async fn thumbnail(&self, _id: &RemoteId, _size: u32) -> Result<Option<Vec<u8>>, RemoteError> {
|
|
Ok(None)
|
|
}
|
|
}
|
|
|
|
/// TRACES: FR-NC-4 | FR-NC-12
|
|
/// Which sync strategy the engine should use for a backend.
|
|
///
|
|
/// Derived from capabilities at connect time. Reported to the user so a slow
|
|
/// backend is visibly slow rather than mysteriously slow (FR-NC-12).
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub enum SyncStrategy {
|
|
/// Ask the server what changed. Cheapest.
|
|
Delta,
|
|
/// Probe the root ETag; recurse only where it differs. One request when
|
|
/// nothing changed.
|
|
EtagPruning,
|
|
/// Enumerate the tree, using per-entry ETags to avoid re-downloading.
|
|
FullListing,
|
|
/// Enumerate and compare modification times. Degraded — clock skew and
|
|
/// second-granularity timestamps both cause misses.
|
|
TimestampCompare,
|
|
}
|
|
|
|
impl SyncStrategy {
|
|
pub fn for_capabilities(caps: &Capabilities) -> Self {
|
|
match caps.change_detection {
|
|
ChangeDetection::DeltaCursor => SyncStrategy::Delta,
|
|
ChangeDetection::PropagatingEtags => SyncStrategy::EtagPruning,
|
|
ChangeDetection::LocalEtags => SyncStrategy::FullListing,
|
|
ChangeDetection::Timestamps => SyncStrategy::TimestampCompare,
|
|
}
|
|
}
|
|
|
|
/// A short description for the UI.
|
|
pub fn describe(self) -> &'static str {
|
|
match self {
|
|
SyncStrategy::Delta => "server change feed",
|
|
SyncStrategy::EtagPruning => "incremental (ETag pruning)",
|
|
SyncStrategy::FullListing => "full listing, cached by ETag",
|
|
SyncStrategy::TimestampCompare => "full listing by timestamp (degraded)",
|
|
}
|
|
}
|
|
|
|
/// Whether this strategy can prove "nothing changed" cheaply.
|
|
///
|
|
/// Where false, every sync costs at least one request per folder, which
|
|
/// the UI should warn about on large libraries.
|
|
pub fn has_cheap_noop(self) -> bool {
|
|
matches!(self, SyncStrategy::Delta | SyncStrategy::EtagPruning)
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn strategy_follows_capability() {
|
|
let mut caps = Capabilities::minimal();
|
|
assert_eq!(
|
|
SyncStrategy::for_capabilities(&caps),
|
|
SyncStrategy::TimestampCompare
|
|
);
|
|
|
|
caps.change_detection = ChangeDetection::PropagatingEtags;
|
|
assert_eq!(
|
|
SyncStrategy::for_capabilities(&caps),
|
|
SyncStrategy::EtagPruning
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn only_delta_and_pruning_have_cheap_noop() {
|
|
assert!(SyncStrategy::Delta.has_cheap_noop());
|
|
assert!(SyncStrategy::EtagPruning.has_cheap_noop());
|
|
// These cost at least one request per folder, every time.
|
|
assert!(!SyncStrategy::FullListing.has_cheap_noop());
|
|
assert!(!SyncStrategy::TimestampCompare.has_cheap_noop());
|
|
}
|
|
}
|