Run the library from local data when the server is unreachable
Also carries in-flight work that shared these files: the zoom structure-key fix in the adjust pipeline, nearest-neighbour filtering past 1:1, the timeline scrub marker correction, the 423-Locked retry in the metadata sweep, and the thumbnail size-class migration. # Offline mode (FR-CAT-9) The app previously assumed the server was reachable and treated its absence as a series of unrelated per-operation failures. A launch without a connection produced an empty grid, even with a complete catalog on disk and every thumbnail already in the shards. Reachability is now inferred from traffic the app was already making, rather than probed for. `RemoteError::indicates_offline` draws the line that makes this possible: a dead connection is offline, a 403 or a 500 is not — the server answered, so blanking the library over one forbidden file would be a worse error than the one being reported. `Reachability` turns those outcomes into a state, so a library browsing happily never issues a probe at all. Going offline takes one failure, because the user is already experiencing it. Coming back requires evidence — a completed scan or a fetched thumbnail — with a capped exponential backoff behind the manual retry, so twelve sweep lanes failing together do not schedule twelve immediate probes. What keeps working: the catalog opens even when the scan that normally provides it failed, so the grid fills from the last successful scan. Thumbnails come from the shards. Rating, flagging and collecting are catalog writes that never touched the network. What stops is opening an original that was never stored locally, and it now says so in those words instead of reporting "network error: connection refused" over a photograph. Work that is pure network is refused rather than left to fail slowly: the metadata sweep, derived sync, and sidecar writes. The sweep would otherwise spend a timeout per image across the whole library while the progress bar implied something was happening. Deferring sidecars is a real gap rather than a hidden one — a rating made offline reaches its sidecar only when that image is judged again while connected — and it is recorded as such at the call site. # The "On this device" filter A chip beside the rating filters, narrowing the grid to images whose original is held locally. It composes with the rating terms rather than replacing them, so "five-star frames I can actually edit on this train" is one filter. The predicate is SQL, like the rating terms and for the same reason: the count in the header has to agree with the cells drawn. It reads `image_cache.tier_actual`, which nothing writes yet — the next commit fills it. Until then the chip honestly reports zero. `Tier` gains an explicit on-disk encoding. The variants are ordered by generosity and the derived `Ord` invites reordering them, which would silently reinterpret every cached row; the round-trip test is what holds the two in agreement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -27,6 +27,45 @@ async fn main() {
|
||||
Err(e) => println!("READ FAILED: {e}"),
|
||||
}
|
||||
|
||||
// What is actually in the derived folder on the server?
|
||||
{
|
||||
let dir = RemotePath::new(format!("{}/.darkroom-derived", session.root));
|
||||
match backend.list(&dir, None).await {
|
||||
Ok(entries) => {
|
||||
println!("\n[derived] {} entry/entries on the server:", entries.len());
|
||||
for e in &entries {
|
||||
println!(" {:<28} {:>10} bytes", e.path.name(), e.size);
|
||||
}
|
||||
}
|
||||
Err(e) => println!("\n[derived] listing failed: {e}"),
|
||||
}
|
||||
}
|
||||
|
||||
// Probe a file the sweep reported as 423 Locked: is it the file, the
|
||||
// range request, or the folder?
|
||||
{
|
||||
let c = dr_sync_nextcloud::http_client("DarkRoom").unwrap();
|
||||
let base = format!("{}/remote.php/dav/files/{}",
|
||||
creds.server.trim_end_matches('/'), session.user_id);
|
||||
let f = "PhotosRaw/Darktable/20230629_no_name/20230629_0030.jpeg";
|
||||
let enc: String = f.split('/').map(|seg| {
|
||||
seg.bytes().map(|b| match b {
|
||||
b'A'..=b'Z'|b'a'..=b'z'|b'0'..=b'9'|b'-'|b'_'|b'.'|b'~' => (b as char).to_string(),
|
||||
_ => format!("%{b:02X}"),
|
||||
}).collect::<String>()
|
||||
}).collect::<Vec<_>>().join("/");
|
||||
let url = format!("{base}/{enc}");
|
||||
|
||||
for (what, range) in [("ranged 0-256k", Some("bytes=0-262143")), ("whole file", None)] {
|
||||
let mut rq = c.get(&url).basic_auth(&creds.login_name, Some(&creds.app_password));
|
||||
if let Some(r) = range { rq = rq.header("Range", r); }
|
||||
match rq.send().await {
|
||||
Ok(r) => println!("GET {what}: {}", r.status()),
|
||||
Err(e) => println!("GET {what}: transport {e}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Raw HTTP, to see the status the connector maps away.
|
||||
{
|
||||
let url = format!("{}/remote.php/dav/files/{}/{}/.darkroom-write-test",
|
||||
|
||||
@@ -92,6 +92,114 @@ impl NextcloudBackend {
|
||||
}
|
||||
}
|
||||
|
||||
/// TRACES: FR-NC-7
|
||||
/// Upload a large body with chunked upload v2.
|
||||
///
|
||||
/// `MKCOL` an upload directory, `PUT` each chunk into it under a numeric
|
||||
/// name, then `MOVE` the `.file` pseudo-entry to the destination, which is
|
||||
/// where the server assembles them.
|
||||
///
|
||||
/// The alternative — refusing anything over the single-shot threshold —
|
||||
/// is what blocked thumbnail shards from ever reaching the server: they
|
||||
/// are 25 MB by design.
|
||||
///
|
||||
/// `OC-Total-Length` is sent on every chunk so quota is checked up front
|
||||
/// rather than at assembly, when the bytes have already been transferred.
|
||||
async fn put_chunked(
|
||||
&self,
|
||||
path: &RemotePath,
|
||||
body: Vec<u8>,
|
||||
) -> Result<Validator, RemoteError> {
|
||||
let total = body.len() as u64;
|
||||
// Named from the destination so a resumed or abandoned upload is
|
||||
// identifiable, and so two uploads cannot collide in one directory.
|
||||
let token: String = path
|
||||
.as_str()
|
||||
.bytes()
|
||||
.map(|b| match b {
|
||||
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' => (b as char).to_string(),
|
||||
_ => "-".to_string(),
|
||||
})
|
||||
.collect();
|
||||
let dir = format!(
|
||||
"{}/remote.php/dav/uploads/{}/{token}",
|
||||
self.server, self.login
|
||||
);
|
||||
|
||||
self.mkcol_url(&dir).await?;
|
||||
|
||||
// Chunks are numbered from 1 and must sort correctly as strings, which
|
||||
// is why they are zero-padded rather than bare integers.
|
||||
let chunk_size = CHUNKS.min_chunk as usize;
|
||||
for (i, chunk) in body.chunks(chunk_size).enumerate() {
|
||||
if i + 1 > CHUNKS.max_chunks as usize {
|
||||
return Err(RemoteError::Protocol(format!(
|
||||
"{total} bytes exceeds {} chunks", CHUNKS.max_chunks
|
||||
)));
|
||||
}
|
||||
let resp = self
|
||||
.client
|
||||
.put(format!("{dir}/{:05}", i + 1))
|
||||
.basic_auth(&self.login, Some(&self.password))
|
||||
.header("OC-Total-Length", total.to_string())
|
||||
.body(chunk.to_vec())
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| RemoteError::Network(e.to_string()))?;
|
||||
map_status(resp.status(), path.as_str())?;
|
||||
}
|
||||
|
||||
// Assemble. The destination is an absolute URL in the Destination
|
||||
// header, and `Overwrite: T` because a re-uploaded shard replaces the
|
||||
// one already there.
|
||||
let resp = self
|
||||
.client
|
||||
.request(
|
||||
reqwest::Method::from_bytes(b"MOVE").expect("valid method"),
|
||||
format!("{dir}/.file"),
|
||||
)
|
||||
.basic_auth(&self.login, Some(&self.password))
|
||||
.header("Destination", self.url_for(path))
|
||||
.header("Overwrite", "T")
|
||||
.header("OC-Total-Length", total.to_string())
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| RemoteError::Network(e.to_string()))?;
|
||||
map_status(resp.status(), path.as_str())?;
|
||||
|
||||
// The MOVE response carries the assembled file's ETag on Nextcloud,
|
||||
// but not on every version; fall back to asking rather than failing an
|
||||
// upload that in fact succeeded.
|
||||
if let Some(v) = resp
|
||||
.headers()
|
||||
.get(reqwest::header::ETAG)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
{
|
||||
return Ok(Validator::new(v));
|
||||
}
|
||||
self.dir_validator(path).await
|
||||
}
|
||||
|
||||
/// `MKCOL` at an absolute URL, treating "already there" as success.
|
||||
async fn mkcol_url(&self, url: &str) -> Result<(), RemoteError> {
|
||||
let resp = self
|
||||
.client
|
||||
.request(
|
||||
reqwest::Method::from_bytes(b"MKCOL").expect("valid method"),
|
||||
url,
|
||||
)
|
||||
.basic_auth(&self.login, Some(&self.password))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| RemoteError::Network(e.to_string()))?;
|
||||
|
||||
// 405 is "already exists", which is exactly what we want.
|
||||
if resp.status() == 405 {
|
||||
return Ok(());
|
||||
}
|
||||
map_status(resp.status(), url)
|
||||
}
|
||||
|
||||
async fn propfind(
|
||||
&self,
|
||||
path: &RemotePath,
|
||||
@@ -209,10 +317,14 @@ impl RemoteBackend for NextcloudBackend {
|
||||
) -> Result<Validator, RemoteError> {
|
||||
// Chunked upload is an implementation detail of put, chosen by size —
|
||||
// exposing it on the trait would leak this protocol (ARCH §8.3).
|
||||
if body.len() as u64 >= CHUNKS.single_shot_below {
|
||||
return Err(RemoteError::Unsupported(
|
||||
"chunked upload v2 not implemented yet",
|
||||
));
|
||||
//
|
||||
// A precondition cannot ride on a chunked upload: the guard belongs to
|
||||
// the assembling MOVE, not to the individual chunks, and Nextcloud
|
||||
// does not honour `If-Match` there. Large bodies are shards and
|
||||
// catalog snapshots, which are written whole and never merged, so
|
||||
// there is no conflict to guard against.
|
||||
if body.len() as u64 >= CHUNKS.single_shot_below && precond.is_none() {
|
||||
return self.put_chunked(path, body).await;
|
||||
}
|
||||
|
||||
let mut req = self
|
||||
|
||||
Reference in New Issue
Block a user