Keep originals on this device, by pin and by use
Build and test / Desktop (Linux) (push) Failing after 1s
Build and test / Android (aarch64) (push) Failing after 0s
Build and test / Layer separation (push) Failing after 1s
Traceability / Requirement traces (push) Failing after 2s

Fills in `image_cache`, which the previous commit's "On this device" filter
read but nothing wrote. Also carries in-flight work that shared these files:
the Android TLS root store, the settings page, and a regenerated
traceability report.

# Two populations, deliberately separate

An original is kept here for one of two reasons, and conflating them produces
the exact failure the feature exists to prevent.

**Pinned** originals were asked for. Pinning a collection before a trip is a
promise, so pinned rows are never evicted and never counted against the
budget — a cap that could silently delete a pinned trip would make pinning
worthless, because it could not be relied on without checking.

**Passively cached** originals are a side effect of working: develop already
downloads the whole file, so keeping it costs no bandwidth and saves the
entire transfer next time. This population is what the budget bounds, evicted
least-recently-used, because it otherwise grows until a day of culling fills
a disk.

Sharing one budget would let a large pin starve the passive cache, or let
browsing evict a pin. They are separate.

# What was built

`dr_catalog::cache` owns the bookkeeping — held tier, size, last use, pinned
— and writes the bytes; deciding to download stays with the caller, which is
what keeps a crate with no network out of the network's business. Files are
written to a temporary and renamed, so a dropped connection cannot leave a
truncated file recorded as a complete original. They are named by image id,
not filename: `Photos/IMG_0001.CR2` and `Trips/IMG_0001.CR2` are different
photographs, and a flat cache keyed on the name would serve one for the other.

`spawn_full_fetch` became read-through. A hit is a disk read; a miss stores
what it downloads and enforces the budget. A cache that cannot be opened is a
miss, not a failure to open the photograph.

Pinning writes intent — `tier_desired` — without downloading, so the button
responds immediately, and `spawn_pin_fetch` fills it in sequentially
afterwards. Sequential because these are tens of megabytes each: the lanes
that make the thumbnail sweep fast buy little against one connection's
bandwidth and cost a great deal of memory. A pin interrupted by a lost
connection resumes from where it stopped.

Schema v5 adds `pinned` and `path`. `pinned` is a column rather than something
inferred from `pinned_by_rule`, which is ON DELETE SET NULL and so cannot
answer for an image whose rule was deleted. A v4 catalog migrates in place;
existing rows default to unpinned, the safe direction.

The budget and "keep opened originals" come from the settings page rather than
a constant, and are applied at startup rather than only on change — a cache
capped at 2 GB last session would otherwise spend this one filling to the
default. Turning off keeping leaves what is already cached readable: those
bytes are paid for, and refusing them would re-download images sitting right
there, including pinned ones.

Also removes a doubled `#[test]` introduced in the previous commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-11 21:12:01 +02:00
co-authored by Claude Opus 5
parent cd75e5a4c6
commit fa12afed18
22 changed files with 4032 additions and 86 deletions
+485 -27
View File
@@ -12,6 +12,7 @@
//! timers, which is the same shape [`crate::launch_ui`] uses for login.
use std::cell::RefCell;
use std::path::PathBuf;
use std::rc::Rc;
use std::sync::mpsc::Receiver;
@@ -83,7 +84,19 @@ pub struct LibraryController {
window: RefCell<usize>,
/// Which rows already have a thumbnail fetch issued, so scrolling back
/// does not refetch what is already on screen.
requested: RefCell<std::collections::HashSet<usize>>,
/// Which images have a fetch in flight or already served, keyed on
/// `(image_id, size class)`.
///
/// **Identity, not row index.** The grid is a window over the catalog, so
/// row 7 means a different photograph after every scroll — a set of
/// indices had to be cleared on each window move, which made every visible
/// cell look unrequested and re-issued the whole screenful on every
/// scroll.
///
/// The size class is part of the key because the two resolutions are
/// fetched independently: holding the 256px one says nothing about whether
/// the large one has been asked for.
requested: RefCell<std::collections::HashSet<(i64, dr_thumbs::ThumbSize)>>,
scan_timer: RefCell<Option<slint::Timer>>,
thumb_timer: RefCell<Option<slint::Timer>>,
/// The whole-library sweep, which outlives any one grid window.
@@ -157,6 +170,10 @@ pub struct LibraryController {
/// judgement, and carrying the old one over would report a server down
/// that was never contacted.
reachability: RefCell<dr_sync::Reachability>,
/// TRACES: FR-NC-6a
/// Drains the pin downloader. Held so a second pin replaces the timer
/// rather than leaving two draining the same finished channel.
pin_timer: RefCell<Option<slint::Timer>>,
/// Narrow the grid to images whose original is stored locally.
///
/// A `Cell` beside `filter` rather than a field inside it: the rating
@@ -164,6 +181,23 @@ pub struct LibraryController {
/// predicate over the cache, and folding two different joins into one type
/// would put the cache schema inside a rating concept.
local_only: std::cell::Cell<bool>,
/// TRACES: FR-NC-6a
/// The ceiling on passively cached originals, from the settings page.
///
/// Held here rather than read from `SettingsStore` at each use because the
/// workers that need it run on threads with no config access — the same
/// reason [`library::CacheContext`] takes it as a field. A `Cell` because
/// the settings page can move it while a library is open, and the next
/// fetch must use the new figure rather than one captured at open.
cache_budget: std::cell::Cell<dr_catalog::Budget>,
/// TRACES: FR-NC-6a
/// Whether opening an image in develop keeps its original on disk.
///
/// Held beside the budget and for the same reason. Distinct from a zero
/// budget: this switches the passive population off entirely, while a
/// small budget still keeps a working set. A metered or small-disk device
/// wants the first.
keep_opened: std::cell::Cell<bool>,
}
impl LibraryController {
@@ -194,10 +228,56 @@ impl LibraryController {
sidecar_timer: RefCell::new(None),
generation: std::cell::Cell::new(0),
reachability: RefCell::new(dr_sync::Reachability::new()),
pin_timer: RefCell::new(None),
local_only: std::cell::Cell::new(false),
// The catalog's own floor until the settings page reports what the
// user has stored, which it does at startup before any fetch.
cache_budget: std::cell::Cell::new(dr_catalog::Budget::default()),
keep_opened: std::cell::Cell::new(
dr_types::CacheSettings::default().keep_opened_originals,
),
})
}
/// TRACES: FR-NC-6a
/// Whether a develop open should keep the original it downloads.
///
/// Turning it off leaves what is already cached alone: those bytes are
/// paid for, and deleting them would make the switch destructive when it
/// only means "stop adding to this".
pub fn set_keep_opened_originals(&self, keep: bool) {
self.keep_opened.set(keep);
}
/// TRACES: FR-NC-6a
/// Set the ceiling on passively cached originals, and apply it now.
///
/// Applied immediately rather than only to later downloads: a user who has
/// just lowered the limit expects the space back, and a budget that took
/// effect only on the next fetch would leave the cache over its stated
/// ceiling for as long as they browsed nothing new.
pub fn set_cache_budget(&self, bytes: Option<u64>) {
let budget = match bytes {
Some(n) => dr_catalog::Budget::bytes(n),
None => dr_catalog::Budget::unlimited(),
};
self.cache_budget.set(budget);
// Best-effort: no library open means no cache to trim, and a failure
// here must not stop the setting from being stored — the ceiling still
// applies to every fetch from now on.
let Some(dir) = self.cache_dir() else { return };
let borrow = self.catalog.borrow();
let Some(catalog) = borrow.as_ref() else { return };
match dr_catalog::Cache::open(&dir, budget)
.and_then(|cache| cache.enforce(catalog.connection()))
{
Ok(0) => {}
Ok(n) => log::info!("cache budget changed: evicted {n} original(s)"),
Err(e) => log::warn!("applying the new cache budget: {e}"),
}
}
/// TRACES: FR-CAT-9
/// Whether the app currently believes the server is unreachable.
pub fn is_offline(&self) -> bool {
@@ -209,6 +289,83 @@ impl LibraryController {
self.local_only.get()
}
/// TRACES: FR-NC-6a
/// The catalog id for a remote path currently in the grid.
///
/// The cache is keyed on `ImageId` because that is what survives a
/// server-side rename (FR-NC-5), while the develop callback carries only a
/// path. `paths` and `image_ids` are parallel to the model, so this is the
/// join between the two.
pub fn image_id_for_path(&self, path: &str) -> Option<dr_types::ImageId> {
let index = self.paths.borrow().iter().position(|p| p == path)?;
self.image_ids
.borrow()
.get(index)
.map(|id| dr_types::ImageId(*id as u64))
}
/// TRACES: FR-NC-6a
/// Where cached originals live for the open library.
///
/// Beside the catalog rather than under a system cache directory: the two
/// are per-account and are discarded together, and a cached original whose
/// catalog row is gone is unreachable anyway.
pub fn cache_dir(&self) -> Option<PathBuf> {
let borrow = self.session.borrow();
let (_, session, _) = borrow.as_ref()?;
library::catalog_path(&session.server, &session.user_id)
.parent()
.map(|p| p.join("originals"))
}
/// Open the originals cache for the current library.
///
/// Used by pinning, which writes intent against the catalog directly
/// rather than going through a fetch.
pub fn cache(&self) -> Option<dr_catalog::Cache> {
let dir = self.cache_dir()?;
match dr_catalog::Cache::open(&dir, self.cache_budget.get()) {
Ok(c) => Some(c),
Err(e) => {
// Not fatal: without a cache every open is a download, which
// is exactly the behaviour that existed before this.
log::warn!("originals cache unavailable: {e}");
None
}
}
}
/// TRACES: FR-NC-6a
/// Everything a full fetch needs to read and write the cache.
///
/// Assembled here because the develop callback holds only a path, while
/// the cache is keyed on `ImageId` and lives beside a catalog whose
/// location comes from the session. `None` where any part is missing — a
/// grid row that has scrolled away, or no library open — and the fetch
/// then simply goes to the network.
pub fn cache_context(&self, path: &str) -> Option<library::CacheContext> {
let borrow = self.session.borrow();
let (_, session, _) = borrow.as_ref()?;
let catalog_path = library::catalog_path(&session.server, &session.user_id);
let dir = catalog_path.parent()?.join("originals");
drop(borrow);
Some(library::CacheContext {
dir,
catalog_path,
image: self.image_id_for_path(path)?,
// The user's ceiling, not the catalog's floor: read at each fetch
// so a budget changed mid-session takes effect on the next one.
budget: self.cache_budget.get(),
// Gates the *write* only. Reading stays enabled either way: bytes
// already on disk were paid for, and refusing to use them because
// the user has since stopped adding new ones would re-download
// images that are sitting right there — and would make a pinned
// collection unopenable offline.
store: self.keep_opened.get(),
})
}
/// Toggle the local-only filter, resetting the window.
///
/// The offset is cleared for the same reason a scope change clears it: the
@@ -541,6 +698,243 @@ fn start_rescan(
drain_scan(window.as_weak(), ctl.clone(), coll_ctl.clone(), rx, path);
}
/// TRACES: FR-NC-6a
/// Pin the scoped collection for offline use, or release it.
///
/// Pinning is two separate things, and keeping them separate is what makes the
/// button feel immediate: recording the *intent* is a local catalog write that
/// completes at once, and downloading the bytes is a background transfer that
/// may take a very long time. The button reflects the first.
fn toggle_pin_scope(window: &AppWindow, ctl: &Rc<LibraryController>) {
let Some(scope) = *ctl.scope.borrow() else {
return;
};
let Some(cache) = ctl.cache() else {
window.set_library_error("No cache directory for this library.".into());
return;
};
let borrow = ctl.catalog.borrow();
let Some(catalog) = borrow.as_ref() else {
return;
};
// Descendants, matching what the grid shows when scoped to a set: pinning
// a parent whose children hold the photographs must pin the photographs,
// or the button would appear to do nothing.
let ids = match dr_catalog::collections::descendants(catalog.connection(), scope) {
Ok(ids) => ids,
Err(e) => {
window.set_library_error(format!("resolving collection: {e}").into());
return;
}
};
let images = collection_images(catalog, &ids);
if images.is_empty() {
window.set_library_error("Nothing in that collection to keep offline.".into());
return;
}
let pinning = !window.get_library_scope_pinned();
let result = if pinning {
cache.pin(catalog.connection(), &images)
} else {
cache.unpin(catalog.connection(), &images)
};
if let Err(e) = result {
window.set_library_error(format!("pinning: {e}").into());
return;
}
window.set_library_scope_pinned(pinning);
window.set_library_error(slint::SharedString::new());
drop(borrow);
if pinning {
log::info!("pinned {} image(s) for offline use", images.len());
start_pin_fetch(window, ctl);
} else {
// The bytes stay until the budget needs the room, so there is nothing
// to run here — unpinning withdraws a guarantee rather than deleting.
log::info!("released the pin on {} image(s)", images.len());
window.set_library_pin_total(0);
window.set_library_pin_done(0);
refresh_local_count(window, ctl);
}
}
/// Every image in the given collections, deduplicated.
///
/// An image in both a parent and a child is one photograph and must be pinned
/// once, exactly as the grid draws it once.
fn collection_images(catalog: &Catalog, ids: &[dr_types::CollectionId]) -> Vec<dr_types::ImageId> {
if ids.is_empty() {
return Vec::new();
}
let placeholders = std::iter::repeat_n("?", ids.len())
.collect::<Vec<_>>()
.join(",");
let sql = format!(
"SELECT DISTINCT image_id FROM collection_members
WHERE collection_id IN ({placeholders})"
);
let params: Vec<rusqlite::types::Value> = ids
.iter()
.map(|c| rusqlite::types::Value::Integer(c.0 as i64))
.collect();
let Ok(mut stmt) = catalog.connection().prepare(&sql) else {
return Vec::new();
};
let rows = stmt.query_map(rusqlite::params_from_iter(params.iter()), |r| {
Ok(dr_types::ImageId(r.get::<_, i64>(0)? as u64))
});
match rows {
Ok(rows) => rows.flatten().collect(),
Err(e) => {
log::debug!("listing collection images: {e}");
Vec::new()
}
}
}
/// TRACES: FR-NC-6a
/// Download whatever the pins still want, reporting progress.
fn start_pin_fetch(window: &AppWindow, ctl: &Rc<LibraryController>) {
let Some((creds, session, _)) = ctl.session.borrow().clone() else {
return;
};
let Some(cache_dir) = ctl.cache_dir() else {
return;
};
// Offline, there is nothing to download from. The pin is already recorded,
// so it resumes on reconnect rather than being lost.
if ctl.is_offline() {
log::info!("offline: the pin is recorded and will download on reconnect");
return;
}
let rx = library::spawn_pin_fetch(
creds,
session.user_id.clone(),
library::catalog_path(&session.server, &session.user_id),
cache_dir,
// Pinned originals are exempt from the budget, but a pin fetch also
// stores passively when it finds an image already cached, so the worker
// still needs the user's ceiling rather than the catalog's floor.
ctl.cache_budget.get(),
);
let timer = slint::Timer::default();
let weak = window.as_weak();
let ctl_cb = ctl.clone();
timer.start(
slint::TimerMode::Repeated,
std::time::Duration::from_millis(300),
move || {
let Some(w) = weak.upgrade() else { return };
loop {
let msg = match rx.try_recv() {
Ok(m) => m,
Err(std::sync::mpsc::TryRecvError::Empty) => return,
Err(std::sync::mpsc::TryRecvError::Disconnected) => {
w.set_library_pin_total(0);
stop(&ctl_cb.pin_timer);
return;
}
};
match msg {
library::PinMessage::Planned { total } => {
w.set_library_pin_total(total as i32);
w.set_library_pin_done(0);
}
library::PinMessage::Stored { done } => {
w.set_library_pin_done(done as i32);
// The "On this device" count grows as they land, so
// the chip agrees with the progress line beside it.
refresh_local_count(&w, &ctl_cb);
}
library::PinMessage::Done { stored, bytes } => {
log::info!(
"pin complete: {stored} original(s), {:.1} MB",
bytes as f64 / 1_048_576.0
);
w.set_library_pin_total(0);
w.set_library_pin_done(0);
refresh_local_count(&w, &ctl_cb);
stop(&ctl_cb.pin_timer);
return;
}
library::PinMessage::Failed { message, offline } => {
log::warn!("pin fetch stopped: {message}");
w.set_library_pin_total(0);
if offline {
ctl_cb
.reachability
.borrow_mut()
.mark_unreachable(message, std::time::Instant::now());
refresh_offline(&w, &ctl_cb);
} else {
w.set_library_error(format!("keeping offline: {message}").into());
}
refresh_local_count(&w, &ctl_cb);
stop(&ctl_cb.pin_timer);
return;
}
}
}
},
);
*ctl.pin_timer.borrow_mut() = Some(timer);
}
/// Refresh the "On this device" count from the catalog.
fn refresh_local_count(window: &AppWindow, ctl: &Rc<LibraryController>) {
let borrow = ctl.catalog.borrow();
let Some(catalog) = borrow.as_ref() else {
return;
};
window.set_library_local_count(library::local_original_count(catalog).unwrap_or(0) as i32);
}
/// TRACES: FR-NC-6a
/// Whether every image in the scoped collection is pinned.
///
/// Read from the catalog rather than remembered, because a pin outlives the
/// session that made it: reopening the library must show the button already
/// active, or the user would pin the same collection twice.
fn scope_is_pinned(catalog: &Catalog, images: &[dr_types::ImageId]) -> bool {
if images.is_empty() {
return false;
}
let placeholders = std::iter::repeat_n("?", images.len())
.collect::<Vec<_>>()
.join(",");
let params: Vec<rusqlite::types::Value> = images
.iter()
.map(|i| rusqlite::types::Value::Integer(i.0 as i64))
.collect();
let pinned: i64 = catalog
.connection()
.query_row(
&format!(
"SELECT count(*) FROM image_cache
WHERE pinned = 1 AND image_id IN ({placeholders})"
),
rusqlite::params_from_iter(params.iter()),
|r| r.get(0),
)
.unwrap_or(0);
pinned as usize == images.len()
}
/// TRACES: FR-CAT-9
/// Paint the connectivity state into the window.
///
@@ -646,6 +1040,19 @@ fn load_window(window: &AppWindow, ctl: &Rc<LibraryController>) {
// query rather than another predicate threaded through the scoped one.
let trash = ctl.viewing_trash.get();
// TRACES: FR-NC-6a
// Whether the newly scoped collection is already pinned. Read here rather
// than remembered, because a pin outlives the session that made it — on
// reopening the library the button has to show what the catalog says, not
// what this run happens to have done.
window.set_library_scope_pinned(match scope {
Some(id) => dr_catalog::collections::descendants(catalog.connection(), id)
.map(|ids| collection_images(catalog, &ids))
.map(|images| scope_is_pinned(catalog, &images))
.unwrap_or(false),
None => false,
});
let total = if trash {
library::total_trashed(catalog).unwrap_or(0)
} else {
@@ -1097,18 +1504,26 @@ fn request_thumbnails(window: &AppWindow, ctl: &Rc<LibraryController>) {
paths
.iter()
.enumerate()
.filter(|(i, _)| requested.insert(*i))
.map(|(i, p)| library::ThumbnailRequest {
.filter_map(|(i, p)| {
let image_id = *image_ids.get(i)?;
// Chosen from how large the cell is actually drawn, so a
// zoomed grid asks for detail a 256px thumbnail cannot give
// and a wall of small cells does not pay for it.
thumb_size: dr_thumbs::ThumbSize::for_cell(cell_pixels),
row: i,
path: p.clone(),
file_id: file_ids.get(i).copied().flatten(),
size: sizes.get(i).copied().unwrap_or(0),
image_id: image_ids.get(i).copied().unwrap_or(0),
needs_metadata: needs_md.get(i).copied().unwrap_or(false),
let thumb_size = dr_thumbs::ThumbSize::for_cell(cell_pixels);
// Keyed on the photograph, so scrolling back over a cell that
// has already been served does not ask for it again.
if !requested.insert((image_id, thumb_size)) {
return None;
}
Some(library::ThumbnailRequest {
row: i,
path: p.clone(),
file_id: file_ids.get(i).copied().flatten(),
size: sizes.get(i).copied().unwrap_or(0),
image_id,
needs_metadata: needs_md.get(i).copied().unwrap_or(false),
thumb_size,
})
})
.collect()
};
@@ -1777,8 +2192,6 @@ fn scrub_to(window: &AppWindow, ctl: &Rc<LibraryController>, when: i64) {
// the viewport sits at row 0 shows an empty grid until the user scrolls.
window.set_library_scroll_to(position as i32);
window.set_library_scroll_token(window.get_library_scroll_token() + 1);
// A new window means new rows; nothing already fetched applies to them.
ctl.requested.borrow_mut().clear();
load_window(window, ctl);
}
@@ -1893,14 +2306,10 @@ where
}
w.set_library_cell_size(next);
// Crossing the class boundary means the visible cells now want a
// resolution the store may not hold, and the window's capacity
// changed with the cell size. Both are answered by reloading.
let was = dr_thumbs::ThumbSize::for_cell(current as u32);
let now = dr_thumbs::ThumbSize::for_cell(next as u32);
if was != now {
ctl.requested.borrow_mut().clear();
}
// No need to forget anything on a class change: the class is part
// of the request key, so cells that now want the large resolution
// simply miss and ask for it, while the 256px ones they already
// hold stay served.
load_window(&w, &ctl);
});
}
@@ -1942,9 +2351,6 @@ where
return;
}
*ctl.window.borrow_mut() = capacity;
// The window's extent changed, so rows outside the old one were
// never requested and rows inside it still hold their thumbnails.
ctl.requested.borrow_mut().clear();
load_window(&w, &ctl);
});
}
@@ -2028,9 +2434,6 @@ where
}
*ctl.offset.borrow_mut() = desired;
// A different window means different rows; nothing already
// requested applies to them.
ctl.requested.borrow_mut().clear();
load_window(&w, &ctl);
});
}
@@ -2173,7 +2576,6 @@ where
// from the restored position has loaded rows above it.
let window_size = *ctl.window.borrow();
*ctl.offset.borrow_mut() = resume.saturating_sub(window_size / 4);
ctl.requested.borrow_mut().clear();
load_window(&w, &ctl);
// Before the grid is shown, not after: the markup gates it on
@@ -2324,6 +2726,17 @@ where
});
}
// TRACES: FR-NC-6a
// Pin or unpin the collection the grid is scoped to.
{
let weak = window.as_weak();
let ctl = ctl.clone();
window.on_library_toggle_pin_scope(move || {
let Some(w) = weak.upgrade() else { return };
toggle_pin_scope(&w, &ctl);
});
}
// TRACES: FR-CAT-9
// Retry now, rather than waiting out the backoff. A user who has just
// reconnected their wifi knows something the backoff does not.
@@ -2527,6 +2940,51 @@ mod tests {
assert_eq!(ThumbSize::for_cell(zoom_cell(281.0, -1) as u32), ThumbSize::Grid);
}
#[test]
fn a_request_key_survives_the_window_moving() {
use dr_thumbs::ThumbSize;
use std::collections::HashSet;
// Keyed on the photograph, not its position. The grid is a window over
// the catalog, so row 7 is a different image after every scroll — a
// set of row indices had to be cleared on each move, and every visible
// cell then looked unrequested and was re-issued.
let mut requested: HashSet<(i64, ThumbSize)> = HashSet::new();
// A screenful at rows 0..3, holding images 100..103.
for id in 100..103 {
assert!(requested.insert((id, ThumbSize::Grid)), "first sight");
}
// Scrolled: the same photographs now occupy different rows.
for id in 100..103 {
assert!(
!requested.insert((id, ThumbSize::Grid)),
"image {id} must not be requested twice"
);
}
// A genuinely new photograph still is.
assert!(requested.insert((200, ThumbSize::Grid)));
}
#[test]
fn the_two_size_classes_are_requested_independently() {
use dr_thumbs::ThumbSize;
use std::collections::HashSet;
// Holding the 256px version says nothing about the large one, so
// zooming past the boundary must still ask.
let mut requested: HashSet<(i64, ThumbSize)> = HashSet::new();
assert!(requested.insert((1, ThumbSize::Grid)));
assert!(
requested.insert((1, ThumbSize::Large)),
"the large class is a separate request"
);
assert!(!requested.insert((1, ThumbSize::Grid)));
}
#[test]
fn zoom_zero_is_the_whole_library() {
let full = (1_000, 2_000);