Keep originals on this device, by pin and by use
Fills in `image_cache`, which the previous commit's "On this device" filter read but nothing wrote. Also carries in-flight work that shared these files: the Android TLS root store, the settings page, and a regenerated traceability report. # Two populations, deliberately separate An original is kept here for one of two reasons, and conflating them produces the exact failure the feature exists to prevent. **Pinned** originals were asked for. Pinning a collection before a trip is a promise, so pinned rows are never evicted and never counted against the budget — a cap that could silently delete a pinned trip would make pinning worthless, because it could not be relied on without checking. **Passively cached** originals are a side effect of working: develop already downloads the whole file, so keeping it costs no bandwidth and saves the entire transfer next time. This population is what the budget bounds, evicted least-recently-used, because it otherwise grows until a day of culling fills a disk. Sharing one budget would let a large pin starve the passive cache, or let browsing evict a pin. They are separate. # What was built `dr_catalog::cache` owns the bookkeeping — held tier, size, last use, pinned — and writes the bytes; deciding to download stays with the caller, which is what keeps a crate with no network out of the network's business. Files are written to a temporary and renamed, so a dropped connection cannot leave a truncated file recorded as a complete original. They are named by image id, not filename: `Photos/IMG_0001.CR2` and `Trips/IMG_0001.CR2` are different photographs, and a flat cache keyed on the name would serve one for the other. `spawn_full_fetch` became read-through. A hit is a disk read; a miss stores what it downloads and enforces the budget. A cache that cannot be opened is a miss, not a failure to open the photograph. Pinning writes intent — `tier_desired` — without downloading, so the button responds immediately, and `spawn_pin_fetch` fills it in sequentially afterwards. Sequential because these are tens of megabytes each: the lanes that make the thumbnail sweep fast buy little against one connection's bandwidth and cost a great deal of memory. A pin interrupted by a lost connection resumes from where it stopped. Schema v5 adds `pinned` and `path`. `pinned` is a column rather than something inferred from `pinned_by_rule`, which is ON DELETE SET NULL and so cannot answer for an image whose rule was deleted. A v4 catalog migrates in place; existing rows default to unpinned, the safe direction. The budget and "keep opened originals" come from the settings page rather than a constant, and are applied at startup rather than only on change — a cache capped at 2 GB last session would otherwise spend this one filling to the default. Turning off keeping leaves what is already cached readable: those bytes are paid for, and refusing them would re-download images sitting right there, including pinned ones. Also removes a doubled `#[test]` introduced in the previous commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,782 @@
|
||||
//! TRACES: FR-NC-6a | FR-CAT-9 | NFR-RES-4
|
||||
//! Which originals are kept on this device, and which may be evicted.
|
||||
//!
|
||||
//! # Two populations, one table
|
||||
//!
|
||||
//! An original ends up here two ways, and conflating them produces exactly the
|
||||
//! failure the whole feature exists to prevent.
|
||||
//!
|
||||
//! **Pinned** originals were asked for. A user pins a collection before a trip
|
||||
//! and expects those photographs to be there when there is no connection —
|
||||
//! that is a promise, so pinned rows are never evicted and never counted
|
||||
//! against the budget. A cap that could silently delete a pinned trip would
|
||||
//! make pinning worthless, because the user could not rely on it without
|
||||
//! checking.
|
||||
//!
|
||||
//! **Passively cached** originals are a side effect of working: opening an
|
||||
//! image in develop downloads it, so keeping the bytes costs nothing extra and
|
||||
//! saves the whole transfer next time. This population is bounded by
|
||||
//! [`Budget`] and evicted least-recently-used, because it grows without limit
|
||||
//! otherwise — a day of culling would fill a disk.
|
||||
//!
|
||||
//! The two budgets are separate rather than shared. Sharing them means a large
|
||||
//! pin starves the passive cache, or worse, that browsing evicts a pin.
|
||||
//!
|
||||
//! # What this module does and does not own
|
||||
//!
|
||||
//! It owns the *bookkeeping*: which images are held, at what tier, how large,
|
||||
//! when last used, and which are pinned. The bytes are files under a cache
|
||||
//! directory, and [`store`](Cache::store) writes them; but deciding to
|
||||
//! download something is the caller's business, because that needs a network
|
||||
//! and this crate has none.
|
||||
//!
|
||||
//! # Why `tier_actual` is the truth
|
||||
//!
|
||||
//! `tier_desired` is what a pin asks for; `tier_actual` is what is on disk.
|
||||
//! Only the second answers "can this be opened right now", which is the
|
||||
//! question offline mode asks (FR-CAT-9). A pinned image whose download has
|
||||
//! not run yet is precisely the one that would fail, so it must not report as
|
||||
//! available.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use dr_types::{ImageId, Tier};
|
||||
use rusqlite::Connection;
|
||||
|
||||
use crate::error::CatalogError;
|
||||
|
||||
/// Default ceiling for passively cached originals.
|
||||
///
|
||||
/// 1 GB holds roughly 30 full-frame RAWs — a working session's worth, which is
|
||||
/// what this cache is for. It is deliberately modest: the passive cache is a
|
||||
/// convenience that should not quietly consume a disk, and a user who wants
|
||||
/// more kept is better served by pinning, which says so explicitly and is not
|
||||
/// subject to eviction at all.
|
||||
pub const DEFAULT_BUDGET_BYTES: u64 = 1024 * 1024 * 1024;
|
||||
|
||||
/// How much disk the passive cache may use.
|
||||
///
|
||||
/// A newtype rather than a bare `u64` so a byte count cannot be passed where a
|
||||
/// budget belongs, and to give the "unlimited" case a name — some users have a
|
||||
/// large disk and would rather never re-download.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct Budget(Option<u64>);
|
||||
|
||||
impl Default for Budget {
|
||||
fn default() -> Self {
|
||||
Self::bytes(DEFAULT_BUDGET_BYTES)
|
||||
}
|
||||
}
|
||||
|
||||
impl Budget {
|
||||
pub fn bytes(n: u64) -> Self {
|
||||
Self(Some(n))
|
||||
}
|
||||
|
||||
/// No ceiling: nothing is ever evicted for space.
|
||||
pub fn unlimited() -> Self {
|
||||
Self(None)
|
||||
}
|
||||
|
||||
pub fn limit(self) -> Option<u64> {
|
||||
self.0
|
||||
}
|
||||
|
||||
/// How much must be freed to fit `used` within this budget.
|
||||
fn overage(self, used: u64) -> u64 {
|
||||
self.0.map_or(0, |cap| used.saturating_sub(cap))
|
||||
}
|
||||
}
|
||||
|
||||
/// What is held for one image.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Entry {
|
||||
pub image: ImageId,
|
||||
/// What is actually on disk.
|
||||
pub tier: Tier,
|
||||
/// What a pin has asked for, which may be ahead of `tier`.
|
||||
pub desired: Tier,
|
||||
pub bytes: u64,
|
||||
/// Unix seconds, or `None` if never read back since being stored.
|
||||
pub last_used: Option<i64>,
|
||||
pub pinned: bool,
|
||||
/// Path relative to the cache directory.
|
||||
pub path: Option<String>,
|
||||
}
|
||||
|
||||
/// How the cache is currently filled.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub struct Usage {
|
||||
/// Bytes held by pinned originals. Not subject to the budget.
|
||||
pub pinned_bytes: u64,
|
||||
/// Bytes held by passively cached originals. What the budget bounds.
|
||||
pub passive_bytes: u64,
|
||||
pub pinned_count: usize,
|
||||
pub passive_count: usize,
|
||||
}
|
||||
|
||||
impl Usage {
|
||||
pub fn total_bytes(self) -> u64 {
|
||||
self.pinned_bytes + self.passive_bytes
|
||||
}
|
||||
}
|
||||
|
||||
/// The on-disk cache of originals, rooted at a directory.
|
||||
pub struct Cache {
|
||||
dir: PathBuf,
|
||||
budget: Budget,
|
||||
}
|
||||
|
||||
impl Cache {
|
||||
/// Open a cache rooted at `dir`, creating it if needed.
|
||||
pub fn open(dir: &Path, budget: Budget) -> Result<Self, CatalogError> {
|
||||
std::fs::create_dir_all(dir)
|
||||
.map_err(|e| CatalogError::Io(format!("creating {}: {e}", dir.display())))?;
|
||||
Ok(Self {
|
||||
dir: dir.to_path_buf(),
|
||||
budget,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn dir(&self) -> &Path {
|
||||
&self.dir
|
||||
}
|
||||
|
||||
pub fn budget(&self) -> Budget {
|
||||
self.budget
|
||||
}
|
||||
|
||||
/// Absolute path for a cached original.
|
||||
///
|
||||
/// Named by image id rather than by the remote filename: two folders on
|
||||
/// the server may hold `IMG_0001.CR2`, and a flat cache keyed on the name
|
||||
/// would have them overwrite each other. The extension is preserved so the
|
||||
/// decoder's format probe sees what it expects.
|
||||
fn relative_path(image: ImageId, source_ref: &str) -> String {
|
||||
let ext = source_ref
|
||||
.rsplit_once('.')
|
||||
.map(|(_, e)| e.to_ascii_lowercase())
|
||||
.filter(|e| !e.is_empty() && e.len() <= 8 && e.chars().all(|c| c.is_ascii_alphanumeric()))
|
||||
.unwrap_or_else(|| "bin".to_string());
|
||||
format!("{}.{ext}", image.0)
|
||||
}
|
||||
|
||||
/// Store an original's bytes and record it.
|
||||
///
|
||||
/// `pinned` says which population this belongs to. Storing an image that
|
||||
/// is already present updates it rather than duplicating — the same
|
||||
/// photograph opened twice is one cache entry, and the second store simply
|
||||
/// refreshes the bytes and the timestamp.
|
||||
///
|
||||
/// Does **not** evict. The caller runs [`enforce`](Self::enforce) once it
|
||||
/// has finished storing, so a batch of downloads is trimmed once rather
|
||||
/// than after every file.
|
||||
pub fn store(
|
||||
&self,
|
||||
conn: &Connection,
|
||||
image: ImageId,
|
||||
source_ref: &str,
|
||||
bytes: &[u8],
|
||||
pinned: bool,
|
||||
now: i64,
|
||||
) -> Result<(), CatalogError> {
|
||||
let rel = Self::relative_path(image, source_ref);
|
||||
let abs = self.dir.join(&rel);
|
||||
|
||||
// Written to a temporary and renamed, so a crash or a dropped
|
||||
// connection mid-write cannot leave a truncated file that the catalog
|
||||
// records as a complete original — which would then fail to decode
|
||||
// with no indication that the *cache* was at fault rather than the
|
||||
// photograph.
|
||||
let tmp = abs.with_extension("partial");
|
||||
std::fs::write(&tmp, bytes)
|
||||
.map_err(|e| CatalogError::Io(format!("writing {}: {e}", tmp.display())))?;
|
||||
std::fs::rename(&tmp, &abs)
|
||||
.map_err(|e| CatalogError::Io(format!("renaming {}: {e}", abs.display())))?;
|
||||
|
||||
// `pinned` is OR-ed rather than assigned: an image that was already
|
||||
// pinned must not be demoted to evictable because it happened to be
|
||||
// opened in develop, which is a passive store.
|
||||
conn.execute(
|
||||
"INSERT INTO image_cache
|
||||
(image_id, tier_actual, tier_desired, bytes, last_used, pinned, path)
|
||||
VALUES (?1, ?2, ?2, ?3, ?4, ?5, ?6)
|
||||
ON CONFLICT(image_id) DO UPDATE SET
|
||||
tier_actual = ?2,
|
||||
tier_desired = max(tier_desired, ?2),
|
||||
bytes = ?3,
|
||||
last_used = ?4,
|
||||
pinned = max(pinned, ?5),
|
||||
path = ?6",
|
||||
rusqlite::params![
|
||||
image.0 as i64,
|
||||
Tier::Original.stored(),
|
||||
bytes.len() as i64,
|
||||
now,
|
||||
i64::from(pinned),
|
||||
rel,
|
||||
],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Read a cached original back, if it is here.
|
||||
///
|
||||
/// Touches `last_used`, which is what makes the eviction order reflect
|
||||
/// actual use rather than download order. A read that finds the row but
|
||||
/// not the file repairs the catalog rather than returning bytes it does
|
||||
/// not have — the two can diverge if a user clears the directory by hand.
|
||||
pub fn load(
|
||||
&self,
|
||||
conn: &Connection,
|
||||
image: ImageId,
|
||||
now: i64,
|
||||
) -> Result<Option<Vec<u8>>, CatalogError> {
|
||||
let path: Option<String> = conn
|
||||
.query_row(
|
||||
"SELECT path FROM image_cache
|
||||
WHERE image_id = ?1 AND tier_actual >= ?2",
|
||||
rusqlite::params![image.0 as i64, Tier::Original.stored()],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.ok()
|
||||
.flatten();
|
||||
|
||||
let Some(rel) = path else { return Ok(None) };
|
||||
let abs = self.dir.join(&rel);
|
||||
|
||||
match std::fs::read(&abs) {
|
||||
Ok(bytes) => {
|
||||
conn.execute(
|
||||
"UPDATE image_cache SET last_used = ?2 WHERE image_id = ?1",
|
||||
rusqlite::params![image.0 as i64, now],
|
||||
)?;
|
||||
Ok(Some(bytes))
|
||||
}
|
||||
Err(e) => {
|
||||
// The file is gone but the row says it is here. Believing the
|
||||
// row would report the image as locally available for ever
|
||||
// while every open failed.
|
||||
log::debug!("cached original {} missing, forgetting it: {e}", abs.display());
|
||||
self.forget(conn, &[image])?;
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether an image's original is on this device.
|
||||
pub fn holds_original(&self, conn: &Connection, image: ImageId) -> bool {
|
||||
conn.query_row(
|
||||
"SELECT 1 FROM image_cache
|
||||
WHERE image_id = ?1 AND tier_actual >= ?2",
|
||||
rusqlite::params![image.0 as i64, Tier::Original.stored()],
|
||||
|_| Ok(()),
|
||||
)
|
||||
.is_ok()
|
||||
}
|
||||
|
||||
/// Mark images as pinned, so they are kept regardless of the budget.
|
||||
///
|
||||
/// Pinning records the *intent* — `tier_desired` — without downloading
|
||||
/// anything: the download needs a network, which belongs to the caller.
|
||||
/// An image already cached passively becomes pinned in place, keeping its
|
||||
/// bytes rather than re-fetching them.
|
||||
pub fn pin(&self, conn: &Connection, images: &[ImageId]) -> Result<usize, CatalogError> {
|
||||
self.set_pinned(conn, images, true)
|
||||
}
|
||||
|
||||
/// Release a pin, returning those images to the evictable population.
|
||||
///
|
||||
/// The bytes stay until eviction needs the room. Deleting immediately
|
||||
/// would make unpinning destructive, when it is meant only to withdraw a
|
||||
/// guarantee.
|
||||
pub fn unpin(&self, conn: &Connection, images: &[ImageId]) -> Result<usize, CatalogError> {
|
||||
self.set_pinned(conn, images, false)
|
||||
}
|
||||
|
||||
fn set_pinned(
|
||||
&self,
|
||||
conn: &Connection,
|
||||
images: &[ImageId],
|
||||
pinned: bool,
|
||||
) -> Result<usize, CatalogError> {
|
||||
if images.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let tx = conn.unchecked_transaction()?;
|
||||
let mut n = 0;
|
||||
for image in images {
|
||||
n += tx.execute(
|
||||
"INSERT INTO image_cache (image_id, tier_actual, tier_desired, bytes, pinned)
|
||||
VALUES (?1, ?2, ?3, 0, ?4)
|
||||
ON CONFLICT(image_id) DO UPDATE SET
|
||||
pinned = ?4,
|
||||
-- A pin raises the target; releasing one lowers it back to
|
||||
-- whatever is actually held, so a released image is not
|
||||
-- left permanently claiming it wants an original.
|
||||
tier_desired = CASE WHEN ?4 = 1 THEN ?3 ELSE tier_actual END",
|
||||
rusqlite::params![
|
||||
image.0 as i64,
|
||||
Tier::Metadata.stored(),
|
||||
Tier::Original.stored(),
|
||||
i64::from(pinned),
|
||||
],
|
||||
)?;
|
||||
}
|
||||
tx.commit()?;
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
/// Images a pin wants but which are not yet downloaded.
|
||||
///
|
||||
/// The work list for whatever fetches originals. Ordered by id for a
|
||||
/// stable, resumable sequence rather than an arbitrary one.
|
||||
pub fn pending_pins(&self, conn: &Connection) -> Result<Vec<ImageId>, CatalogError> {
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT image_id FROM image_cache
|
||||
WHERE pinned = 1 AND tier_actual < tier_desired
|
||||
ORDER BY image_id",
|
||||
)?;
|
||||
let rows = stmt
|
||||
.query_map([], |r| Ok(ImageId(r.get::<_, i64>(0)? as u64)))?
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
/// How full the cache is, split by population.
|
||||
///
|
||||
/// Counts only rows that actually hold an original: a pin that has not
|
||||
/// downloaded yet occupies no disk, and counting its intent would evict
|
||||
/// real files to make room for bytes that do not exist.
|
||||
pub fn usage(&self, conn: &Connection) -> Result<Usage, CatalogError> {
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT pinned, count(*), coalesce(sum(bytes), 0)
|
||||
FROM image_cache
|
||||
WHERE tier_actual >= ?1
|
||||
GROUP BY pinned",
|
||||
)?;
|
||||
let mut usage = Usage::default();
|
||||
let rows = stmt.query_map(rusqlite::params![Tier::Original.stored()], |r| {
|
||||
Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?, r.get::<_, i64>(2)?))
|
||||
})?;
|
||||
for row in rows {
|
||||
let (pinned, count, bytes) = row?;
|
||||
if pinned == 1 {
|
||||
usage.pinned_count = count as usize;
|
||||
usage.pinned_bytes = bytes as u64;
|
||||
} else {
|
||||
usage.passive_count = count as usize;
|
||||
usage.passive_bytes = bytes as u64;
|
||||
}
|
||||
}
|
||||
Ok(usage)
|
||||
}
|
||||
|
||||
/// Evict least-recently-used passive entries until the budget is met.
|
||||
///
|
||||
/// Returns how many images were dropped. Pinned entries are never
|
||||
/// candidates, which is the guarantee that makes a pin worth making.
|
||||
///
|
||||
/// A row whose file has already vanished is still dropped from the
|
||||
/// catalog: it frees no disk, but leaving it would let a phantom entry
|
||||
/// hold the cache permanently over budget and evict real files in its
|
||||
/// place.
|
||||
pub fn enforce(&self, conn: &Connection) -> Result<usize, CatalogError> {
|
||||
let usage = self.usage(conn)?;
|
||||
let mut over = self.budget.overage(usage.passive_bytes);
|
||||
if over == 0 {
|
||||
return Ok(0);
|
||||
}
|
||||
|
||||
// Oldest first. `last_used IS NULL` sorts first deliberately: a row
|
||||
// that has never been read back is the least valuable thing here.
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT image_id, bytes, path FROM image_cache
|
||||
WHERE pinned = 0 AND tier_actual >= ?1
|
||||
ORDER BY last_used IS NULL DESC, last_used ASC",
|
||||
)?;
|
||||
let candidates = stmt
|
||||
.query_map(rusqlite::params![Tier::Original.stored()], |r| {
|
||||
Ok((
|
||||
ImageId(r.get::<_, i64>(0)? as u64),
|
||||
r.get::<_, i64>(1)? as u64,
|
||||
r.get::<_, Option<String>>(2)?,
|
||||
))
|
||||
})?
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
|
||||
let mut evicted = Vec::new();
|
||||
for (image, bytes, path) in candidates {
|
||||
if over == 0 {
|
||||
break;
|
||||
}
|
||||
if let Some(rel) = path {
|
||||
let abs = self.dir.join(rel);
|
||||
if let Err(e) = std::fs::remove_file(&abs) {
|
||||
// Already gone is the common case and not a failure; the
|
||||
// row still has to go, or it accounts for space nothing
|
||||
// occupies.
|
||||
log::debug!("evicting {}: {e}", abs.display());
|
||||
}
|
||||
}
|
||||
over = over.saturating_sub(bytes);
|
||||
evicted.push(image);
|
||||
}
|
||||
|
||||
let n = evicted.len();
|
||||
self.forget(conn, &evicted)?;
|
||||
Ok(n)
|
||||
}
|
||||
|
||||
/// Drop cache rows, without touching files.
|
||||
///
|
||||
/// The row is reduced to `Metadata` rather than deleted, so a pin recorded
|
||||
/// against it survives: unpinning is the only thing that should clear a
|
||||
/// pin, and eviction of the bytes is not unpinning.
|
||||
fn forget(&self, conn: &Connection, images: &[ImageId]) -> Result<(), CatalogError> {
|
||||
if images.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let tx = conn.unchecked_transaction()?;
|
||||
for image in images {
|
||||
tx.execute(
|
||||
"UPDATE image_cache
|
||||
SET tier_actual = ?2, bytes = 0, path = NULL
|
||||
WHERE image_id = ?1",
|
||||
rusqlite::params![image.0 as i64, Tier::Metadata.stored()],
|
||||
)?;
|
||||
}
|
||||
tx.commit()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Everything currently held, newest use first. For a cache management view.
|
||||
pub fn entries(&self, conn: &Connection) -> Result<Vec<Entry>, CatalogError> {
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT image_id, tier_actual, tier_desired, bytes, last_used, pinned, path
|
||||
FROM image_cache
|
||||
WHERE tier_actual >= ?1
|
||||
ORDER BY last_used IS NULL, last_used DESC",
|
||||
)?;
|
||||
let rows = stmt
|
||||
.query_map(rusqlite::params![Tier::Original.stored()], |r| {
|
||||
Ok(Entry {
|
||||
image: ImageId(r.get::<_, i64>(0)? as u64),
|
||||
tier: Tier::from_stored(r.get(1)?),
|
||||
desired: Tier::from_stored(r.get(2)?),
|
||||
bytes: r.get::<_, i64>(3)? as u64,
|
||||
last_used: r.get(4)?,
|
||||
pinned: r.get::<_, i64>(5)? == 1,
|
||||
path: r.get(6)?,
|
||||
})
|
||||
})?
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
Ok(rows)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::Catalog;
|
||||
|
||||
/// Distinguishes concurrent fixtures. The harness runs tests in parallel,
|
||||
/// and a shared directory would have one test's eviction delete another's
|
||||
/// files.
|
||||
static SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
|
||||
|
||||
/// A scratch directory that is fresh for each call.
|
||||
fn tempdir() -> PathBuf {
|
||||
let base = std::env::temp_dir().join(format!(
|
||||
"dr-cache-test-{}-{}",
|
||||
std::process::id(),
|
||||
SEQ.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
|
||||
));
|
||||
let _ = std::fs::remove_dir_all(&base);
|
||||
std::fs::create_dir_all(&base).unwrap();
|
||||
base
|
||||
}
|
||||
|
||||
/// A catalog with `n` images, and a cache in a scratch directory.
|
||||
fn fixture(n: usize) -> (Catalog, Cache, PathBuf, Vec<ImageId>) {
|
||||
fixture_with(n, Budget::bytes(1000))
|
||||
}
|
||||
|
||||
fn fixture_with(n: usize, budget: Budget) -> (Catalog, Cache, PathBuf, Vec<ImageId>) {
|
||||
let catalog = Catalog::in_memory().unwrap();
|
||||
catalog
|
||||
.connection()
|
||||
.execute(
|
||||
"INSERT INTO roots (id, kind, label) VALUES (1, 'remote', 'test')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let mut ids = Vec::new();
|
||||
for i in 0..n {
|
||||
catalog
|
||||
.connection()
|
||||
.execute(
|
||||
"INSERT INTO images (root_id, source_ref, added_at)
|
||||
VALUES (1, ?1, 0)",
|
||||
rusqlite::params![format!("Photos/img{i:03}.CR2")],
|
||||
)
|
||||
.unwrap();
|
||||
ids.push(ImageId(
|
||||
catalog.connection().last_insert_rowid() as u64
|
||||
));
|
||||
}
|
||||
let dir = tempdir();
|
||||
let cache = Cache::open(&dir, budget).unwrap();
|
||||
(catalog, cache, dir, ids)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_stored_original_reads_back() {
|
||||
let (cat, cache, _dir, ids) = fixture(1);
|
||||
cache
|
||||
.store(cat.connection(), ids[0], "a.CR2", b"raw bytes", false, 10)
|
||||
.unwrap();
|
||||
|
||||
assert!(cache.holds_original(cat.connection(), ids[0]));
|
||||
assert_eq!(
|
||||
cache.load(cat.connection(), ids[0], 20).unwrap().as_deref(),
|
||||
Some(&b"raw bytes"[..])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_image_never_stored_is_absent() {
|
||||
let (cat, cache, _dir, ids) = fixture(1);
|
||||
assert!(!cache.holds_original(cat.connection(), ids[0]));
|
||||
assert_eq!(cache.load(cat.connection(), ids[0], 0).unwrap(), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn eviction_takes_the_least_recently_used_first() {
|
||||
let (cat, cache, _dir, ids) = fixture(3);
|
||||
// 400 each against a 1000 budget: storing the third puts it 200 over.
|
||||
let bytes = vec![0u8; 400];
|
||||
cache.store(cat.connection(), ids[0], "a.CR2", &bytes, false, 10).unwrap();
|
||||
cache.store(cat.connection(), ids[1], "b.CR2", &bytes, false, 20).unwrap();
|
||||
cache.store(cat.connection(), ids[2], "c.CR2", &bytes, false, 30).unwrap();
|
||||
|
||||
// Touch the oldest so it is no longer the least recently used.
|
||||
cache.load(cat.connection(), ids[0], 40).unwrap();
|
||||
|
||||
assert_eq!(cache.enforce(cat.connection()).unwrap(), 1);
|
||||
// ids[1] was the stalest by the time eviction ran.
|
||||
assert!(!cache.holds_original(cat.connection(), ids[1]));
|
||||
assert!(cache.holds_original(cat.connection(), ids[0]));
|
||||
assert!(cache.holds_original(cat.connection(), ids[2]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_pinned_original_is_never_evicted() {
|
||||
// The guarantee the whole feature rests on: a pinned trip must still
|
||||
// be there after a day of browsing pushes the cache over its cap.
|
||||
let (cat, cache, _dir, ids) = fixture(3);
|
||||
let bytes = vec![0u8; 800];
|
||||
|
||||
cache.store(cat.connection(), ids[0], "a.CR2", &bytes, true, 10).unwrap();
|
||||
cache.store(cat.connection(), ids[1], "b.CR2", &bytes, false, 20).unwrap();
|
||||
cache.store(cat.connection(), ids[2], "c.CR2", &bytes, false, 30).unwrap();
|
||||
|
||||
cache.enforce(cat.connection()).unwrap();
|
||||
|
||||
assert!(
|
||||
cache.holds_original(cat.connection(), ids[0]),
|
||||
"the pinned original survives even though it is the oldest"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pinned_bytes_do_not_count_against_the_budget() {
|
||||
// Otherwise a large pin starves the passive cache into evicting
|
||||
// everything, and browsing becomes uncacheable the moment a trip is
|
||||
// pinned.
|
||||
let (cat, cache, _dir, ids) = fixture(2);
|
||||
cache
|
||||
.store(cat.connection(), ids[0], "a.CR2", &vec![0u8; 5000], true, 10)
|
||||
.unwrap();
|
||||
cache
|
||||
.store(cat.connection(), ids[1], "b.CR2", &vec![0u8; 500], false, 20)
|
||||
.unwrap();
|
||||
|
||||
// Pinned use is far past the 1000 budget, but the passive 500 fits.
|
||||
assert_eq!(cache.enforce(cat.connection()).unwrap(), 0);
|
||||
assert!(cache.holds_original(cat.connection(), ids[1]));
|
||||
|
||||
let usage = cache.usage(cat.connection()).unwrap();
|
||||
assert_eq!(usage.pinned_bytes, 5000);
|
||||
assert_eq!(usage.passive_bytes, 500);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unlimited_budget_evicts_nothing() {
|
||||
let (cat, cache, _dir, ids) = fixture_with(2, Budget::unlimited());
|
||||
for (i, id) in ids.iter().enumerate() {
|
||||
cache
|
||||
.store(cat.connection(), *id, "a.CR2", &vec![0u8; 100_000], false, i as i64)
|
||||
.unwrap();
|
||||
}
|
||||
assert_eq!(cache.enforce(cat.connection()).unwrap(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pinning_records_intent_without_bytes() {
|
||||
// A pin is not a download: it says what should be here, and something
|
||||
// with a network makes it so.
|
||||
let (cat, cache, _dir, ids) = fixture(2);
|
||||
cache.pin(cat.connection(), &ids).unwrap();
|
||||
|
||||
assert!(!cache.holds_original(cat.connection(), ids[0]));
|
||||
assert_eq!(cache.pending_pins(cat.connection()).unwrap(), ids);
|
||||
assert_eq!(cache.usage(cat.connection()).unwrap().pinned_bytes, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_downloaded_pin_stops_being_pending() {
|
||||
let (cat, cache, _dir, ids) = fixture(2);
|
||||
cache.pin(cat.connection(), &ids).unwrap();
|
||||
cache
|
||||
.store(cat.connection(), ids[0], "a.CR2", b"bytes", true, 10)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(cache.pending_pins(cat.connection()).unwrap(), vec![ids[1]]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pinning_an_already_cached_image_keeps_its_bytes() {
|
||||
// Re-downloading something already on disk because the user pinned it
|
||||
// would be the most visible possible waste.
|
||||
let (cat, cache, _dir, ids) = fixture(1);
|
||||
cache
|
||||
.store(cat.connection(), ids[0], "a.CR2", b"raw bytes", false, 10)
|
||||
.unwrap();
|
||||
cache.pin(cat.connection(), &ids).unwrap();
|
||||
|
||||
assert!(cache.pending_pins(cat.connection()).unwrap().is_empty());
|
||||
assert_eq!(
|
||||
cache.load(cat.connection(), ids[0], 20).unwrap().as_deref(),
|
||||
Some(&b"raw bytes"[..])
|
||||
);
|
||||
assert_eq!(cache.usage(cat.connection()).unwrap().pinned_bytes, 9);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn opening_a_pinned_image_does_not_unpin_it() {
|
||||
// The develop path stores passively. If that overwrote `pinned`, then
|
||||
// simply *looking at* a pinned photograph would silently make it
|
||||
// evictable — the pin would decay through use.
|
||||
let (cat, cache, _dir, ids) = fixture(1);
|
||||
cache.pin(cat.connection(), &ids).unwrap();
|
||||
cache
|
||||
.store(cat.connection(), ids[0], "a.CR2", b"bytes", false, 10)
|
||||
.unwrap();
|
||||
|
||||
let entries = cache.entries(cat.connection()).unwrap();
|
||||
assert!(entries[0].pinned, "still pinned after a passive store");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unpinning_keeps_the_bytes_but_makes_them_evictable() {
|
||||
let (cat, cache, _dir, ids) = fixture(2);
|
||||
cache
|
||||
.store(cat.connection(), ids[0], "a.CR2", &vec![0u8; 800], true, 10)
|
||||
.unwrap();
|
||||
cache.unpin(cat.connection(), &ids[0..1]).unwrap();
|
||||
|
||||
// Still here — unpinning withdraws a guarantee, it does not delete.
|
||||
assert!(cache.holds_original(cat.connection(), ids[0]));
|
||||
|
||||
// But now it is a candidate.
|
||||
cache
|
||||
.store(cat.connection(), ids[1], "b.CR2", &vec![0u8; 800], false, 20)
|
||||
.unwrap();
|
||||
assert_eq!(cache.enforce(cat.connection()).unwrap(), 1);
|
||||
assert!(!cache.holds_original(cat.connection(), ids[0]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_file_is_forgotten_rather_than_reported_present() {
|
||||
// A user clearing the cache directory by hand must not leave every
|
||||
// image claiming to be local while every open fails.
|
||||
let (cat, cache, dir, ids) = fixture_with(1, Budget::bytes(1000));
|
||||
cache
|
||||
.store(cat.connection(), ids[0], "a.CR2", b"bytes", false, 10)
|
||||
.unwrap();
|
||||
|
||||
for entry in std::fs::read_dir(&dir).unwrap() {
|
||||
std::fs::remove_file(entry.unwrap().path()).unwrap();
|
||||
}
|
||||
|
||||
assert_eq!(cache.load(cat.connection(), ids[0], 20).unwrap(), None);
|
||||
assert!(!cache.holds_original(cat.connection(), ids[0]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn storing_the_same_image_twice_is_one_entry() {
|
||||
let (cat, cache, _dir, ids) = fixture(1);
|
||||
cache.store(cat.connection(), ids[0], "a.CR2", b"first", false, 10).unwrap();
|
||||
cache.store(cat.connection(), ids[0], "a.CR2", b"second try", false, 20).unwrap();
|
||||
|
||||
let usage = cache.usage(cat.connection()).unwrap();
|
||||
assert_eq!(usage.passive_count, 1);
|
||||
assert_eq!(usage.passive_bytes, 10, "the later size, not the sum");
|
||||
assert_eq!(
|
||||
cache.load(cat.connection(), ids[0], 30).unwrap().as_deref(),
|
||||
Some(&b"second try"[..])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_images_with_the_same_filename_do_not_collide() {
|
||||
// `Photos/IMG_0001.CR2` and `Trips/IMG_0001.CR2` are different
|
||||
// photographs; a cache keyed on the filename would serve one for the
|
||||
// other, which is the worst failure this cache could have.
|
||||
let (cat, cache, _dir, ids) = fixture(2);
|
||||
cache
|
||||
.store(cat.connection(), ids[0], "Photos/IMG_0001.CR2", b"first", false, 10)
|
||||
.unwrap();
|
||||
cache
|
||||
.store(cat.connection(), ids[1], "Trips/IMG_0001.CR2", b"second", false, 20)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
cache.load(cat.connection(), ids[0], 30).unwrap().as_deref(),
|
||||
Some(&b"first"[..])
|
||||
);
|
||||
assert_eq!(
|
||||
cache.load(cat.connection(), ids[1], 30).unwrap().as_deref(),
|
||||
Some(&b"second"[..])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn eviction_stops_once_the_budget_is_met() {
|
||||
// Evicting everything on a small overage would throw away a working
|
||||
// set to reclaim a few bytes.
|
||||
let (cat, cache, _dir, ids) = fixture(3);
|
||||
for (i, id) in ids.iter().enumerate() {
|
||||
cache
|
||||
.store(cat.connection(), *id, "a.CR2", &vec![0u8; 400], false, i as i64)
|
||||
.unwrap();
|
||||
}
|
||||
// 1200 held against 1000: dropping one 400-byte entry suffices.
|
||||
assert_eq!(cache.enforce(cat.connection()).unwrap(), 1);
|
||||
assert_eq!(cache.usage(cat.connection()).unwrap().passive_count, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_extensionless_source_still_gets_a_path() {
|
||||
let (cat, cache, _dir, ids) = fixture(1);
|
||||
cache
|
||||
.store(cat.connection(), ids[0], "Photos/no-extension", b"bytes", false, 10)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
cache.load(cat.connection(), ids[0], 20).unwrap().as_deref(),
|
||||
Some(&b"bytes"[..])
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -30,6 +30,7 @@ use std::path::Path;
|
||||
use dr_types::{Availability, ImageId};
|
||||
use rusqlite::Connection;
|
||||
|
||||
pub mod cache;
|
||||
pub mod collections;
|
||||
pub mod error;
|
||||
pub mod jobs;
|
||||
@@ -41,6 +42,7 @@ pub mod schema;
|
||||
pub mod sync;
|
||||
pub mod trash;
|
||||
|
||||
pub use cache::{Budget, Cache, DEFAULT_BUDGET_BYTES};
|
||||
pub use collections::{Collection, CollectionKind, TreeRow};
|
||||
pub use error::CatalogError;
|
||||
pub use jobs::{Job, JobKind, Priority};
|
||||
|
||||
@@ -15,7 +15,7 @@ use rusqlite::Connection;
|
||||
use crate::error::CatalogError;
|
||||
|
||||
/// Schema version this build writes and understands.
|
||||
pub const SCHEMA_VERSION: i64 = 4;
|
||||
pub const SCHEMA_VERSION: i64 = 5;
|
||||
|
||||
/// Apply migrations up to [`SCHEMA_VERSION`].
|
||||
///
|
||||
@@ -60,6 +60,12 @@ pub fn migrate(conn: &Connection) -> Result<i64, CatalogError> {
|
||||
tx.pragma_update(None, "user_version", 4)?;
|
||||
tx.commit()?;
|
||||
}
|
||||
if from < 5 {
|
||||
let tx = conn.unchecked_transaction()?;
|
||||
tx.execute_batch(V5)?;
|
||||
tx.pragma_update(None, "user_version", 5)?;
|
||||
tx.commit()?;
|
||||
}
|
||||
|
||||
Ok(from)
|
||||
}
|
||||
@@ -220,6 +226,34 @@ fn stem_of(path: &str) -> &str {
|
||||
}
|
||||
}
|
||||
|
||||
const V5: &str = r#"
|
||||
-- TRACES: FR-NC-6a | FR-CAT-9 | NFR-RES-4
|
||||
-- Offline availability: what is kept, why it is kept, and where it lives.
|
||||
--
|
||||
-- `pinned` separates a promise from a convenience, and the distinction has to
|
||||
-- be a *column* rather than something inferred from `pinned_by_rule`. A pin is
|
||||
-- the user saying "this collection comes with me"; a passively cached original
|
||||
-- is the app noticing they opened something. Only the second is evictable, so
|
||||
-- the eviction query has to be able to ask the question directly — and it has
|
||||
-- to keep answering correctly for an image whose pinning rule was since
|
||||
-- deleted, which `pinned_by_rule` alone cannot do because it is
|
||||
-- ON DELETE SET NULL.
|
||||
ALTER TABLE image_cache ADD COLUMN pinned INTEGER NOT NULL DEFAULT 0;
|
||||
|
||||
-- Where the cached original actually is, relative to the cache directory.
|
||||
-- Relative rather than absolute: the library moves between machines and
|
||||
-- between an app sandbox and a user directory, and an absolute path baked in
|
||||
-- at download time would break on every one of those.
|
||||
ALTER TABLE image_cache ADD COLUMN path TEXT;
|
||||
|
||||
-- Eviction reads exactly this: unpinned rows, oldest use first. Partial on
|
||||
-- `pinned = 0` because pinned rows are never candidates and including them
|
||||
-- would make the index proportional to the whole library rather than to the
|
||||
-- passive cache.
|
||||
CREATE INDEX image_cache_evictable ON image_cache(last_used)
|
||||
WHERE pinned = 0;
|
||||
"#;
|
||||
|
||||
const V4: &str = r#"
|
||||
-- TRACES: FR-CAT-15
|
||||
-- Soft delete. A trashed image is a real file that has been *moved* to a trash
|
||||
@@ -596,6 +630,51 @@ mod tests {
|
||||
assert_eq!(backfilled(&c, "shadowed_by"), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_v4_catalog_gains_the_pinning_columns() {
|
||||
// TRACES: FR-NC-6a
|
||||
// An existing library must not have to be rescanned to gain offline
|
||||
// pinning. The rows are already there; only the columns are new.
|
||||
let c = mem();
|
||||
c.execute_batch(V1).unwrap();
|
||||
c.execute_batch(V2).unwrap();
|
||||
c.execute_batch(V3).unwrap();
|
||||
c.execute_batch(V4).unwrap();
|
||||
c.pragma_update(None, "user_version", 4).unwrap();
|
||||
c.execute(
|
||||
"INSERT INTO roots(id, kind, label) VALUES (1, 'remote', 'lib')",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
c.execute(
|
||||
"INSERT INTO images(id, root_id, source_ref, added_at)
|
||||
VALUES (7, 1, 'IMG_7.CR2', 0)",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
// A cache row written before pinning existed.
|
||||
c.execute(
|
||||
"INSERT INTO image_cache(image_id, tier_actual, bytes) VALUES (7, 2, 100)",
|
||||
[],
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(migrate(&c).unwrap(), 4, "migrated from v4");
|
||||
|
||||
// The pre-existing row survives, and defaults to unpinned — the safe
|
||||
// direction, since claiming a pin nobody made would exempt it from
|
||||
// eviction for ever.
|
||||
let (pinned, bytes): (i64, i64) = c
|
||||
.query_row(
|
||||
"SELECT pinned, bytes FROM image_cache WHERE image_id = 7",
|
||||
[],
|
||||
|r| Ok((r.get(0)?, r.get(1)?)),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(pinned, 0);
|
||||
assert_eq!(bytes, 100, "the existing row is untouched");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stems_ignore_directories_containing_dots() {
|
||||
assert_eq!(stem_of("2026.08/IMG_1.CR2"), "IMG_1");
|
||||
|
||||
Reference in New Issue
Block a user