Backfill the catalog once per state, not on every open

Catalog::open ran schema::backfill every time, and every worker thread
opens its own connection. A develop landing made five opens, and each
paid the RAW/JPEG pairing, the default-version anti-join over every
image, the uuid pass over every default version and the keyword check:
17 ms of CPU an open on a copy of the reference catalog, ~80 ms a
landing, to confirm that nothing had changed since the open before.

Everything the backfill repairs is a row some write added: an image a
scan inserted, a version or keyword assignment a merge brought in. So
the open now reads a stamp - user_version, max(id) of images and
versions, max(rowid) of keywords, and the file's device and inode - and
skips the backfill when the stamp matches the one recorded at this
path's last backfill in this process. The maxima are each the last page
of a b-tree; an open that skips costs ~1 ms.

The backfill still runs:
- on the first open in a process (nothing recorded yet);
- on any open that migrated the schema, unconditionally;
- after a pull: merge_remote forgets the path, so the next open
  backfills even when every incoming row collided and nothing moved;
- when the file is replaced under its name: the inode is in the stamp,
  and recovery::set_aside, the first step of a restore and a rebuild,
  forgets the path;
- when another process or thread adds rows, because the stamp is read
  from the file, not from anything this process did.

The stamp is taken before the backfill, not after. Read after, it would
describe the backfill's own inserts, and could record an image another
connection inserted in between as covered when it was not. Read before,
the worst case is one redundant pass after a backfill that did real work.

Kept in memory rather than in the catalog: a stamp row would need a
table an older build does not have and would travel in the sync
snapshot, where a flag from another device's catalog says nothing about
this one. No schema version bump, so the tablet on 0.16.0 still reads
the snapshot. Tests cover the skip, a scan's new image, a migration, a
pull and a replaced file.
This commit is contained in:
2026-09-26 14:26:10 -04:00
parent 2226d543f9
commit ffdd640170
5 changed files with 324 additions and 4 deletions
+13 -2
View File
@@ -37,6 +37,7 @@ use dr_types::{Availability, ImageId};
use rusqlite::Connection;
pub mod albums;
mod backfilled;
pub mod bursts;
pub mod cache;
pub mod collections;
@@ -249,8 +250,18 @@ impl Catalog {
// A migration adds a column; it cannot know what the value should be
// for rows that already existed. Backfilling on open is what stops
// those rows being silently partial.
for (what, n) in schema::backfill(&conn)? {
log::info!("backfilled {what} for {n} row(s) (schema was v{from})");
//
// Once per catalog state rather than once per open (NFR-P9): every
// worker thread opens its own connection, and a develop landing made
// five, each paying the whole backfill to confirm nothing had changed.
// [`backfilled`] says what "changed" means and why it is enough. A
// migration always backfills, stamp or no stamp.
let stamp = backfilled::stamp(&conn, path)?;
if from < schema::SCHEMA_VERSION || !backfilled::is_current(path, &stamp) {
for (what, n) in schema::backfill(&conn)? {
log::info!("backfilled {what} for {n} row(s) (schema was v{from})");
}
backfilled::record(path, stamp);
}
Ok(Catalog { conn })
}