Backfill the catalog once per state, not on every open

Catalog::open ran schema::backfill every time, and every worker thread
opens its own connection. A develop landing made five opens, and each
paid the RAW/JPEG pairing, the default-version anti-join over every
image, the uuid pass over every default version and the keyword check:
17 ms of CPU an open on a copy of the reference catalog, ~80 ms a
landing, to confirm that nothing had changed since the open before.

Everything the backfill repairs is a row some write added: an image a
scan inserted, a version or keyword assignment a merge brought in. So
the open now reads a stamp - user_version, max(id) of images and
versions, max(rowid) of keywords, and the file's device and inode - and
skips the backfill when the stamp matches the one recorded at this
path's last backfill in this process. The maxima are each the last page
of a b-tree; an open that skips costs ~1 ms.

The backfill still runs:
- on the first open in a process (nothing recorded yet);
- on any open that migrated the schema, unconditionally;
- after a pull: merge_remote forgets the path, so the next open
  backfills even when every incoming row collided and nothing moved;
- when the file is replaced under its name: the inode is in the stamp,
  and recovery::set_aside, the first step of a restore and a rebuild,
  forgets the path;
- when another process or thread adds rows, because the stamp is read
  from the file, not from anything this process did.

The stamp is taken before the backfill, not after. Read after, it would
describe the backfill's own inserts, and could record an image another
connection inserted in between as covered when it was not. Read before,
the worst case is one redundant pass after a backfill that did real work.

Kept in memory rather than in the catalog: a stamp row would need a
table an older build does not have and would travel in the sync
snapshot, where a flag from another device's catalog says nothing about
this one. No schema version bump, so the tablet on 0.16.0 still reads
the snapshot. Tests cover the skip, a scan's new image, a migration, a
pull and a replaced file.
This commit is contained in:
2026-09-26 14:26:10 -04:00
parent 2226d543f9
commit ffdd640170
5 changed files with 324 additions and 4 deletions
+7
View File
@@ -330,6 +330,13 @@ pub fn merge_remote(conn: &Connection, remote: &Path) -> Result<MergeReport, Cat
let result = merge::merge_all(conn);
// The merge brings in rows the backfill exists for — assignments whose
// word this device has no term for, from a remote older than v6 — so the
// next open must run it, whether or not the stamp happened to move.
if let Some(path) = conn.path().filter(|p| !p.is_empty()) {
crate::backfilled::forget(Path::new(path));
}
// Detach even if the merge failed, or the next attempt errors with
// "database remote_cat is already in use".
let detach = conn.execute(&format!("DETACH DATABASE {REMOTE_SCHEMA}"), []);