Backfill the catalog once per state, not on every open
Catalog::open ran schema::backfill every time, and every worker thread opens its own connection. A develop landing made five opens, and each paid the RAW/JPEG pairing, the default-version anti-join over every image, the uuid pass over every default version and the keyword check: 17 ms of CPU an open on a copy of the reference catalog, ~80 ms a landing, to confirm that nothing had changed since the open before. Everything the backfill repairs is a row some write added: an image a scan inserted, a version or keyword assignment a merge brought in. So the open now reads a stamp - user_version, max(id) of images and versions, max(rowid) of keywords, and the file's device and inode - and skips the backfill when the stamp matches the one recorded at this path's last backfill in this process. The maxima are each the last page of a b-tree; an open that skips costs ~1 ms. The backfill still runs: - on the first open in a process (nothing recorded yet); - on any open that migrated the schema, unconditionally; - after a pull: merge_remote forgets the path, so the next open backfills even when every incoming row collided and nothing moved; - when the file is replaced under its name: the inode is in the stamp, and recovery::set_aside, the first step of a restore and a rebuild, forgets the path; - when another process or thread adds rows, because the stamp is read from the file, not from anything this process did. The stamp is taken before the backfill, not after. Read after, it would describe the backfill's own inserts, and could record an image another connection inserted in between as covered when it was not. Read before, the worst case is one redundant pass after a backfill that did real work. Kept in memory rather than in the catalog: a stamp row would need a table an older build does not have and would travel in the sync snapshot, where a flag from another device's catalog says nothing about this one. No schema version bump, so the tablet on 0.16.0 still reads the snapshot. Tests cover the skip, a scan's new image, a migration, a pull and a replaced file.
This commit is contained in:
@@ -330,6 +330,13 @@ pub fn merge_remote(conn: &Connection, remote: &Path) -> Result<MergeReport, Cat
|
||||
|
||||
let result = merge::merge_all(conn);
|
||||
|
||||
// The merge brings in rows the backfill exists for — assignments whose
|
||||
// word this device has no term for, from a remote older than v6 — so the
|
||||
// next open must run it, whether or not the stamp happened to move.
|
||||
if let Some(path) = conn.path().filter(|p| !p.is_empty()) {
|
||||
crate::backfilled::forget(Path::new(path));
|
||||
}
|
||||
|
||||
// Detach even if the merge failed, or the next attempt errors with
|
||||
// "database remote_cat is already in use".
|
||||
let detach = conn.execute(&format!("DETACH DATABASE {REMOTE_SCHEMA}"), []);
|
||||
|
||||
Reference in New Issue
Block a user