From 0f7ea741d89bb8846b878c8de308f178dcc01c47 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Sat, 26 Sep 2026 14:55:08 -0400 Subject: [PATCH] Record the open's backfill stamp and the first-use indexes in catalog.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #75 and the develop-landing work changed how the catalog is opened and queried without touching its design document. §2 now lists the two indexes made on first use, keywords_term_version and faces_box, beside the tables made that way, and says why each exists. A paragraph states what the backfill stamp in backfilled.rs holds, why it records the newest rows by content rather than by id, and when the backfill still runs. §6.2's example of coalescing was a thumbnail job, a kind nothing enqueues since #73. --- docs/dev/catalog.md | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/docs/dev/catalog.md b/docs/dev/catalog.md index 61a85ec..0bf900b 100644 --- a/docs/dev/catalog.md +++ b/docs/dev/catalog.md @@ -138,7 +138,27 @@ EXISTS` where it is first used, and an older build that meets it ignores it: - `dedup_probes` (FR-CAT-11a, §3.5); - the albums (FR-EXP-10, `core/dr-catalog/src/albums.rs`): `albums`, `album_exports` — one row per file written into an album, keyed on the file name, since two crops of one photograph are two - files — and `album_folders`, this device's folder for each (§8.2). + files — and `album_folders`, this device's folder for each (§8.2); +- `keywords_term_version (keyword, version_id)`, made by `keywords::list`, which counts each word's + photographs on every selection change and without it read a `keywords` row per assignment to + learn its version; +- `faces_box (image_id, model_id, x, y, w, h)`, made by the merge's `match_faces`, which reads every + local face's box and model and without it opened each ~8 KB `faces` row to do so. + +A failure to make one of the indexes — a read-only or busy catalog — is logged and the query runs +without it, as it did before. + +**Opening does not repeat the backfill.** `schema::backfill` repairs what a write left owing — an +image a scan inserted without its default version, the RAW/JPEG pairing, a merged version's uuid, a +keyword assignment whose word has no term — and it used to run in every `Catalog::open`. Every +worker opens its own connection, so a develop landing paid it five times (~80 ms of CPU on the +reference catalog) to learn that nothing had changed. `core/dr-catalog/src/backfilled.rs` now +records, per path and per process, a stamp read before each backfill: the schema version, the +file's device and inode, and the newest image, version and keyword assignment by content as well as +id — because none of those tables is `AUTOINCREMENT`, a freed newest id is handed out again, and the +row that takes it is exactly one the backfill is owed. An open whose stamp matches skips it (~1 ms); +the first open in a process, a migration, a pull and a replaced file always run it. Kept in memory +rather than in the catalog, so nothing about it travels in the sync snapshot. --- @@ -457,7 +477,7 @@ and fails if any kind is enqueued that no handler or claim names. ### 6.2 Coalescing is the point `UNIQUE(kind, subject_id)` on `jobs` means enqueueing is idempotent: an image touched five times -during a scan has one thumbnail job, not five. Enqueue is +has one job of a kind, not five. Enqueue is `INSERT … ON CONFLICT DO UPDATE SET priority = max(priority, excluded.priority)`, so a re-request at higher priority promotes the existing row rather than duplicating it.