Drop retired Thumbnail jobs whenever a catalog is opened
Stopping the enqueue leaves the rows already queued: 23,582 on the reference catalog, about 1 MB of table and indexes that every query over jobs pays for. A migration would be the usual tool and is the wrong one here. A schema bump makes an older build refuse the synced catalog snapshot, and the tablet is on 0.16.0. So the rows are dropped at runtime instead, by jobs::drop_retired over a new JobKind::RETIRED list, from runner::recover - which already runs exactly once per catalog open, before any worker. It runs every open rather than once because an older build sharing the catalog queues them again on its next scan. kind leads the UNIQUE(kind, subject_id) index, so with nothing left it is one index probe. Measured on a copy of the reference catalog: 23,582 rows dropped in 40 ms on the first open, 0.07 ms after. Thumbnail stays in the enum so its number is never reused for a kind that would then inherit old rows. The runner tests that call recover move to a live kind; the jobs.rs tests of queue mechanics never call it and are unchanged. Refs #73
This commit is contained in:
@@ -810,8 +810,9 @@ fn adopt_catalog(
|
||||
// the callers' early return on an already-open catalog is what makes it
|
||||
// once. The job queue is durable, so a run that was killed mid-job left its
|
||||
// row marked `Running` with nobody holding it; recovery hands those back to
|
||||
// be resumed rather than lost, and drops jobs naming photographs that have
|
||||
// since been deleted.
|
||||
// be resumed rather than lost, drops jobs naming photographs that have
|
||||
// since been deleted, and drops rows of retired kinds — the thumbnail jobs
|
||||
// 0.16.0 and earlier queued per photograph, which nothing claims (#73).
|
||||
//
|
||||
// Here rather than wherever a runner starts, because there is no owner
|
||||
// column in `jobs`: a second recovery pass while a worker held a claim
|
||||
@@ -819,9 +820,11 @@ fn adopt_catalog(
|
||||
match dr_catalog::runner::recover(cat.connection()) {
|
||||
Ok(r) if r.did_anything() => log::info!(
|
||||
"job queue: {} interrupted job(s) resumed, \
|
||||
{} for deleted photographs dropped",
|
||||
{} for deleted photographs dropped, \
|
||||
{} of a retired kind dropped",
|
||||
r.reclaimed,
|
||||
r.reaped
|
||||
r.reaped,
|
||||
r.retired
|
||||
),
|
||||
Ok(_) => {}
|
||||
// Not surfaced. The queue is rebuildable like everything else in the
|
||||
|
||||
Reference in New Issue
Block a user