Write a scan's findings with prepared statements, and its jobs in the same commit
`persist` runs after every scan, for every photograph the scan listed. On a settled library that is the folders whose ETag changed -- a sidecar written there by a rating is enough -- so one relisted folder of 1,600 images is an ordinary pass, and a first scan is all 24,000. Per photograph it prepared four statements from their SQL (a folder lookup, the image upsert, the id read-back, the remote upsert) and then, after the commit, found the image again by path and enqueued its thumbnail job as an autocommitting statement of its own -- a commit per photograph, for rows that were almost all already queued. Now the statements are prepared once per pass, a folder's id is looked up once per folder rather than once per photograph in it, and the job is enqueued inside the transaction with the id already in hand. That also makes the job atomic with the row it points at, which is what the old ordering after the commit was trying to guarantee. `jobs::enqueue` uses a cached statement for the same reason. persist_bench on a copy of the reference catalog, CPU, best of runs: largest folder (1,589 images) 102-118 ms -> 10-13 ms whole library (23,582 images) 1.55-2.19 s -> 188-192 ms The fingerprint of images, remote, jobs and folders after the run is the same for both builds.
This commit is contained in:
@@ -185,7 +185,8 @@ pub fn enqueue(
|
||||
priority: Priority,
|
||||
payload: Option<&str>,
|
||||
) -> Result<(), CatalogError> {
|
||||
conn.execute(
|
||||
// Cached: a scan enqueues one per photograph it lists.
|
||||
conn.prepare_cached(
|
||||
"INSERT INTO jobs(kind, subject_id, priority, state, payload)
|
||||
VALUES (?1, ?2, ?3, 0, ?4)
|
||||
ON CONFLICT(kind, subject_id) DO UPDATE SET
|
||||
@@ -195,8 +196,13 @@ pub fn enqueue(
|
||||
state = CASE WHEN jobs.state = 2 THEN 0 ELSE jobs.state END,
|
||||
attempts = CASE WHEN jobs.state = 2 THEN 0 ELSE jobs.attempts END,
|
||||
not_before = CASE WHEN jobs.state = 2 THEN 0 ELSE jobs.not_before END",
|
||||
rusqlite::params![kind as i64, subject_id, priority as i64, payload],
|
||||
)?;
|
||||
)?
|
||||
.execute(rusqlite::params![
|
||||
kind as i64,
|
||||
subject_id,
|
||||
priority as i64,
|
||||
payload
|
||||
])?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user