perf(db): reads no longer wait behind writes; pages answer from cache
A series page took about a second to show its seasons on a phone, every visit, although they were cached. Three things stacked up: - One SQLite connection behind one mutex served the whole app, so every read queued behind every write. The database now has one owner: a writer thread for writes and a pool of read-only WAL connections for reads. synchronous = NORMAL and a busy timeout on every connection. - The listing query built the set of every available item in the database before filtering to the parent (~80 ms on a desktop for a 100k-item cache), then fetched user data one row at a time. It now checks availability per row, uses the hierarchy indexes (1.5 ms on the same benchmark) and batches the user-data lookup. - A cache read that missed the 100 ms fast path was set aside until the server answered. It is now raced against the server; whichever answers first with content wins. On the Fairphone, Frasier's season and episode lists now come from cache in 34-133 ms (was 600-1030 ms waiting on the server). Fixes found on the way, each with a test that failed first: - sync_queue_mutation could return another mutation's row id: the id came from a second trip to the shared connection. insert() reads it in the same job. - save_to_cache switched foreign keys off on the shared connection across its awaits, so concurrent writes ran unchecked. The toggle now lives inside one writer job, and a page is one transaction instead of one commit per row. Also: thumbnail LRU touches no longer block the lookup; unused tokio-rusqlite dropped. Design and invariants in docs/architecture/08-database-design.md (Connection ownership, Listing query shape) and 03-data-flow.md.
This commit is contained in:
@@ -49,6 +49,19 @@ pub async fn sync_queue_mutation(
|
||||
Arc::new(database.service())
|
||||
};
|
||||
|
||||
enqueue_mutation(&*db_service, user_id, operation, item_id, payload).await
|
||||
}
|
||||
|
||||
/// Insert one pending mutation and return the id of *that* row.
|
||||
///
|
||||
/// TRACES: UR-002, UR-017 | DR-014
|
||||
pub(crate) async fn enqueue_mutation<S: DatabaseService>(
|
||||
db_service: &S,
|
||||
user_id: String,
|
||||
operation: String,
|
||||
item_id: Option<String>,
|
||||
payload: Option<String>,
|
||||
) -> Result<i64, String> {
|
||||
let query = Query::with_params(
|
||||
"INSERT INTO sync_queue (user_id, operation, item_id, payload, status, created_at)
|
||||
VALUES (?, ?, ?, ?, 'pending', CURRENT_TIMESTAMP)",
|
||||
@@ -60,13 +73,9 @@ pub async fn sync_queue_mutation(
|
||||
],
|
||||
);
|
||||
|
||||
db_service.execute(query).await.map_err(|e| e.to_string())?;
|
||||
let id = db_service
|
||||
.last_insert_rowid()
|
||||
.await
|
||||
.map_err(|e| e.to_string())?;
|
||||
|
||||
Ok(id)
|
||||
// `insert`, not `execute` + `last_insert_rowid`: the id must be read in
|
||||
// the same job as the insert, or a concurrent write hands us its row.
|
||||
db_service.insert(query).await
|
||||
}
|
||||
|
||||
/// Get all pending sync operations for a user
|
||||
@@ -387,4 +396,58 @@ mod tests {
|
||||
assert!(item.retry_count == i);
|
||||
}
|
||||
}
|
||||
|
||||
/// Each queued mutation must get back the id of its *own* row.
|
||||
///
|
||||
/// The id used to come from a separate `last_insert_rowid()` call — a
|
||||
/// second trip to the shared connection — so another insert landing in
|
||||
/// between handed this mutation someone else's id, and marking it synced
|
||||
/// later completed the wrong row.
|
||||
///
|
||||
/// TRACES: UR-002, UR-017 | DR-014 | UT-014
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 8)]
|
||||
async fn concurrent_enqueues_each_get_their_own_row_id() {
|
||||
let database = crate::storage::Database::open_in_memory().unwrap();
|
||||
let service = Arc::new(database.service());
|
||||
service
|
||||
.execute(Query::new(
|
||||
"INSERT INTO servers (id, name, url) VALUES ('s', 'S', 'http://s')",
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
service
|
||||
.execute(Query::new(
|
||||
"INSERT INTO users (id, server_id, username) VALUES ('u', 's', 'u')",
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let tasks: Vec<_> = (0..200)
|
||||
.map(|i| {
|
||||
let service = Arc::clone(&service);
|
||||
tokio::spawn(async move {
|
||||
let op = format!("op-{i}");
|
||||
let id = enqueue_mutation(&*service, "u".into(), op.clone(), None, None)
|
||||
.await
|
||||
.unwrap();
|
||||
(op, id)
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
|
||||
for task in tasks {
|
||||
let (op, id) = task.await.unwrap();
|
||||
let stored: String = service
|
||||
.query_one(
|
||||
Query::with_params(
|
||||
"SELECT operation FROM sync_queue WHERE id = ?",
|
||||
vec![QueryParam::Int64(id)],
|
||||
),
|
||||
|row| row.get(0),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(stored, op, "mutation {op} was handed row {id}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user