dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 50 Wiki Activity

perf(db): reads no longer wait behind writes; pages answer from cache

Browse Source 
A series page took about a second to show its seasons on a phone, every
visit, although they were cached. Three things stacked up:

- One SQLite connection behind one mutex served the whole app, so every
  read queued behind every write. The database now has one owner: a
  writer thread for writes and a pool of read-only WAL connections for
  reads. synchronous = NORMAL and a busy timeout on every connection.
- The listing query built the set of every available item in the
  database before filtering to the parent (~80 ms on a desktop for a
  100k-item cache), then fetched user data one row at a time. It now
  checks availability per row, uses the hierarchy indexes (1.5 ms on
  the same benchmark) and batches the user-data lookup.
- A cache read that missed the 100 ms fast path was set aside until the
  server answered. It is now raced against the server; whichever answers
  first with content wins.

On the Fairphone, Frasier's season and episode lists now come from
cache in 34-133 ms (was 600-1030 ms waiting on the server).

Fixes found on the way, each with a test that failed first:
- sync_queue_mutation could return another mutation's row id: the id
  came from a second trip to the shared connection. insert() reads it in
  the same job.
- save_to_cache switched foreign keys off on the shared connection
  across its awaits, so concurrent writes ran unchecked. The toggle now
  lives inside one writer job, and a page is one transaction instead of
  one commit per row.

Also: thumbnail LRU touches no longer block the lookup; unused
tokio-rusqlite dropped. Design and invariants in
docs/architecture/08-database-design.md (Connection ownership, Listing
query shape) and 03-data-flow.md.
This commit is contained in:
dtourolle
2026-09-24 03:58:04 +02:00
parent 1fb5f070c8
commit 21f24dd998
12 changed files with 1722 additions and 751 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/02-svelte-frontend.md
+11 -46
View File
@@ -314,57 +314,22 @@ pub struct PlaybackModeManager {
**Location**: `src-tauri/src/storage/db_service.rs`
Async database interface wrapping synchronous `rusqlite` to prevent blocking the Tokio runtime:
Async database interface over `rusqlite`. `RusqliteService` owns the
database: writes run as jobs on one writer thread, reads on a pool of read-only
WAL connections, so a read never waits for a write. Callers only see the
trait (`execute`, `insert`, `execute_detached`, `query_one` / `query_optional` /
`query_many`, `transaction`, `transaction_without_foreign_keys`) and build
queries with `Query` + `QueryParam`, which keeps values out of the SQL string.
```rust
#[async_trait]
pub trait DatabaseService: Send + Sync {
async fn execute(&self, query: Query) -> Result<usize, DatabaseError>;
async fn execute_batch(&self, queries: Vec<Query>) -> Result<(), DatabaseError>;
async fn query_one<T, F>(&self, query: Query, mapper: F) -> Result<T, DatabaseError>
where F: FnOnce(&Row) -> Result<T> + Send + 'static;
async fn query_optional<T, F>(&self, query: Query, mapper: F) -> Result<Option<T>, DatabaseError>
where F: FnOnce(&Row) -> Result<T> + Send + 'static;
async fn query_many<T, F>(&self, query: Query, mapper: F) -> Result<Vec<T>, DatabaseError>
where F: Fn(&Row) -> Result<T> + Send + 'static;
async fn transaction<F, T>(&self, f: F) -> Result<T, DatabaseError>
where F: FnOnce(Transaction) -> Result<T> + Send + 'static;
}
pub struct RusqliteService {
connection: Arc<Mutex<Connection>>,
}
impl DatabaseService for RusqliteService {
async fn execute(&self, query: Query) -> Result<usize, DatabaseError> {
let conn = self.connection.clone();
tokio::task::spawn_blocking(move || {
// Execute query on blocking thread pool
}).await?
}
// ... other methods use spawn_blocking
}
```
**Key Benefits:**
- **No Freezing**: All blocking DB ops run in thread pool via `spawn_blocking`
- **Type Safety**: `QueryParam` enum prevents SQL injection
- **Future Proof**: Easy to swap to native async DB (tokio-rusqlite)
- **Testable**: Can mock DatabaseService for tests
**Usage Pattern:**
```rust
// Before (blocking - causes UI freeze)
let conn = database.connection();
let conn = conn.lock().unwrap(); // BLOCKS
conn.query_row(...) // BLOCKS
// After (async - no freezing)
let db_service = database.service();
let db_service = database.service(); // cheap clone of the shared owner
let query = Query::with_params("SELECT ...", vec![...]);
db_service.query_one(query, |row| {...}).await // spawn_blocking internally
db_service.query_one(query, |row| {...}).await // runs on a reader
```
Ownership model, invariants and the reasons for them:
[08-database-design.md → Connection ownership](08-database-design.md#connection-ownership).
## Component Hierarchy
```mermaid
docs/architecture/03-data-flow.md
+22 -7
View File
@@ -28,12 +28,17 @@ sequenceDiagram
Server->>Conn: mark_unreachable() (debounced)
end
alt Cache returns with content
alt Cache answers first with content (inside 100ms, or later but before the server)
Cache-->>Hybrid: Result with items
Hybrid-->>Rust: Return cache result
else Cache timeout or empty
Server-->>Hybrid: Fresh result (later)
Hybrid->>Cache: save_to_cache() in background
else Server answers first, or cache is empty
Server-->>Hybrid: Fresh result
Hybrid-->>Rust: Return server result
else Server fails
Cache-->>Hybrid: Whatever the cache has (waited for)
Hybrid-->>Rust: Return cache result, else the server error
end
Rust-->>Client: SearchResult
@@ -42,11 +47,21 @@ sequenceDiagram
```
**Key Points:**
- Cache queries have 100ms timeout for responsiveness
- Server queries always run for fresh data
- Cache wins if it has meaningful content
- Automatic fallback to server if cache is empty/stale
- Background cache updates (planned)
- Both legs start together. A cache answer with content inside 100 ms
(`CACHE_FAST_PATH`) returns at once.
- **The deadline does not decide the race.** A cache read still running at
100 ms is raced against the server (`HybridRepository::race_slow_cache`), and
whichever answers first *with content* wins. It used to be that a read past
the deadline was only consulted if the server failed, so a page whose cache
read took 150 ms always paid the full server round trip — about a second on a
phone, on every visit.
- An empty or failed cache answer is not a win; the server decides. A failed
server falls back to whatever the cache said, waiting for it if necessary.
- On a cache win the server's page is still cached in the background when it
arrives, so per-user state (positions, favourites) keeps up.
- A `get_items` leg that takes 250 ms or more is logged at INFO with its row
count, so a slow page can be attributed to the cache or the server from a
device log alone.
- **Connectivity side-effect**: each server request feeds the `ConnectivityMonitor`, which is the source of truth for the offline/online banner (see [07-connectivity.md](07-connectivity.md)). A server-answered error (401/404/5xx) still counts as *reachable* — only network failures, sustained past a debounce window, flip the app to offline.
### Listing order is decided in Rust
docs/architecture/06-downloads-and-offline.md
+5 -4
View File
@@ -224,10 +224,11 @@ them:
the current user — local path, direct play, item id as media source (a download
names no source, so the server served its default, which carries the item's id)
— and `HybridRepository::get_playback_info` consults it **first**.
- **A slow cache read is waited for, never discarded.** The cache is one SQLite
connection behind one mutex, so any write in progress (the catalog sync that
starts at every launch, a download finishing) pushes a read past the 100 ms fast
path. `get_items`, the library list, genres and playlist items used to discard
- **A slow cache read is waited for, never discarded.** Reads no longer queue
behind writes (see [Connection ownership](08-database-design.md#connection-ownership)),
but a big query, a cold page cache or a busy reader pool can still push a read
past the 100 ms fast path. Such a read is raced against the server rather
than set aside (see [03-data-flow.md](03-data-flow.md)). `get_items`, the library list, genres and playlist items used to discard
such a read, wait for the server, and — offline — return its error over data on
disk; "More info" on a downloaded show failed that way. They now start the read
with `cache_try` (which keeps it running) and `settle` on it when the server
docs/architecture/08-database-design.md
+93
View File
@@ -594,6 +594,99 @@ flowchart LR
| Episode | ~3 KB | ~100 KB | 300 MB - 2 GB |
| Full music library (5000 songs) | ~10 MB | ~250 MB | 25-75 GB |
## Connection ownership
`storage::Database` is opened once at startup and owns the database for the
life of the app; nothing else opens the file. Everything goes through
`Database::service()`, which returns a clone of one `RusqliteService`
(`storage/db_service.rs`) — callers never hold a `Connection`.
```
┌─────────────────────────────┐
execute / insert / │ writer thread ("db-writer") │ one read-write connection,
transaction / ────►│ jobs run in arrival order │ foreign_keys = ON
execute_detached └─────────────────────────────┘
┌─────────────────────────────┐
query_one / │ reader pool (3 connections) │ query_only = ON; WAL gives
query_optional / ─►│ via spawn_blocking │ each the last committed
query_many └─────────────────────────────┘ snapshot
```
Pragmas: `journal_mode = WAL`, `synchronous = NORMAL`, `busy_timeout = 5 s` on
every connection.
**Why.** It used to be one connection behind one `std::sync::Mutex`. WAL was on,
but with a single connection its one benefit — readers running beside a writer —
never applied: every read queued behind every write. A series page's background
refresh cached hundreds of episodes one autocommit (and one fsync) at a time,
and library pages, thumbnail lookups and settings reads all waited behind the
pile; the same page measured 1.7 s or 3.95 s depending on what was queued.
**Invariants a change must keep:**
- **Writes go to the writer, reads to the pool.** A reader is `query_only`, so a
write sent through `query_*` fails loudly rather than racing the writer. A
read that must see a write *in the same unit of work* belongs inside the
`transaction` closure, which runs on the writer.
- **Connection-wide state is set inside one writer job.** `PRAGMA foreign_keys`
is per connection and ignored inside a transaction. `save_to_cache` used to
switch it off with one `execute` and back on with another, so it stayed off
for every other write that ran across the save's awaits (they did — nearly
all of 1,600 FK-violating writes got through in the regression test).
`transaction_without_foreign_keys` flips it around `BEGIN`/`COMMIT` in a single
job. Any future pragma toggle must work the same way.
- **An insert's rowid comes from the same job.** `insert()` returns it; there is
deliberately no standalone `last_insert_rowid()`, which returned whichever row
the last *anyone* inserted.
- **Batch writes into one transaction.** A commit is a queue slot on the
writer; a page of items is one `transaction`, not one `execute` per row.
- **Best-effort bookkeeping does not wait.** `execute_detached` queues a write
(the thumbnail LRU access time) in order with the rest and returns
immediately; failures are only logged.
- **The writer survives a panicking job.** The job's caller gets an error; the
loop rolls back any transaction the job left open and restores
`foreign_keys = ON`, then carries on.
**`synchronous = NORMAL`** is corruption-safe in WAL mode and survives an app
crash; only a power cut can lose the last few commits. Everything stored is
either re-fetchable from the server or (the sync queue, local positions)
recoverable at that granularity.
**In-memory databases** (tests) cannot be shared between connections, so
`RusqliteService::new` has no pool and routes reads through the writer — the
old serialized behaviour.
**Not done, and why:** grouping consecutive small `execute` jobs into one
commit on the writer. With `synchronous = NORMAL` a commit no longer fsyncs, so
the gain is small, and it would change a failed statement's error semantics
for the jobs grouped with it.
### Listing query shape
`OfflineRepository::get_items` (`items_listing_sql`) is the hot read: every
library, series, season and album page goes through it, and it must fit the
100 ms cache fast path on a phone.
- **Availability is checked per row, with `EXISTS`** (cached for browsing,
downloaded, or a container with a downloaded child). It used to be a CTE
that built the id of *every* available item in the database before filtering
to the parent: ~80 ms on a desktop for a 100k-item cache, whatever the parent.
- **A parent that is not a library is matched on the hierarchy columns alone**
(`parent_id`/`album_id`/`season_id`/`series_id`), which SQLite answers with a
multi-index `OR`. Whether the parent is a library is looked up first; the
library clause can only match for a library, and inside the same `OR` it
forced a full scan.
- **`+i.server_id`**: the app never runs `ANALYZE`, and without statistics the
planner prefers the `server_id` index — which every row shares — over the
hierarchy indexes. The unary `+` takes it out of consideration.
- **User data is fetched in batches** (`with_user_data`, one `IN (…)` query per
500 rows), not once per row.
Result: 80 ms → 1.5 ms on the desktop benchmark;
`listing_a_non_library_parent_uses_the_hierarchy_indexes` asserts the plan.
`storage::tests::write_bench_database` (ignored) writes a phone-sized
catalogue for timing queries with the `sqlite3` CLI.
## Rust Module Structure
```
src-tauri/Cargo.lock
Generated
-12
View File
@@ -2315,7 +2315,6 @@ dependencies = [
"tempfile",
"tiny_http",
"tokio",
"tokio-rusqlite",
"tokio-util",
"urlencoding",
"uuid",
@@ -5340,17 +5339,6 @@ dependencies = [
"syn 2.0.112",
]
[[package]]
name = "tokio-rusqlite"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b65501378eb676f400c57991f42cbd0986827ab5c5200c53f206d710fb32a945"
dependencies = [
"crossbeam-channel",
"rusqlite",
"tokio",
]
[[package]]
name = "tokio-rustls"
version = "0.26.4"
src-tauri/Cargo.toml
-1
View File
@@ -53,7 +53,6 @@ futures-util = "0.3"
async-trait = "0.1"
# SQLite for offline storage
tokio-rusqlite = "0.6"
rusqlite = { version = "0.32", features = ["bundled"] }
chrono = { version = "0.4", features = ["serde"] }
directories = "5"
src-tauri/src/commands/sync.rs
+70 -7
View File
@@ -49,6 +49,19 @@ pub async fn sync_queue_mutation(
Arc::new(database.service())
};
enqueue_mutation(&*db_service, user_id, operation, item_id, payload).await
}
/// Insert one pending mutation and return the id of *that* row.
///
/// TRACES: UR-002, UR-017 | DR-014
pub(crate) async fn enqueue_mutation<S: DatabaseService>(
db_service: &S,
user_id: String,
operation: String,
item_id: Option<String>,
payload: Option<String>,
) -> Result<i64, String> {
let query = Query::with_params(
"INSERT INTO sync_queue (user_id, operation, item_id, payload, status, created_at)
VALUES (?, ?, ?, ?, 'pending', CURRENT_TIMESTAMP)",
@@ -60,13 +73,9 @@ pub async fn sync_queue_mutation(
],
);
db_service.execute(query).await.map_err(|e| e.to_string())?;
let id = db_service
.last_insert_rowid()
.await
.map_err(|e| e.to_string())?;
Ok(id)
// `insert`, not `execute` + `last_insert_rowid`: the id must be read in
// the same job as the insert, or a concurrent write hands us its row.
db_service.insert(query).await
}
/// Get all pending sync operations for a user
@@ -387,4 +396,58 @@ mod tests {
assert!(item.retry_count == i);
}
}
/// Each queued mutation must get back the id of its *own* row.
///
/// The id used to come from a separate `last_insert_rowid()` call — a
/// second trip to the shared connection — so another insert landing in
/// between handed this mutation someone else's id, and marking it synced
/// later completed the wrong row.
///
/// TRACES: UR-002, UR-017 | DR-014 | UT-014
#[tokio::test(flavor = "multi_thread", worker_threads = 8)]
async fn concurrent_enqueues_each_get_their_own_row_id() {
let database = crate::storage::Database::open_in_memory().unwrap();
let service = Arc::new(database.service());
service
.execute(Query::new(
"INSERT INTO servers (id, name, url) VALUES ('s', 'S', 'http://s')",
))
.await
.unwrap();
service
.execute(Query::new(
"INSERT INTO users (id, server_id, username) VALUES ('u', 's', 'u')",
))
.await
.unwrap();
let tasks: Vec<_> = (0..200)
.map(|i| {
let service = Arc::clone(&service);
tokio::spawn(async move {
let op = format!("op-{i}");
let id = enqueue_mutation(&*service, "u".into(), op.clone(), None, None)
.await
.unwrap();
(op, id)
})
})
.collect();
for task in tasks {
let (op, id) = task.await.unwrap();
let stored: String = service
.query_one(
Query::with_params(
"SELECT operation FROM sync_queue WHERE id = ?",
vec![QueryParam::Int64(id)],
),
|row| row.get(0),
)
.await
.unwrap();
assert_eq!(stored, op, "mutation {op} was handed row {id}");
}
}
}
src-tauri/src/repository/hybrid.rs
+312 -86
View File
@@ -50,6 +50,42 @@ impl<T> CacheLeg<T> {
}
}
/// Log a `get_items` leg that took long enough to be felt, so a slow page can
/// be attributed to the cache or the server from a device log alone.
fn log_slow_leg(
leg: &str,
parent_id: &str,
started: std::time::Instant,
result: &Result<SearchResult, RepoError>,
) {
let ms = started.elapsed().as_millis();
let outcome = match result {
Ok(data) => format!("{} rows", data.items.len()),
Err(e) => format!("error: {e:?}"),
};
let parent = &parent_id[..8.min(parent_id.len())];
if ms >= 250 {
log::info!("[HybridRepo] get_items {leg} leg for {parent} took {ms} ms ({outcome})");
} else {
debug!("[HybridRepo] get_items {leg} leg for {parent} took {ms} ms ({outcome})");
}
}
/// Outcome of racing a cache read that missed the fast path against the
/// server. See [`HybridRepository::race_slow_cache`].
enum Raced<T> {
/// The cache answered first with something to show (exclusions applied).
Cache(T),
/// The server answered first, or the cache had nothing. Raw: the caller
/// caches the full page and applies exclusions to what it returns.
Server(T),
/// The server failed; this is what the cache said instead (exclusions
/// applied), possibly an empty listing — which still beats an error.
Fallback(T),
/// Neither could answer; the server's error.
Failed(RepoError),
}
/// Hybrid repository combining online and offline data sources
///
/// Uses cache-first parallel racing strategy:
@@ -397,6 +433,64 @@ impl HybridRepository {
}
}
/// Race a cache read that missed the fast path against the server:
/// whichever answers first *with something to show* wins.
///
/// The fast-path deadline bounds how long a cache hit may delay the UI; it
/// must not also decide the race. It used to: past 100 ms the query waited
/// for the server even when the cache answered moments later, so a page
/// whose cache read took 150 ms always paid the full server round trip
/// (about a second on a phone). An empty or failed cache answer is not a
/// win — the server decides then — and a failed server falls back to
/// whatever the cache said.
///
/// TRACES: UR-002 | DR-013
async fn race_slow_cache<T, F>(
mut slow: tokio::task::JoinHandle<Result<T, RepoError>>,
server: F,
) -> Raced<T>
where
T: MeaningfulContent + ExcludeHidden,
F: std::future::Future<Output = Result<T, RepoError>>,
{
tokio::pin!(server);
// `biased`, server first: if both are ready at once, the fresher
// answer wins.
tokio::select! {
biased;
server_result = &mut server => match server_result {
Ok(data) => Raced::Server(data),
Err(e) => {
debug!("[HybridRepo] Server failed; waiting for the slow cache query");
match slow.await {
Ok(Ok(data)) => Raced::Fallback(data.without_excluded()),
_ => Raced::Failed(e),
}
}
},
cache_result = &mut slow => {
let cache = cache_result
.unwrap_or_else(|join| Err(RepoError::Database {
message: format!("Cache query failed: {join}"),
}))
.map(ExcludeHidden::without_excluded);
match cache {
Ok(data) if data.has_content() => {
debug!("[HybridRepo] Slow cache answered before the server");
Raced::Cache(data)
}
other => match server.await {
Ok(data) => Raced::Server(data),
Err(e) => match other {
Ok(data) => Raced::Fallback(data),
Err(_) => Raced::Failed(e),
},
},
}
}
}
}
/// Cache-first query: try cache, fall back to server on miss.
///
/// 1. Check cache (100ms fast path, via `cache_leg`; a slow read keeps running)
@@ -459,33 +553,28 @@ impl HybridRepository {
}
}
debug!("[HybridRepo] Cache miss or slow, querying server");
// Still running: race it against the server rather than waiting the
// server out. If the server then fails, the cache is the only thing
// that can answer — offline, a cache read slowed by a concurrent write
// used to surface as a network error.
if let Some(handle) = slow {
return match Self::race_slow_cache(handle, server_future).await {
Raced::Cache(data) => {
on_cache_hit();
Ok(data)
}
Raced::Server(data) => Ok(data.without_excluded()),
Raced::Fallback(data) => Ok(data),
Raced::Failed(e) => Err(e),
};
}
debug!("[HybridRepo] Cache miss, querying server");
match server_future.await {
Ok(data) => Ok(data.without_excluded()),
Err(e) => {
// The server cannot answer. If the cache is still working, it is
// now the only thing that can, so wait it out rather than
// reporting the server's failure over data we are about to hold.
// This is the offline path: a cache read slowed by a concurrent
// write used to surface as a network error.
if let Some(handle) = slow {
debug!("[HybridRepo] Server failed; waiting for the slow cache query");
return match handle.await {
Ok(Ok(data)) => Ok(data.without_excluded()),
Ok(Err(cache_err)) => {
debug!("[HybridRepo] Slow cache query also failed: {cache_err}");
Err(e)
}
Err(join) => {
debug!("[HybridRepo] Slow cache query panicked: {join}");
Err(e)
}
};
}
// Cache answered in time but had nothing: return that, so an
// empty-but-valid cached listing still beats a network error.
fast.unwrap_or(Err(e))
}
// Cache answered in time but had nothing: return that, so an
// empty-but-valid cached listing still beats a network error.
Err(e) => fast.unwrap_or(Err(e)),
}
}
@@ -498,10 +587,10 @@ impl HybridRepository {
///
/// Missing the deadline does **not** cancel the query — it keeps running on
/// its own task and [`CacheLeg::settle`] can still collect it. That
/// distinction is the whole point. The database is one SQLite connection
/// behind one mutex, so a concurrent write (a sync drain, a bulk
/// `save_to_cache`) blocks reads for its duration and this deadline trips
/// routinely on slow storage. Treating that as "the cache is empty" while
/// distinction is the whole point. A read can miss the deadline for many
/// reasons — a large listing, slow storage, a busy reader pool (and, before
/// reads had their own connections, any write in progress). Treating that
/// as "the cache is empty" while
/// throwing the answer away meant that offline — where the server leg also
/// fails — browsing surfaced a network error instead of the cached content
/// sitting right there on disk.
@@ -532,9 +621,9 @@ impl HybridRepository {
/// if it made it, and otherwise the read itself, still running.
///
/// The cache-then-server queries used to *discard* a read that missed the
/// deadline. The database is one SQLite connection behind one mutex, so any
/// write in progress — the catalog sync that starts at every launch, a
/// download finishing — pushes a read past 100 ms routinely; offline the
/// deadline. When reads shared one connection with writes, any write in
/// progress — the catalog sync that starts at every launch, a download
/// finishing — pushed a read past 100 ms routinely; offline the
/// server then failed too, and the page reported a network error over data
/// sitting on disk. Keeping the read lets [`Self::settle`] wait for it.
///
@@ -576,6 +665,48 @@ impl HybridRepository {
fast.or(Err(server_err))
}
/// Cache the server's page once it arrives, without holding up the
/// caller, who has already answered from the cache. A failed or empty
/// server answer leaves the existing cache alone.
fn save_when_server_answers(
server: tokio::task::JoinHandle<Result<SearchResult, RepoError>>,
offline: Arc<OfflineRepository>,
parent_id: String,
) {
tokio::spawn(async move {
if let Ok(Ok(server_data)) = server.await {
Self::save_in_background(offline, parent_id, &server_data);
}
});
}
/// Cache a server page in the background (one transaction; see
/// `OfflineRepository::save_to_cache`).
fn save_in_background(
offline: Arc<OfflineRepository>,
parent_id: String,
server_data: &SearchResult,
) {
if server_data.items.is_empty() {
return;
}
let items = server_data.items.clone();
tokio::spawn(async move {
match offline.save_to_cache(&parent_id, &items).await {
Ok(_) => debug!(
"[HybridRepo] Cached {} items for parent {}",
items.len(),
&parent_id[..8.min(parent_id.len())]
),
Err(e) => warn!(
"[HybridRepo] Failed to cache {} items: {:?}",
items.len(),
e
),
}
});
}
/// [`Self::settle`] for `get_items`, whose slow read has not yet had
/// exclusions applied.
async fn cache_or(
@@ -653,8 +784,12 @@ impl MediaRepository for HybridRepository {
let opts_clone = options.clone();
// Start server request in background (non-blocking)
let server_handle =
tokio::spawn(async move { online.get_items(&parent_id_clone, options).await });
let mut server_handle = tokio::spawn(async move {
let started = std::time::Instant::now();
let result = online.get_items(&parent_id_clone, options).await;
log_slow_leg("server", &parent_id_clone, started, &result);
result
});
// Check cache first (fast, 100ms timeout).
//
@@ -671,8 +806,13 @@ impl MediaRepository for HybridRepository {
// is what made a downloaded show fail offline ("Failed to load item")
// whenever a write held the database past 100 ms — the catalog sync
// that starts at every launch does, routinely. TRACES: UR-002 | DR-294
let (cache_result, slow_cache) =
Self::cache_try(async move { offline.get_items(&parent_id, opts_clone).await }).await;
let (cache_result, slow_cache) = Self::cache_try(async move {
let started = std::time::Instant::now();
let result = offline.get_items(&parent_id, opts_clone).await;
log_slow_leg("cache", &parent_id, started, &result);
result
})
.await;
let cache_result = cache_result.map(ExcludeHidden::without_excluded);
// Downloads-only gate: when the "Show all server media" toggle is off
@@ -702,54 +842,45 @@ impl MediaRepository for HybridRepository {
"[HybridRepo] Cache hit for get_items, returning immediately for parent {}",
&parent_id_for_save[..8.min(parent_id_for_save.len())]
);
// Background: save server result to cache when it arrives
tokio::spawn(async move {
match server_handle.await {
Ok(Ok(server_data)) if !server_data.items.is_empty() => {
if let Err(e) = offline_for_save
.save_to_cache(&parent_id_for_save, &server_data.items)
.await
{
warn!("[HybridRepo] Background cache update failed: {:?}", e);
} else {
debug!(
"[HybridRepo] Background updated {} cached items for parent {}",
server_data.items.len(),
&parent_id_for_save[..8.min(parent_id_for_save.len())]
);
}
}
_ => {} // Server failed or returned empty — keep existing cache
}
});
Self::save_when_server_answers(server_handle, offline_for_save, parent_id_for_save);
return Ok(data.clone());
}
}
// Cache miss — wait for server result
// The cache missed the fast path but is still reading: race it against
// the server instead of waiting the server out (DR-013).
if let Some(slow) = slow_cache {
let server = async {
(&mut server_handle).await.unwrap_or_else(|join| {
Err(RepoError::Network {
message: format!("Server task failed: {}", join),
})
})
};
return match Self::race_slow_cache(slow, server).await {
Raced::Cache(data) => {
// The server is still in flight: cache its answer when it
// lands, exactly as on a fast-path hit.
Self::save_when_server_answers(
server_handle,
offline_for_save,
parent_id_for_save,
);
Ok(data)
}
Raced::Server(server_data) => {
Self::save_in_background(offline_for_save, parent_id_for_save, &server_data);
Ok(server_data.without_excluded())
}
Raced::Fallback(data) => Ok(data),
Raced::Failed(e) => Err(e),
};
}
// Cache answered in time with nothing — wait for the server.
match server_handle.await {
Ok(Ok(server_data)) => {
if !server_data.items.is_empty() {
let items_clone = server_data.items.clone();
tokio::spawn(async move {
if let Err(e) = offline_for_save
.save_to_cache(&parent_id_for_save, &items_clone)
.await
{
warn!(
"[HybridRepo] Failed to save {} items to cache: {:?}",
items_clone.len(),
e
);
} else {
debug!(
"[HybridRepo] Saved {} items to cache for parent {}",
items_clone.len(),
&parent_id_for_save[..8.min(parent_id_for_save.len())]
);
}
});
}
Self::save_in_background(offline_for_save, parent_id_for_save, &server_data);
// The cache keeps the server's full page (above) — an exclusion
// is a view preference and can be undone, so hiding items from
// the *cache* would make un-hiding them require a re-crawl. Only
@@ -757,12 +888,12 @@ impl MediaRepository for HybridRepository {
// TRACES: UR-076 | DR-209
Ok(server_data.without_excluded())
}
Ok(Err(e)) => Self::cache_or(cache_result, slow_cache, e).await,
Ok(Err(e)) => Self::cache_or(cache_result, None, e).await,
Err(join_err) => {
let e = RepoError::Network {
message: format!("Server task failed: {}", join_err),
};
Self::cache_or(cache_result, slow_cache, e).await
Self::cache_or(cache_result, None, e).await
}
}
}
@@ -771,9 +902,9 @@ impl MediaRepository for HybridRepository {
/// background so the stored copy keeps up with the server.
///
/// The background refresh is what carries per-user state home: caching an
/// item runs `mirror_user_data`, which is the only path by which a watch
/// position set on another device reaches the local `user_data` row the
/// resume check reads. Without it a cache hit returned this device's own
/// item also writes its `user_data_mirror_query` row, the only path by
/// which a watch position set on another device reaches the local
/// `user_data` row the resume check reads. Without it a cache hit returned this device's own
/// stale position forever and cross-device resume silently did nothing —
/// `get_items` already refreshes this way, so browsing a season worked
/// while opening the episode directly did not.
@@ -1385,9 +1516,9 @@ mod tests {
/// Offline, a cache read slowed past the fast path must still answer.
///
/// The database is one SQLite connection behind one mutex, so a concurrent
/// write blocks reads for its duration and the 100 ms fast path trips on
/// slow storage. The deadline used to *cancel* the read and report it as a
/// The 100 ms fast path trips routinely on slow storage (and, while reads
/// shared one connection with writes, behind any write). The deadline used
/// to *cancel* the read and report it as a
/// miss; with the server leg also failing (offline), the user got a network
/// error over cached content that was sitting on disk.
///
@@ -1464,6 +1595,101 @@ mod tests {
assert!(matches!(err, RepoError::Network { .. }), "got {err:?}");
}
/// A cache read that misses the fast path but lands before the server must
/// be what the user sees.
///
/// The 100 ms deadline used to decide the race outright: past it, the page
/// waited for the server even when the cache answered a few milliseconds
/// later — on a phone, a series page showed its seasons after the ~1 s
/// server round trip instead of the ~150 ms cache read, on every visit.
///
/// TRACES: UR-002 | DR-013
#[tokio::test]
async fn a_slow_cache_that_beats_the_server_wins() {
let cache = HybridRepository::cache_leg(async {
tokio::time::sleep(Duration::from_millis(150)).await;
Ok(vec![MediaItem {
id: "slow-cache".to_string(),
..Default::default()
}])
})
.await;
let server = async {
tokio::time::sleep(Duration::from_millis(1500)).await;
Ok(vec![MediaItem {
id: "server".to_string(),
..Default::default()
}])
};
let started = std::time::Instant::now();
let got = HybridRepository::parallel_race(cache, server)
.await
.unwrap();
assert_eq!(
got[0].id, "slow-cache",
"waited for the server over a cache answer"
);
assert!(started.elapsed() < Duration::from_millis(1000));
}
/// A server that beats a slow cache still answers first.
///
/// TRACES: UR-002 | DR-013
#[tokio::test]
async fn a_server_that_beats_a_slow_cache_wins() {
let cache = HybridRepository::cache_leg(async {
tokio::time::sleep(Duration::from_millis(1500)).await;
Ok(vec![MediaItem {
id: "slow-cache".to_string(),
..Default::default()
}])
})
.await;
let server = async {
tokio::time::sleep(Duration::from_millis(150)).await;
Ok(vec![MediaItem {
id: "server".to_string(),
..Default::default()
}])
};
let started = std::time::Instant::now();
let got = HybridRepository::parallel_race(cache, server)
.await
.unwrap();
assert_eq!(got[0].id, "server");
assert!(started.elapsed() < Duration::from_millis(1000));
}
/// A slow cache that comes back *empty* is not an answer: the server
/// decides.
///
/// TRACES: UR-002 | DR-013
#[tokio::test]
async fn an_empty_slow_cache_defers_to_the_server() {
let cache = HybridRepository::cache_leg(async {
tokio::time::sleep(Duration::from_millis(150)).await;
Ok(Vec::<MediaItem>::new())
})
.await;
let server = async {
tokio::time::sleep(Duration::from_millis(400)).await;
Ok(vec![MediaItem {
id: "server".to_string(),
..Default::default()
}])
};
let got = HybridRepository::parallel_race(cache, server)
.await
.unwrap();
assert_eq!(got[0].id, "server");
}
/// Mock offline repository that tracks queries and saves
struct MockOfflineRepo {
items: Arc<Mutex<Vec<MediaItem>>>,
src-tauri/src/repository/offline.rs
+668 -446
View File
File diff suppressed because it is too large Load Diff
src-tauri/src/storage/db_service.rs
+268 -106
View File
@@ -1,16 +1,31 @@
//! Database service abstraction layer
//! Database service: the single owner of the SQLite database.
//!
//! This module provides an async database interface that abstracts away
//! the underlying database implementation. This makes it easy to:
//! - Switch between sync (rusqlite) and async (tokio-rusqlite) implementations
//! - Prevent blocking the async runtime with synchronous database calls
//! - Test with different database backends
//! - Migrate to other database systems in the future
//! Every query in the app goes through [`RusqliteService`], which owns the
//! connections and hands work to them — callers never touch a `Connection`.
//!
//! - **Writes** (`execute`, `insert`, `transaction`, …) are sent as jobs to one
//! dedicated writer thread that owns the read-write connection. SQLite allows
//! one writer at a time anyway; owning it on one thread makes that explicit,
//! keeps connection-wide state (pragmas) out of reach of concurrent callers,
//! and parks no tokio blocking threads on a mutex while writes queue up.
//! - **Reads** (`query_*`) run on a small pool of read-only connections. The
//! database is in WAL mode, so readers see the last committed state and never
//! wait for the writer — a large catalog-cache transaction no longer stalls
//! library pages, thumbnail lookups or settings reads.
//!
//! A service built with [`RusqliteService::new`] has no reader pool (in-memory
//! databases cannot be shared between connections) and routes reads through
//! the writer, which is the old single-connection behaviour tests rely on.
//!
//! See `docs/architecture/08-database-design.md` → "Connection ownership".
use crate::utils::lock::MutexSafe;
use async_trait::async_trait;
use log::{debug, error};
use rusqlite::{params_from_iter, Connection, Result as SqliteResult, Row};
use std::sync::{Arc, Mutex};
use std::panic::AssertUnwindSafe;
use std::sync::mpsc;
use std::sync::{Arc, Condvar, Mutex};
/// Database query result type
pub type DbResult<T> = Result<T, String>;
@@ -84,8 +99,28 @@ pub trait DatabaseService: Send + Sync {
F: FnOnce(&mut Transaction) -> DbResult<T> + Send + 'static,
T: Send + 'static;
/// Get the row ID of the most recent successful INSERT
async fn last_insert_rowid(&self) -> DbResult<i64>;
/// Run a transaction with foreign-key enforcement switched off for its
/// duration only.
///
/// `PRAGMA foreign_keys` is per connection and is a no-op inside a
/// transaction, so it has to be flipped around the `BEGIN`/`COMMIT` — and
/// all of that must happen as one job on the writer, or any other write
/// that got in between would run unchecked too.
async fn transaction_without_foreign_keys<F, T>(&self, f: F) -> DbResult<T>
where
F: FnOnce(&mut Transaction) -> DbResult<T> + Send + 'static,
T: Send + 'static;
/// Execute an INSERT and return the rowid of the row it inserted.
///
/// The rowid is read in the same job as the insert. Reading it with a
/// second call would race every other write, returning someone else's id.
async fn insert(&self, query: Query) -> DbResult<i64>;
/// Queue a write without waiting for it — for best-effort bookkeeping (an
/// LRU access time) that must not hold up the caller. It still runs in
/// order with every other write; failures are only logged.
fn execute_detached(&self, query: Query);
}
/// Transaction handle for batching multiple operations
@@ -110,48 +145,183 @@ impl<'a> Transaction<'a> {
}
}
/// Rusqlite-based database service implementation
///
/// This implementation wraps synchronous rusqlite operations in tokio::task::spawn_blocking
/// to prevent blocking the async runtime.
type Job = Box<dyn FnOnce(&Connection) + Send>;
/// The thread that owns the read-write connection. Jobs run one at a time, in
/// the order they were sent; the thread exits when the last service handle
/// (and so the last sender) is dropped.
struct Writer {
jobs: mpsc::Sender<Job>,
}
impl Writer {
fn spawn(conn: Arc<Mutex<Connection>>) -> Self {
let (jobs, queue) = mpsc::channel::<Job>();
std::thread::Builder::new()
.name("db-writer".into())
.spawn(move || {
for job in queue {
// The connection stays behind a mutex only so migrations and
// tests can reach it; in the app this thread is its sole user.
// `lock_safe` so a poisoned lock is recovered, not fatal.
let conn = conn.lock_safe();
// A panicking row mapper must not take the owner down with
// it: the job's reply channel drops, its caller gets an
// error, and the next job runs normally. The guard lives
// outside the unwind, so the mutex is not poisoned either.
if std::panic::catch_unwind(AssertUnwindSafe(|| job(&conn))).is_err() {
error!("[db] a database job panicked; the writer carries on");
// Undo whatever connection state the job was midway
// through: an open transaction would make the next
// job's BEGIN fail, and a job that switched foreign
// keys off would leave them off for everyone.
if !conn.is_autocommit() {
let _ = conn.execute_batch("ROLLBACK");
}
let _ = conn.execute_batch("PRAGMA foreign_keys = ON");
}
}
})
.expect("failed to spawn the database writer thread");
Self { jobs }
}
async fn run<T, F>(&self, f: F) -> DbResult<T>
where
T: Send + 'static,
F: FnOnce(&Connection) -> DbResult<T> + Send + 'static,
{
let (reply, result) = tokio::sync::oneshot::channel();
self.jobs
.send(Box::new(move |conn| {
let _ = reply.send(f(conn));
}))
.map_err(|_| "database writer has stopped".to_string())?;
result
.await
.map_err(|_| "database job panicked".to_string())?
}
fn run_detached(&self, f: impl FnOnce(&Connection) + Send + 'static) {
if self.jobs.send(Box::new(f)).is_err() {
debug!("[db] writer stopped; dropped a detached write");
}
}
}
/// Read-only connections, checked out one per query. WAL gives each a
/// snapshot of the last commit, so they never wait for the writer.
struct ReaderPool {
idle: Mutex<Vec<Connection>>,
returned: Condvar,
}
impl ReaderPool {
/// Blocking: waits for a free connection. Call from `spawn_blocking`.
fn run<T>(&self, f: impl FnOnce(&Connection) -> T) -> T {
let conn = {
let mut idle = self.idle.lock_safe();
loop {
if let Some(conn) = idle.pop() {
break conn;
}
idle = self
.returned
.wait(idle)
.unwrap_or_else(|poisoned| poisoned.into_inner());
}
};
// Returned on drop, so a panicking mapper does not leak the connection.
let checkout = Checkout {
pool: self,
conn: Some(conn),
};
f(checkout.conn.as_ref().expect("checked-out connection"))
}
}
struct Checkout<'a> {
pool: &'a ReaderPool,
conn: Option<Connection>,
}
impl Drop for Checkout<'_> {
fn drop(&mut self) {
if let Some(conn) = self.conn.take() {
self.pool.idle.lock_safe().push(conn);
self.pool.returned.notify_one();
}
}
}
/// Rusqlite-based database service: a cheap, cloneable handle to the writer
/// thread and reader pool. See the module docs.
#[derive(Clone)]
pub struct RusqliteService {
conn: Arc<Mutex<Connection>>,
writer: Arc<Writer>,
readers: Option<Arc<ReaderPool>>,
}
impl RusqliteService {
/// A service over a single connection: writes *and* reads go through the
/// writer thread. Used for in-memory databases, which cannot be shared
/// between connections.
#[cfg_attr(not(test), allow(dead_code))]
pub fn new(conn: Arc<Mutex<Connection>>) -> Self {
Self { conn }
Self {
writer: Arc::new(Writer::spawn(conn)),
readers: None,
}
}
/// A service whose reads run on `readers` — read-only connections to the
/// same (file-backed, WAL-mode) database — alongside the writer.
pub fn with_readers(conn: Arc<Mutex<Connection>>, readers: Vec<Connection>) -> Self {
let readers = (!readers.is_empty()).then(|| {
Arc::new(ReaderPool {
idle: Mutex::new(readers),
returned: Condvar::new(),
})
});
Self {
writer: Arc::new(Writer::spawn(conn)),
readers,
}
}
async fn read<T, F>(&self, f: F) -> DbResult<T>
where
T: Send + 'static,
F: FnOnce(&Connection) -> DbResult<T> + Send + 'static,
{
match &self.readers {
Some(pool) => {
let pool = Arc::clone(pool);
tokio::task::spawn_blocking(move || pool.run(f))
.await
.map_err(|e| format!("Task join error: {}", e))?
}
None => self.writer.run(f).await,
}
}
}
#[async_trait]
impl DatabaseService for RusqliteService {
async fn execute(&self, query: Query) -> DbResult<usize> {
let conn = Arc::clone(&self.conn);
tokio::task::spawn_blocking(move || {
// `lock_safe`, not `lock`: this is the busiest lock in the app and a
// panic under the guard would otherwise poison it, failing every
// later query with "poisoned lock" until the process restarts.
let conn = conn.lock_safe();
execute_query(&conn, query)
})
.await
.map_err(|e| format!("Task join error: {}", e))?
self.writer
.run(move |conn| execute_query(conn, query))
.await
}
async fn execute_batch(&self, sql: &str) -> DbResult<()> {
let conn = Arc::clone(&self.conn);
let sql = sql.to_string();
tokio::task::spawn_blocking(move || {
// `lock_safe`, not `lock`: this is the busiest lock in the app and a
// panic under the guard would otherwise poison it, failing every
// later query with "poisoned lock" until the process restarts.
let conn = conn.lock_safe();
conn.execute_batch(&sql)
.map_err(|e| format!("Execute batch failed: {}", e))
})
.await
.map_err(|e| format!("Task join error: {}", e))?
self.writer
.run(move |conn| {
conn.execute_batch(&sql)
.map_err(|e| format!("Execute batch failed: {}", e))
})
.await
}
async fn query_one<T, F>(&self, query: Query, mapper: F) -> DbResult<T>
@@ -159,16 +329,7 @@ impl DatabaseService for RusqliteService {
T: Send + 'static,
F: Fn(&Row) -> SqliteResult<T> + Send + 'static,
{
let conn = Arc::clone(&self.conn);
tokio::task::spawn_blocking(move || {
// `lock_safe`, not `lock`: this is the busiest lock in the app and a
// panic under the guard would otherwise poison it, failing every
// later query with "poisoned lock" until the process restarts.
let conn = conn.lock_safe();
query_one(&conn, query, mapper)
})
.await
.map_err(|e| format!("Task join error: {}", e))?
self.read(move |conn| query_one(conn, query, mapper)).await
}
async fn query_optional<T, F>(&self, query: Query, mapper: F) -> DbResult<Option<T>>
@@ -176,16 +337,8 @@ impl DatabaseService for RusqliteService {
T: Send + 'static,
F: Fn(&Row) -> SqliteResult<T> + Send + 'static,
{
let conn = Arc::clone(&self.conn);
tokio::task::spawn_blocking(move || {
// `lock_safe`, not `lock`: this is the busiest lock in the app and a
// panic under the guard would otherwise poison it, failing every
// later query with "poisoned lock" until the process restarts.
let conn = conn.lock_safe();
query_optional(&conn, query, mapper)
})
.await
.map_err(|e| format!("Task join error: {}", e))?
self.read(move |conn| query_optional(conn, query, mapper))
.await
}
async fn query_many<T, F>(&self, query: Query, mapper: F) -> DbResult<Vec<T>>
@@ -193,16 +346,7 @@ impl DatabaseService for RusqliteService {
T: Send + 'static,
F: Fn(&Row) -> SqliteResult<T> + Send + 'static,
{
let conn = Arc::clone(&self.conn);
tokio::task::spawn_blocking(move || {
// `lock_safe`, not `lock`: this is the busiest lock in the app and a
// panic under the guard would otherwise poison it, failing every
// later query with "poisoned lock" until the process restarts.
let conn = conn.lock_safe();
query_many(&conn, query, mapper)
})
.await
.map_err(|e| format!("Task join error: {}", e))?
self.read(move |conn| query_many(conn, query, mapper)).await
}
async fn transaction<F, T>(&self, f: F) -> DbResult<T>
@@ -210,47 +354,65 @@ impl DatabaseService for RusqliteService {
F: FnOnce(&mut Transaction) -> DbResult<T> + Send + 'static,
T: Send + 'static,
{
let conn = Arc::clone(&self.conn);
tokio::task::spawn_blocking(move || {
// `lock_safe`, not `lock`: this is the busiest lock in the app and a
// panic under the guard would otherwise poison it, failing every
// later query with "poisoned lock" until the process restarts.
let conn = conn.lock_safe();
conn.execute("BEGIN TRANSACTION", [])
.map_err(|e| format!("Failed to begin transaction: {}", e))?;
let mut transaction = Transaction::new(&conn);
let result = f(&mut transaction);
match result {
Ok(value) => {
conn.execute("COMMIT", [])
.map_err(|e| format!("Failed to commit transaction: {}", e))?;
Ok(value)
}
Err(e) => {
conn.execute("ROLLBACK", [])
.map_err(|e| format!("Failed to rollback transaction: {}", e))?;
Err(e)
}
}
})
.await
.map_err(|e| format!("Task join error: {}", e))?
self.writer.run(move |conn| run_transaction(conn, f)).await
}
async fn last_insert_rowid(&self) -> DbResult<i64> {
let conn = Arc::clone(&self.conn);
tokio::task::spawn_blocking(move || {
// `lock_safe`, not `lock`: this is the busiest lock in the app and a
// panic under the guard would otherwise poison it, failing every
// later query with "poisoned lock" until the process restarts.
let conn = conn.lock_safe();
Ok(conn.last_insert_rowid())
})
.await
.map_err(|e| format!("Task join error: {}", e))?
async fn transaction_without_foreign_keys<F, T>(&self, f: F) -> DbResult<T>
where
F: FnOnce(&mut Transaction) -> DbResult<T> + Send + 'static,
T: Send + 'static,
{
self.writer
.run(move |conn| {
conn.execute_batch("PRAGMA foreign_keys = OFF")
.map_err(|e| format!("Failed to disable foreign keys: {}", e))?;
let result = run_transaction(conn, f);
// Always restored, whatever the transaction did.
if let Err(e) = conn.execute_batch("PRAGMA foreign_keys = ON") {
error!("[db] failed to re-enable foreign keys: {}", e);
}
result
})
.await
}
async fn insert(&self, query: Query) -> DbResult<i64> {
self.writer
.run(move |conn| {
execute_query(conn, query)?;
Ok(conn.last_insert_rowid())
})
.await
}
fn execute_detached(&self, query: Query) {
self.writer.run_detached(move |conn| {
if let Err(e) = execute_query(conn, query) {
debug!("[db] detached write failed: {}", e);
}
});
}
}
fn run_transaction<F, T>(conn: &Connection, f: F) -> DbResult<T>
where
F: FnOnce(&mut Transaction) -> DbResult<T>,
{
conn.execute("BEGIN TRANSACTION", [])
.map_err(|e| format!("Failed to begin transaction: {}", e))?;
let mut transaction = Transaction::new(conn);
match f(&mut transaction) {
Ok(value) => {
conn.execute("COMMIT", [])
.map_err(|e| format!("Failed to commit transaction: {}", e))?;
Ok(value)
}
Err(e) => {
conn.execute("ROLLBACK", [])
.map_err(|e| format!("Failed to rollback transaction: {}", e))?;
Err(e)
}
}
}
src-tauri/src/storage/mod.rs
+266 -26
View File
@@ -17,9 +17,27 @@ use rusqlite::{Connection, Result as SqliteResult};
pub use db_service::{DatabaseService, RusqliteService};
use schema::MIGRATIONS;
/// Database connection wrapper with thread-safe access
/// How long a connection retries a locked database before erroring — covers a
/// reader meeting a WAL checkpoint, or the writer meeting a reader's snapshot.
const BUSY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
/// How many read-only connections serve queries alongside the writer. Reads
/// are short; a few cover a library page's parallel fetches plus background
/// work without holding many file handles.
const READER_CONNECTIONS: usize = 3;
/// The database: opened once at startup and owned for the life of the app.
///
/// All access goes through [`Database::service`], which hands out clones of one
/// [`RusqliteService`] — a writer thread plus a pool of read-only connections
/// (see `db_service`). Nothing else opens the database file.
pub struct Database {
/// The read-write connection. Owned by the service's writer thread; kept
/// here only for migrations (which run before that thread starts taking
/// work) and for tests.
#[cfg_attr(not(test), allow(dead_code))]
conn: Arc<Mutex<Connection>>,
service: RusqliteService,
path: PathBuf,
}
@@ -36,18 +54,36 @@ impl Database {
// Enable foreign keys
conn.execute_batch("PRAGMA foreign_keys = ON;")?;
// Enable WAL mode for better concurrent access
// WAL lets the reader connections run alongside the writer.
conn.execute_batch("PRAGMA journal_mode = WAL;")?;
// In WAL mode NORMAL is corruption-safe and skips the fsync FULL pays on
// every commit (a power cut can lose the last commits, an app crash
// cannot). On Android flash that fsync dominated every small write.
conn.execute_batch("PRAGMA synchronous = NORMAL;")?;
conn.busy_timeout(BUSY_TIMEOUT)?;
let db = Self {
conn: Arc::new(Mutex::new(conn)),
let conn = Arc::new(Mutex::new(conn));
Self::migrate_connection(&conn, MIGRATIONS)?;
// Readers open after migrations, so they only ever see the final schema.
let readers = (0..READER_CONNECTIONS)
.map(|_| Self::open_reader(path))
.collect::<SqliteResult<Vec<_>>>()?;
Ok(Self {
service: RusqliteService::with_readers(Arc::clone(&conn), readers),
conn,
path: path.clone(),
};
})
}
// Run migrations
db.migrate()?;
Ok(db)
/// A connection that can only read. `query_only` makes an accidental write
/// routed to the pool fail loudly instead of racing the writer.
fn open_reader(path: &PathBuf) -> SqliteResult<Connection> {
let conn = Connection::open(path)?;
conn.busy_timeout(BUSY_TIMEOUT)?;
conn.execute_batch("PRAGMA query_only = ON;")?;
Ok(conn)
}
/// Open an in-memory database (for testing)
@@ -58,15 +94,16 @@ impl Database {
// Enable foreign keys
conn.execute_batch("PRAGMA foreign_keys = ON;")?;
let db = Self {
conn: Arc::new(Mutex::new(conn)),
let conn = Arc::new(Mutex::new(conn));
Self::migrate_connection(&conn, MIGRATIONS)?;
// An in-memory database cannot be shared between connections, so this
// one has no reader pool: reads go through the writer.
Ok(Self {
service: RusqliteService::new(Arc::clone(&conn)),
conn,
path: PathBuf::from(":memory:"),
};
// Run migrations
db.migrate()?;
Ok(db)
})
}
/// Get connection (for testing)
@@ -75,11 +112,19 @@ impl Database {
Arc::clone(&self.conn)
}
/// Run all pending migrations.
/// Re-run all migrations against an open database (tests only; `open` runs
/// them before the service starts).
#[cfg(test)]
pub fn migrate(&self) -> SqliteResult<()> {
self.migrate_with(MIGRATIONS)
}
/// Test seam: inject a failing migration. See [`Self::migrate_connection`].
#[cfg(test)]
fn migrate_with(&self, migrations: &[(&str, &str)]) -> SqliteResult<()> {
Self::migrate_connection(&self.conn, migrations)
}
/// Apply `migrations` in order, skipping ones `_migrations` already records.
///
/// **Each migration is one transaction, and the `_migrations` row is written
@@ -96,12 +141,16 @@ impl Database {
/// Every migration is pure DDL/DML, which SQLite runs transactionally — a
/// `PRAGMA` or `VACUUM` added to one would not roll back and must not be.
///
/// Split out from [`Self::migrate`] so tests can inject a failing migration.
/// Runs on the bare connection, before the writer thread takes it, so
/// tests can also inject a failing migration through `migrate_with`.
///
/// TRACES: UR-002 | DR-012 | UT-014
fn migrate_with(&self, migrations: &[(&str, &str)]) -> SqliteResult<()> {
fn migrate_connection(
conn: &Mutex<Connection>,
migrations: &[(&str, &str)],
) -> SqliteResult<()> {
info!("Starting database migrations...");
let conn = self.conn.lock_safe();
let conn = conn.lock_safe();
// Create migrations table if it doesn't exist
debug!("Creating _migrations table if it doesn't exist...");
@@ -160,12 +209,10 @@ impl Database {
Ok(())
}
/// Get a database service for async-safe operations
///
/// This wraps all blocking database operations in spawn_blocking to prevent
/// freezing the async runtime.
/// A handle to the database service. Cheap: every call returns a clone of
/// the same writer thread and reader pool.
pub fn service(&self) -> RusqliteService {
RusqliteService::new(Arc::clone(&self.conn))
self.service.clone()
}
/// Get the database file path
@@ -880,4 +927,197 @@ mod tests {
assert_eq!(user_id, "user2");
assert_eq!(username, "recent_user");
}
/// A read must not queue behind a long write.
///
/// The database is in WAL mode precisely so readers can run alongside a
/// writer, but every query used to go through one connection behind one
/// mutex — so a big catalog-cache transaction stalled every library page,
/// thumbnail lookup and settings read in the app until it committed.
///
/// TRACES: UR-002 | DR-012 | UT-014
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn reads_do_not_wait_for_an_in_flight_write() {
use crate::storage::db_service::{DatabaseService, Query};
use std::time::{Duration, Instant};
let dir = tempfile::tempdir().unwrap();
let db = Database::open(&dir.path().join("jellytau.db")).unwrap();
let service = db.service();
let writer = service.clone();
let write = tokio::spawn(async move {
writer
.transaction(|_tx| {
std::thread::sleep(Duration::from_millis(600));
Ok(())
})
.await
});
// Let the write take the connection first.
tokio::time::sleep(Duration::from_millis(100)).await;
let started = Instant::now();
let servers: i64 = service
.query_one(Query::new("SELECT COUNT(*) FROM servers"), |r| r.get(0))
.await
.unwrap();
let waited = started.elapsed();
assert_eq!(servers, 0);
assert!(
waited < Duration::from_millis(250),
"a read waited {waited:?} for an unrelated write to commit"
);
write.await.unwrap().unwrap();
}
/// Commit cost: in WAL mode `synchronous = NORMAL` is corruption-safe and
/// skips the per-commit fsync that FULL (the default) pays — on Android
/// flash that is the dominant cost of every small write. A busy timeout
/// lets the reader connections ride out a checkpoint instead of failing.
///
/// TRACES: UR-002 | DR-012 | UT-014
#[test]
fn open_configures_wal_for_interactive_use() {
let dir = tempfile::tempdir().unwrap();
let db = Database::open(&dir.path().join("jellytau.db")).unwrap();
let conn = db.connection();
let conn = conn.lock_safe();
let mode: String = conn
.query_row("PRAGMA journal_mode", [], |r| r.get(0))
.unwrap();
let synchronous: i64 = conn
.query_row("PRAGMA synchronous", [], |r| r.get(0))
.unwrap();
let busy_timeout: i64 = conn
.query_row("PRAGMA busy_timeout", [], |r| r.get(0))
.unwrap();
assert_eq!(mode, "wal");
assert_eq!(synchronous, 1, "expected synchronous = NORMAL");
assert!(busy_timeout > 0, "expected a busy timeout");
}
/// Writes a phone-sized catalogue to `$JELLYTAU_BENCH_DB` for timing
/// queries with the `sqlite3` CLI. Not a test; run explicitly with
/// `--ignored`.
#[test]
#[ignore]
fn write_bench_database() {
let Ok(path) = std::env::var("JELLYTAU_BENCH_DB") else {
return;
};
let _ = std::fs::remove_file(&path);
let db = Database::open(&PathBuf::from(&path)).unwrap();
let conn = db.connection();
let conn = conn.lock_safe();
conn.execute_batch(
"BEGIN;
INSERT INTO servers (id, name, url) VALUES ('srv', 'S', 'http://s');
INSERT INTO users (id, server_id, username) VALUES ('u', 'srv', 'u');
INSERT INTO libraries (id, server_id, name, collection_type) VALUES ('tv', 'srv', 'TV', 'tvshows');
INSERT INTO libraries (id, server_id, name, collection_type) VALUES ('music', 'srv', 'Music', 'music');",
)
.unwrap();
let now = "2026-09-23T00:00:00Z";
let mut item = conn
.prepare(
"INSERT INTO items (id, server_id, library_id, parent_id, name, sort_name, item_type,
series_id, season_id, album_id, synced_at)
VALUES (?1, 'srv', ?2, ?3, ?4, ?4, ?5, ?6, ?7, ?8, ?9)",
)
.unwrap();
let none: Option<String> = None;
for s in 0..300 {
let series = format!("series-{s}");
item.execute(rusqlite::params![
series,
"tv",
none,
format!("Show {s}"),
"Series",
none,
none,
none,
now
])
.unwrap();
for n in 0..11 {
let season = format!("{series}-s{n}");
item.execute(rusqlite::params![
season,
"tv",
series,
format!("Season {n}"),
"Season",
series,
none,
none,
now
])
.unwrap();
for e in 0..24 {
let ep = format!("{season}-e{e}");
item.execute(rusqlite::params![
ep,
"tv",
season,
format!("A Title {e}"),
"Episode",
series,
season,
none,
now
])
.unwrap();
conn.execute(
"INSERT INTO user_data (user_id, item_id, playback_position_ticks) VALUES ('u', ?1, 5)",
[&ep],
)
.unwrap();
}
}
}
for a in 0..2000 {
let album = format!("album-{a}");
item.execute(rusqlite::params![
album,
"music",
none,
format!("Album {a}"),
"MusicAlbum",
none,
none,
none,
now
])
.unwrap();
for t in 0..12 {
item.execute(rusqlite::params![
format!("{album}-t{t}"),
"music",
album,
format!("Track {t}"),
"Audio",
none,
none,
album,
now
])
.unwrap();
}
}
for d in 0..400 {
conn.execute(
"INSERT INTO downloads (item_id, user_id, file_path, status) VALUES (?1, 'u', '/x', 'completed')",
[format!("series-{}-s1-e{}", d % 300, d % 24)],
)
.unwrap();
}
drop(item);
// No ANALYZE: the app never runs it, so the planner works without stats.
conn.execute_batch("COMMIT;").unwrap();
}
}
src-tauri/src/thumbnail/cache.rs
+7 -10
View File
@@ -122,7 +122,7 @@ impl ThumbnailCache {
{
let path = PathBuf::from(&path_str);
if path.exists() {
self.touch(&db, item_id, image_type, Some(tag)).await;
self.touch(&db, item_id, image_type, Some(tag));
return Some(path);
}
// File gone — drop the stale row and fall through to the tag-agnostic
@@ -157,7 +157,7 @@ impl ThumbnailCache {
let path_str: String = db.query_optional(any_tag, |row| row.get(0)).await.ok()??;
let path = PathBuf::from(&path_str);
if path.exists() {
self.touch(&db, item_id, image_type, None).await;
self.touch(&db, item_id, image_type, None);
Some(path)
} else {
None
@@ -166,13 +166,7 @@ impl ThumbnailCache {
/// Update `last_accessed` for LRU tracking. When `tag` is `Some`, scope to
/// that exact row; when `None`, touch every row for the item + type.
async fn touch(
&self,
db: &Arc<RusqliteService>,
item_id: &str,
image_type: &str,
tag: Option<&str>,
) {
fn touch(&self, db: &Arc<RusqliteService>, item_id: &str, image_type: &str, tag: Option<&str>) {
let query = match tag {
Some(tag) => Query::with_params(
"UPDATE thumbnails SET last_accessed = CURRENT_TIMESTAMP
@@ -192,7 +186,10 @@ impl ThumbnailCache {
],
),
};
let _ = db.execute(query).await;
// Detached: an LRU timestamp is bookkeeping, and a grid scroll does
// one of these per visible poster — awaiting each write held every
// thumbnail lookup behind the writer queue.
db.execute_detached(query);
}
/// Save thumbnail to cache