perf(db): one logical container per item, indexed; no whole-table reads
Listings matched children on four columns at once (parent_id, album_id, season_id, series_id) because Jellyfin's ParentId is the storage parent, not the logical one. The OR defeated the planner into a full scan, and it was wrong: every episode carries its series id, so a series listed all its episodes beside its seasons (on both the browse and Downloads surfaces). - Migration 027 adds items.container_id, a VIRTUAL generated column (episode -> season/series/parent, season -> series, track -> album, else parent) indexed with (sort_name, name), so a listing is one ordered index range and every write path is covered untouched. - Containers never cached (an episode that arrived via Next Up) get placeholders named from the child's own fields, in the migration and on every cache write, so offline navigation stays series -> season. - The six queries that built the set of every downloaded item in a CTE (get_item, latest, recently played, search, favourites, by-person, Downloads) now check availability per row with one shared predicate. - PRAGMA optimize at open gives the planner statistics. Benchmark (~110k items, desktop): series listing ~80 ms -> <1 ms; migration 027 upgrades that database in ~0.1 s (0.2 s on the Fairphone). Tests first: a series listing its episodes, and the Downloads series drill, both failed before the change.
This commit is contained in:
@@ -243,9 +243,14 @@ CREATE TABLE items (
|
||||
last_sync DATETIME,
|
||||
|
||||
UNIQUE(jellyfin_id, server_id)
|
||||
|
||||
-- Logical container (migration 027): episode → season/series, season →
|
||||
-- series, track → album, else parent. See "Listing query shape".
|
||||
-- container_id TEXT GENERATED ALWAYS AS (CASE item_type … END) VIRTUAL
|
||||
);
|
||||
|
||||
-- Performance indexes
|
||||
CREATE INDEX idx_items_container ON items(container_id, sort_name, name);
|
||||
CREATE INDEX idx_items_server ON items(server_id);
|
||||
CREATE INDEX idx_items_library ON items(library_id);
|
||||
CREATE INDEX idx_items_parent ON items(parent_id);
|
||||
@@ -613,7 +618,8 @@ life of the app; nothing else opens the file. Everything goes through
|
||||
```
|
||||
|
||||
Pragmas: `journal_mode = WAL`, `synchronous = NORMAL`, `busy_timeout = 5 s` on
|
||||
every connection.
|
||||
every connection. `PRAGMA optimize` runs once at open, after migrations, so the planner has
|
||||
statistics (see "Listing query shape").
|
||||
|
||||
**Why.** It used to be one connection behind one `std::sync::Mutex`. WAL was on,
|
||||
but with a single connection its one benefit — readers running beside a writer —
|
||||
@@ -665,27 +671,52 @@ for the jobs grouped with it.
|
||||
|
||||
`OfflineRepository::get_items` (`items_listing_sql`) is the hot read: every
|
||||
library, series, season and album page goes through it, and it must fit the
|
||||
100 ms cache fast path on a phone.
|
||||
100 ms cache fast path on a phone. Every other cached read follows the same
|
||||
rules.
|
||||
|
||||
- **Availability is checked per row, with `EXISTS`** (cached for browsing,
|
||||
downloaded, or a container with a downloaded child). It used to be a CTE
|
||||
that built the id of *every* available item in the database before filtering
|
||||
to the parent: ~80 ms on a desktop for a 100k-item cache, whatever the parent.
|
||||
- **A parent that is not a library is matched on the hierarchy columns alone**
|
||||
(`parent_id`/`album_id`/`season_id`/`series_id`), which SQLite answers with a
|
||||
multi-index `OR`. Whether the parent is a library is looked up first; the
|
||||
library clause can only match for a library, and inside the same `OR` it
|
||||
forced a full scan.
|
||||
- **`+i.server_id`**: the app never runs `ANALYZE`, and without statistics the
|
||||
planner prefers the `server_id` index — which every row shares — over the
|
||||
hierarchy indexes. The unary `+` takes it out of consideration.
|
||||
- **Children are matched on `items.container_id`** (migration 027), a VIRTUAL
|
||||
generated column holding the item's *logical* container: an episode's season
|
||||
(else series, else parent), a season's series, a track's album, otherwise the
|
||||
parent. Jellyfin's `ParentId` is the storage parent, not the logical one — in
|
||||
a series without season folders an episode's `ParentId` is the series while
|
||||
its `SeasonId` names a virtual season — so listings used to match on four
|
||||
columns at once. That `OR` defeated the planner into walking the whole table,
|
||||
and it was wrong: every episode carries its series id, so a series listed all
|
||||
its episodes beside its seasons. Being generated, the column covers every
|
||||
write path (cache, downloads, catalog crawl) without any of them knowing, and
|
||||
cannot drift from the columns it is computed from.
|
||||
- **`idx_items_container (container_id, sort_name, name)`** serves a listing as
|
||||
one index range already in display order — no sort step. `sort_name` is
|
||||
usually NULL in the cache (the cache never writes it), hence `name` in the
|
||||
index too.
|
||||
- **Containers exist even when never browsed.** A series lists its seasons, so
|
||||
an episode whose season row was never cached (it arrived through Next Up or
|
||||
Latest) would be unreachable from its series offline. `save_to_cache` — and
|
||||
migration 027 for rows already on disk — inserts placeholders named from the
|
||||
child's own fields (`season_name`, `series_name`, `album_name`) with
|
||||
`synced_at` NULL, so they show only when a download makes them available; the
|
||||
server's real row replaces them wholesale on the next browse.
|
||||
- **Availability is a per-row `EXISTS`** (`downloaded_sql` / `available_sql`):
|
||||
cached for browsing (only with the catalog-browse flag), downloaded, or a
|
||||
container with a downloaded *descendant* — which is why that one check still
|
||||
looks at all four link columns (a series is available through an episode two
|
||||
levels down). It used to be a CTE that built the id of every available item
|
||||
in the database before filtering, paying for the whole table on every call.
|
||||
- **The library clause is added only for a library parent** (`is_library`),
|
||||
never `OR`ed into an ordinary listing, where it forces a full scan.
|
||||
- **`+i.server_id`** keeps the planner off the server index, which every row
|
||||
shares. `PRAGMA optimize` at open (`analysis_limit = 400`) gives the planner
|
||||
statistics, but even with them it chose that index for the old `OR`; the `+`
|
||||
is the guarantee.
|
||||
- **User data is fetched in batches** (`with_user_data`, one `IN (…)` query per
|
||||
500 rows), not once per row.
|
||||
|
||||
Result: 80 ms → 1.5 ms on the desktop benchmark;
|
||||
`listing_a_non_library_parent_uses_the_hierarchy_indexes` asserts the plan.
|
||||
`storage::tests::write_bench_database` (ignored) writes a phone-sized
|
||||
catalogue for timing queries with the `sqlite3` CLI.
|
||||
Measured on a ~110k-item benchmark catalogue (desktop): a series listing went
|
||||
from ~80 ms to under 1 ms; migration 027 upgrades an existing database of that
|
||||
size in ~0.1 s. `listing_a_non_library_parent_uses_the_container_index` and
|
||||
migration 027's tests assert the plans; `storage::tests::write_bench_database`
|
||||
(ignored; set `JELLYTAU_BENCH_DB`) writes the benchmark catalogue for the
|
||||
`sqlite3` CLI.
|
||||
|
||||
## Rust Module Structure
|
||||
|
||||
|
||||
Reference in New Issue
Block a user