perf(db): one logical container per item, indexed; no whole-table reads

Listings matched children on four columns at once (parent_id, album_id,
season_id, series_id) because Jellyfin's ParentId is the storage parent,
not the logical one. The OR defeated the planner into a full scan, and it
was wrong: every episode carries its series id, so a series listed all its
episodes beside its seasons (on both the browse and Downloads surfaces).

- Migration 027 adds items.container_id, a VIRTUAL generated column
  (episode -> season/series/parent, season -> series, track -> album,
  else parent) indexed with (sort_name, name), so a listing is one
  ordered index range and every write path is covered untouched.
- Containers never cached (an episode that arrived via Next Up) get
  placeholders named from the child's own fields, in the migration and
  on every cache write, so offline navigation stays series -> season.
- The six queries that built the set of every downloaded item in a CTE
  (get_item, latest, recently played, search, favourites, by-person,
  Downloads) now check availability per row with one shared predicate.
- PRAGMA optimize at open gives the planner statistics.

Benchmark (~110k items, desktop): series listing ~80 ms -> <1 ms;
migration 027 upgrades that database in ~0.1 s (0.2 s on the Fairphone).
Tests first: a series listing its episodes, and the Downloads series
drill, both failed before the change.
This commit is contained in:
2026-09-24 04:45:44 +02:00
parent e21ddae737
commit c0545a245f
4 changed files with 679 additions and 302 deletions
+49 -18
View File
@@ -243,9 +243,14 @@ CREATE TABLE items (
last_sync DATETIME,
UNIQUE(jellyfin_id, server_id)
-- Logical container (migration 027): episode → season/series, season →
-- series, track → album, else parent. See "Listing query shape".
-- container_id TEXT GENERATED ALWAYS AS (CASE item_type … END) VIRTUAL
);
-- Performance indexes
CREATE INDEX idx_items_container ON items(container_id, sort_name, name);
CREATE INDEX idx_items_server ON items(server_id);
CREATE INDEX idx_items_library ON items(library_id);
CREATE INDEX idx_items_parent ON items(parent_id);
@@ -613,7 +618,8 @@ life of the app; nothing else opens the file. Everything goes through
```
Pragmas: `journal_mode = WAL`, `synchronous = NORMAL`, `busy_timeout = 5 s` on
every connection.
every connection. `PRAGMA optimize` runs once at open, after migrations, so the planner has
statistics (see "Listing query shape").
**Why.** It used to be one connection behind one `std::sync::Mutex`. WAL was on,
but with a single connection its one benefit — readers running beside a writer —
@@ -665,27 +671,52 @@ for the jobs grouped with it.
`OfflineRepository::get_items` (`items_listing_sql`) is the hot read: every
library, series, season and album page goes through it, and it must fit the
100 ms cache fast path on a phone.
100 ms cache fast path on a phone. Every other cached read follows the same
rules.
- **Availability is checked per row, with `EXISTS`** (cached for browsing,
downloaded, or a container with a downloaded child). It used to be a CTE
that built the id of *every* available item in the database before filtering
to the parent: ~80 ms on a desktop for a 100k-item cache, whatever the parent.
- **A parent that is not a library is matched on the hierarchy columns alone**
(`parent_id`/`album_id`/`season_id`/`series_id`), which SQLite answers with a
multi-index `OR`. Whether the parent is a library is looked up first; the
library clause can only match for a library, and inside the same `OR` it
forced a full scan.
- **`+i.server_id`**: the app never runs `ANALYZE`, and without statistics the
planner prefers the `server_id` index — which every row shares — over the
hierarchy indexes. The unary `+` takes it out of consideration.
- **Children are matched on `items.container_id`** (migration 027), a VIRTUAL
generated column holding the item's *logical* container: an episode's season
(else series, else parent), a season's series, a track's album, otherwise the
parent. Jellyfin's `ParentId` is the storage parent, not the logical one — in
a series without season folders an episode's `ParentId` is the series while
its `SeasonId` names a virtual season — so listings used to match on four
columns at once. That `OR` defeated the planner into walking the whole table,
and it was wrong: every episode carries its series id, so a series listed all
its episodes beside its seasons. Being generated, the column covers every
write path (cache, downloads, catalog crawl) without any of them knowing, and
cannot drift from the columns it is computed from.
- **`idx_items_container (container_id, sort_name, name)`** serves a listing as
one index range already in display order — no sort step. `sort_name` is
usually NULL in the cache (the cache never writes it), hence `name` in the
index too.
- **Containers exist even when never browsed.** A series lists its seasons, so
an episode whose season row was never cached (it arrived through Next Up or
Latest) would be unreachable from its series offline. `save_to_cache` — and
migration 027 for rows already on disk — inserts placeholders named from the
child's own fields (`season_name`, `series_name`, `album_name`) with
`synced_at` NULL, so they show only when a download makes them available; the
server's real row replaces them wholesale on the next browse.
- **Availability is a per-row `EXISTS`** (`downloaded_sql` / `available_sql`):
cached for browsing (only with the catalog-browse flag), downloaded, or a
container with a downloaded *descendant* — which is why that one check still
looks at all four link columns (a series is available through an episode two
levels down). It used to be a CTE that built the id of every available item
in the database before filtering, paying for the whole table on every call.
- **The library clause is added only for a library parent** (`is_library`),
never `OR`ed into an ordinary listing, where it forces a full scan.
- **`+i.server_id`** keeps the planner off the server index, which every row
shares. `PRAGMA optimize` at open (`analysis_limit = 400`) gives the planner
statistics, but even with them it chose that index for the old `OR`; the `+`
is the guarantee.
- **User data is fetched in batches** (`with_user_data`, one `IN (…)` query per
500 rows), not once per row.
Result: 80 ms → 1.5 ms on the desktop benchmark;
`listing_a_non_library_parent_uses_the_hierarchy_indexes` asserts the plan.
`storage::tests::write_bench_database` (ignored) writes a phone-sized
catalogue for timing queries with the `sqlite3` CLI.
Measured on a ~110k-item benchmark catalogue (desktop): a series listing went
from ~80 ms to under 1 ms; migration 027 upgrades an existing database of that
size in ~0.1 s. `listing_a_non_library_parent_uses_the_container_index` and
migration 027's tests assert the plans; `storage::tests::write_bench_database`
(ignored; set `JELLYTAU_BENCH_DB`) writes the benchmark catalogue for the
`sqlite3` CLI.
## Rust Module Structure