Files
jellytau/docs/architecture/06-downloads-and-offline.md
T
dtourolle 21f24dd998 perf(db): reads no longer wait behind writes; pages answer from cache
A series page took about a second to show its seasons on a phone, every
visit, although they were cached. Three things stacked up:

- One SQLite connection behind one mutex served the whole app, so every
  read queued behind every write. The database now has one owner: a
  writer thread for writes and a pool of read-only WAL connections for
  reads. synchronous = NORMAL and a busy timeout on every connection.
- The listing query built the set of every available item in the
  database before filtering to the parent (~80 ms on a desktop for a
  100k-item cache), then fetched user data one row at a time. It now
  checks availability per row, uses the hierarchy indexes (1.5 ms on
  the same benchmark) and batches the user-data lookup.
- A cache read that missed the 100 ms fast path was set aside until the
  server answered. It is now raced against the server; whichever answers
  first with content wins.

On the Fairphone, Frasier's season and episode lists now come from
cache in 34-133 ms (was 600-1030 ms waiting on the server).

Fixes found on the way, each with a test that failed first:
- sync_queue_mutation could return another mutation's row id: the id
  came from a second trip to the shared connection. insert() reads it in
  the same job.
- save_to_cache switched foreign keys off on the shared connection
  across its awaits, so concurrent writes ran unchecked. The toggle now
  lives inside one writer job, and a page is one transaction instead of
  one commit per row.

Also: thumbnail LRU touches no longer block the lookup; unused
tokio-rusqlite dropped. Design and invariants in
docs/architecture/08-database-design.md (Connection ownership, Listing
query shape) and 03-data-flow.md.
2026-09-24 03:58:04 +02:00

397 lines
14 KiB
Markdown

# Download Manager & Offline Architecture
## Overview
**Location**: `src-tauri/src/download/`
The download manager provides offline media support with priority-based queue management, progress tracking, retry logic, and smart caching.
```mermaid
flowchart TB
subgraph Frontend["Frontend"]
DownloadButton["DownloadButton.svelte"]
DownloadsPage["/downloads"]
DownloadsStore["downloads.ts store"]
end
subgraph Backend["Rust Backend"]
Commands["Download Commands"]
DownloadManager["DownloadManager"]
DownloadWorker["DownloadWorker"]
SmartCache["SmartCache Engine"]
end
subgraph Storage["Storage"]
SQLite[("SQLite DB")]
MediaFiles[("Downloaded Files")]
end
DownloadButton -->|"invoke('download_item')"| Commands
DownloadsPage -->|"invoke('get_downloads')"| Commands
Commands --> DownloadManager
DownloadManager --> DownloadWorker
DownloadManager --> SmartCache
DownloadWorker -->|"HTTP Stream"| MediaFiles
DownloadWorker -->|"Events"| DownloadsStore
Commands <--> SQLite
SmartCache <--> SQLite
```
## Download Worker
**Location**: `src-tauri/src/download/worker.rs`
The download worker handles HTTP streaming with retry logic and resume support:
```rust
pub struct DownloadWorker {
client: reqwest::Client,
max_retries: u32,
}
pub struct DownloadTask {
pub id: i64,
pub item_id: String,
pub user_id: String,
pub priority: i32,
pub url: String,
pub target_path: PathBuf,
pub mime_type: Option<String>,
pub expected_size: Option<i64>,
}
```
**Retry Strategy**:
- Exponential backoff: 5s, 15s, 45s
- Maximum 3 retry attempts
- HTTP Range requests for resume support
- Progress events emitted every 1MB
**Download Flow**:
```mermaid
sequenceDiagram
participant UI
participant Command as download_item
participant DB as SQLite
participant Worker as DownloadWorker
participant Jellyfin as Jellyfin Server
participant Store as downloads store
UI->>Command: download_item(itemId, userId)
Command->>DB: INSERT INTO downloads
Command->>Worker: Start download task
Worker->>Jellyfin: GET /Items/{id}/Download
loop Progress Updates
Jellyfin->>Worker: Stream chunks
Worker->>Worker: Write to .part file
Worker->>Store: Emit progress event
Store->>UI: Update progress bar
end
Worker->>Worker: Rename .part to final
Worker->>DB: UPDATE status='completed'
Worker->>Store: Emit completed event
Store->>UI: Show completed
```
## Smart Caching Engine
**Location**: `src-tauri/src/download/cache.rs`
The smart caching system provides predictive downloads based on listening patterns:
```rust
pub struct SmartCache {
config: Arc<Mutex<CacheConfig>>,
album_play_history: Arc<Mutex<HashMap<String, Vec<String>>>>,
}
pub struct CacheConfig {
pub queue_precache_enabled: bool,
pub queue_precache_count: usize, // Default: 5
pub album_affinity_enabled: bool,
pub album_affinity_threshold: usize, // Default: 3
pub storage_limit: u64, // Default: 10GB
pub wifi_only: bool, // Default: true
}
```
**Caching Strategies**:
1. **Queue Pre-caching**: Auto-download next 5 tracks when playing (WiFi only)
2. **Album Affinity**: If user plays 3+ tracks from album, cache entire album
3. **LRU Eviction**: Remove least recently accessed when storage limit reached
```mermaid
flowchart TB
Play["Track Played"] --> CheckQueue{"Queue<br/>Pre-cache?"}
CheckQueue -->|"Yes"| CacheNext5["Download<br/>Next 5 Tracks"]
Play --> TrackHistory["Track Play History"]
TrackHistory --> CheckAlbum{"3+ Tracks<br/>from Album?"}
CheckAlbum -->|"Yes"| CacheAlbum["Download<br/>Full Album"]
CacheNext5 --> CheckStorage{"Storage<br/>Limit?"}
CacheAlbum --> CheckStorage
CheckStorage -->|"Exceeded"| EvictLRU["Evict LRU Items"]
CheckStorage -->|"OK"| Download["Queue Download"]
```
## One Storage Model: Cache Entries Are Downloads
**TRACES**: UR-071 | DR-126, DR-127
A cache entry **is** a download with a shorter life: the same `downloads` row and
the same file handling, distinguished by `download_source` plus an expiry. There
is one storage model rather than a cache and a download library that can
disagree about what is on disk.
| `download_source` | Life | Reclaimed by |
|-------------------|------|--------------|
| `'auto'` (temporary) | Expiry, or eviction under space pressure | Both |
| `'user'` (permanent) | No expiry | Neither |
**Eviction only reclaims the temporary tier.** `evict_lru_async` originally
selected every completed download ordered by `completed_at ASC` with no source
filter, so hitting the storage limit deleted the *oldest* download — typically a
film saved deliberately for offline — to make room for a newly precached track.
It now evicts only `COALESCE(download_source, 'user') = 'auto'` rows.
`COALESCE` rather than a bare equality is load-bearing: rows predating the
migration can be NULL, and **unknown provenance must be treated as the user's,
never as disposable**. Freeing less than requested is the correct outcome when
only user downloads remain — the caller reports "unable to free enough".
A temporary row can be **promoted** to permanent when the user chooses to keep
it. That only clears the expiry and flips the source; the bytes never move.
## Offline Catalog Visibility
**TRACES**: UR-052 | DR-078, DR-079, DR-080
Offline, a library page shows **only media on the device**. A "Show all server
media" toggle additionally reveals the cached server catalog, greyed out and
queueable for download on reconnect.
The gate is a process-global `INCLUDE_CATALOG_BROWSE` in
`repository/offline.rs`, written by the `set_show_server_catalog` command. It
gates the synced-catalog leg of `get_items`; without it the toggle rendered but
every server item still appeared, which is the defect the spec was written for.
`isConnected` derives from backend-reported reachability alone (DR-079) — see
[07-connectivity.md](07-connectivity.md).
Per-item disk usage comes from `repository_get_download_disk_usage`
(`DownloadDiskUsage`), aggregated from `downloads.file_size` — used by the
Downloaded browse cards, detail pages, the device total and the remove
confirmation (DR-085).
## What a Video Download Fetches
**TRACES**: UR-071, UR-004 | DR-171, DR-293
An `original`-quality download is the server's untouched file (`Static=true`)
unless its audio cannot be decoded by **the renderer that will play it**
`renderer_can_decode_audio`, DR-234's per-platform answer. Only then is the
server asked to re-encode the audio on the way down (`allowVideoStreamCopy`
keeps the picture byte-for-byte).
The distinction matters because a transcode is generated as it is sent: no
`Content-Length`, `Range` ignored. It measured ~1 MB/s and restarted from byte
zero on every network blip, against a direct copy that moved a 910 MB episode in
94 s with no retries. On Android the renderer is ExoPlayer with the FFmpeg
extension ([05-platform-backends.md](05-platform-backends.md)), which decodes
AC-3/E-AC-3/DTS/TrueHD, so Android downloads are always the direct copy. On
Linux the webview still renders video and the transcode still applies.
The policy used to judge against the *webview's* codec list on every platform
(DR-171), because a download outlives the native-video setting that was active
when it arrived. That reasoning is why Android's webview video path was removed
rather than merely defaulted off: a file downloaded as the original must never
meet a renderer that cannot decode it.
## Offline Means No Network
**TRACES**: UR-002, UR-071 | DR-294
Three defects made "offline" depend on the network; the invariants that replace
them:
- **A download plays without the server.** Playing a downloaded item asked the
server for its `PlaybackInfo` only to read the media-source id; offline that
retried for seven seconds, failed, and the file was never opened.
`OfflineRepository::local_playback_info` answers for any completed download of
the current user — local path, direct play, item id as media source (a download
names no source, so the server served its default, which carries the item's id)
— and `HybridRepository::get_playback_info` consults it **first**.
- **A slow cache read is waited for, never discarded.** Reads no longer queue
behind writes (see [Connection ownership](08-database-design.md#connection-ownership)),
but a big query, a cold page cache or a busy reader pool can still push a read
past the 100 ms fast path. Such a read is raced against the server rather
than set aside (see [03-data-flow.md](03-data-flow.md)). `get_items`, the library list, genres and playlist items used to discard
such a read, wait for the server, and — offline — return its error over data on
disk; "More info" on a downloaded show failed that way. They now start the read
with `cache_try` (which keeps it running) and `settle` on it when the server
fails. Cache-only reads (search, favourites) have no server to fall back from,
so they simply await the cache.
- **Server-only sections degrade, they do not fail a page.** Next Up went only to
the server, and the TV landing page loads it in one `Promise.all`, so offline it
blanked the whole page. It now falls back to the cache when the server cannot
answer.
What still needs the server, deliberately: streaming anything not downloaded,
live TV and channels, reporting playback, and edits (favourites, playlists,
played state).
## Download Commands
**Location**: `src-tauri/src/commands/download/``mod.rs` (the commands below), `pinning.rs`, `smart_cache.rs`
| Command | Parameters | Description |
|---------|------------|-------------|
| `download_item` | `item_id, user_id, file_path` | Queue single item download |
| `download_album` | `album_id, user_id` | Queue all tracks in album |
| `get_downloads` | `user_id, status_filter` | Get download list |
| `pause_download` | `download_id` | Pause active download |
| `resume_download` | `download_id` | Resume paused download |
| `cancel_download` | `download_id` | Cancel and delete partial |
| `delete_download` | `download_id` | Delete completed download |
| `download_video` / `download_series` / `download_season` | item ids | Queue video content |
| `get_download_storage_stats` | `user_id` | Device totals for the downloads screen |
| `delete_album_downloads` / `delete_downloads_under` / `delete_all_downloads` | container id | Bulk removal |
| `pin_item` / `unpin_item` / `is_item_pinned` | `item_id` | Protect metadata from a cache clear |
| `set_max_concurrent_downloads` | `max` | Worker concurrency (3 by default) |
## Offline Commands
**Location**: `src-tauri/src/commands/offline.rs`
| Command | Parameters | Description |
|---------|------------|-------------|
| `offline_is_available` | `item_id` | Check if item downloaded |
| `offline_get_items` | `user_id` | Get all offline items |
| `offline_search` | `user_id, query` | Search downloaded items |
## Player Integration
**Location**: `src-tauri/src/commands/player.rs` (modified)
The player checks for local downloads before streaming:
```rust
fn create_media_item(req: PlayItemRequest, db: Option<&DatabaseWrapper>) -> MediaItem {
let local_path = db.and_then(|db_wrapper| {
check_for_local_download(db_wrapper, &jellyfin_id).ok().flatten()
});
let source = if let Some(path) = local_path {
MediaSource::Local {
file_path: PathBuf::from(path),
jellyfin_item_id: Some(jellyfin_id.clone())
}
} else {
MediaSource::Remote {
stream_url: req.stream_url,
jellyfin_item_id: jellyfin_id.clone()
}
};
MediaItem { source, /* ... */ }
}
```
## Frontend Downloads Store
**Location**: `src/lib/stores/downloads.ts`
```typescript
interface DownloadsState {
downloads: Record<number, DownloadInfo>;
activeCount: number;
queuedCount: number;
}
const downloads = createDownloadsStore();
// Actions
downloads.downloadItem(itemId, userId, filePath)
downloads.downloadAlbum(albumId, userId)
downloads.pause(downloadId)
downloads.resume(downloadId)
downloads.cancel(downloadId)
downloads.delete(downloadId)
downloads.refresh(userId, statusFilter)
// Derived stores
export const activeDownloads = derived(downloads, ($d) =>
Object.values($d.downloads).filter((d) => d.status === 'downloading')
);
```
**Event Handling**:
The store listens to Tauri events for real-time updates:
```typescript
listen<DownloadEvent>('download-event', (event) => {
const payload = event.payload;
switch (payload.type) {
case 'started':
// Update status to 'downloading'
case 'progress':
// Update progress and bytes_downloaded
case 'completed':
// Update status to 'completed', progress to 1.0
case 'failed':
// Update status to 'failed', store error message
}
});
```
## Download UI Components
**DownloadButton** (`src/lib/components/library/DownloadButton.svelte`):
- Multiple states: available, downloading, completed, failed, paused
- Circular progress ring during download
- Size variants: sm, md, lg
- Integrated into TrackList with `showDownload={true}` prop
**DownloadItem** (`src/lib/components/downloads/DownloadItem.svelte`):
- Individual download list item with progress bar
- Action buttons: pause, resume, cancel, delete
- Status indicators with color coding
**Downloads Page** (`src/routes/downloads/+page.svelte`):
- Active/Completed tabs
- Bulk actions: Pause All, Resume All, Clear Completed
- Empty states with helpful instructions
## Database Schema
**downloads table**:
```sql
CREATE TABLE downloads (
id INTEGER PRIMARY KEY AUTOINCREMENT,
item_id TEXT NOT NULL,
user_id TEXT NOT NULL,
file_path TEXT,
file_size INTEGER,
mime_type TEXT,
status TEXT DEFAULT 'pending', -- pending, downloading, completed, failed, paused
progress REAL DEFAULT 0.0,
bytes_downloaded INTEGER DEFAULT 0,
priority INTEGER DEFAULT 0,
error_message TEXT,
retry_count INTEGER DEFAULT 0,
queued_at TEXT DEFAULT CURRENT_TIMESTAMP,
started_at TEXT,
completed_at TEXT
);
CREATE INDEX idx_downloads_queue
ON downloads(status, priority DESC, queued_at ASC)
WHERE status IN ('pending', 'downloading');
```