A series page took about a second to show its seasons on a phone, every visit, although they were cached. Three things stacked up: - One SQLite connection behind one mutex served the whole app, so every read queued behind every write. The database now has one owner: a writer thread for writes and a pool of read-only WAL connections for reads. synchronous = NORMAL and a busy timeout on every connection. - The listing query built the set of every available item in the database before filtering to the parent (~80 ms on a desktop for a 100k-item cache), then fetched user data one row at a time. It now checks availability per row, uses the hierarchy indexes (1.5 ms on the same benchmark) and batches the user-data lookup. - A cache read that missed the 100 ms fast path was set aside until the server answered. It is now raced against the server; whichever answers first with content wins. On the Fairphone, Frasier's season and episode lists now come from cache in 34-133 ms (was 600-1030 ms waiting on the server). Fixes found on the way, each with a test that failed first: - sync_queue_mutation could return another mutation's row id: the id came from a second trip to the shared connection. insert() reads it in the same job. - save_to_cache switched foreign keys off on the shared connection across its awaits, so concurrent writes ran unchecked. The toggle now lives inside one writer job, and a page is one transaction instead of one commit per row. Also: thumbnail LRU touches no longer block the lookup; unused tokio-rusqlite dropped. Design and invariants in docs/architecture/08-database-design.md (Connection ownership, Listing query shape) and 03-data-flow.md.
397 lines
14 KiB
Markdown
397 lines
14 KiB
Markdown
# Download Manager & Offline Architecture
|
|
|
|
## Overview
|
|
|
|
**Location**: `src-tauri/src/download/`
|
|
|
|
The download manager provides offline media support with priority-based queue management, progress tracking, retry logic, and smart caching.
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph Frontend["Frontend"]
|
|
DownloadButton["DownloadButton.svelte"]
|
|
DownloadsPage["/downloads"]
|
|
DownloadsStore["downloads.ts store"]
|
|
end
|
|
|
|
subgraph Backend["Rust Backend"]
|
|
Commands["Download Commands"]
|
|
DownloadManager["DownloadManager"]
|
|
DownloadWorker["DownloadWorker"]
|
|
SmartCache["SmartCache Engine"]
|
|
end
|
|
|
|
subgraph Storage["Storage"]
|
|
SQLite[("SQLite DB")]
|
|
MediaFiles[("Downloaded Files")]
|
|
end
|
|
|
|
DownloadButton -->|"invoke('download_item')"| Commands
|
|
DownloadsPage -->|"invoke('get_downloads')"| Commands
|
|
Commands --> DownloadManager
|
|
DownloadManager --> DownloadWorker
|
|
DownloadManager --> SmartCache
|
|
DownloadWorker -->|"HTTP Stream"| MediaFiles
|
|
DownloadWorker -->|"Events"| DownloadsStore
|
|
Commands <--> SQLite
|
|
SmartCache <--> SQLite
|
|
```
|
|
|
|
## Download Worker
|
|
|
|
**Location**: `src-tauri/src/download/worker.rs`
|
|
|
|
The download worker handles HTTP streaming with retry logic and resume support:
|
|
|
|
```rust
|
|
pub struct DownloadWorker {
|
|
client: reqwest::Client,
|
|
max_retries: u32,
|
|
}
|
|
|
|
pub struct DownloadTask {
|
|
pub id: i64,
|
|
pub item_id: String,
|
|
pub user_id: String,
|
|
pub priority: i32,
|
|
pub url: String,
|
|
pub target_path: PathBuf,
|
|
pub mime_type: Option<String>,
|
|
pub expected_size: Option<i64>,
|
|
}
|
|
```
|
|
|
|
**Retry Strategy**:
|
|
- Exponential backoff: 5s, 15s, 45s
|
|
- Maximum 3 retry attempts
|
|
- HTTP Range requests for resume support
|
|
- Progress events emitted every 1MB
|
|
|
|
**Download Flow**:
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant UI
|
|
participant Command as download_item
|
|
participant DB as SQLite
|
|
participant Worker as DownloadWorker
|
|
participant Jellyfin as Jellyfin Server
|
|
participant Store as downloads store
|
|
|
|
UI->>Command: download_item(itemId, userId)
|
|
Command->>DB: INSERT INTO downloads
|
|
Command->>Worker: Start download task
|
|
Worker->>Jellyfin: GET /Items/{id}/Download
|
|
|
|
loop Progress Updates
|
|
Jellyfin->>Worker: Stream chunks
|
|
Worker->>Worker: Write to .part file
|
|
Worker->>Store: Emit progress event
|
|
Store->>UI: Update progress bar
|
|
end
|
|
|
|
Worker->>Worker: Rename .part to final
|
|
Worker->>DB: UPDATE status='completed'
|
|
Worker->>Store: Emit completed event
|
|
Store->>UI: Show completed
|
|
```
|
|
|
|
## Smart Caching Engine
|
|
|
|
**Location**: `src-tauri/src/download/cache.rs`
|
|
|
|
The smart caching system provides predictive downloads based on listening patterns:
|
|
|
|
```rust
|
|
pub struct SmartCache {
|
|
config: Arc<Mutex<CacheConfig>>,
|
|
album_play_history: Arc<Mutex<HashMap<String, Vec<String>>>>,
|
|
}
|
|
|
|
pub struct CacheConfig {
|
|
pub queue_precache_enabled: bool,
|
|
pub queue_precache_count: usize, // Default: 5
|
|
pub album_affinity_enabled: bool,
|
|
pub album_affinity_threshold: usize, // Default: 3
|
|
pub storage_limit: u64, // Default: 10GB
|
|
pub wifi_only: bool, // Default: true
|
|
}
|
|
```
|
|
|
|
**Caching Strategies**:
|
|
|
|
1. **Queue Pre-caching**: Auto-download next 5 tracks when playing (WiFi only)
|
|
2. **Album Affinity**: If user plays 3+ tracks from album, cache entire album
|
|
3. **LRU Eviction**: Remove least recently accessed when storage limit reached
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
Play["Track Played"] --> CheckQueue{"Queue<br/>Pre-cache?"}
|
|
CheckQueue -->|"Yes"| CacheNext5["Download<br/>Next 5 Tracks"]
|
|
|
|
Play --> TrackHistory["Track Play History"]
|
|
TrackHistory --> CheckAlbum{"3+ Tracks<br/>from Album?"}
|
|
CheckAlbum -->|"Yes"| CacheAlbum["Download<br/>Full Album"]
|
|
|
|
CacheNext5 --> CheckStorage{"Storage<br/>Limit?"}
|
|
CacheAlbum --> CheckStorage
|
|
CheckStorage -->|"Exceeded"| EvictLRU["Evict LRU Items"]
|
|
CheckStorage -->|"OK"| Download["Queue Download"]
|
|
```
|
|
|
|
## One Storage Model: Cache Entries Are Downloads
|
|
|
|
**TRACES**: UR-071 | DR-126, DR-127
|
|
|
|
A cache entry **is** a download with a shorter life: the same `downloads` row and
|
|
the same file handling, distinguished by `download_source` plus an expiry. There
|
|
is one storage model rather than a cache and a download library that can
|
|
disagree about what is on disk.
|
|
|
|
| `download_source` | Life | Reclaimed by |
|
|
|-------------------|------|--------------|
|
|
| `'auto'` (temporary) | Expiry, or eviction under space pressure | Both |
|
|
| `'user'` (permanent) | No expiry | Neither |
|
|
|
|
**Eviction only reclaims the temporary tier.** `evict_lru_async` originally
|
|
selected every completed download ordered by `completed_at ASC` with no source
|
|
filter, so hitting the storage limit deleted the *oldest* download — typically a
|
|
film saved deliberately for offline — to make room for a newly precached track.
|
|
It now evicts only `COALESCE(download_source, 'user') = 'auto'` rows.
|
|
`COALESCE` rather than a bare equality is load-bearing: rows predating the
|
|
migration can be NULL, and **unknown provenance must be treated as the user's,
|
|
never as disposable**. Freeing less than requested is the correct outcome when
|
|
only user downloads remain — the caller reports "unable to free enough".
|
|
|
|
A temporary row can be **promoted** to permanent when the user chooses to keep
|
|
it. That only clears the expiry and flips the source; the bytes never move.
|
|
|
|
## Offline Catalog Visibility
|
|
|
|
**TRACES**: UR-052 | DR-078, DR-079, DR-080
|
|
|
|
Offline, a library page shows **only media on the device**. A "Show all server
|
|
media" toggle additionally reveals the cached server catalog, greyed out and
|
|
queueable for download on reconnect.
|
|
|
|
The gate is a process-global `INCLUDE_CATALOG_BROWSE` in
|
|
`repository/offline.rs`, written by the `set_show_server_catalog` command. It
|
|
gates the synced-catalog leg of `get_items`; without it the toggle rendered but
|
|
every server item still appeared, which is the defect the spec was written for.
|
|
`isConnected` derives from backend-reported reachability alone (DR-079) — see
|
|
[07-connectivity.md](07-connectivity.md).
|
|
|
|
Per-item disk usage comes from `repository_get_download_disk_usage`
|
|
(`DownloadDiskUsage`), aggregated from `downloads.file_size` — used by the
|
|
Downloaded browse cards, detail pages, the device total and the remove
|
|
confirmation (DR-085).
|
|
|
|
## What a Video Download Fetches
|
|
|
|
**TRACES**: UR-071, UR-004 | DR-171, DR-293
|
|
|
|
An `original`-quality download is the server's untouched file (`Static=true`)
|
|
unless its audio cannot be decoded by **the renderer that will play it** —
|
|
`renderer_can_decode_audio`, DR-234's per-platform answer. Only then is the
|
|
server asked to re-encode the audio on the way down (`allowVideoStreamCopy`
|
|
keeps the picture byte-for-byte).
|
|
|
|
The distinction matters because a transcode is generated as it is sent: no
|
|
`Content-Length`, `Range` ignored. It measured ~1 MB/s and restarted from byte
|
|
zero on every network blip, against a direct copy that moved a 910 MB episode in
|
|
94 s with no retries. On Android the renderer is ExoPlayer with the FFmpeg
|
|
extension ([05-platform-backends.md](05-platform-backends.md)), which decodes
|
|
AC-3/E-AC-3/DTS/TrueHD, so Android downloads are always the direct copy. On
|
|
Linux the webview still renders video and the transcode still applies.
|
|
|
|
The policy used to judge against the *webview's* codec list on every platform
|
|
(DR-171), because a download outlives the native-video setting that was active
|
|
when it arrived. That reasoning is why Android's webview video path was removed
|
|
rather than merely defaulted off: a file downloaded as the original must never
|
|
meet a renderer that cannot decode it.
|
|
|
|
## Offline Means No Network
|
|
|
|
**TRACES**: UR-002, UR-071 | DR-294
|
|
|
|
Three defects made "offline" depend on the network; the invariants that replace
|
|
them:
|
|
|
|
- **A download plays without the server.** Playing a downloaded item asked the
|
|
server for its `PlaybackInfo` only to read the media-source id; offline that
|
|
retried for seven seconds, failed, and the file was never opened.
|
|
`OfflineRepository::local_playback_info` answers for any completed download of
|
|
the current user — local path, direct play, item id as media source (a download
|
|
names no source, so the server served its default, which carries the item's id)
|
|
— and `HybridRepository::get_playback_info` consults it **first**.
|
|
- **A slow cache read is waited for, never discarded.** Reads no longer queue
|
|
behind writes (see [Connection ownership](08-database-design.md#connection-ownership)),
|
|
but a big query, a cold page cache or a busy reader pool can still push a read
|
|
past the 100 ms fast path. Such a read is raced against the server rather
|
|
than set aside (see [03-data-flow.md](03-data-flow.md)). `get_items`, the library list, genres and playlist items used to discard
|
|
such a read, wait for the server, and — offline — return its error over data on
|
|
disk; "More info" on a downloaded show failed that way. They now start the read
|
|
with `cache_try` (which keeps it running) and `settle` on it when the server
|
|
fails. Cache-only reads (search, favourites) have no server to fall back from,
|
|
so they simply await the cache.
|
|
- **Server-only sections degrade, they do not fail a page.** Next Up went only to
|
|
the server, and the TV landing page loads it in one `Promise.all`, so offline it
|
|
blanked the whole page. It now falls back to the cache when the server cannot
|
|
answer.
|
|
|
|
What still needs the server, deliberately: streaming anything not downloaded,
|
|
live TV and channels, reporting playback, and edits (favourites, playlists,
|
|
played state).
|
|
|
|
## Download Commands
|
|
|
|
**Location**: `src-tauri/src/commands/download/` — `mod.rs` (the commands below), `pinning.rs`, `smart_cache.rs`
|
|
|
|
| Command | Parameters | Description |
|
|
|---------|------------|-------------|
|
|
| `download_item` | `item_id, user_id, file_path` | Queue single item download |
|
|
| `download_album` | `album_id, user_id` | Queue all tracks in album |
|
|
| `get_downloads` | `user_id, status_filter` | Get download list |
|
|
| `pause_download` | `download_id` | Pause active download |
|
|
| `resume_download` | `download_id` | Resume paused download |
|
|
| `cancel_download` | `download_id` | Cancel and delete partial |
|
|
| `delete_download` | `download_id` | Delete completed download |
|
|
| `download_video` / `download_series` / `download_season` | item ids | Queue video content |
|
|
| `get_download_storage_stats` | `user_id` | Device totals for the downloads screen |
|
|
| `delete_album_downloads` / `delete_downloads_under` / `delete_all_downloads` | container id | Bulk removal |
|
|
| `pin_item` / `unpin_item` / `is_item_pinned` | `item_id` | Protect metadata from a cache clear |
|
|
| `set_max_concurrent_downloads` | `max` | Worker concurrency (3 by default) |
|
|
|
|
## Offline Commands
|
|
|
|
**Location**: `src-tauri/src/commands/offline.rs`
|
|
|
|
| Command | Parameters | Description |
|
|
|---------|------------|-------------|
|
|
| `offline_is_available` | `item_id` | Check if item downloaded |
|
|
| `offline_get_items` | `user_id` | Get all offline items |
|
|
| `offline_search` | `user_id, query` | Search downloaded items |
|
|
|
|
## Player Integration
|
|
|
|
**Location**: `src-tauri/src/commands/player.rs` (modified)
|
|
|
|
The player checks for local downloads before streaming:
|
|
|
|
```rust
|
|
fn create_media_item(req: PlayItemRequest, db: Option<&DatabaseWrapper>) -> MediaItem {
|
|
let local_path = db.and_then(|db_wrapper| {
|
|
check_for_local_download(db_wrapper, &jellyfin_id).ok().flatten()
|
|
});
|
|
|
|
let source = if let Some(path) = local_path {
|
|
MediaSource::Local {
|
|
file_path: PathBuf::from(path),
|
|
jellyfin_item_id: Some(jellyfin_id.clone())
|
|
}
|
|
} else {
|
|
MediaSource::Remote {
|
|
stream_url: req.stream_url,
|
|
jellyfin_item_id: jellyfin_id.clone()
|
|
}
|
|
};
|
|
|
|
MediaItem { source, /* ... */ }
|
|
}
|
|
```
|
|
|
|
## Frontend Downloads Store
|
|
|
|
**Location**: `src/lib/stores/downloads.ts`
|
|
|
|
```typescript
|
|
interface DownloadsState {
|
|
downloads: Record<number, DownloadInfo>;
|
|
activeCount: number;
|
|
queuedCount: number;
|
|
}
|
|
|
|
const downloads = createDownloadsStore();
|
|
|
|
// Actions
|
|
downloads.downloadItem(itemId, userId, filePath)
|
|
downloads.downloadAlbum(albumId, userId)
|
|
downloads.pause(downloadId)
|
|
downloads.resume(downloadId)
|
|
downloads.cancel(downloadId)
|
|
downloads.delete(downloadId)
|
|
downloads.refresh(userId, statusFilter)
|
|
|
|
// Derived stores
|
|
export const activeDownloads = derived(downloads, ($d) =>
|
|
Object.values($d.downloads).filter((d) => d.status === 'downloading')
|
|
);
|
|
```
|
|
|
|
**Event Handling**:
|
|
|
|
The store listens to Tauri events for real-time updates:
|
|
|
|
```typescript
|
|
listen<DownloadEvent>('download-event', (event) => {
|
|
const payload = event.payload;
|
|
|
|
switch (payload.type) {
|
|
case 'started':
|
|
// Update status to 'downloading'
|
|
case 'progress':
|
|
// Update progress and bytes_downloaded
|
|
case 'completed':
|
|
// Update status to 'completed', progress to 1.0
|
|
case 'failed':
|
|
// Update status to 'failed', store error message
|
|
}
|
|
});
|
|
```
|
|
|
|
## Download UI Components
|
|
|
|
**DownloadButton** (`src/lib/components/library/DownloadButton.svelte`):
|
|
- Multiple states: available, downloading, completed, failed, paused
|
|
- Circular progress ring during download
|
|
- Size variants: sm, md, lg
|
|
- Integrated into TrackList with `showDownload={true}` prop
|
|
|
|
**DownloadItem** (`src/lib/components/downloads/DownloadItem.svelte`):
|
|
- Individual download list item with progress bar
|
|
- Action buttons: pause, resume, cancel, delete
|
|
- Status indicators with color coding
|
|
|
|
**Downloads Page** (`src/routes/downloads/+page.svelte`):
|
|
- Active/Completed tabs
|
|
- Bulk actions: Pause All, Resume All, Clear Completed
|
|
- Empty states with helpful instructions
|
|
|
|
## Database Schema
|
|
|
|
**downloads table**:
|
|
|
|
```sql
|
|
CREATE TABLE downloads (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
item_id TEXT NOT NULL,
|
|
user_id TEXT NOT NULL,
|
|
file_path TEXT,
|
|
file_size INTEGER,
|
|
mime_type TEXT,
|
|
status TEXT DEFAULT 'pending', -- pending, downloading, completed, failed, paused
|
|
progress REAL DEFAULT 0.0,
|
|
bytes_downloaded INTEGER DEFAULT 0,
|
|
priority INTEGER DEFAULT 0,
|
|
error_message TEXT,
|
|
retry_count INTEGER DEFAULT 0,
|
|
queued_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
|
started_at TEXT,
|
|
completed_at TEXT
|
|
);
|
|
|
|
CREATE INDEX idx_downloads_queue
|
|
ON downloads(status, priority DESC, queued_at ASC)
|
|
WHERE status IN ('pending', 'downloading');
|
|
```
|