dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 38 Wiki Activity

feat(search): answer search from a local index; tier downloads by lifetime

Browse Source 
Search's instant leg read only downloaded items, so with no downloads it
returned nothing and every keystroke fell through to a full Recursive=true
server query. It now reads the whole synced catalog through the same
availability CTE get_items uses, gated on the same include_catalog_browse
flag so search and browse cannot diverge. (UR-065, DR-108)

Also fixes three defects found while confirming that:

- items_fts grew by a full duplicate index every catalog pass. INSERT OR
  REPLACE fires no AFTER DELETE trigger without recursive_triggers, so the
  old index row was orphaned, and a TEXT PRIMARY KEY meant the replacement
  took a fresh rowid and inserted a second entry. Now a real upsert, with
  migration 021 rebuilding existing indexes. (DR-110)
- DELETE FROM items existed nowhere, so server-side deletions never
  propagated. Adds a post-crawl mark-and-sweep, scoped to crawled types,
  skipping downloaded items, and refusing to run after a partial crawl
  because items.parent_id cascades. (DR-110)
- The index omitted MusicArtist, Playlist and People, which search groups
  results by. Adds them plus people_fts (migration 022). (DR-111)

Re-indexing moves from a frontend startup call to a Rust background task
with a 6h TTL, so a long session no longer searches a stale catalog and a
restart no longer forces a crawl regardless of freshness. (DR-109, IR-030)

Downloads gain a lifetime tier. Eviction selected every completed row by
age with no download_source filter, so hitting the storage limit deleted
the oldest download -- typically one saved deliberately for offline -- to
make room for a precached track. It now reclaims only 'auto' rows, and
expired ones are reclaimed first, before live cache is evicted.
(DR-126, DR-127)

Downloaded video and audio-only handoffs now play from disk instead of
streaming; the video path had never consulted downloads at all. No
transcode is involved: MPV runs video=no and ExoPlayer has no surface for
an Audio item. (DR-123 in part, DR-128)

FTS queries are built as quoted phrases so apostrophes, hyphens and
slashes are data rather than operator syntax, and the item-type filter is
bound rather than interpolated.

Specs: docs/specs/catalog-index-search.md,
docs/specs/read-through-media-cache.md

Includes concurrently-developed favourites browsing and background-audio
stream-end handling; the two workstreams share offline.rs, lib.rs and
online.rs, so no subset of files builds independently.
This commit is contained in:
dtourolle
2026-08-04 17:35:17 +02:00
parent c55ff45692
commit 62873cab3d
52 changed files with 6110 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src-tauri/src/commands/catalog.rs
+315 -5
View File
@@ -14,10 +14,12 @@
//! that were queued offline (they have `stream_url IS NULL`), mirroring the
//! heal-and-pump pattern in `player_preload_upcoming`.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
use log::{info, warn};
use tauri::State;
use tauri::{Emitter, Manager, State};
use crate::commands::download::{pump_download_queue, DownloadManagerWrapper};
use crate::commands::repository::RepositoryManagerWrapper;
@@ -29,16 +31,79 @@ use crate::storage::db_service::{DatabaseService, Query, QueryParam};
/// full-catalog sync.
const LAST_CATALOG_SYNC_KEY: &str = "last_catalog_sync";
/// How long an index stays fresh before a re-index is due.
///
/// This lives in Rust rather than being a frontend constant because it decides
/// *whether the local cache is authoritative* — the same class of decision as
/// `include_catalog_browse`, and squarely the "sync policy" the spec review
/// checklist keeps out of the presentation layer. If it later becomes
/// user-configurable it stays a Rust-owned setting edited through a command.
const CATALOG_INDEX_TTL: Duration = Duration::from_secs(6 * 60 * 60);
/// How often the scheduler wakes to *check* staleness. Far shorter than the TTL
/// because a tick is nearly free — one indexed `app_settings` lookup — and it is
/// what makes the indexer responsive to events it cannot subscribe to: signing
/// in, and coming back online. The TTL, not the tick, decides whether a crawl
/// actually happens.
const CATALOG_INDEX_TICK: Duration = Duration::from_secs(5 * 60);
/// Delay before the first staleness check, to let sign-in complete and the
/// repository be registered. Without it the first check runs against an empty
/// repository manager and a fresh install would sit unindexed until the next
/// tick.
const CATALOG_INDEX_FIRST_CHECK: Duration = Duration::from_secs(15);
/// Kebab-case, per the project's event convention.
pub const CATALOG_INDEX_EVENT: &str = "catalog-index-event";
/// Guards against two passes running at once. Replaces the frontend's
/// `syncInProgress` boolean in `offlineCatalog.ts`, which could not see a pass
/// started by the scheduler.
static INDEX_IN_PROGRESS: AtomicBool = AtomicBool::new(false);
/// Clears [`INDEX_IN_PROGRESS`] however the pass leaves — including on the `?`
/// early return when `get_libraries` fails, which a plain store at the end of
/// the function would leak.
struct IndexPassGuard;
impl Drop for IndexPassGuard {
fn drop(&mut self) {
INDEX_IN_PROGRESS.store(false, Ordering::SeqCst);
}
}
/// Progress of a background index pass, for the staleness hint in the UI.
#[derive(specta::Type, Debug, Clone, serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct CatalogIndexEvent {
/// `started` | `finished` | `failed`
pub state: String,
pub items_cached: usize,
pub items_pruned: usize,
pub libraries_failed: usize,
/// Present on `failed`.
pub error: Option<String>,
}
/// Item types worth caching for offline browsing: containers the library
/// landing pages render plus the playable leaves users queue for download.
/// `MusicArtist` and `Playlist` are here because search groups results by them
/// (UR-060's Artists group). Without them in the crawl, the local index can
/// never answer an artist query and those groups can only ever be filled by the
/// server leg. Keep this in step with what `prune_stale_catalog` is allowed to
/// sweep — the crawl is only authoritative for the types it asks for.
///
/// TRACES: UR-065, UR-060 | DR-111
const CATALOG_ITEM_TYPES: &[&str] = &[
"MusicAlbum",
"MusicArtist",
"Movie",
"Series",
"Season",
"Episode",
"Audio",
"BoxSet",
"Playlist",
];
#[derive(specta::Type, Debug, Clone, serde::Serialize, serde::Deserialize)]
@@ -48,6 +113,9 @@ pub struct CatalogSyncResult {
pub items_cached: usize,
/// Libraries that failed to sync (e.g. server hiccup); best-effort.
pub libraries_failed: usize,
/// Entries removed because the server no longer has them. Always 0 when any
/// library failed, since a partial crawl cannot prove an item is gone.
pub items_pruned: usize,
}
#[derive(specta::Type, Debug, Clone, serde::Serialize, serde::Deserialize)]
@@ -71,8 +139,6 @@ pub async fn sync_full_catalog(
db: State<'_, DatabaseWrapper>,
handle: String,
) -> Result<CatalogSyncResult, String> {
use crate::repository::MediaRepository;
let repo = repository.0.get(&handle).ok_or("Repository not found")?;
let db_service = {
@@ -80,6 +146,27 @@ pub async fn sync_full_catalog(
Arc::new(database.service())
};
run_index_pass(repo, db_service).await
}
/// One full-catalog indexing pass, shared by the [`sync_full_catalog`] command
/// and the background scheduler (DR-109) so there is exactly one implementation
/// and one concurrency guard.
///
/// TRACES: UR-065 | DR-109, DR-110
pub(crate) async fn run_index_pass(
repo: Arc<crate::repository::HybridRepository>,
db_service: Arc<crate::storage::db_service::RusqliteService>,
) -> Result<CatalogSyncResult, String> {
use crate::repository::MediaRepository;
// One pass at a time. The command and the scheduler can both land here, and
// two concurrent crawls would double the server load and race on the sweep.
if INDEX_IN_PROGRESS.swap(true, Ordering::SeqCst) {
return Err("A catalog index pass is already running".to_string());
}
let _guard = IndexPassGuard;
let libraries = repo.get_libraries().await.map_err(|e| e.to_string())?;
info!(
"[Catalog] Full sync starting across {} libraries",
@@ -88,6 +175,10 @@ pub async fn sync_full_catalog(
let include_types: Vec<String> = CATALOG_ITEM_TYPES.iter().map(|s| s.to_string()).collect();
// Taken before the crawl: every row the crawl writes gets a `synced_at`
// newer than this, so anything still older afterwards is gone server-side.
let pass_started_at = chrono::Utc::now().to_rfc3339();
let mut items_cached = 0usize;
let mut libraries_failed = 0usize;
@@ -118,6 +209,35 @@ pub async fn sync_full_catalog(
}
}
// Propagate server-side deletions — but only after a *complete* crawl.
// `sync_full_catalog` is best-effort per library, and `items.parent_id` is
// ON DELETE CASCADE, so sweeping when a library failed to fetch could
// cascade a whole series away because one request timed out.
let mut items_pruned = 0usize;
if libraries_failed == 0 && !libraries.is_empty() {
match repo
.prune_stale_catalog(&pass_started_at, &include_types)
.await
{
Ok(removed) => {
items_pruned = removed;
if removed > 0 {
info!(
"[Catalog] Pruned {} entries no longer on the server",
removed
);
}
}
Err(e) => warn!("[Catalog] Prune of stale catalog entries failed: {:?}", e),
}
} else if libraries_failed > 0 {
info!(
"[Catalog] Skipping stale-entry prune: {} librar{} failed to sync, so the crawl is not authoritative",
libraries_failed,
if libraries_failed == 1 { "y" } else { "ies" }
);
}
// Record the sync time so callers can skip re-syncing too eagerly.
let now = chrono::Utc::now().to_rfc3339();
let upsert = Query::with_params(
@@ -133,16 +253,169 @@ pub async fn sync_full_catalog(
}
info!(
"[Catalog] Full sync complete: {} items cached, {} libraries failed",
items_cached, libraries_failed
"[Catalog] Full sync complete: {} items cached, {} pruned, {} libraries failed",
items_cached, items_pruned, libraries_failed
);
Ok(CatalogSyncResult {
items_cached,
libraries_failed,
items_pruned,
})
}
/// Whether an index pass is due, given when one last completed.
///
/// Pure so the policy is unit-testable without a clock, a server, or a database.
/// `None` (never indexed) and an unparseable stored value both mean "due" — a
/// corrupt timestamp should trigger a re-index, not silently freeze the catalog.
///
/// TRACES: UR-065 | DR-109 | UT-115
pub(crate) fn index_is_due(
last_synced_at: Option<&str>,
now: chrono::DateTime<chrono::Utc>,
ttl: Duration,
) -> bool {
let Some(raw) = last_synced_at else {
return true;
};
let Ok(last) = chrono::DateTime::parse_from_rfc3339(raw) else {
return true;
};
now.signed_duration_since(last.with_timezone(&chrono::Utc))
.to_std()
.map(|elapsed| elapsed >= ttl)
// Negative elapsed => the stored stamp is in the future (clock skew).
// Not due; a future stamp will age into due-ness on its own.
.unwrap_or(false)
}
/// Read the last-sync timestamp straight from `app_settings`.
async fn read_last_sync(
db_service: &Arc<crate::storage::db_service::RusqliteService>,
) -> Option<String> {
db_service
.query_optional(
Query::with_params(
"SELECT value FROM app_settings WHERE key = ?",
vec![QueryParam::String(LAST_CATALOG_SYNC_KEY.to_string())],
),
|row| row.get(0),
)
.await
.ok()
.flatten()
}
/// Start the background catalog indexer.
///
/// Replaces the frontend's startup-only `syncCatalog()` call: index freshness is
/// sync policy and belongs in Rust (see the layer assignment in
/// docs/specs/catalog-index-search.md). Ticks every [`CATALOG_INDEX_TICK`] and
/// runs a pass when a repository exists, the server is reachable, and the index
/// is older than [`CATALOG_INDEX_TTL`].
///
/// TRACES: UR-065 | DR-109, IR-030
pub fn spawn_catalog_indexer(app: tauri::AppHandle) {
tauri::async_runtime::spawn(async move {
// Check shortly after launch, then on every tick — not tick-then-check,
// which would leave a fresh install unindexed for a full tick.
tokio::time::sleep(CATALOG_INDEX_FIRST_CHECK).await;
loop {
if let Err(e) = maybe_run_scheduled_pass(&app).await {
// Never fatal — a failed pass leaves the existing index in place
// and we retry on the next tick.
warn!("[Catalog] Scheduled index pass skipped: {}", e);
}
tokio::time::sleep(CATALOG_INDEX_TICK).await;
}
});
}
/// One scheduler tick: check the preconditions, then index if due.
async fn maybe_run_scheduled_pass(app: &tauri::AppHandle) -> Result<(), String> {
if INDEX_IN_PROGRESS.load(Ordering::SeqCst) {
return Ok(());
}
let db_service = {
let db = app.state::<DatabaseWrapper>();
let database = db.0.lock().map_err(|e| e.to_string())?;
Arc::new(database.service())
};
if !index_is_due(
read_last_sync(&db_service).await.as_deref(),
chrono::Utc::now(),
CATALOG_INDEX_TTL,
) {
return Ok(());
}
// Offline: leave the index alone. The crawl would fail every library and,
// more importantly, a partial crawl must never reach the deletion sweep.
{
let monitor = app.state::<crate::commands::connectivity::ConnectivityMonitorWrapper>();
let monitor = monitor.0.lock().await;
if !monitor.get_status().await.is_server_reachable {
return Ok(());
}
}
let repo = {
let manager = app.state::<RepositoryManagerWrapper>();
let handles = manager.0.handles();
let Some(handle) = handles.first() else {
// Not signed in yet.
return Ok(());
};
manager.0.get(handle).ok_or("Repository not found")?
};
info!("[Catalog] Index is stale; starting a scheduled pass");
let _ = app.emit(
CATALOG_INDEX_EVENT,
CatalogIndexEvent {
state: "started".to_string(),
items_cached: 0,
items_pruned: 0,
libraries_failed: 0,
error: None,
},
);
match run_index_pass(repo, db_service).await {
Ok(result) => {
let _ = app.emit(
CATALOG_INDEX_EVENT,
CatalogIndexEvent {
state: "finished".to_string(),
items_cached: result.items_cached,
items_pruned: result.items_pruned,
libraries_failed: result.libraries_failed,
error: None,
},
);
Ok(())
}
Err(e) => {
let _ = app.emit(
CATALOG_INDEX_EVENT,
CatalogIndexEvent {
state: "failed".to_string(),
items_cached: 0,
items_pruned: 0,
libraries_failed: 0,
error: Some(e.clone()),
},
);
Err(e)
}
}
}
/// Report the last-synced timestamp so the UI can show a hint / decide whether
/// to trigger a fresh sync.
#[tauri::command]
@@ -378,6 +651,43 @@ mod tests {
use rusqlite::Connection;
use std::sync::Mutex;
/// The re-index policy. Pure, so it is testable without a clock, a server or
/// a database — which is the reason it was factored out of the scheduler.
///
/// TRACES: UR-065 | DR-109 | UT-115
#[test]
fn test_index_is_due() {
let ttl = Duration::from_secs(6 * 60 * 60);
let now = chrono::DateTime::parse_from_rfc3339("2026-08-04T12:00:00+00:00")
.unwrap()
.with_timezone(&chrono::Utc);
// Never indexed => due. This is the first-run case.
assert!(index_is_due(None, now, ttl));
// Indexed 7 hours ago => past the 6h TTL => due.
assert!(index_is_due(Some("2026-08-04T05:00:00+00:00"), now, ttl));
// Indexed 1 hour ago => fresh => not due. This is what stops the
// scheduler re-crawling every tick.
assert!(!index_is_due(Some("2026-08-04T11:00:00+00:00"), now, ttl));
// Exactly at the TTL boundary counts as due.
assert!(index_is_due(Some("2026-08-04T06:00:00+00:00"), now, ttl));
// A corrupt stored value must trigger a re-index, not freeze the
// catalog forever behind an unparseable timestamp.
assert!(index_is_due(Some("not-a-timestamp"), now, ttl));
assert!(index_is_due(Some(""), now, ttl));
// A timestamp in the future (clock skew, or a restored backup) is not
// due — it ages into due-ness rather than causing a crawl every tick.
assert!(!index_is_due(Some("2026-08-05T00:00:00+00:00"), now, ttl));
// Offsets other than UTC are compared as instants, not as strings.
assert!(!index_is_due(Some("2026-08-04T13:30:00+02:00"), now, ttl));
}
fn test_db() -> Arc<RusqliteService> {
let conn = Connection::open_in_memory().unwrap();
conn.execute_batch(