# DarkRoom — Catalog, library view, and background work **Status:** Draft v0.1 · 2026-08-09 **Companion to:** [requirements.md](requirements.md), [architecture.md](architecture.md) Specifies `dr-catalog`: the index the library view queries, how it stays current without rescanning everything, and how thumbnails get made. [architecture.md §6.2](architecture.md) sketches the schema in eight lines; this expands it to the point of implementability and fills the two gaps that sketch leaves open — **incremental local scan** and **the job queue**. Sync's remote side is already designed ([architecture.md §8](architecture.md)): ETag pruning turns a no-op sync of 50k images into one request. Nothing equivalent existed for a local root, which is the central problem this document solves. --- ## 1. What this must not do Stated first because every design choice below follows from it. | Must not | Why | |---|---| | Stat 50k files to open the catalog | NFR-P1: catalog open < 2 s desktop, < 4 s Android. SAF `DocumentsContract` queries are far slower than `stat` (spike S10). | | Re-derive thumbnails for unchanged images | NFR-P3 throughput is for *new* work; redoing it on every connect makes first paint unbounded. | | Fetch previews for remote images nobody looks at | A 50k remote library at 1–3 MB per range-extract is 50–150 GB. FR-NC-6 forbids bulk transfer by default. | | Evaluate cache rules per grid cell | ARCH §9.5 already answers this: `tier_desired` is materialised. | | Block the UI executor on any of it | NFR-P9, NFR-ARCH-1. | The unifying principle: **work is proportional to what changed, or to what the user is looking at — never to library size.** --- ## 2. Schema Extends [architecture.md §6.2](architecture.md). Additions beyond that sketch are marked ⊕. ```sql -- Roots ----------------------------------------------------------------- roots( id INTEGER PRIMARY KEY, kind TEXT, -- 'local' | 'saf' | 'remote' grant_blob BLOB, -- SAF persisted permission; NULL on Linux label TEXT, last_seen INTEGER, scan_generation INTEGER -- ⊕ bumped per completed scan; see §3.4 ); -- Folders: the unit of change detection, local and remote alike --------- folders( id INTEGER PRIMARY KEY, root_id INTEGER NOT NULL REFERENCES roots(id), parent_id INTEGER REFERENCES folders(id), path TEXT NOT NULL, etag TEXT, -- remote: propagating ETag (ARCH §8.4) mtime INTEGER, -- ⊕ local: directory mtime entry_count INTEGER, -- ⊕ local: direct children, mtime's blind spot scanned_generation INTEGER, -- ⊕ deletion sweep; see §3.4 UNIQUE(root_id, path) ); -- Images ---------------------------------------------------------------- images( id INTEGER PRIMARY KEY, root_id INTEGER NOT NULL REFERENCES roots(id), folder_id INTEGER REFERENCES folders(id), -- ⊕ folder filter without LIKE source_ref TEXT NOT NULL, content_hash TEXT, -- NULL until hashed; see §3.5 format TEXT, w INTEGER, h INTEGER, captured_at INTEGER, -- UTC seconds; NULL if EXIF absent captured_offset INTEGER, -- ⊕ minutes east of UTC; see §4.2 camera TEXT, lens TEXT, iso INTEGER, aperture REAL, shutter REAL, availability INTEGER, file_size INTEGER, -- ⊕ cheap change signal alongside mtime file_mtime INTEGER, -- ⊕ metadata_state INTEGER, -- ⊕ 0=none 1=stat-only 2=full EXIF; §3.5 sidecar_mtime INTEGER, UNIQUE(root_id, source_ref) ); -- Versions, keywords, remote, cache: per ARCH §6.2, unchanged ----------- -- Collections ⊕ --------------------------------------------------------- collections( id INTEGER PRIMARY KEY, name TEXT NOT NULL, parent_id INTEGER REFERENCES collections(id), -- collection sets kind INTEGER NOT NULL, -- 0 = manual, 1 = smart selector_json TEXT, -- smart only; the §5 Selector created INTEGER ); collection_members( collection_id INTEGER NOT NULL REFERENCES collections(id) ON DELETE CASCADE, image_id INTEGER NOT NULL REFERENCES images(id) ON DELETE CASCADE, position INTEGER, -- manual ordering; NULL = by capture time PRIMARY KEY(collection_id, image_id) ); -- Jobs ⊕ ---------------------------------------------------------------- jobs( id INTEGER PRIMARY KEY, kind INTEGER NOT NULL, subject_id INTEGER, -- image or folder, per kind priority INTEGER NOT NULL, state INTEGER NOT NULL, -- 0=pending 1=running 2=failed attempts INTEGER NOT NULL DEFAULT 0, not_before INTEGER, -- retry backoff payload TEXT, UNIQUE(kind, subject_id) -- coalescing; see §6.2 ); ``` Indices that exist for a stated query, not speculatively: ```sql CREATE INDEX images_captured ON images(captured_at); -- §4 timeline CREATE INDEX images_folder ON images(folder_id); CREATE INDEX images_hash ON images(content_hash) WHERE content_hash IS NOT NULL; CREATE INDEX folders_parent ON folders(parent_id); CREATE INDEX jobs_ready ON jobs(state, priority DESC, not_before); CREATE INDEX versions_image ON versions(image_id); CREATE INDEX members_image ON collection_members(image_id); ``` `content_hash` is indexed *partially*. It is NULL for most rows most of the time (§3.5), and a partial index over the non-NULL subset is both smaller and what FR-CAT-9's reconnection-by-hash and FR-CAT-11's duplicate detection actually query. --- ## 3. Incremental scan ### 3.1 The local analogue of ETag pruning Nextcloud propagates ETags up the tree, so one request proves a whole library unchanged ([architecture.md §8.4](architecture.md)). A filesystem offers no such guarantee — a directory's mtime changes when its *direct* entries change, and not when a grandchild does. There is no cheap "did anything below here change" probe. So local scan prunes at each level rather than at the root: ``` scan(folder): (mtime, count) = stat(folder) if (mtime, count) == stored: # This directory's own entries are unchanged. Its files need no # examination at all — but subdirectories may still have changed # internally, so recurse into known children without listing. for child in stored_children(folder): scan(child) else: entries = list(folder) # the expensive call reconcile(folder, entries) # §3.3 for child in entries.dirs: scan(child) mark scanned(folder, current_generation) ``` Cost is **one `stat` per directory** when nothing changed, versus one per *file*. A 50k-image library in ~2k folders costs 2k stats — a few milliseconds locally, and the difference between meeting and missing NFR-P1 on SAF. The recursion into unchanged directories is not redundant: it is what makes a change to one deep file detectable at all, given no upward propagation. What it avoids is the *listing* — on SAF a `DocumentsContract` query returning 200 rows costs far more than a metadata probe on the directory itself. ### 3.2 Why entry-count as well as mtime Directory mtime alone misses a real case: delete one file and create another within the same timestamp granularity, and mtime can be unchanged while contents differ. Some filesystems and most SAF providers report coarse timestamps, which widens the window. Storing `(mtime, entry_count)` closes the common form of this — a paired add and remove changes neither, but that is rarer than a bare add or remove, and both of those move the count. It is a cheap narrowing, not a proof. **Where correctness must not depend on it,** the user gets an explicit *Rescan folder* action (FR-CAT-1), and reconnection matches by content hash (FR-CAT-9). Sync's remote path is unaffected — ETags are authoritative there. ### 3.3 Reconciling a changed directory For each entry in a listing: | Situation | Action | |---|---| | Not in catalog | Insert with `metadata_state = 1`; enqueue `ExtractMetadata` | | In catalog, `(size, mtime)` match | Nothing — the common case | | In catalog, `(size, mtime)` differ | Re-enqueue `ExtractMetadata` and `Thumbnail`; clear `content_hash` | | In catalog, absent from listing | Deletion candidate — §3.4 | | Placeholder (`*.nextcloud`) | Catalogue as the image it stands for; `Availability::Offline` (ARCH §9.0) | Sidecars are examined in the same pass: a `.drsc` whose mtime exceeds `images.sidecar_mtime` enqueues a `ReadSidecar` job. This is how an edit made on another device — landed by the Nextcloud client, not by us — reaches the catalog. ### 3.4 Deletion without a full sweep A file removed outside the app appears only as an *absence*, which a pruned scan cannot see: the folder it vanished from has a changed mtime and is listed, but a folder never visited is never compared. Generation counting handles this without a full pass. Each scan bumps `roots.scan_generation`, and every folder reached — whether listed or skipped — records it. After the walk: ```sql -- Folders never reached: their parent no longer lists them. DELETE FROM folders WHERE root_id = ?1 AND scanned_generation < ?2; ``` Images under a deleted folder cascade. Images missing from a *listed* folder are caught directly in §3.3. Together these cover deletion with no additional traversal. Deletion here means **removing the catalog row for a source proven absent**, which FR-CAT-9 sharply distinguishes from a source merely unreachable. A root that fails to open at all — unplugged drive, revoked SAF grant — aborts the scan and marks the root offline. It never runs the sweep, because every folder would look unreached and the sweep would delete the entire library. That guard is the single most dangerous line in this design, and it is stated as an invariant: **the deletion sweep runs only after a scan that completed without a root-level access error.** ### 3.5 Metadata in two passes Full EXIF extraction requires opening and parsing each file. At 50k images that is minutes, and it must not stand between the user and a usable grid. `metadata_state` records how far each image has got: | State | Holds | Cost | |---|---|---| | 0 — none | Row exists, nothing read | — | | 1 — stat-only | Name, size, mtime, format from extension | Free, from the listing | | 2 — full | EXIF: capture time, camera, lens, exposure, dimensions | One open + parse | The grid is usable at state 1: it can show filenames, sort by filename or file mtime, and display placeholder cells. Promotion to state 2 runs as background jobs, prioritised by what is on screen (§6.3), so visible images get real capture times within a frame or two of being scrolled to. **Capture-time filtering (§4) needs state 2**, so a freshly scanned library's timeline is incomplete until the pass finishes. The UI states this plainly — a progress affordance on the timeline, not a silently wrong filter. Which is the FR-NC-6c principle applied to metadata rather than pixels: say what you actually have. `content_hash` is a *third*, still lazier tier. It requires reading the whole file, so it is computed only when something needs it: import duplicate detection (FR-CAT-11), or reconnecting a moved source (FR-CAT-9). Never during a routine scan. --- ## 4. The library view ### 4.1 Query model The UI never assembles SQL. It hands the catalog a `Query` and receives a stable, windowable result: ```rust pub struct Query { pub filter: Selector, // §5 — same type cache rules use pub sort: Sort, pub descending: bool, } pub enum Sort { CapturedAt, Added, FileName, Rating, /// Manual order within a collection; falls back to CapturedAt elsewhere. CollectionPosition, } ``` Results are fetched by window, never wholesale — FR-CAT-4 requires memory bounded independently of catalog size: ```rust impl Catalog { fn count(&self, q: &Query) -> Result; fn window(&self, q: &Query, range: Range) -> Result, CatalogError>; } ``` `GridRow` carries exactly what a cell draws — id, thumbnail key, availability, rating, flag, capture time — and nothing that would require a join per cell. Availability badges read `tier_desired` directly (ARCH §9.5), so no rule evaluation happens on the render path. A `LIMIT/OFFSET` window degrades at high offsets, since SQLite must walk the skipped rows. Scrolling is overwhelmingly *sequential*, so the catalog keeps a keyset cursor for forward and backward paging and falls back to OFFSET only for a scrollbar jump. Jumps are rare and single; scrolling is continuous. ### 4.2 Time Capture time is the spine of a photo library, and it has one persistent trap: **a photograph's timestamp is local to where it was taken.** Store UTC alone and a shoot that ran 09:00–17:00 in Tokyo displays as spanning two days in Paris. Store local time alone and ordering across a timezone change is wrong. So both: `captured_at` in UTC for ordering, `captured_offset` in minutes for display and for day-bucketing. EXIF `OffsetTimeOriginal` supplies it where present; where absent — common on older bodies — the offset is NULL and the catalog falls back to the library's configured display timezone, flagged so the UI can show it as inferred. Day, month, and year buckets are computed against **local** time. "Everything from 3 August" means the photographer's 3 August. The timeline affordance is a histogram of counts per bucket, which the grid uses for scrubbing: ```rust pub enum Granularity { Year, Month, Day, Hour } pub struct TimeBucket { pub start: i64, // UTC seconds, bucket start pub count: u32, } fn timeline(&self, q: &Query, g: Granularity) -> Result, CatalogError>; ``` This is one grouped aggregate over the `images_captured` index, not 50k rows into the UI. It is what makes "drag across two years to find the trip" work, and it is the cheapest useful thing a library view can offer over a flat grid. ### 4.3 Filtering interactively FR-CAT-6 requires filter results to update interactively on 50k images. Three things make that hold: 1. **Filters compile to indexed predicates.** A `Selector` becomes a WHERE clause over indexed columns. Keyword and collection membership become `EXISTS` subqueries against their own indices. 2. **Count and first window are one round trip.** The grid needs a row count to size its scrollbar and the first screenful to paint; the catalog returns both together. 3. **A filter change cancels the one in flight.** Typing in a search box issues a query per keystroke; each supersedes the last (NFR-ARCH-3). Without this the UI queues work it will discard. --- ## 5. Selectors: one type, three uses [architecture.md §9.2](architecture.md) defines `Selector` for cache rules. The same type expresses library filters and smart collections. This is deliberate and worth stating as a design decision, because three near-identical predicate languages is a classic way for a catalog to rot. | Use | Meaning | |---|---| | Library filter | What the grid shows now | | Smart collection | A saved, named filter (FR-CAT-7) | | Cache rule | What is kept locally, at which tier (FR-NC-6a) | One consequence is directly useful: any filter the user has narrowed to can be saved as a smart collection, and any collection can be pinned offline, with no conversion step. "Show me 5-star images from the last 90 days" → save as a collection → pin it for the trip. Three features, one mechanism. `Selector` moves to `dr-types` so `dr-catalog` and `dr-sync` share it without either depending on the other. It gains variants the cache-rule sketch did not need: ```rust pub enum Selector { All, // ⊕ the empty filter Collection(CollectionId), Folder { root: RootId, path: String, recursive: bool }, DateRange(DateSelector), Rating { min: u8 }, Label(ColourLabel), Flag(FlagState), Keyword(String), Camera(String), // ⊕ FR-CAT-6 indexed field Lens(String), // ⊕ IsoRange { min: u32, max: u32 }, // ⊕ Availability(Availability), // ⊕ "what can I edit right now" Text(String), // ⊕ filename/keyword substring All_(Vec), Any(Vec), Not(Box), } ``` `Availability` as a selector earns its place: on a tablet the most useful filter is often "what do I actually have here", and it is also the natural thing to *pin* — "keep everything I've flagged that isn't already local". Compilation is a straightforward recursive walk producing SQL with bound parameters. **Nothing user-supplied is ever interpolated into SQL text.** `Text` becomes a bound `LIKE` pattern with `%`, `_`, and the escape character escaped. --- ## 6. Background work ### 6.1 Job kinds ```rust pub enum JobKind { ScanFolder, // §3, recursive from a folder ExtractMetadata, // state 1 → 2 Thumbnail, // §7 ReadSidecar, // external sidecar change detected WriteSidecar, // local edit → disk, debounced (ARCH §6.1) ContentHash, // on demand only FetchPreview, // remote range-extract (FR-NC-3) FetchOriginal, // pinned or explicitly requested } ``` ### 6.2 Coalescing is the point `UNIQUE(kind, subject_id)` on `jobs` means enqueueing is idempotent: an image touched five times during a scan has one thumbnail job, not five. Enqueue is `INSERT … ON CONFLICT DO UPDATE SET priority = max(priority, excluded.priority)`, so a re-request at higher priority promotes the existing row rather than duplicating it. This is what makes "regenerate on update" safe to call liberally. Every code path that notices a change can just enqueue; the table absorbs the redundancy. ### 6.3 Priority Reuses the existing GPU scheduler classes ([architecture.md §5.3](architecture.md)) so one notion of priority governs the whole app: | Class | Jobs | Preempts | |---|---|---| | `Interactive` | Metadata and thumbnails for visible cells; preview for the open image | everything | | `Prefetch` | The scroll margin; next image in culling | Background | | `Background` | Bulk metadata, rule-driven fetches, hashing | — | Visible-cell work is enqueued by the grid as it scrolls, at `Interactive`. The effect is that a freshly scanned library fills in *where the user is looking* first, and grinds through the rest behind them. ### 6.4 Durability and failure Jobs live in the catalog, so they survive process death — which on Android is routine, not exceptional (FR-PLAT-AND-3). On startup, rows in state `running` revert to `pending`: the process that owned them is gone. Failures increment `attempts` and set `not_before` to an exponential backoff. After a bounded retry count the job is marked failed and attached to its image as a typed error (NFR-ARCH-4) — one corrupt file does not stall the queue, and the user can see which files failed and why. **A job runner never touches the UI executor**, and `Interactive` work runs on the decode pool with the I/O pool behind it (ARCH §7.1). --- ## 7. Thumbnails ### 7.1 When Not "on first connect" as a bulk operation. Thumbnails are generated: - **On demand**, for cells entering the viewport plus the prefetch margin — at `Interactive` - **On change**, when §3.3 sees a differing `(size, mtime)` - **On rule**, for images a cache rule pins at `Preview` or above — at `Background` - **Never** for a remote image nobody has looked at and no rule covers For a local library this converges on "everything, eventually", because scrolling reaches everything and the background pass has nothing else to do. For a remote library it converges on "what you actually browsed". **Measured on a real 17,185-RAW library, 2026-08-09:** cataloguing it by whole-file fetch would move roughly **370 GB**; the range-extract path moves a few MB for the images actually viewed. This is the single largest cost difference in the design, and it is why §7.1 is a list of narrow triggers rather than "generate them all on connect". ### 7.2 How, by availability | Availability | Source | Cost | |---|---|---| | `Original`, local | Embedded JPEG via `dr-decode` preview path | ~200 KB read, no demosaic | | `Original`, no embedded preview | Full decode, downscale | Expensive — `Background` only | | Remote | Range-extract embedded JPEG (FR-NC-3) | 1–3 MB vs 25–100 MB — **measured: 262 KB of a 21.5 MB DNG, 119 ms, 1.22% of the file** | | Placeholder / `Offline` | None — render the offline affordance | 0 | The remote path deliberately does **not** ask the Nextcloud client to hydrate the file. ARCH §9.0 established hydration is whole-file, so it costs ~100× what the range extract does. Hydration stays reserved for the original tier, where the user has asked for the actual image. Server previews (`/core/preview`) are tried only where PROPFIND reported `nc:has-preview`. ARCH §6.7 verified stock Nextcloud ships no RAW preview provider, so for RAW this is nearly always absent — it is an opportunistic saving, never the mechanism. ### 7.3 Storage Thumbnails are content-addressed by `(content_hash | source_ref, size_class)` and stored as files under the platform cache directory, with the `cache` table holding the index. Files, not BLOBs: SQLite handles small blobs well but a 50k-image thumbnail cache is gigabytes, and mixing it into the catalog would bloat the file the app must open in under two seconds. Two size classes at v1 — grid (256px) and filmstrip/loupe (1024px) — both long-edge, both JPEG. The cache is LRU-capped per NFR-RES-4, and thumbnails evict before proxies and long after sidecars, which never evict at all (FR-NC-6b). --- ## 8. Syncing the catalog file Decided 2026-08-09. **This qualifies [architecture.md §6.12](architecture.md)** — the catalog remains a rebuildable index, but the file itself now travels to Nextcloud. The qualification is worth stating precisely, because the sidecar-authoritative model is load-bearing and this is the one place it bends. ### 8.1 Why collections forced this Every other thing the catalog holds has authoritative backing outside it. Ratings, labels, keywords, and edit graphs live in sidecars next to the images, so a rebuild recovers them. **Collections do not.** A manual collection is a set of images the user assembled by hand; nothing in the filesystem records it. Losing the catalog loses them, and no rescan brings them back. So collections need to be durable across devices somehow. Syncing the catalog file is the chosen mechanism. ### 8.2 What the file sync does and does not carry Only **collections and their membership** merge. The rest of a catalog describes *local* state — folder mtimes, cache file paths, job rows, `tier_actual` — and importing another device's version of those would be actively wrong. The downloaded remote is read for its collections and discarded. This is what keeps §6.12 substantially intact: nothing here makes the local database authoritative for anything a rebuild could not recover. The catalog is still deletable. What syncs is one table pair that had no other home. ### 8.3 Two hazards the implementation must handle **A WAL database is not one file.** Committed transactions can sit in `catalog.sqlite-wal` with the main file lagging, so copying `catalog.sqlite` alone uploads a torn snapshot — internally consistent as of some older point, silently missing everything since. Upload therefore runs a `TRUNCATE` checkpoint and then SQLite's backup API, which serialises against concurrent writers rather than racing them. It never copies the live file. **Integer primary keys are not identities.** Two devices each allocate `collections.id = 1` for different collections, so a row-level merge keyed on the integer id would collide them. Collections therefore carry a **UUID**, and membership maps across devices by **image content hash**. The integer ids stay local and are never compared across catalogs. ### 8.4 Merge rules | Concern | Rule | Why | |---|---|---| | Which collection wins | Higher `revision` — a counter bumped per local edit. `modified` only breaks an exact tie | A device with a skewed clock cannot silently overwrite real work. The same reason FR-NC-9 avoids mtime for sidecars | | Membership | **Set union**, not last-writer-wins | Two devices adding different images to one collection keep both. The exception — a removal racing an addition — resolves toward the addition, which is recoverable by removing it again. A lost addition is not | | Deletion | Tombstone (`deleted = 1`) carrying a revision | Without it, merging against a device that still holds the collection resurrects it. With a revision, deletion competes on equal footing with a rename | | An image the remote has and we do not | Skip the membership row | It joins on a later merge, once a scan has catalogued the file. Not an error | | A remote from a newer schema | Decline before attaching | Attempting it would fail mid-transaction rather than declining cleanly | Merging is idempotent: running it twice reports no changes the second time. That property is tested, because a merge that oscillates would upload on every sync forever. ### 8.5 What was rejected **Replace-if-newer.** The literal reading of "sync the file and take the newer one". Rejected because it is not a merge: whichever device syncs second loses every collection the first did not have. Binary SQLite files do not merge, so "newer wins" means "older is destroyed". **A `collections.drsc` sidecar at the library root.** The alternative that would have kept §6.12 untouched, merging as text the way edit sidecars do. Viable, and cheaper in machinery, but it means a second serialisation format and a second merge implementation for the same data. Recorded here because if the SQLite path proves troublesome, this is the fallback with a known shape. --- ## 9. What this document does not settle - **FTS.** `Selector::Text` is a `LIKE` scan over filename and keywords. Adequate at 50k; if free text over description and title becomes a real workflow, an FTS5 table is the answer, and it is additive. - **Smart collection materialisation.** Currently evaluated on read. If a smart collection's membership needs to be *stable* — for manual ordering, or for a pinned set that must not shift under the user — it needs materialising with an invalidation rule. Deferred until there is a concrete need. - **Multi-root capture-time collisions.** FR-CAT-11 detects duplicates on import; the same image catalogued under two roots is a related but distinct case, not yet specified. - **Timeline granularity selection.** Which bucket size the UI picks for a given zoom is a UI concern, but the catalog should probably suggest one from the query's date span rather than have the UI guess. --- ## 9. Requirements touched | ID | How this document addresses it | |---|---| | FR-CAT-1 | §3 incremental scan, cancellable and resumable via §6 jobs | | FR-CAT-3 | §7 thumbnail pyramid, two size classes, embedded-preview fast path | | FR-CAT-4 | §4.1 windowed queries, memory independent of catalog size | | FR-CAT-5 | §3.5 two-pass metadata | | FR-CAT-6 | §4.3 indexed filter compilation, §5 selectors | | FR-CAT-7 | §2 collections schema, §5 manual and smart | | FR-CAT-9 | §3.4 the offline/deleted distinction and the sweep guard | | FR-CAT-11 | §3.5 lazy content hashing | | FR-NC-3 | §7.2 range-extract for remote thumbnails | | FR-NC-6a | §5 shared selector type | | FR-NC-6c | §3.5 metadata honesty, §7.2 availability-driven sourcing | | NFR-P1 | §3.1 one stat per directory, not per file | | NFR-P3 | §7.1 on-demand generation | | NFR-ARCH-2 | §6.3 priority classes shared with the GPU scheduler | | NFR-ARCH-3 | §4.3 query cancellation, §6 job cancellation | | NFR-RES-4 | §7.3 LRU cap, eviction order |