DarkRoom/docs/catalog.md

DarkRoom — Catalog, library view, and background work

Status: Draft v0.1 · 2026-08-09 Companion to: requirements.md, architecture.md

Specifies dr-catalog: the index the library view queries, how it stays current without rescanning everything, and how thumbnails get made. architecture.md §6.2 sketches the schema in eight lines; this expands it to the point of implementability and fills the two gaps that sketch leaves open — incremental local scan and the job queue.

Sync's remote side is already designed (architecture.md §8): ETag pruning turns a no-op sync of 50k images into one request. Nothing equivalent existed for a local root, which is the central problem this document solves.

---

1. What this must not do

Stated first because every design choice below follows from it.

Must not	Why
Stat 50k files to open the catalog	NFR-P1: catalog open < 2 s desktop, < 4 s Android. SAF DocumentsContract queries are far slower than stat (spike S10).
Re-derive thumbnails for unchanged images	NFR-P3 throughput is for new work; redoing it on every connect makes first paint unbounded.
Fetch previews for remote images nobody looks at	A 50k remote library at 1–3 MB per range-extract is 50–150 GB. FR-NC-6 forbids bulk transfer by default.
Evaluate cache rules per grid cell	ARCH §9.5 already answers this: tier_desired is materialised.
Block the UI executor on any of it	NFR-P9, NFR-ARCH-1.

The unifying principle: work is proportional to what changed, or to what the user is looking at — never to library size.

---

2. Schema

Extends architecture.md §6.2. Additions beyond that sketch are marked ⊕.

-- Roots -----------------------------------------------------------------
roots(
  id INTEGER PRIMARY KEY,
  kind TEXT,                    -- 'local' | 'saf' | 'remote'
  grant_blob BLOB,              -- SAF persisted permission; NULL on Linux
  label TEXT,
  last_seen INTEGER,
  scan_generation INTEGER       -- ⊕ bumped per completed scan; see §3.4
);

-- Folders: the unit of change detection, local and remote alike ---------
folders(
  id INTEGER PRIMARY KEY,
  root_id INTEGER NOT NULL REFERENCES roots(id),
  parent_id INTEGER REFERENCES folders(id),
  path TEXT NOT NULL,
  etag TEXT,                    -- remote: propagating ETag (ARCH §8.4)
  mtime INTEGER,                -- ⊕ local: directory mtime
  entry_count INTEGER,          -- ⊕ local: direct children, mtime's blind spot
  scanned_generation INTEGER,   -- ⊕ deletion sweep; see §3.4
  UNIQUE(root_id, path)
);

-- Images ----------------------------------------------------------------
images(
  id INTEGER PRIMARY KEY,
  root_id INTEGER NOT NULL REFERENCES roots(id),
  folder_id INTEGER REFERENCES folders(id),     -- ⊕ folder filter without LIKE
  source_ref TEXT NOT NULL,
  content_hash TEXT,            -- NULL until hashed; see §3.5
  format TEXT,
  w INTEGER, h INTEGER,
  captured_at INTEGER,          -- UTC seconds; NULL if EXIF absent
  captured_offset INTEGER,      -- ⊕ minutes east of UTC; see §4.2
  camera TEXT, lens TEXT,
  iso INTEGER, aperture REAL, shutter REAL,
  availability INTEGER,
  file_size INTEGER,            -- ⊕ cheap change signal alongside mtime
  file_mtime INTEGER,           -- ⊕
  metadata_state INTEGER,       -- ⊕ 0=none 1=stat-only 2=full EXIF; §3.5
  sidecar_mtime INTEGER,
  UNIQUE(root_id, source_ref)
);

-- Versions, keywords, remote, cache: per ARCH §6.2, unchanged -----------

-- Collections ⊕ ---------------------------------------------------------
collections(
  id INTEGER PRIMARY KEY,
  name TEXT NOT NULL,
  parent_id INTEGER REFERENCES collections(id),  -- collection sets
  kind INTEGER NOT NULL,        -- 0 = manual, 1 = smart
  selector_json TEXT,           -- smart only; the §5 Selector
  created INTEGER
);

collection_members(
  collection_id INTEGER NOT NULL REFERENCES collections(id) ON DELETE CASCADE,
  image_id INTEGER NOT NULL REFERENCES images(id) ON DELETE CASCADE,
  position INTEGER,             -- manual ordering; NULL = by capture time
  PRIMARY KEY(collection_id, image_id)
);

-- Jobs ⊕ ----------------------------------------------------------------
jobs(
  id INTEGER PRIMARY KEY,
  kind INTEGER NOT NULL,
  subject_id INTEGER,           -- image or folder, per kind
  priority INTEGER NOT NULL,
  state INTEGER NOT NULL,       -- 0=pending 1=running 2=failed
  attempts INTEGER NOT NULL DEFAULT 0,
  not_before INTEGER,           -- retry backoff
  payload TEXT,
  UNIQUE(kind, subject_id)      -- coalescing; see §6.2
);

Indices that exist for a stated query, not speculatively:

CREATE INDEX images_captured   ON images(captured_at);        -- §4 timeline
CREATE INDEX images_folder     ON images(folder_id);
CREATE INDEX images_hash       ON images(content_hash) WHERE content_hash IS NOT NULL;
CREATE INDEX folders_parent    ON folders(parent_id);
CREATE INDEX jobs_ready        ON jobs(state, priority DESC, not_before);
CREATE INDEX versions_image    ON versions(image_id);
CREATE INDEX members_image     ON collection_members(image_id);

content_hash is indexed partially. It is NULL for most rows most of the time (§3.5), and a partial index over the non-NULL subset is both smaller and what FR-CAT-9's reconnection-by-hash and FR-CAT-11's duplicate detection actually query.

---

3. Incremental scan

3.1 The local analogue of ETag pruning

Nextcloud propagates ETags up the tree, so one request proves a whole library unchanged (architecture.md §8.4). A filesystem offers no such guarantee — a directory's mtime changes when its direct entries change, and not when a grandchild does. There is no cheap "did anything below here change" probe.

So local scan prunes at each level rather than at the root:

scan(folder):
    (mtime, count) = stat(folder)
    if (mtime, count) == stored:
        # This directory's own entries are unchanged. Its files need no
        # examination at all — but subdirectories may still have changed
        # internally, so recurse into known children without listing.
        for child in stored_children(folder):
            scan(child)
    else:
        entries = list(folder)                  # the expensive call
        reconcile(folder, entries)              # §3.3
        for child in entries.dirs: scan(child)
    mark scanned(folder, current_generation)

Cost is one stat per directory when nothing changed, versus one per file. A 50k-image library in ~2k folders costs 2k stats — a few milliseconds locally, and the difference between meeting and missing NFR-P1 on SAF.

The recursion into unchanged directories is not redundant: it is what makes a change to one deep file detectable at all, given no upward propagation. What it avoids is the listing — on SAF a DocumentsContract query returning 200 rows costs far more than a metadata probe on the directory itself.

3.2 Why entry-count as well as mtime

Directory mtime alone misses a real case: delete one file and create another within the same timestamp granularity, and mtime can be unchanged while contents differ. Some filesystems and most SAF providers report coarse timestamps, which widens the window.

Storing (mtime, entry_count) closes the common form of this — a paired add and remove changes neither, but that is rarer than a bare add or remove, and both of those move the count. It is a cheap narrowing, not a proof.

Where correctness must not depend on it, the user gets an explicit Rescan folder action (FR-CAT-1), and reconnection matches by content hash (FR-CAT-9). Sync's remote path is unaffected — ETags are authoritative there.

3.3 Reconciling a changed directory

For each entry in a listing:

Situation	Action
Not in catalog	Insert with metadata_state = 1; enqueue ExtractMetadata
In catalog, (size, mtime) match	Nothing — the common case
In catalog, (size, mtime) differ	Re-enqueue ExtractMetadata and Thumbnail; clear content_hash
In catalog, absent from listing	Deletion candidate — §3.4
Placeholder (*.nextcloud)	Catalogue as the image it stands for; Availability::Offline (ARCH §9.0)

Sidecars are examined in the same pass: a .drsc whose mtime exceeds images.sidecar_mtime enqueues a ReadSidecar job. This is how an edit made on another device — landed by the Nextcloud client, not by us — reaches the catalog.

3.4 Deletion without a full sweep

A file removed outside the app appears only as an absence, which a pruned scan cannot see: the folder it vanished from has a changed mtime and is listed, but a folder never visited is never compared.

Generation counting handles this without a full pass. Each scan bumps roots.scan_generation, and every folder reached — whether listed or skipped — records it. After the walk:

-- Folders never reached: their parent no longer lists them.
DELETE FROM folders
 WHERE root_id = ?1 AND scanned_generation < ?2;

Images under a deleted folder cascade. Images missing from a listed folder are caught directly in §3.3. Together these cover deletion with no additional traversal.

Deletion here means removing the catalog row for a source proven absent, which FR-CAT-9 sharply distinguishes from a source merely unreachable. A root that fails to open at all — unplugged drive, revoked SAF grant — aborts the scan and marks the root offline. It never runs the sweep, because every folder would look unreached and the sweep would delete the entire library.

That guard is the single most dangerous line in this design, and it is stated as an invariant: the deletion sweep runs only after a scan that completed without a root-level access error.

3.5 Metadata in two passes

Full EXIF extraction requires opening and parsing each file. At 50k images that is minutes, and it must not stand between the user and a usable grid.

metadata_state records how far each image has got:

State	Holds	Cost
0 — none	Row exists, nothing read	—
1 — stat-only	Name, size, mtime, format from extension	Free, from the listing
2 — full	EXIF: capture time, camera, lens, exposure, dimensions	One open + parse

The grid is usable at state 1: it can show filenames, sort by filename or file mtime, and display placeholder cells. Promotion to state 2 runs as background jobs, prioritised by what is on screen (§6.3), so visible images get real capture times within a frame or two of being scrolled to.

Capture-time filtering (§4) needs state 2, so a freshly scanned library's timeline is incomplete until the pass finishes. The UI states this plainly — a progress affordance on the timeline, not a silently wrong filter. Which is the FR-NC-6c principle applied to metadata rather than pixels: say what you actually have.

content_hash is a third, still lazier tier. It requires reading the whole file, so it is computed only when something needs it: import duplicate detection (FR-CAT-11), or reconnecting a moved source (FR-CAT-9). Never during a routine scan.

---

4. The library view

4.1 Query model

The UI never assembles SQL. It hands the catalog a Query and receives a stable, windowable result:

pub struct Query {
    pub filter: Selector,        // §5 — same type cache rules use
    pub sort: Sort,
    pub descending: bool,
}

pub enum Sort {
    CapturedAt,
    Added,
    FileName,
    Rating,
    /// Manual order within a collection; falls back to CapturedAt elsewhere.
    CollectionPosition,
}

Results are fetched by window, never wholesale — FR-CAT-4 requires memory bounded independently of catalog size:

impl Catalog {
    fn count(&self, q: &Query) -> Result<usize, CatalogError>;
    fn window(&self, q: &Query, range: Range<usize>) -> Result<Vec<GridRow>, CatalogError>;
}

GridRow carries exactly what a cell draws — id, thumbnail key, availability, rating, flag, capture time — and nothing that would require a join per cell. Availability badges read tier_desired directly (ARCH §9.5), so no rule evaluation happens on the render path.

A LIMIT/OFFSET window degrades at high offsets, since SQLite must walk the skipped rows. Scrolling is overwhelmingly sequential, so the catalog keeps a keyset cursor for forward and backward paging and falls back to OFFSET only for a scrollbar jump. Jumps are rare and single; scrolling is continuous.

4.2 Time

Capture time is the spine of a photo library, and it has one persistent trap: a photograph's timestamp is local to where it was taken. Store UTC alone and a shoot that ran 09:00–17:00 in Tokyo displays as spanning two days in Paris. Store local time alone and ordering across a timezone change is wrong.

So both: captured_at in UTC for ordering, captured_offset in minutes for display and for day-bucketing. EXIF OffsetTimeOriginal supplies it where present; where absent — common on older bodies — the offset is NULL and the catalog falls back to the library's configured display timezone, flagged so the UI can show it as inferred.

Day, month, and year buckets are computed against local time. "Everything from 3 August" means the photographer's 3 August.

The timeline affordance is a histogram of counts per bucket, which the grid uses for scrubbing:

pub enum Granularity { Year, Month, Day, Hour }

pub struct TimeBucket {
    pub start: i64,       // UTC seconds, bucket start
    pub count: u32,
}

fn timeline(&self, q: &Query, g: Granularity) -> Result<Vec<TimeBucket>, CatalogError>;

This is one grouped aggregate over the images_captured index, not 50k rows into the UI. It is what makes "drag across two years to find the trip" work, and it is the cheapest useful thing a library view can offer over a flat grid.

4.3 Filtering interactively

FR-CAT-6 requires filter results to update interactively on 50k images. Three things make that hold:

1. Filters compile to indexed predicates. A Selector becomes a WHERE clause over indexed columns. Keyword and collection membership become EXISTS subqueries against their own indices.
2. Count and first window are one round trip. The grid needs a row count to size its scrollbar and the first screenful to paint; the catalog returns both together.
3. A filter change cancels the one in flight. Typing in a search box issues a query per keystroke; each supersedes the last (NFR-ARCH-3). Without this the UI queues work it will discard.

---

5. Selectors: one type, three uses

architecture.md §9.2 defines Selector for cache rules. The same type expresses library filters and smart collections. This is deliberate and worth stating as a design decision, because three near-identical predicate languages is a classic way for a catalog to rot.

Use	Meaning
Library filter	What the grid shows now
Smart collection	A saved, named filter (FR-CAT-7)
Cache rule	What is kept locally, at which tier (FR-NC-6a)

One consequence is directly useful: any filter the user has narrowed to can be saved as a smart collection, and any collection can be pinned offline, with no conversion step. "Show me 5-star images from the last 90 days" → save as a collection → pin it for the trip. Three features, one mechanism.

Selector moves to dr-types so dr-catalog and dr-sync share it without either depending on the other. It gains variants the cache-rule sketch did not need:

pub enum Selector {
    All,                                  // ⊕ the empty filter
    Collection(CollectionId),
    Folder { root: RootId, path: String, recursive: bool },
    DateRange(DateSelector),
    Rating { min: u8 },
    Label(ColourLabel),
    Flag(FlagState),
    Keyword(String),
    Camera(String),                       // ⊕ FR-CAT-6 indexed field
    Lens(String),                         // ⊕
    IsoRange { min: u32, max: u32 },      // ⊕
    Availability(Availability),           // ⊕ "what can I edit right now"
    Text(String),                         // ⊕ filename/keyword substring
    All_(Vec<Selector>),
    Any(Vec<Selector>),
    Not(Box<Selector>),
}

Availability as a selector earns its place: on a tablet the most useful filter is often "what do I actually have here", and it is also the natural thing to pin — "keep everything I've flagged that isn't already local".

Compilation is a straightforward recursive walk producing SQL with bound parameters. Nothing user-supplied is ever interpolated into SQL text. Text becomes a bound LIKE pattern with %, _, and the escape character escaped.

---

6. Background work

6.1 Job kinds

pub enum JobKind {
    ScanFolder,        // §3, recursive from a folder
    ExtractMetadata,   // state 1 → 2
    Thumbnail,         // §7
    ReadSidecar,       // external sidecar change detected
    WriteSidecar,      // local edit → disk, debounced (ARCH §6.1)
    ContentHash,       // on demand only
    FetchPreview,      // remote range-extract (FR-NC-3)
    FetchOriginal,     // pinned or explicitly requested
}

6.2 Coalescing is the point

UNIQUE(kind, subject_id) on jobs means enqueueing is idempotent: an image touched five times during a scan has one thumbnail job, not five. Enqueue is INSERT … ON CONFLICT DO UPDATE SET priority = max(priority, excluded.priority), so a re-request at higher priority promotes the existing row rather than duplicating it.

This is what makes "regenerate on update" safe to call liberally. Every code path that notices a change can just enqueue; the table absorbs the redundancy.

6.3 Priority

Reuses the existing GPU scheduler classes (architecture.md §5.3) so one notion of priority governs the whole app:

Class	Jobs	Preempts
Interactive	Metadata and thumbnails for visible cells; preview for the open image	everything
Prefetch	The scroll margin; next image in culling	Background
Background	Bulk metadata, rule-driven fetches, hashing	—

Visible-cell work is enqueued by the grid as it scrolls, at Interactive. The effect is that a freshly scanned library fills in where the user is looking first, and grinds through the rest behind them.

6.4 Durability and failure

Jobs live in the catalog, so they survive process death — which on Android is routine, not exceptional (FR-PLAT-AND-3). On startup, rows in state running revert to pending: the process that owned them is gone.

Failures increment attempts and set not_before to an exponential backoff. After a bounded retry count the job is marked failed and attached to its image as a typed error (NFR-ARCH-4) — one corrupt file does not stall the queue, and the user can see which files failed and why.

A job runner never touches the UI executor, and Interactive work runs on the decode pool with the I/O pool behind it (ARCH §7.1).

---

7. Thumbnails

7.1 When

Not "on first connect" as a bulk operation. Thumbnails are generated:

• On demand, for cells entering the viewport plus the prefetch margin — at Interactive
• On change, when §3.3 sees a differing (size, mtime)
• On rule, for images a cache rule pins at Preview or above — at Background
• Never for a remote image nobody has looked at and no rule covers

For a local library this converges on "everything, eventually", because scrolling reaches everything and the background pass has nothing else to do. For a remote library it converges on "what you actually browsed".

Measured on a real 17,185-RAW library, 2026-08-09: cataloguing it by whole-file fetch would move roughly 370 GB; the range-extract path moves a few MB for the images actually viewed. This is the single largest cost difference in the design, and it is why §7.1 is a list of narrow triggers rather than "generate them all on connect".

7.2 How, by availability

Availability	Source	Cost
Original, local	Embedded JPEG via dr-decode preview path	~200 KB read, no demosaic
Original, no embedded preview	Full decode, downscale	Expensive — Background only
Remote	Range-extract embedded JPEG (FR-NC-3)	1–3 MB vs 25–100 MB — measured: 262 KB of a 21.5 MB DNG, 119 ms, 1.22% of the file
Placeholder / Offline	None — render the offline affordance	0

The remote path deliberately does not ask the Nextcloud client to hydrate the file. ARCH §9.0 established hydration is whole-file, so it costs ~100× what the range extract does. Hydration stays reserved for the original tier, where the user has asked for the actual image.

Server previews (/core/preview) are tried only where PROPFIND reported nc:has-preview. ARCH §6.7 verified stock Nextcloud ships no RAW preview provider, so for RAW this is nearly always absent — it is an opportunistic saving, never the mechanism.

7.3 Storage

Thumbnails are content-addressed by (content_hash | source_ref, size_class) and stored as files under the platform cache directory, with the cache table holding the index. Files, not BLOBs: SQLite handles small blobs well but a 50k-image thumbnail cache is gigabytes, and mixing it into the catalog would bloat the file the app must open in under two seconds.

Two size classes at v1 — grid (256px) and filmstrip/loupe (1024px) — both long-edge, both JPEG. The cache is LRU-capped per NFR-RES-4, and thumbnails evict before proxies and long after sidecars, which never evict at all (FR-NC-6b).

---

8. Syncing the catalog file

Decided 2026-08-09. This qualifies architecture.md §6.12 — the catalog remains a rebuildable index, but the file itself now travels to Nextcloud. The qualification is worth stating precisely, because the sidecar-authoritative model is load-bearing and this is the one place it bends.

8.1 Why collections forced this

Every other thing the catalog holds has authoritative backing outside it. Ratings, labels, keywords, and edit graphs live in sidecars next to the images, so a rebuild recovers them. Collections do not. A manual collection is a set of images the user assembled by hand; nothing in the filesystem records it. Losing the catalog loses them, and no rescan brings them back.

So collections need to be durable across devices somehow. Syncing the catalog file is the chosen mechanism.

8.2 What the file sync does and does not carry

Only collections and their membership merge. The rest of a catalog describes local state — folder mtimes, cache file paths, job rows, tier_actual — and importing another device's version of those would be actively wrong. The downloaded remote is read for its collections and discarded.

This is what keeps §6.12 substantially intact: nothing here makes the local database authoritative for anything a rebuild could not recover. The catalog is still deletable. What syncs is one table pair that had no other home.

8.3 Two hazards the implementation must handle

A WAL database is not one file. Committed transactions can sit in catalog.sqlite-wal with the main file lagging, so copying catalog.sqlite alone uploads a torn snapshot — internally consistent as of some older point, silently missing everything since. Upload therefore runs a TRUNCATE checkpoint and then SQLite's backup API, which serialises against concurrent writers rather than racing them. It never copies the live file.

Integer primary keys are not identities. Two devices each allocate collections.id = 1 for different collections, so a row-level merge keyed on the integer id would collide them. Collections therefore carry a UUID, and membership maps across devices by image content hash. The integer ids stay local and are never compared across catalogs.

8.4 Merge rules

Concern	Rule	Why
Which collection wins	Higher revision — a counter bumped per local edit. modified only breaks an exact tie	A device with a skewed clock cannot silently overwrite real work. The same reason FR-NC-9 avoids mtime for sidecars
Membership	Set union, not last-writer-wins	Two devices adding different images to one collection keep both. The exception — a removal racing an addition — resolves toward the addition, which is recoverable by removing it again. A lost addition is not
Deletion	Tombstone (deleted = 1) carrying a revision	Without it, merging against a device that still holds the collection resurrects it. With a revision, deletion competes on equal footing with a rename
An image the remote has and we do not	Skip the membership row	It joins on a later merge, once a scan has catalogued the file. Not an error
A remote from a newer schema	Decline before attaching	Attempting it would fail mid-transaction rather than declining cleanly

Merging is idempotent: running it twice reports no changes the second time. That property is tested, because a merge that oscillates would upload on every sync forever.

8.5 What was rejected

Replace-if-newer. The literal reading of "sync the file and take the newer one". Rejected because it is not a merge: whichever device syncs second loses every collection the first did not have. Binary SQLite files do not merge, so "newer wins" means "older is destroyed".

A collections.drsc sidecar at the library root. The alternative that would have kept §6.12 untouched, merging as text the way edit sidecars do. Viable, and cheaper in machinery, but it means a second serialisation format and a second merge implementation for the same data. Recorded here because if the SQLite path proves troublesome, this is the fallback with a known shape.

---

9. What this document does not settle

• FTS. Selector::Text is a LIKE scan over filename and keywords. Adequate at 50k; if free text over description and title becomes a real workflow, an FTS5 table is the answer, and it is additive.
• Smart collection materialisation. Currently evaluated on read. If a smart collection's membership needs to be stable — for manual ordering, or for a pinned set that must not shift under the user — it needs materialising with an invalidation rule. Deferred until there is a concrete need.
• Multi-root capture-time collisions. FR-CAT-11 detects duplicates on import; the same image catalogued under two roots is a related but distinct case, not yet specified.
• Timeline granularity selection. Which bucket size the UI picks for a given zoom is a UI concern, but the catalog should probably suggest one from the query's date span rather than have the UI guess.

---

9. Requirements touched

ID	How this document addresses it
FR-CAT-1	§3 incremental scan, cancellable and resumable via §6 jobs
FR-CAT-3	§7 thumbnail pyramid, two size classes, embedded-preview fast path
FR-CAT-4	§4.1 windowed queries, memory independent of catalog size
FR-CAT-5	§3.5 two-pass metadata
FR-CAT-6	§4.3 indexed filter compilation, §5 selectors
FR-CAT-7	§2 collections schema, §5 manual and smart
FR-CAT-9	§3.4 the offline/deleted distinction and the sweep guard
FR-CAT-11	§3.5 lazy content hashing
FR-NC-3	§7.2 range-extract for remote thumbnails
FR-NC-6a	§5 shared selector type
FR-NC-6c	§3.5 metadata honesty, §7.2 availability-driven sourcing
NFR-P1	§3.1 one stat per directory, not per file
NFR-P3	§7.1 on-demand generation
NFR-ARCH-2	§6.3 priority classes shared with the GPU scheduler
NFR-ARCH-3	§4.3 query cancellation, §6 job cancellation
NFR-RES-4	§7.3 LRU cap, eviction order