dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

23,582 Thumbnail jobs are queued and nothing consumes them #73

New Issue
Closed
opened 2026-09-26 14:43:22 +00:00 by dtourolle · 1 comment
dtourolle commented 2026-09-26 14:43:22 +00:00
Owner
Copy Link
Copy Source

The reference catalog has 23,582 Thumbnail jobs queued and nothing consumes that kind. The job queue in dr-catalog has no consumer for it at all, and the scan's persist adds another job per photograph on every scan.

Why

Found in the 2026-09-25 performance pass. v0.16.0 made the enqueue cheap (it moved inside the scan's transaction) but kept the behaviour. So the table grows with the library, and every query over jobs pays for rows nobody will read.

Deliverable

Decide whether a thumbnail job queue is the design (then wire a consumer) or a leftover (then stop enqueueing, and drop the queued rows in a migration). Record the decision in docs/dev/catalog.md.

Acceptance

  • No job kind is enqueued without a consumer (a test that lists kinds enqueued against kinds claimed)
  • The reference catalog's jobs table after one scan holds only jobs something will run
**The reference catalog has 23,582 `Thumbnail` jobs queued and nothing consumes that kind.** The job queue in `dr-catalog` has no consumer for it at all, and the scan's `persist` adds another job per photograph on every scan. ## Why Found in the 2026-09-25 performance pass. v0.16.0 made the enqueue cheap (it moved inside the scan's transaction) but kept the behaviour. So the table grows with the library, and every query over `jobs` pays for rows nobody will read. ## Deliverable Decide whether a thumbnail job queue is the design (then wire a consumer) or a leftover (then stop enqueueing, and drop the queued rows in a migration). Record the decision in docs/dev/catalog.md. ## Acceptance - [ ] No job kind is enqueued without a consumer (a test that lists kinds enqueued against kinds claimed) - [ ] The reference catalog's `jobs` table after one scan holds only jobs something will run
dtourolle added the catalogsize:Sperformance labels 2026-09-26 14:43:22 +00:00
dtourolle commented 2026-09-26 20:20:17 +00:00
Author
Owner
Copy Link
Copy Source

Done in 5da2858, 5fcd375, 6e67ef4, b3dbf4a and 441f6f1, released in v0.17.0. The Thumbnail job kind was a leftover: the runner deliberately registered no handler for it (846a249), because the grid worker and the sweep serve thumbnails from what the store lacks. Scans no longer enqueue it. Pending rows are dropped at runtime whenever a catalog is opened (23,582 rows, 1 MB, on the reference catalog), which tolerates older peers re-enqueueing them, with no schema bump. core/dr-catalog/tests/every_queued_kind_has_a_consumer.rs refuses any kind enqueued without a consumer. The decision is recorded in catalog.md §6.1.

Done in 5da2858, 5fcd375, 6e67ef4, b3dbf4a and 441f6f1, released in v0.17.0. The Thumbnail job kind was a leftover: the runner deliberately registered no handler for it (846a249), because the grid worker and the sweep serve thumbnails from what the store lacks. Scans no longer enqueue it. Pending rows are dropped at runtime whenever a catalog is opened (23,582 rows, 1 MB, on the reference catalog), which tolerates older peers re-enqueueing them, with no schema bump. `core/dr-catalog/tests/every_queued_kind_has_a_consumer.rs` refuses any kind enqueued without a consumer. The decision is recorded in catalog.md §6.1.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
a11y
Accessibility
 android
apps/darkroom-android and the platform contract
 blocked
Cannot start until something else lands
 catalog
core/dr-catalog and the data model
 culling
Selection, comparison and the culling surfaces
 decision
Needs a decision recorded, not code
 develop
The develop view and its pipeline
 docs
An edit to the register of record
 ergonomics
How the work feels to do, not what it computes
 gpu
core/dr-gpu and the render path
 i18n
Localisation
 linux
Desktop packaging, portals and distribution
 masks
Local adjustments and selection
 performance
Budgets, benchmarks and frame timing
 pipeline
core/dr-pipeline
 plugins
The plugin host and the declarative format
 post-v1
Specified as the design of record; outside the v1 count
 size:L
A subsystem, not a feature
 size:M
Days
 size:S
Hours to a day
 spike
A validation spike from requirements §9
 sync
dr-sync and the Nextcloud backends
 ui
ui/dr-ui
 unmet-requirement
An existing requirement that is specified but not built
No labels catalog performance size:S
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
dtourolle (Duncan Tourolle)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dtourolle/DarkRoom#73