FR-CAT-11 — Consolidate duplicate originals into one RAW, keeping every edit #67

New Issue

2026-09-26T00:44:57Z

dtourolle commented

2026-09-26 00:44:57 +00:00

Consolidate copies of the same RAW into one original, keeping every edit, rating, keyword and collection membership, and moving the extra files to DarkRoom's trash.

Why

The library holds the same frame several times over: the camera's file, a copy in a backup folder, and copies renamed by other tools. On the reference catalog (23,582 images) there are 2,776 moments with more than one visible copy, e.g. one Canon EOS 6D frame at 1687602063 exists four times:

PhotosRaw/2023/2023-06-24/_MG_4623.CR2
PhotosRaw/2023/bck/_MG_4623.CR2
PhotosRaw/Darktable/20230628_no_name/20230628_0642.CR2
PhotosRaw/alps trip/Raw/20230628_0642.CR2

"All photographs" shows each of them, and the "tomb" collection holds all four. Import-time dedup (FR-CAT-11, dedup.rs) only stops new copies arriving; nothing reconciles the ones already there. Burst grouping never folds them either (no perceptual hashes on this catalog), and folding would hide the problem rather than remove it. The photographer wants one RAW per photograph, with different processing expressed as virtual copies (#11) rather than as duplicate files.

Deliverable

Find candidate groups from the catalog alone: same camera, same captured_at, same file_size, none trashed, JPEG shadows excluded. This is cheap (one grouped query, no file reads).
Prove sameness before touching a file: a content hash per member (dedup::set_content_hash). For remote storage, hash in the background under the job runner and show progress. A group whose hashes disagree is not a duplicate and is dropped from the plan, and the review says so.
Pick a survivor by a stated rule the photographer can override per group, e.g. prefer a path not under a bck/backup-looking folder, then the camera's own filename over a renamed one, then the oldest added_at.
Merge catalog state onto the survivor, in one transaction per group:
- collection memberships (union, keeping the survivor's position)
- keywords (union)
- rating, flag and label (the highest rating; a flag or label only where the members agree, otherwise the survivor's, and the review lists the conflict)
- faces and identities
- develop edits: identical or empty edits collapse. Edits that differ become virtual copies (#11) on the survivor, so no processing is lost. Until #11 exists, a group whose members carry different edits is shown and skipped, not merged.
Move the other files to DarkRoom's trash (trash.rs), so the step is recoverable on local and Nextcloud storage alike. Never delete outright.
UI: a "Duplicate originals: N" repair alongside the existing repairs, opening a review sheet: each group with thumbnails, paths, the chosen survivor, what will merge, and conflicts; per-group include/exclude; then "Consolidate N groups". A dry-run summary comes before anything moves.

Acceptance

candidate groups come from one grouped query (COUNT or GROUP BY, not a per-row loop; see CLAUDE.md "Catalog reads")
nothing is moved unless every member's content hash matches
after consolidation the survivor carries the union of collections and keywords, and no edit is lost (a differing edit becomes a virtual copy, or the group is skipped while #11 is unbuilt)
extra files land in DarkRoom's trash and can be restored from it
one transaction per group; an interrupted run leaves every group either fully merged or untouched
reachable from the UI on desktop and tablet, with the review sheet, and documented in the manual

**Consolidate copies of the same RAW into one original, keeping every edit, rating, keyword and collection membership, and moving the extra files to DarkRoom's trash.** ## Why The library holds the same frame several times over: the camera's file, a copy in a backup folder, and copies renamed by other tools. On the reference catalog (23,582 images) there are **2,776 moments** with more than one visible copy, e.g. one Canon EOS 6D frame at 1687602063 exists four times: - `PhotosRaw/2023/2023-06-24/_MG_4623.CR2` - `PhotosRaw/2023/bck/_MG_4623.CR2` - `PhotosRaw/Darktable/20230628_no_name/20230628_0642.CR2` - `PhotosRaw/alps trip/Raw/20230628_0642.CR2` "All photographs" shows each of them, and the "tomb" collection holds all four. Import-time dedup (FR-CAT-11, `dedup.rs`) only stops *new* copies arriving; nothing reconciles the ones already there. Burst grouping never folds them either (no perceptual hashes on this catalog), and folding would hide the problem rather than remove it. The photographer wants one RAW per photograph, with different processing expressed as virtual copies (#11) rather than as duplicate files. ## Deliverable 1. **Find candidate groups** from the catalog alone: same camera, same `captured_at`, same `file_size`, none trashed, JPEG shadows excluded. This is cheap (one grouped query, no file reads). 2. **Prove sameness before touching a file:** a content hash per member (`dedup::set_content_hash`). For remote storage, hash in the background under the job runner and show progress. A group whose hashes disagree is not a duplicate and is dropped from the plan, and the review says so. 3. **Pick a survivor** by a stated rule the photographer can override per group, e.g. prefer a path not under a `bck`/backup-looking folder, then the camera's own filename over a renamed one, then the oldest `added_at`. 4. **Merge catalog state onto the survivor**, in one transaction per group: - collection memberships (union, keeping the survivor's position) - keywords (union) - rating, flag and label (the highest rating; a flag or label only where the members agree, otherwise the survivor's, and the review lists the conflict) - faces and identities - develop edits: identical or empty edits collapse. **Edits that differ become virtual copies (#11) on the survivor**, so no processing is lost. Until #11 exists, a group whose members carry different edits is shown and skipped, not merged. 5. **Move the other files to DarkRoom's trash** (`trash.rs`), so the step is recoverable on local and Nextcloud storage alike. Never delete outright. 6. **UI:** a "Duplicate originals: N" repair alongside the existing repairs, opening a review sheet: each group with thumbnails, paths, the chosen survivor, what will merge, and conflicts; per-group include/exclude; then "Consolidate N groups". A dry-run summary comes before anything moves. ## Acceptance - [ ] candidate groups come from one grouped query (COUNT or GROUP BY, not a per-row loop; see CLAUDE.md "Catalog reads") - [ ] nothing is moved unless every member's content hash matches - [ ] after consolidation the survivor carries the union of collections and keywords, and no edit is lost (a differing edit becomes a virtual copy, or the group is skipped while #11 is unbuilt) - [ ] extra files land in DarkRoom's trash and can be restored from it - [ ] one transaction per group; an interrupted run leaves every group either fully merged or untouched - [ ] reachable from the UI on desktop and tablet, with the review sheet, and documented in the manual

dtourolle added the catalog size:L labels 2026-09-26 00:44:57 +00:00

dtourolle added a new dependency 2026-09-26 00:45:03 +00:00

#11 FR-DEV-15 — More than one edit variant per photograph

dtourolle referenced this issue

2026-09-26 00:45:04 +00:00

FR-DEV-15 — More than one edit variant per photograph #11

dtourolle referenced this issue from a commit

2026-09-26 13:47:07 +00:00

Find catalog duplicates and fold a group onto one copy in one transaction

dtourolle referenced this issue from a commit

2026-09-26 13:47:07 +00:00

Prove duplicate originals the same and consolidate them on workers

dtourolle referenced this issue from a commit

2026-09-26 13:47:07 +00:00

Describe consolidating duplicate originals in the manual and the register

dtourolle commented

2026-09-26 13:47:50 +00:00

Shipped in v0.16.0: 3c2eacb, 8d4ecb7, 220e9af, 54aee50, 7c9a4ee, 6640ce0, plus the manual picture in b1d36b9.

Finding groups: one grouped query on the same root, camera, capture time and file size (not trashed, not shadowed). On the reference catalog it finds 1,836 groups holding 5,215 files, so 3,379 copies would be trashed. The query takes 10–35 ms.
Proving sameness: a stored content_hash where every member has one; otherwise SHA-256 of the first and last 1 MiB of each file, read by byte range through the storage backend. Results are kept in dedup_probes, keyed on size and mtime. Edits are compared through each copy's .drsc. A group is skipped, with the reason shown, if its files differ, a copy can't be read, or its edits differ.
Survivor: not under a backup-named folder, then the camera's own filename, then the oldest, with an override per group.
Merge: one transaction per group: the highest rating, union of keywords and collections, flag and label where the copies agree (conflicts shown), faces and names carried over, and a single edit with its snapshots written into the survivor's sidecar. The extras go to DarkRoom's trash and can be restored; an interrupted run is finished by the next one.
UI: a "Duplicate originals N" row in the sidebar and a button in Settings open a paginated review. You check the groups first, then "Move N copies to trash".

Against the acceptance list: a grouped query ✅; nothing moves without matching hashes ✅ (first and last MiB rather than a full hash, which is the stated trade-off for remote storage); union and no edit lost ✅ (groups with differing edits are skipped); extras in trash and restorable ✅; one transaction per group, with the recovery path ✅; UI on desktop and tablet ✅; manual ✅.

Still open:

Groups whose copies carry different edits wait for virtual copies (#11).
Copies of the same moment with different file sizes (~940 on the reference catalog) are never offered.
Nothing has run against the real library yet. The check reads about 10 GB over ~10k ranged requests.

Shipped in **v0.16.0**: 3c2eacb, 8d4ecb7, 220e9af, 54aee50, 7c9a4ee, 6640ce0, plus the manual picture in b1d36b9. - **Finding groups:** one grouped query on the same root, camera, capture time and file size (not trashed, not shadowed). On the reference catalog it finds 1,836 groups holding 5,215 files, so 3,379 copies would be trashed. The query takes 10–35 ms. - **Proving sameness:** a stored `content_hash` where every member has one; otherwise SHA-256 of the first and last 1 MiB of each file, read by byte range through the storage backend. Results are kept in `dedup_probes`, keyed on size and mtime. Edits are compared through each copy's `.drsc`. A group is skipped, with the reason shown, if its files differ, a copy can't be read, or its edits differ. - **Survivor:** not under a backup-named folder, then the camera's own filename, then the oldest, with an override per group. - **Merge:** one transaction per group: the highest rating, union of keywords and collections, flag and label where the copies agree (conflicts shown), faces and names carried over, and a single edit with its snapshots written into the survivor's sidecar. The extras go to DarkRoom's trash and can be restored; an interrupted run is finished by the next one. - **UI:** a "Duplicate originals N" row in the sidebar and a button in Settings open a paginated review. You check the groups first, then "Move N copies to trash". Against the acceptance list: a grouped query ✅; nothing moves without matching hashes ✅ (first and last MiB rather than a full hash, which is the stated trade-off for remote storage); union and no edit lost ✅ (groups with differing edits are skipped); extras in trash and restorable ✅; one transaction per group, with the recovery path ✅; UI on desktop and tablet ✅; manual ✅. **Still open:** - Groups whose copies carry *different* edits wait for virtual copies (#11). - Copies of the same moment with different file sizes (~940 on the reference catalog) are never offered. - Nothing has run against the real library yet. The check reads about 10 GB over ~10k ranged requests.

dtourolle referenced this issue

2026-09-26 13:48:33 +00:00

FR-CAT-11a — Consolidate duplicate groups whose copies carry different edits #68

dtourolle removed a dependency 2026-09-26 13:48:34 +00:00

#11 FR-DEV-15 — More than one edit variant per photograph

dtourolle commented

2026-09-26 13:48:34 +00:00

Closing: the first version shipped in v0.16.0 (details above). The remaining case, groups whose copies carry different edits, moves to #68, which waits on virtual copies (#11).

dtourolle closed this issue

2026-09-26 13:48:34 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dtourolle/DarkRoom#67