dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Compare commits

base: dtourolle/DarkRoom:v0.15.0
Branches Tags
dtourolle/DarkRoom:master dtourolle/DarkRoom:release/0.21 dtourolle/DarkRoom:feat/macos-inference
dtourolle/DarkRoom:v0.24.0 dtourolle/DarkRoom:v0.23.0 dtourolle/DarkRoom:v0.22.1 dtourolle/DarkRoom:v0.22.0 dtourolle/DarkRoom:v0.21.0 dtourolle/DarkRoom:v0.20.0 dtourolle/DarkRoom:v0.19.4 dtourolle/DarkRoom:v0.19.3 dtourolle/DarkRoom:v0.19.2 dtourolle/DarkRoom:v0.19.1 dtourolle/DarkRoom:v0.19.0 dtourolle/DarkRoom:v0.18.2 dtourolle/DarkRoom:v0.18.1 dtourolle/DarkRoom:v0.18.0 dtourolle/DarkRoom:v0.17.0 dtourolle/DarkRoom:v0.16.0 dtourolle/DarkRoom:v0.15.0 dtourolle/DarkRoom:v0.14.1 dtourolle/DarkRoom:v0.14.0 dtourolle/DarkRoom:v0.13.6 dtourolle/DarkRoom:v0.13.5 dtourolle/DarkRoom:v0.13.4 dtourolle/DarkRoom:v0.13.3 dtourolle/DarkRoom:v0.13.2 dtourolle/DarkRoom:v0.13.1 dtourolle/DarkRoom:v0.13.0 dtourolle/DarkRoom:v0.12.1 dtourolle/DarkRoom:v0.12.0 dtourolle/DarkRoom:v0.11.0 dtourolle/DarkRoom:v0.10.0 dtourolle/DarkRoom:v0.8.0 dtourolle/DarkRoom:v0.7.0 dtourolle/DarkRoom:v0.6.0 dtourolle/DarkRoom:v0.5.0 dtourolle/DarkRoom:v0.3.0
..
compare: dtourolle/DarkRoom:v0.18.0
Branches Tags
dtourolle/DarkRoom:master dtourolle/DarkRoom:release/0.21 dtourolle/DarkRoom:feat/macos-inference
dtourolle/DarkRoom:v0.24.0 dtourolle/DarkRoom:v0.23.0 dtourolle/DarkRoom:v0.22.1 dtourolle/DarkRoom:v0.22.0 dtourolle/DarkRoom:v0.21.0 dtourolle/DarkRoom:v0.20.0 dtourolle/DarkRoom:v0.19.4 dtourolle/DarkRoom:v0.19.3 dtourolle/DarkRoom:v0.19.2 dtourolle/DarkRoom:v0.19.1 dtourolle/DarkRoom:v0.19.0 dtourolle/DarkRoom:v0.18.2 dtourolle/DarkRoom:v0.18.1 dtourolle/DarkRoom:v0.18.0 dtourolle/DarkRoom:v0.17.0 dtourolle/DarkRoom:v0.16.0 dtourolle/DarkRoom:v0.15.0 dtourolle/DarkRoom:v0.14.1 dtourolle/DarkRoom:v0.14.0 dtourolle/DarkRoom:v0.13.6 dtourolle/DarkRoom:v0.13.5 dtourolle/DarkRoom:v0.13.4 dtourolle/DarkRoom:v0.13.3 dtourolle/DarkRoom:v0.13.2 dtourolle/DarkRoom:v0.13.1 dtourolle/DarkRoom:v0.13.0 dtourolle/DarkRoom:v0.12.1 dtourolle/DarkRoom:v0.12.0 dtourolle/DarkRoom:v0.11.0 dtourolle/DarkRoom:v0.10.0 dtourolle/DarkRoom:v0.8.0 dtourolle/DarkRoom:v0.7.0 dtourolle/DarkRoom:v0.6.0 dtourolle/DarkRoom:v0.5.0 dtourolle/DarkRoom:v0.3.0
107 Commits
v0.15.0 .. v0.18.0
Author SHA1 Message Date
dtourolle 515d4eb59e Release 0.18.0
Benchmarks / CPU and I/O (per commit) (push) Successful in 5m56s
Details
Benchmarks / Frame budget (on demand) (push) Skipped
Details
Traceability / Requirement traces (push) Successful in 45s
Details
Build and test / Android (aarch64) (push) Successful in 17m8s
Details
Build and test / android-image (push) Successful in 2s
Details
🐳 Android image / Build and push (push) Successful in 2s
Details
Build and test / Desktop (Linux) (push) Successful in 48m1s
Details
Build and test / windows-image (push) Successful in 2s
Details
🐳 Windows image / Build and push (push) Successful in 2s
Details
Build and test / Layer separation (push) Successful in 36s
Details
Build and test / Windows (x86_64, cross) (push) Successful in 21m35s
Details
Build and test / Publish the release (push) Successful in 1m12s
Details
2026-09-26 19:13:06 -04:00
dtourolle b2f3936a53 Say how synced faces and same-name people merge since #77 and #78 
catalog.md §8.2 still said confirmed and rejected faces were matched to
local faces "by box", and that the merge reads only a remote face's box
and model; faces.md said `match_faces` "still matches by overlap alone
across devices". Since #77 the match falls back to embeddings, on the
photographs where a box leaves a remote face over, and since #78
`dedup_people` folds people of one name whose faces agree after every
sync. Both now say so, with the thresholds and the reason a less
decisive pair stays unmatched, taken from the code's own documentation.
 2026-09-26 18:30:07 -04:00
dtourolle ec7a8c07ee Add a dedup_people example to measure the job on a catalog copy 
`dedup_people COPY.sqlite` prints the listed people and faces before
and after, what the first run merged and kept apart and why, and the
time of three runs. The second and third runs are the cost the job
adds to every sync. `--peer PEER_COPY.sqlite` then plays two sync
round trips. The peer merges with the previous release's code path and
no job, this side merges back through sync::merge_remote, and the named
people each side lists are compared after every step.

On the reference pair: the desktop merges Claudine, Jessie x2, Mathias
and Noemi (80 -> 75 named). The tablet also merges its empty second
Ian (80 -> 74). The first run takes 1.2 s on the desktop, which is
building faces_box; the merge builds it first in practice. Later runs
take 8-15 ms.
 2026-09-26 17:50:39 -04:00
dtourolle 78df4211b0 Run the people deduplication after every catalog merge (#78) 
A merge is where two devices' people meet: the same name typed on each,
or a redirect one of them made. So dedup_people::run now follows every
successful sync::merge_remote. It runs on the sync worker, never the
UI thread, before the snapshot is pushed, so what it folds reaches the
server on the same pass.

It runs in its own transaction, and a failure is logged, not
returned. What the merge took is committed and valid either way, and
the next pass tries again. Once a catalog is clean it costs 8-15 ms on
the reference library (19k faces, 26k people rows). On copies of the
two real catalogs, a round trip with a peer running the previous merge
converges on 75 listed named people on both sides and stays there
over a second round. merge_remote with the job takes 75-90 ms there.
 2026-09-26 17:50:38 -04:00
dtourolle c78b798cf0 Fold people of one name whose faces agree, and faces held twice (#78) 
Seven names are two or three live people on both devices: Ian (756
confirmed faces, and a second Ian with none), Jessie three times,
Claudine, Mathias, Noemi, Pascal and PJ. Each was typed on its own
device and carried across by sync, which keys people on their uuid and
so keeps both. Each half of a person shows half their photographs.

dedup_people::run, in one transaction:

- Same-name people (trimmed, case-folded as the Identity screen folds
  them) merge into the one with the most confirmed faces, ties to the
  smaller uuid, through faces::merge_people_within, so confirmations,
  rejections and the survivor's name are kept. A person holding no
  faces at all merges: there is nothing to compare or to carry. Anyone
  else needs >= 2 confirmed faces per shared embedder on both sides and
  centroids at cosine >= 0.7 in each. A face confirmed as one and
  rejected as the other keeps them apart. Unnamed and set-aside people
  are never merged by name.
- Faces held twice (one image, one embedder, IoU >= 0.5, cosine >= 0.7)
  keep the stronger detector's row (FaceDetector::outranks), then the
  confirmed one, then the older. The survivor takes the confirmed
  assignment and both rows' rejections. A pair confirmed as two
  different people is left and counted.
- Judgements still on a merged-away person move to the person at the
  end of its redirects, and a redirect cycle (two devices merging one
  pair in opposite directions) is broken at the smaller uuid.

Measured on copies of the desktop catalog and the tablet's server
snapshot, w600k_mbf, confirmed faces only:
- Centroids of differently named people: 2,699 pairs, median 0.02,
  99.9th percentile 0.41. One pair reaches 0.70 (0.700 desktop, 0.705
  tablet), "Michelle Casanonve" and "Michelle Casanova", one person
  typed two ways. Next is 0.62/0.64, "Boris Jost" and "Boris". The
  highest pair that is plainly two people is 0.43/0.44.
- One person split in random halves: minimum 0.69, median 0.91 over 72
  people. Four faces against twenty-two reach 0.7 in 97% of draws.
  One face against twenty of somebody else's reached 0.74 in 3,000
  draws, and two faces reached 0.61, hence the two-face minimum.
- Pascal (22 and 4 confirmed) is at 0.57 and PJ (14 and 7) at 0.50,
  under 0.7 on both devices, so both pairs stay apart and are logged.
  The desktop's second Ian holds 4 suggestions and no confirmations,
  at 0.38 against Ian's centroid, and stays apart. On the tablet it
  holds nothing and merges.

Why a merge made here survives a peer on 0.17.0: the merged-away
person stays as a merged_into redirect with a bumped revision, which
the catalog merge has always taken on revision. The peer hides the
duplicate and never sends it back as a live person. Its own
confirmations of that person stay on the redirect, because a merge
never overwrites a local confirmation. The manual merge has always
left them there too. They follow the redirect when the peer runs this
job. A test syncs two catalog files through the previous merge code
and back, and the people converge and stay converged.

Once a catalog is clean the job reads 80 redirects, the named people,
and the face boxes from the covering faces_box index. That is ~10 ms
on the reference library. There is no schema change. The index is
created IF NOT EXISTS, as the merge already does.
 2026-09-26 17:50:28 -04:00
dtourolle 6450f54199 Carry rejections when two people are merged 
merge_people moved a person's faces onto the target and left the
"not this person" rejections on the redirect. A rejection there binds
nothing: once Annie is Anna, the grouping pass is free to suggest the
face the user pushed away from Annie as Anna, which is the behaviour
rejections exist to prevent. The Identity screen's merge has done this
since it was written, and the deduplication job for #78 merges through
the same function, so it would have done it for every same-name pair.

The source's rejections now move to the target (INSERT OR IGNORE, so
one the target already holds is not doubled). Where the two halves
disagree about one face, confirmed as one and rejected as the other,
the confirmation stands, as `confirm` already rules for one face; and a
moved rejection withdraws a suggestion of the same face, as `reject`
already does. Confirmations and names are unchanged.

The body is split into merge_people_within, taking the caller's
transaction, so a job that merges several pairs commits once
(unchecked_transaction cannot nest). merge_people keeps its signature
and its one transaction.
 2026-09-26 17:41:55 -04:00
dtourolle 1479e45637 Keep one person to one face per photograph in the catalog merge 
The grouping pass never puts two faces of one photograph in the same
group (the cannot-link in dr_face::cluster). The merge did not check
this. When the two devices disagree about which face in a frame is a
person, merge_people_within applied the remote's confirmation, or the
anchor of a set-aside group, to face X. This device already held the
same person on face Y of the same photograph, so the person ended up on
both faces.

The reference library has 80 such person/photograph pairs on the
desktop and 89 on the tablet: 79/88 unnamed set-aside groups and one
named person confirmed on two faces. There are no duplicate faces (no
pair of faces in one image and embedder with IoU >= 0.5).

An incoming assignment is now refused when another local face of the
same photograph already holds that person. The one exception is an
incoming confirmation against a local suggestion: the suggestion is
withdrawn and the confirmation is applied. Two confirmations stay as
this device has them, the same rule as a local confirmation outranking
a remote one. A face that already holds the person is not a rival to
itself, so a steady-state pass is unaffected. On the reference pair
this refuses 0 assignments and writes the same 8,954 as before; it only
changes what a future disagreement does. The refusals are counted in
MergeReport::faces_one_per_photograph.

Existing pairs are left alone. They are two different faces (cosine
0.31 for the named one), not one face twice, so there is nothing to
fuse, and which face is the wrong one is not the merge's to guess.
 2026-09-26 16:54:05 -04:00
dtourolle b5b30e3750 Match synced faces by embedding where the boxes cannot (#77) 
On the reference library, 631 of the faces in the tablet's snapshot
match no desktop face by box (IoU >= 0.5), so a name on them stays on
one device. Twenty of those are the same face with the box drawn
somewhere else. Whole photographs sit at IoU 0-0.48 with cosines of
0.72-0.96 between the two devices' vectors, and ten of them already
carry the same person on both sides. The merge never read the
1 KB embedding every face row carries.

match_faces now runs two passes. The box pass is unchanged except that
a pair must now be unique on both sides: a remote face with two
overlapping local faces, or a local face overlapped by two remote ones,
is no longer settled by whichever overlap is larger. Only for the
photographs where a remote face is left over, and a local face is still
free, does it read vectors: one json_each statement per side, keyed by
row id. That was 357 photographs on the reference library, not all
19 MB of vectors. A left-over face pairs with the local face it
resembles most when:
- the cosine is >= 0.7,
- the two are each other's best,
- each leads its runner-up by >= 0.2, and
- a box has not already claimed the local face.
Anything less decisive stays unmatched, so a new face stays new.

Why the threshold is safe, measured on both catalogs (w600k_mbf):
- Of 169,548 pairs of different faces in one photograph, 4 reach 0.7
  (lookalikes in one frame) and the maximum is 0.82.
- At 0.6 the rule would claim two pairs that carry different people on
  the two devices. At 0.7 it claims 20, none contradicted and 10
  corroborated, each leading its runner-up by more than 0.5.
- It only compares faces the boxes left unmatched on both sides: 737
  such pairs, so about 0.02 false pairs expected.
- A low cosine never overrules a box. About 150 box-matched pairs fall
  below 0.45, because two detectors cut the same tiny face differently.
  73 of them carry the same person on both devices.
- Faces are compared only within one file_id and one embedder, because
  the same person in another photograph reaches cosine 1.0.
- `Embedding::cosine` refuses a comparison across models.

Before -> after on the reference pair: matched by box 18,348 -> 18,348,
by embedding 0 -> 20, unmatched 631 -> 611, ambiguous 0 -> 0. The
report counts the embedding matches. No schema change.
 2026-09-26 16:54:05 -04:00
dtourolle 884b681c21 Time a merge with another device's catalog in catalog_bench 
The bench merged the catalog with a copy of itself. Every face in that
merge matches its own box, so the pass never reaches the faces the two
devices disagree about. On the reference library that is 631 of the
tablet's 19,052 faces, and it is the work #77 adds to.

`--remote PEER.sqlite` now also times `merge_remote_catalog` against a
copy of the peer file, and prints the first pass's report so two builds
can be checked for agreement. The first run writes what the peer
brought. The runs after it are the steady state, so compare builds from
two fresh copies of one catalog.
 2026-09-26 16:54:05 -04:00
dtourolle 23abfd1827 Ship four presets for a bluer sky 
Blue sky, Deep blue sky, Polariser, and Blue sky with golden land, in a
Skies section after Essentials. Each darkens the colour mixer's azure and
blue bands and adds chroma to them — what a polarising filter does to a
clear sky — and brings the highlights down with it, so a white cloud does
not read as a cut-out against the deeper blue. The stronger ones nudge
azure towards blue and add dehaze.

They are looks and work only on the hues a sky occupies, so an overcast
frame is left nearly alone: there is no blue for them to deepen, and
tinting grey cloud blue would be worse than doing nothing. Tuned by eye on
the demo library's alpine and Manhattan frames, with an overcast Étretat
frame as the control.
 2026-09-26 16:46:46 -04:00
dtourolle 94ea2569ee Tag the SAF export path FR-EXP-10 only, not FR-PLAT-AND-1 
saf.rs and the export path's SAF branch shipped tagged FR-PLAT-AND-1,
and the matrix counted the requirement as covered. Its subject is the
library — reached through SAF grants — and Android still reaches a
library over a Nextcloud account or a folder path. What the SAF code
does is give an album a folder on the tablet, which is FR-EXP-10.

outstanding.md said the figure overstated it and should be read with
this one subtracted; it now says the tags were narrowed, and coverage
reads 161 of 192.
 2026-09-26 16:19:55 -04:00
dtourolle 7a56d16df1 Count the whole library in "All photographs" whatever is scoped (#76) 
The sidebar's "All photographs" row was bound to library-total, which
is the scope's count: the header's "412 images" and the scrollbar's
size. Under a collection, and now under an album, the row read as the
album's size — 4 where the library holds 70.

It reads a separate library-whole-total now: the same number as the
scope's when nothing is scoped (no second count), and otherwise the
unscoped count under the same filter, read only when the view's facts
move, so scrolling inside an album does not recount the library.
 2026-09-26 16:19:54 -04:00
dtourolle 5baaf9bac2 Release 0.17.0
Benchmarks / CPU and I/O (per commit) (push) Successful in 5m22s
Details
Benchmarks / Frame budget (on demand) (push) Skipped
Details
Traceability / Requirement traces (push) Successful in 1m1s
Details
Build and test / Android (aarch64) (push) Successful in 32m18s
Details
Build and test / android-image (push) Successful in 1s
Details
🐳 Android image / Build and push (push) Successful in 1s
Details
Build and test / Desktop (Linux) (push) Successful in 50m25s
Details
Build and test / windows-image (push) Successful in 1s
Details
🐳 Windows image / Build and push (push) Successful in 1s
Details
Build and test / Layer separation (push) Successful in 31s
Details
Build and test / Windows (x86_64, cross) (push) Successful in 38m0s
Details
Build and test / Publish the release (push) Successful in 50s
Details
2026-09-26 16:18:18 -04:00
dtourolle 7428f6f845 Re-record the manual for 0.17.0, with albums and the new preset sheet 
Every scene is recorded again on the 0.17.0 build, because the header
(Export to Exports), the sidebar (Albums) and the develop column had
all moved. The launch pictures showed the typed folder field, and the
export settings "Export to"; the presets picture predated the sections.

New scenes: presets_film scrolls the sheet down through the shipped
sections and applies Ilford HP5 Plus, a look that changes the film and
nothing else; albums exports four New York frames to an album, selects
it to show the originals behind its files, and opens a new album's
sheet before deleting what it made. record.sh now points the profile's
old export folder at DR_HOME/Exports, which the app turns into the
album "Exports" on first open - the one way to have an album without
the portal's dialogue, which Xvfb cannot show. The launch scene signs
out of a remembered folder instead of typing one, so it shows
"Open folder" beside the folder used last.

Not recorded: the server browser's New folder, which needs a Nextcloud
server, and the download screen, which a folder library never reaches
(the original is a local read, over before the first poll). The manual
says so where it describes each. film_reach and duplicates passed;
inference was pinned to the CPU.
 2026-09-26 15:15:51 -04:00
dtourolle 9f95d23ff4 Say that no workflow builds the Flatpak yet 
NFR-COMPAT-2's table has CI building every channel, the Flatpak
included. No workflow in .gitea/workflows/ builds it, and the sweep
before this one records that none has been built by hand either. A
status note under the table says so, so the decision and the state are
not read as the same thing.
 2026-09-26 14:58:46 -04:00
dtourolle 1b8d0e740f Say the benchmark's second open no longer pays the backfill 
benchmarks.md and catalog_open.rs both said schema::backfill runs on
every Catalog::open. Since ffdd640 it runs on the first open of a path
in a process and is skipped while the stamp matches, so in dr-bench
catalog_open_ms still includes it and catalog_open_warm_ms, the second
open in the same process, no longer does. That is what a library
reopened in one session costs, and both now say which figure is which.
The comment keeps its line count, so no tag below it moves.
 2026-09-26 14:58:20 -04:00
dtourolle d2b99bb8c1 Note the folder dialogues in windows.md 
Since 6683c14 every folder on the desktop is chosen through rfd, which
on Windows is the common item dialogue. windows.md's list of what is
already handled now says so, and that nothing has opened one under
Wine or on Windows. "The import flow's path picker" is now the import
page's Browse… button.
 2026-09-26 14:57:35 -04:00
dtourolle f39b88b005 Bring the README, the docs index and CONTRIBUTING up to 0.17.0 
The README said export went "to a folder here or back into the
library", which albums replaced and forbid, and called presets "named
presets" beside a shipped collection of film looks and Lightroom
imports. It now says both, and that a photograph only on the server
opens on its thumbnail with its download's progress. The requirement
count comes from traceability.md's summary: 192, 84% claimed; the
version and release count are left to the release commit.

docs/README.md's manual row gains presets and albums. CONTRIBUTING
said 177 requirements and 826 crates, and the Flatpak manifest 826
crates; the lockfile now holds 849 packages, 822 from the registry.
 2026-09-26 14:56:12 -04:00
dtourolle 2f5f2041ab Record dehaze's two passes in frame-budget.md 
The figures are bee5c58's, quoted as that commit measured them: five
passes to two, 22.9 ms to 9.1 ms at 2560 x 1600 fit and 54.1 ms to
28.1 ms at 4K, the output bit-identical. Under its own heading and
status line, like the 2026-09-25 section, since this file was not
re-run for them.
 2026-09-26 14:55:18 -04:00
dtourolle 0f7ea741d8 Record the open's backfill stamp and the first-use indexes in catalog.md 
#75 and the develop-landing work changed how the catalog is opened and
queried without touching its design document. §2 now lists the two
indexes made on first use, keywords_term_version and faces_box, beside
the tables made that way, and says why each exists. A paragraph states
what the backfill stamp in backfilled.rs holds, why it records the
newest rows by content rather than by id, and when the backfill still
runs.

§6.2's example of coalescing was a thumbnail job, a kind nothing
enqueues since #73.
 2026-09-26 14:55:08 -04:00
dtourolle dee509c6ef Describe albums, the folder pickers and download progress in the designs 
storage.md's trait listing stopped at get; it now has get_reporting,
what the default and the Nextcloud override do, and where develop reads
the figures. A new §5.3 says how folders are chosen — the portal or
Windows dialogue, the server browser whose New folder is create_dir,
SAF on Android — and where an album's files go: a server folder relative
to the account root, with the outbox's third .dest line, or a device
folder that never syncs.

catalog.md §8.2 said only collections merge, which had not been true
since keywords, people and capture metadata joined them, and is less
true with albums; it now lists what merges and why album_folders does
not. §2 records that the album tables, like dedup_probes, are made on
first use rather than by a migration.

outstanding.md said there was no SAF code on Android. There is now,
for album folders only, and it carries TRACES: FR-PLAT-AND-1, which
the entry says overstates a requirement about the library; FR-PLAT-AND-2
and S10's row follow from that.
 2026-09-26 14:54:46 -04:00
dtourolle caaae11d98 Say the folder dialogue is the portal, and what the Flatpak has not proved 
distribution.md §4, outstanding.md's FR-PLAT-LIN-3 entry, the Flatpak
manifest's comment and the README all said a library was chosen by
typing a path and that nothing in the tree called the FileChooser
portal. Since 6683c14 every folder the desktop asks for is chosen
through rfd's xdg-portal backend, so those sentences were false.

What they now say instead is narrower than "it works in the sandbox":
no Flatpak has been built here, so whether the portal's path opens a
library, holds across a restart and takes a sidecar is unobserved, and
volumes() still cannot see a host card. The chooser also landed in ui/
rather than behind the dr-plat seam distribution.md had proposed, and
both documents say so. FR-PLAT-LIN-3 gets a status note to the same
effect.
 2026-09-26 14:52:44 -04:00
dtourolle e57c5b8182 Regenerate the traceability matrix after the rebase 2026-09-26 14:26:10 -04:00
dtourolle 02ddce8d80 Stamp the backfill on the newest rows, not only their ids 
The backfill stamp read max(id) of images and versions and max(rowid)
of keywords. None of those tables is AUTOINCREMENT, so SQLite hands a
freed newest id out again: empty the trash of the newest photograph and
scan a new one, or let a local folder's walk delete a renamed file's row
and insert the new name in the same pass, and the new image takes the
old id. max(id) does not move, nor does count(*), and when a newer
version elsewhere keeps max(versions.id) still too, the stamp matched
and the open skipped the backfill.

That row is exactly one that needs it. Neither scan path creates the
default version: scan::persist and walk insert the image and leave the
version, the RAW/JPEG pairing and the keyword terms to the next open.
Skipped, the image went without them until the app restarted, so a
rating or a pulled sidecar judgement had no version to land on and a
JPEG beside its RAW showed twice.

The stamp now carries the newest row's content: the newest image's id,
path, added time and whether it has a version; the newest version's id
and image; the newest assignment's rowid, version and word. Whether the
newest image has a version is the part that cannot be fooled - after a
backfill every image has one, and a row that has just taken a freed id
has none - so the two stamps differ even when the same file comes back
at the same id in the same second. Still one statement: three reverse
rowid scans that stop at the first row, and one probe of versions_image.
An open that skips still costs ~1 ms on the reference catalog copy.

This closes the hole in the stamp itself rather than by a forget() at
each delete site, so a delete path added later, or one in another
process, cannot reopen it. Two tests delete the newest image and insert
another at the freed id on a separate connection, with a newer version
elsewhere holding max(versions.id); both fail against the old stamp.
 2026-09-26 14:26:10 -04:00
dtourolle faf52f6dbd Ask the prefetch's cache questions on one held connection 
holds_original, the prefetch worker's check that a neighbour's original
is already cached, opened the catalog for every neighbour it asked about.
Its own comment called it a row check; the open around it was four of
the five opens a develop landing made.

The worker now keeps one Catalog for the batch it is serving, opened at
the first check and reopened only if the batch names another catalog
file. The connection runs in autocommit, so each check still sees what
fetch_original committed in between. fetch_original is unchanged.

With the backfill no longer run on every open, a landing whose
neighbours are all cached goes from five opens to two, and from ~80 ms
of CPU to ~1-2 ms on a copy of the reference catalog.
 2026-09-26 14:26:10 -04:00
dtourolle ffdd640170 Backfill the catalog once per state, not on every open 
Catalog::open ran schema::backfill every time, and every worker thread
opens its own connection. A develop landing made five opens, and each
paid the RAW/JPEG pairing, the default-version anti-join over every
image, the uuid pass over every default version and the keyword check:
17 ms of CPU an open on a copy of the reference catalog, ~80 ms a
landing, to confirm that nothing had changed since the open before.

Everything the backfill repairs is a row some write added: an image a
scan inserted, a version or keyword assignment a merge brought in. So
the open now reads a stamp - user_version, max(id) of images and
versions, max(rowid) of keywords, and the file's device and inode - and
skips the backfill when the stamp matches the one recorded at this
path's last backfill in this process. The maxima are each the last page
of a b-tree; an open that skips costs ~1 ms.

The backfill still runs:
- on the first open in a process (nothing recorded yet);
- on any open that migrated the schema, unconditionally;
- after a pull: merge_remote forgets the path, so the next open
  backfills even when every incoming row collided and nothing moved;
- when the file is replaced under its name: the inode is in the stamp,
  and recovery::set_aside, the first step of a restore and a rebuild,
  forgets the path;
- when another process or thread adds rows, because the stamp is read
  from the file, not from anything this process did.

The stamp is taken before the backfill, not after. Read after, it would
describe the backfill's own inserts, and could record an image another
connection inserted in between as covered when it was not. Read before,
the worst case is one redundant pass after a backfill that did real work.

Kept in memory rather than in the catalog: a stamp row would need a
table an older build does not have and would travel in the sync
snapshot, where a flag from another device's catalog says nothing about
this one. No schema version bump, so the tablet on 0.16.0 still reads
the snapshot. Tests cover the skip, a scan's new image, a migration, a
pull and a replaced file.
 2026-09-26 14:26:10 -04:00
dtourolle 2226d543f9 Time a develop landing in catalog_bench 
Landing on a photograph in develop opens the catalog once to fetch the
original and once more per prefetched neighbour to ask whether the cache
already holds it: five opens, each running the whole backfill. The bench
timed one open but not the landing, so the cost of the shape was not
visible and a fix to it could not be measured.

Two figures now, both against an empty cache so the question is asked the
same way whatever the answer: the five-open shape the app had, and the
two-open shape where the prefetch worker keeps one connection for its
batch. On a copy of the reference catalog (23,582 images) under load, the
five-open landing costs ~80 ms of CPU.
 2026-09-26 14:25:46 -04:00
dtourolle bee5c5866f Erode dehaze's window in one pass per axis, and recover in the second 
Dehaze cost 22.9 ms of a 2560x1600 frame on the reference laptop RTX 3050,
and 54.1 ms at 3840x2160, with the memory clock held at 810 MHz by the power
cap (graphics 1762 MHz). It ran five passes: a run and a span erosion along
x, the same along y, and the recovery. At those clocks a detail pass costs
what it reads and writes, not what it taps: a pass with an empty body -
one render-sized rgba16float read and write - measured 4.0 ms, and each
dehaze pass 4.4-4.6 ms, so the taps were about 2 ms of the 22 and the four
hand-offs between passes were the rest.

Each axis is now one pass that takes the minimum over the whole window
directly, and the recovery rides in the y pass, which already holds the
veil and the pixel's own colour. That is 36 texture reads per pixel at
2560x1600 in place of 12, nearly all of them cache hits, and two passes in
place of five.

The picture is the same bits. A minimum is exact in any order, and the
window is the one Split always covered, the surplus pixel on the far side
included (Split::first and Split::width). The veil crossing the removed
hand-offs was already exactly representable in rgba16float - a minimum of
channels read from rgba16float, floored at zero - so storing it between
passes never rounded anything that the fused form now keeps unrounded.

Measured with a scratch probe that renders the synthetic 60 MP frame from
examples/frame_budget.rs, only a detail parameter moving so the fused pass
is reused, 30 frames per scene after six of warm-up, five runs of each
binary alternated, median of the per-run p50:

  scene                   before     after
  dehaze      2560 fit    22.88 ms    9.06 ms
  dehaze      2560 1:1    23.41 ms    9.52 ms
  dehaze      3840 fit    54.09 ms   28.12 ms
  all detail  2560 fit    53.11 ms   39.97 ms  (NR, sharpen, clarity,
  all detail  2560 1:1    67.48 ms   56.42 ms   texture, dehaze)
  every op    2560 fit    57.59 ms   44.19 ms  (with film)
  every op    2560 1:1    71.83 ms   57.93 ms
  controls without dehaze (NR, sharpen, clarity, texture): within +-2%

The rgba8 output hashed identically before and after for every scene -
dehaze alone, all five detail operations, every operation with film, and
each other detail operation alone - at fit and 1:1, at 2560x1600,
3840x2160, 1917x1203 and 333x211: 64 of 64.
 2026-09-26 14:18:42 -04:00
dtourolle 9b580c3720 Satisfy rustfmt and clippy on the album and folder picker changes 
rustfmt over the files the albums work touched, and the album merge's
incoming row as a named struct rather than an eight-field tuple, which
clippy's type_complexity refused.
 2026-09-26 14:13:54 -04:00
dtourolle 1abb18d972 Specify albums and pointing at folders, and describe them in the manual 
FR-EXP-10 is the album: a named export destination beneath the
collections, whose folder holds only the exported files while the
catalog links each back to its original; how albums sync, why a device
folder does not, and why the tables are made on first use rather than
by a migration. FR-EXP-6 now says the destination is an album, never
inside the library, and that folders are chosen by pointing — the
portal or Windows dialogue, SAF's tree picker, the server browser —
each able to make a folder.

The manual's launch and export sections say the same in the words on
screen. Its pictures still show the 0.16.0 launch screen and export
settings; they are re-recorded with the rig, not edited by hand.
 2026-09-26 14:13:53 -04:00
dtourolle 92d4b23bed Give an album a folder on the tablet, through Android's folder picker 
Android's only export destination was the library on the server
(ExportTarget::available), because writing to the device goes through
the Storage Access Framework and nothing did. An album's folder on the
tablet is now chosen in the system's tree picker — which has its own
"Create new folder" — and exports are written into it with
DocumentsContract.

The picker answers through onActivityResult, and the main activity is
NativeActivity, whose result is not ours. FolderPicker is a translucent
activity that only asks: it starts ACTION_OPEN_DOCUMENT_TREE, takes a
persistable grant (a folder is chosen once and exported to for months),
leaves the URI in a static, and finishes. Rust polls it from a Slint
timer — one static call, rather than a registered native method and a
thread to deliver on.

Two things the first build on the tablet got wrong, recorded where they
are fixed:

- Our classes must be loaded through Context.getClassLoader(). The
  class of what ndk_context holds is a framework class from the boot
  loader, which reports every class in the APK as not found.
- What ndk_context holds is the application context, not the activity,
  and starting an activity from it throws without FLAG_ACTIVITY_NEW_TASK.

Saf.write creates the document (or, under Overwrite, reopens the one of
that name with "wt" so a shorter file does not keep the old tail) and
returns the name the provider actually gave it, since SAF renames on a
collision by itself; the album records that name. A tree URI reads in
the sidebar as its folder ("Pictures/Web"), not as a content:// string.
 2026-09-26 14:13:53 -04:00
dtourolle 7cbcacc02e Export to an album instead of a folder in the settings 
Export took a path typed into the settings page, or a folder inside the
library on the server. The first is how exports end up somewhere nobody
looks; the second put JPEGs into the tree a scan catalogues, where they
came back as photographs beside the RAWs they were made from.

The destination is now an album (FR-EXP-10), chosen by name in the
export sheet. Albums are listed under the collections in the sidebar;
"+" there, or "New album…" in the sheet, opens a sheet for its name and
its folder — on this device through the platform's dialogue, or on the
server through the browser with "New folder". A server folder inside
the library is refused, and the sheet says why. Selecting an album
narrows the grid to the photographs behind its files: library::Scope
is Collection or Album, and scope_clause is the one place the two are
spelled, which also retires the two copies of the collection predicate
total_images_scoped and read_cells_scoped had inlined.

A batch resolves the album when it starts, and refuses in words when
none is chosen, it has gone, or its folder is local to another device.
Each item reports the image it came from, and the files written are
recorded against the album in one transaction when the batch ends.

A server album lives outside the library, so its queued uploads are
relative to the account root. That is a third line in the outbox's
.dest record rather than a leading slash, because a record written
before albums may carry a stray slash and must keep the meaning it was
written with.

An export folder set before albums becomes an album called "Exports"
on first open, so upgrading does not lose where exports were going.
The old destination fields stay in ExportSettings so older settings
files still read.
 2026-09-26 14:13:53 -04:00
dtourolle 2eb06b1064 Make a folder from the server browser 
The in-app browser that chooses a library folder on the server could
only open folders that already existed, so a library, or an export
destination, that was not on the server yet had to be made in the
Nextcloud web page first. It now has "New folder": a name, then MKCOL,
then the parent listed again and the new folder walked into — a folder
somebody has just named is the one they mean to choose.

The listing is the server's rather than the name inserted locally: the
server may have normalised or refused it. A name with a slash, "..",
or nothing at all is refused before any request, because a folder
typed with a slash in it is a path the user did not mean.

remote_folders holds the two WebDAV round trips (list, make) off the UI
thread, with the answer delivered through a Slint timer, so the album
sheet can use the same browser.
 2026-09-26 14:13:53 -04:00
dtourolle 6683c14b40 Choose folders in the platform's dialogue, not by typing a path 
Every folder the desktop asked for was a text field: the library folder
at launch, an import's source and second copy, a preset folder brought
over from Lightroom. A typed path is how a destination silently becomes
a new folder nobody meant — one wrong letter three levels down and the
write succeeds somewhere the photographer will never look — and a field
cannot make the folder that is not there yet.

They now open the platform's own dialogue through rfd: the XDG desktop
portal on Linux, the common item dialogue on Windows. The portal rather
than GTK because it reaches the user's files from inside the Flatpak and
needs no GTK in a Slint application, and it draws whichever desktop's
chooser is running, "New folder" included. It is awaited on Slint's
event loop (spawn_local), so the window keeps drawing while it is open,
and parented to the window so it opens over it.

PathRow shows what is chosen, read-only, beside the button. Android has
no filesystem dialogue — only SAF, which returns document trees, not
paths — so there the same rows stay typed fields (Pickers.local-paths).

The launch screen keeps the folder used last on screen with "Open
folder" beside it, so reopening is one press. Presets get two buttons,
a folder and a single .xmp file, because no platform dialogue picks
"a file or a folder" in one go.
 2026-09-26 14:13:53 -04:00
dtourolle 94542371f6 Keep albums in the catalog: export folders and what went into them 
An album is a named export destination. Its folder holds only the
exported files; the catalog records, per file, the image it was
rendered from, so an album can show the originals behind its JPEGs
(FR-EXP-10).

The tables are created on first use (CREATE TABLE IF NOT EXISTS), the
way dedup_probes is, rather than by a schema migration: a new
user_version makes every older build refuse this catalog's snapshot at
sync, and the 0.16.0 tablet would stop merging collections, keywords
and people for a feature it does not have.

Albums merge as collections do: by uuid and revision, tombstones on
delete, exports as a set union keyed on the server's file id (content
hash for a folder library). A folder on the server lives on the album
row and syncs; a folder on this device lives in album_folders, which
the merge never reads and the upload snapshot drops, because a path or
a SAF grant on one device means nothing on another.

Exports are keyed on the file name, not the image: two crops of one
photograph are two files and two rows, and an overwrite re-points the
name at whatever wrote it last.
 2026-09-26 14:13:28 -04:00
dtourolle 715fcf8512 Regenerate the traceability matrix after the rebase 2026-09-26 14:03:27 -04:00
dtourolle 49b7bc2f9d Build the upload snapshot without the face crops instead of stripping them 
Each sync pass spent 0.8-2.0 s of CPU and 1.0-5.4 s wall on the upload
snapshot of the reference catalog (24k images, 18,871 faces), ahead of the
rest of the pass. The upload itself had been crop-less since the crops
moved to the face shards. The cost was in how it got that way. The backup
API copied all 158 MB of the catalog, 96 MB of it the ~5 KB JPEG crop on
every faces row. Then `UPDATE faces SET crop = NULL` rewrote 18.9k rows
and freed their overflow chains, and VACUUM rebuilt the file again. That
wrote the catalog about three times over to upload 50 MB.

The snapshot is now built rather than copied. An empty file attaches the
catalog, creates each table from the catalog's own sqlite_master and fills
it with INSERT ... SELECT, with faces.crop selected as NULL. Indexes,
triggers and views follow, and user_version, application_id, page size and
the WAL header flag are carried over. It all runs in one transaction on the
snapshot's connection, so the catalog is read as of one moment and
concurrent writers are serialised, not raced, as the backup API did. The
build journal is in memory with synchronous off, because the file is
scratch that is rebuilt every pass and quick_check'd before upload. Foreign
keys are off on that connection. The bundled SQLite enables them, and then
a multi-row INSERT into images scans images for children of each new row
(shadowed_by is a self-reference with no index), which cost 1.2 s alone.

Measured on a .backup copy of the reference catalog with catalog_bench,
old and new binaries back to back on a loaded machine:
  before  best 1.0-5.4 s wall, 0.84-1.98 s cpu, 49.8 MB
  after   best 0.40-2.1 s wall, 0.39-0.96 s cpu, 50.4 MB
With the machine quiet the new build takes 0.31-0.43 s.

What a receiving device gets is unchanged. It is the same schema, the same
rows and a NULL crop, which is what 0.16.0 already uploads and merges. The
merge reads only a remote face's box and model (merge::match_faces) and
never writes a local crop. No device adopts a downloaded catalog as its
own, and a fresh one takes faces and crops from the shards. There is no
schema bump, so older builds still merge it. NFR-R2 backups keep using the
backup API and keep their crops.

Tests: the snapshot matches the catalog in schema, row counts, pragmas and
WAL header. A leftover file is replaced. Merging a crop-less snapshot
carries a confirmed name across by box and leaves the local crop
untouched, and does so idempotently.
 2026-09-26 14:03:27 -04:00
dtourolle a59f14c797 Describe shipped presets and looks in the manual and FR-DEV-6 
The presets sheet now lists the shipped collection beside the
photographer's own, a copy under a shipped name overrides it, and
shipped and imported presets are looks that leave a photograph's own
corrections alone. The manual says so where it introduces the sheet,
and FR-DEV-6 states the rule. The sheet's screenshot (media/presets.png)
predates the sections and needs re-recording.
 2026-09-26 13:44:32 -04:00
dtourolle e82190fdf2 Import Lightroom presets as looks 
A Lightroom preset changes the settings it was saved with and leaves every
other one where the photograph had it. Imported as a whole edit, a preset
holding only a grade reset the exposure, white balance and noise reduction
it was put on top of — the opposite of what the photographer had in
Lightroom.

Imported presets now reach only the operations they name
(`Reach::Named`), the rule the shipped presets already follow.
 2026-09-26 13:44:32 -04:00
dtourolle a7b090cf36 Ship presets with the application instead of seeding them 
The six starter presets were copied into the photographer's own library
on a first run and were theirs from then on. That cannot grow into a
real collection: a copy is frozen at the release that wrote it, so an
improved preset reaches nobody who had the old one, and re-seeding would
overwrite a preset someone had tuned.

`dr_pipeline::bundled` now holds the shipped presets as `.drpl` files
compiled into the binary, in sections — Essentials (the former six) and
three sections of film presets, one per measured stock in dr-film,
printed on the paper its profile names — and never writes them to the
user's file. Every shipped preset is a look (`Reach::Named`), so applying
one keeps the corrections a photograph already has.

A name links a photographer's copy to a shipped preset. Saving over a
shipped name makes their version the one that name applies; it is listed
in the shipped section, marked as changed, and deleting it reverts to the
shipped one. Renaming it makes it one of their own and the shipped preset
reappears. Keyed on the name because that is what the photographer sees
and chooses by.

Copies an older first run seeded are forgotten on load where they are
still exactly as seeded — otherwise all six would list as changed and
stay frozen at their old values. A tuned one is kept and now overrides.

The sheet lists "Yours" first, then each shipped section, with headings.
Shipped rows apply and nothing else; a changed row offers Revert where
the photographer's own offer Delete. A dr-ui test checks every shipped
film names a stock this build can bake, on that stock's own paper,
because dr-pipeline does not link the profile database.

The film presets name stocks by id; the measurements behind them are
spektrafilm's (CC BY-SA 4.0), attributed in each file as in dr-film.
 2026-09-26 13:44:32 -04:00
dtourolle 90c0695c05 Let a preset name its film, and let a look reach only what it names 
A preset could not choose a film stock. The stock is a choice of material
rather than a parameter, so `Preset` — a map of `op.param = value` — had
nowhere to hold it, and "Portra 400, printed" could not be saved, copied
or shipped as a look. Worse, the film node's own sliders *were*
parameters: a paste moved one stock's exposure and push onto whatever
stock the target was on, and left the target's tables baked from the
values it had just replaced.

A preset now carries a `FilmRef` beside its parameters. It travels under
whichever scope carries the film node, so the stock and its sliders are
never split, and by the replacement rule every other parameter follows:
applied at that scope, a preset without a film develops the target
without one. `Preset::apply` returns the `FilmRebake` it owes, as
`EditGraph::set_state` already did, because this crate cannot bake a
stock; the develop session pays it before recording the step, and the
batch paste writes the stock into each sidecar through `film_for`. The
library file spells it `film =` / `film_print =`, as a sidecar does, and
an older build keeps those lines as ones it does not understand.

`EditState` keeps the film in its own field only: the parameters it
captures leave it out, so one edit has one place to say which stock it
is on.

Second, a preset now has a reach. Replacement is right for a copy of a
whole edit — "make these match" — and wrong for a look: a stock-only
"Portra 400" applied that way would put the photograph's exposure, white
balance and noise reduction back to default. `Reach::Named` replaces only
the operations a preset names (whole operations, so a look that sets the
blacks resets the whites beside them) and the film only if it names one.
Saved edits and the clipboard keep `Reach::Whole`; the line `reach =
named` is written only for the other, so existing libraries write the
same bytes.
 2026-09-26 13:44:32 -04:00
dtourolle c6d4c1ba62 Regenerate the traceability matrix after the rebase 2026-09-26 13:29:34 -04:00
dtourolle ce6705be89 Merge synced face assignments against what is held, read once per pass 
After 9cff677 the loop over the other device's confirmed and ignored faces
(13,000 on the reference library) still asked three cached statements per
face -- the person by uuid, the face's current assignment, and whether this
pair was rejected here. It was 51 ms of a steady-state merge.

The person is now resolved in the statement that reads the incoming rows,
by the local `people.uuid` key:

  SCAN fp
  SEARCH p USING INTEGER PRIMARY KEY (rowid=?)
  SEARCH lp USING COVERING INDEX sqlite_autoindex_people_1 (uuid=?)

and the local `face_person` (16,800 rows) and `face_person_rejected` are
each read once into memory and looked up there. A write goes to the table
and to the map, so a second remote face matched to the same local face sees
what the first left, as it did when each face re-read the table. The
incoming rows are ordered by face id -- the order the table was already
walked in -- since which of two such faces is applied last decides the
answer. An inner join to `people` drops the rows the old loop skipped for
want of a local person, and the counts in the report are unchanged.

After: the loop 10-12 ms. The merge as a whole, with the two changes before
this, went from 228-231 ms to 135 ms best of 5, and every catalog table
checksums the same after the bench as after the old build's run.
 2026-09-26 13:28:50 -04:00
dtourolle ae0281fedd Match synced faces from an index of their boxes, not from their rows 
`merge::match_faces` reads every local face's box and model to pair the
other device's faces with ours. It took 54 ms of a steady-state merge on the
reference library (19,000 faces).

A `faces` row is eight kilobytes -- the embedding, the crop, the dense
landmarks -- and `model_id` sits past the embedding, so reading it opened
each row's overflow pages:

  SCAN f
  SEARCH r USING INTEGER PRIMARY KEY (rowid=?)

`faces_box (image_id, model_id, x, y, w, h)` holds every column the scan
asks for:

  SCAN f USING COVERING INDEX faces_box
  SEARCH r USING INTEGER PRIMARY KEY (rowid=?)

The local scan went from 38 ms to 8 ms (sqlite3 on a copy, aggregated so
output formatting is not timed), and `match_faces` from 54 ms to 30-37 ms;
what remains is the other device's half. That is read from its snapshot,
which has whatever indexes its build made -- this one will carry
`faces_box` in its uploads -- and whose rows have had their crops stripped.
The bench merges a full copy with crops, so it overstates that half.

Created on first use in `match_faces`, with CREATE INDEX IF NOT EXISTS,
rather than by a migration, for the reason `keywords::ensure_term_index`
gives: a schema version bump makes older builds refuse the snapshot, and an
extra index is invisible to them. The first merge after the upgrade builds
it (about a second, once). Its prefix duplicates `faces_image_model`, which
is left alone; the planner takes either for an (image_id, model_id) probe.
Tables checksum the same after the bench run as after the old build's.
 2026-09-26 13:28:50 -04:00
dtourolle 981022ab1d Ask a synced keyword's tombstone once per merge, not once per assignment 
`merge_remote_catalog` on the reference library (catalog_bench, a copy
merged with itself: the steady state of a sync pass) cost 228-231 ms best
of 5. Timing its phases put 91 ms in the keyword half, not in the faces the
issue named.

Both assignment unions refuse a word this device holds only as a tombstone,
with a correlated `NOT EXISTS (... deleted = 1) OR EXISTS (... deleted = 0)`
per incoming assignment. The `deleted = 1` half has no index to use --
`keyword_terms_name` is partial on `deleted = 0` -- so it scanned the whole
vocabulary for each of the 10,800 rows:

  SCAN rk
  CORRELATED SCALAR SUBQUERY 1
    SCAN t
  CORRELATED SCALAR SUBQUERY 2
    SEARCH t USING COVERING INDEX keyword_terms_name (name=?)

The refused words are one set for the whole statement, so it is asked once:
`rk.keyword NOT IN (tombstoned names EXCEPT live names)`, which is the same
condition -- refused exactly when deleted under some identity and live under
none -- and which SQLite builds as a list before the walk:

  SCAN rk
  LIST SUBQUERY 2
    MERGE (EXCEPT) ...

The file-id union alone went from 72 ms to 11 ms (sqlite3 on a copy), and
the keyword phase of the merge from 91 ms to 28-35 ms. Every table of the
catalog checksums the same after the bench as after the old build's run,
and the merge tests for tombstones and renames pass unchanged.
 2026-09-26 13:28:50 -04:00
dtourolle 87badb6f99 Count the grid by subtracting the hidden burst frames, not probing per image 
The grid's total is read on every scroll reload (`load_window` compares it
to notice a delete). On the reference library it cost 1.3-1.5 ms best-of-50
by catalog_bench, 2-3.6 ms on a busy machine, and the issue measured 4 ms.

`uncollapsed` asked every visible image whether a collapsed burst stands in
for it -- two primary-key probes per image, 19,000 times, on a library with
no bursts at all:

  SCAN i USING INDEX images_grid_order
  CORRELATED SCALAR SUBQUERY
    SEARCH bm USING INTEGER PRIMARY KEY (rowid=?)
    CORRELATED SCALAR SUBQUERY
      SEARCH be USING INTEGER PRIMARY KEY (rowid=?)

`total_images_filtered` now counts what the filter keeps and subtracts the
frames `bursts::collapsed_away_frames` lists, under the same filter:

  SCALAR SUBQUERY: SCAN i USING INDEX images_grid_order
  SCALAR SUBQUERY: SCAN bm; SEARCH be ...; SEARCH i USING INTEGER PRIMARY KEY

The second half walks only `burst_members`. Each image is in it at most
once (it is the key), and the filter is applied to both halves, so the
subtraction removes exactly the rows the predicate used to drop. The new
fragment sits beside `not_collapsed_away` in bursts.rs, and a test holds
the two to the same rows with bursts open and closed.

After: 0.3 ms, the same count (19,152). The cells query keeps the predicate:
it is a window with a LIMIT and needs the rows, not their number. The
rated grid count (3.5-4 ms with a one-star filter) is unchanged: its cost
is the rating subquery per image, and changing how `RatingFilter` spells
it changes every grid and timeline query, which is left for its own change.
 2026-09-26 13:28:50 -04:00
dtourolle d537e4a965 Serve the keyword counts from an index that carries the version 
`keywords::list` is the vocabulary with a per-word photograph count, and
`keywords::for_images` calls it on every selection change to redraw the
keyword panel. On the reference library (58 words, 10,800 assignments) it
cost 3.0-3.5 ms best-of-50 by catalog_bench, `for_images` 3.1-3.6 ms (6 and
5 ms on a busy machine).

Per word, the count walks `keywords_term (keyword)` and, for each
assignment, reads the `keywords` row to learn its version before probing
`versions` for the image:

  SEARCH k USING INDEX keywords_term (keyword=?)
  SEARCH v USING INTEGER PRIMARY KEY (rowid=?)

With `keywords_term_version (keyword, version_id)` the first step is
index-only:

  SEARCH k USING COVERING INDEX keywords_term_version (keyword=?)
  SEARCH v USING INTEGER PRIMARY KEY (rowid=?)

After: `list` 1.3 ms, `for_images` 1.5 ms, with the same answers (digests of
both outputs compared on the reference library).

The index is created on first use by `list`, with CREATE INDEX IF NOT EXISTS,
not by a migration: a new schema version makes every older build refuse this
catalog's snapshot at sync (`sync::remote_is_mergeable` compares
`user_version` and nothing else), and a build that meets an extra index
ignores it. Once the index exists the statement is a schema lookup, 8 us. A
failure to create it -- a read-only or busy catalog -- is logged and the
list is read without it, as before.
 2026-09-26 13:28:50 -04:00
dtourolle fe6e523443 Count the originals on this device from the cache, not from every image 
`library::local_original_count` feeds the "On this device" chip and runs
beside the rating counts on every star keystroke. On the reference library
it cost 1.3-1.4 ms best-of-50 (3 ms on a busy machine) to find 254
originals among 19,000 visible images.

It was a correlated EXISTS per visible image:

  SCAN i USING INDEX images_grid_order
  SEARCH ic EXISTS USING INTEGER PRIMARY KEY (rowid=?)

`image_cache` holds a row only for what has been fetched, so the question
is driven from it: `i.id IN (SELECT image_id FROM image_cache WHERE
tier_actual >= Original)`, which SQLite plans as the list first and a probe
of `images` by id for each entry:

  SEARCH i USING INTEGER PRIMARY KEY (rowid=?)
  LIST SUBQUERY 1
    SCAN image_cache

`image_id` is the cache's primary key, so each image is in the list at most
once and the count is the one the EXISTS gave (254). After: 0.05 ms. On a
library whose every original is cached this is as much work as before,
which is the proportion the rule asks for.

The count stays on the keystroke path: dropping it there would leave the
chip stale after a background download until something else refreshed it,
and at this cost there is nothing left to save. catalog_bench spells the
query as dr-ui does, so its copy changes with it.
 2026-09-26 13:28:50 -04:00
dtourolle 73059f2656 Count the label chips from the labelled versions, not from every image 
`label_histogram` runs on every label keystroke and after every batch of
judgements is saved. On the reference library it cost 7.2-8.4 ms best-of-50
by catalog_bench (13 ms on a busy machine), to report that none of 23,500
images carried a label.

The join was the rating histogram's, with one thing worse: the index does
not carry `label`, so each probe went on to read the version's row.

  SCAN i USING COVERING INDEX images_folder
  SEARCH v USING INDEX versions_judgement (image_id=?) LEFT-JOIN
  USE TEMP B-TREE FOR GROUP BY

It now takes the rating histogram's shape: only labelled default versions
are grouped, and the unlabelled slot is what is left of `judged_rows`.

  SCAN versions USING INDEX versions_judgement
  USE TEMP B-TREE FOR GROUP BY          (the labelled rows only)

That pass still reads each default version's row for `label`, but in the
index's order, which follows the table's; a partial index on the labelled
rows would make it index-only, and was not worth a new index for the
remaining 1 ms. After: 1.7-2.0 ms, the same answer on the reference library,
and a test that compares it with the old join over the awkward states the
rating test uses (a second default's label counted, unknown codes and zero
folded into unlabelled).
 2026-09-26 13:28:50 -04:00
dtourolle 81118728f4 Count the rating chips from the rated versions, not from every image 
`rating_histogram` runs on every star keystroke. On the reference library
(24k images, 1,200 of them rated) it cost 6.3 ms best-of-50 by
catalog_bench, and up to 10-14 ms when the machine is busy.

It was `images LEFT JOIN versions ON ... AND is_default = 1 GROUP BY
rating`. The plan:

  SCAN i USING COVERING INDEX images_folder
  SEARCH v USING COVERING INDEX versions_judgement (image_id=?) LEFT-JOIN
  USE TEMP B-TREE FOR GROUP BY

A probe of the index per image, then a sort of all 23,500 rows, to put
22,000 of them in slot zero.

Now the rated rows are grouped on their own (`rating != 0`: one pass over
`versions_judgement`, a sort of 1,200 rows), and slot zero is what is left
of the join's row count. That count is three index-only aggregates -- the
library size, the default versions, and the images holding one -- so an
image with no version is still unrated, and an image with two default
versions still counts twice, exactly as the join counted it:

  SCAN versions USING COVERING INDEX versions_judgement      (x3)
  SCAN images USING COVERING INDEX images_folder

`count(DISTINCT image_id)` has its own statement because alone it reads the
distinct values off the index order; beside other aggregates SQLite builds a
temporary b-tree for it.

After: 1.2 ms. The histogram is the same on the reference library
([22364, 663, 19, 47, 115, 374]), and a new test compares it with the old
join on a catalog holding every state the schema allows: no version, only a
virtual copy, two defaults, ratings below zero and above five.
 2026-09-26 13:28:50 -04:00
dtourolle 408f189019 Measure what the library screen reads on each keystroke and scroll 
Issue #75 lists catalog reads paid on interactive paths rather than once:
the rating and label chip counts on every judgement keystroke, the "On this
device" count beside them, the keyword panel's vocabulary on every
selection change, and the grid's total on every scroll reload. catalog_bench
now times each of them against a real catalog and prints their answers, so
a change to any of them can be checked for giving the same numbers.

Two of them live in dr-ui's private `library` module; their SQL is spelled
in the bench as it is spelled there, which the module comment says.

Reference library (24k images), best of 50, CPU, on a loaded machine:
rating_histogram 9.0 ms, local_original_count 2.0, label_histogram 10.0,
keywords::list 4.0, keywords::for_images 5.0, grid count 1.9, grid count
with a one-star filter 4.0.
 2026-09-26 13:28:50 -04:00
dtourolle fabc1c5b56 Regenerate the traceability matrix after the rebase 2026-09-26 13:22:08 -04:00
dtourolle 441f6f1404 Record why thumbnails are not queued 
catalog.md §6 still described the design of 2026-08-09, where the grid
enqueued Thumbnail jobs at Interactive and a runner drained them. It was
never built that way: the grid asks a worker directly and the sweep's
work list is what the thumbnail store lacks. The one enqueue that did
exist fed a queue nobody claimed (#73).

§6.1 now states the decision and its evidence: the store is shared
between devices and is the only record that knows a thumbnail exists,
metadata is owed through metadata_state the same way, the retired rows
are dropped at open rather than by a migration so no older device loses
the synced catalog, and every_queued_kind_has_a_consumer holds the rule.
§6.3 notes that the priority ordering is had without the queue.

outstanding.md's FR-PLAT-AND-4 paragraph said the scan's thumbnail jobs
were the one reachable enqueue; it now says nothing enqueues, and that
feeding the runner means a handler and its enqueue in the same change.

Refs #73
 2026-09-26 13:15:18 -04:00
dtourolle b3dbf4a039 Refuse a job kind that is enqueued with nothing to claim it 
The queue coalesces, so a producer with no consumer never fails: it
leaves one row per subject for ever. That is how 23,582 Thumbnail jobs
accumulated unnoticed (#73), and nothing at runtime would have said so.

every_queued_kind_has_a_consumer reads the shipping sources of every
crate under core/, ui/, apps/ and platform/ (cfg(test) items dropped)
and pairs the JobKind named at each enqueue( call with the kinds named
in a fn kinds( body or a claim_next_matching( call. An enqueue that does
not spell its kind is refused, since the pairing could not be checked.

It guards against passing over nothing: the queue's own files and the
scan must have been read. A second test runs the reader over fixed
snippets so a parsing bug shows up as a failure. Run against master's
scan.rs and walk.rs it names all three orphan enqueues.

Refs #73
 2026-09-26 13:15:18 -04:00
dtourolle 6e67ef4467 Drop retired Thumbnail jobs whenever a catalog is opened 
Stopping the enqueue leaves the rows already queued: 23,582 on the
reference catalog, about 1 MB of table and indexes that every query over
jobs pays for.

A migration would be the usual tool and is the wrong one here. A schema
bump makes an older build refuse the synced catalog snapshot, and the
tablet is on 0.16.0. So the rows are dropped at runtime instead, by
jobs::drop_retired over a new JobKind::RETIRED list, from runner::recover
- which already runs exactly once per catalog open, before any worker.

It runs every open rather than once because an older build sharing the
catalog queues them again on its next scan. kind leads the
UNIQUE(kind, subject_id) index, so with nothing left it is one index
probe. Measured on a copy of the reference catalog: 23,582 rows dropped
in 40 ms on the first open, 0.07 ms after.

Thumbnail stays in the enum so its number is never reused for a kind
that would then inherit old rows. The runner tests that call recover
move to a live kind; the jobs.rs tests of queue mechanics never call
it and are unchanged.

Refs #73
 2026-09-26 13:15:18 -04:00
dtourolle 5fcd3752d7 Stop the local walk queueing work nothing claims 
walk::scan_root enqueued an ExtractMetadata and a Thumbnail job for every
image it inserted or found changed. No handler claims either kind. The
walk is only reachable from the scan_local example today, so no real
catalog holds these rows, but it is the same leftover the remote scan
carried (#73) and it is what a local library would inherit.

Both debts are already recorded where their consumers look: an inserted
or changed image is written at metadata_state 1, which is the metadata
sweep's work list, and the thumbnail store answers for itself.

The tests that used job rows as the measure of "this image owes work"
now read metadata_state, which is the record the sweep actually uses;
the no-requeue test marks the first image read before the second scan,
so it still proves an unchanged neighbour is not put back in debt.

Refs #73
 2026-09-26 13:15:17 -04:00
dtourolle 5da28584a4 Stop the scan queueing a thumbnail job per photograph 
The reference catalog held 23,582 Thumbnail jobs, one per image, and
every scan re-coalesced all of them. Nothing has ever claimed that kind:
no JobHandler is registered for it on desktop or Android, and
dr_catalog::sync never merges another device's jobs in.

Thumbnails are owed by the store, not the queue. The grid's worker and
the thumbnail sweep both find their work by asking ThumbStore what it
lacks, and the store is shared between devices, so it is the only record
that knows another device already made one. A queue row was a second,
staler copy of that debt that grew with the library and was read by
nothing.

persist still writes the images and their remote identities in the one
transaction; it just no longer adds a row to jobs for each of them. The
two tests that asserted the rows existed become one that asserts a
repeated scan queues nothing.

Refs #73
 2026-09-26 13:09:44 -04:00
dtourolle 8a1d9c8642 Find the canvas tools by the condition they are gated on now 
The download fix renamed the canvas gates to root.has-photo, and the
canvas-order test still searched for the old spelling, so it panicked
before checking anything. The order it guards is unchanged.
 2026-09-26 11:58:30 -04:00
dtourolle c3b10ed372 Format the download description test 2026-09-26 11:23:09 -04:00
dtourolle 4bec01eaf1 Say a photograph is downloading, and how far, instead of failing 
The develop view reported a remote original on its way through the
error message, so it read "Could not load image" over "Downloading…".
It did so on every step along the roll, including a cached frame that
was ready within a tick, so each step flashed the error.

Waiting is now its own state. On the step, the grid's thumbnail of the
photograph stands in at once. Only when a transfer is really on the
wire does it dim under "Not on this device yet", with a line like
"Downloading — 12.4 of 38.0 MB" and a progress bar.

The bytes come from a new RemoteBackend::get_reporting. The Nextcloud
backend overrides it to read the body chunk by chunk; the default
reports once at the end. Progress is kept in the in-flight registry by
path, because a step usually lands on a frame the prefetcher is already
fetching. The catalog's file length stands in when the server sends no
Content-Length.
 2026-09-26 11:02:11 -04:00
dtourolle 3b97195b37 Keep a stepped-past download from replacing the open photograph 
Opening a photograph from the library starts a download and a timer that
polls for it. Every step along the roll started another, and each one put
its result on screen when it landed, so a frame stepped past earlier
could arrive last and replace the one whose name was showing. Each open
now takes a generation number; a download that lands for an older
generation is recorded in the activity list (its bytes are cached) and
goes no further.

The outgoing session also stayed live until the new download landed.
Its sliders kept working, and a second step before the first landed
saved that session's edit under the new photograph's identity. The
session is now dropped as soon as its edit is saved.
 2026-09-26 11:00:36 -04:00
dtourolle 5794f8c6ea Release 0.16.0
Benchmarks / CPU and I/O (per commit) (push) Successful in 5m57s
Details
Benchmarks / Frame budget (on demand) (push) Skipped
Details
Traceability / Requirement traces (push) Successful in 47s
Details
Build and test / Android (aarch64) (push) Successful in 31m41s
Details
Build and test / android-image (push) Successful in 4s
Details
🐳 Android image / Build and push (push) Successful in 4s
Details
Build and test / Desktop (Linux) (push) Successful in 47m44s
Details
Build and test / windows-image (push) Successful in 4s
Details
🐳 Windows image / Build and push (push) Successful in 3s
Details
Build and test / Layer separation (push) Successful in 45s
Details
Build and test / Windows (x86_64, cross) (push) Successful in 36m16s
Details
Build and test / Publish the release (push) Successful in 1m35s
Details
2026-09-26 09:46:32 -04:00
dtourolle b1d36b9143 Picture the duplicate originals review in the manual 
The manual described the review (7c9a4ee) without a picture, because the
demo library holds no duplicates. The new duplicates scene makes two:
it copies two New York frames into a bck folder beside their own,
restarts the app so the scan finds them, waits for the sidebar row the
sweep's dating brings (54aee50), opens the review from it, presses
Check and takes the page. It deletes the copies and restarts on the
library as it was; record.sh's snapshot restore would remove them too.
It is registered last, so no other scene sees the copies.

The picture shows both groups proved the same file, the camera-named
copy outside bck marked Stays, and "Move 2 copies to trash" ready.
 2026-09-26 08:02:51 -04:00
dtourolle 048d48f532 Re-record the manual on the 0.16.0 interface 
Every scene was recorded again on a release build of this commit with
the automation feature. Master changed what nearly every picture shows
after they were taken: scrollbars on the develop column, the grid, the
sidebar and Settings; a "?" beside Settings in develop's top bar, with
its controls regrouped; the Film row opening its list as a popup.

The film scene pressed the list's rows through the develop column, which
no longer holds them: the list is a popup, and the automation hook
reports its contents relative to it. The scene now opens it with
film_list_open, turns the wheel down it and back so the popup and its
scrollbar are seen scrolling, and clicks Velvia at its popup position
plus the popup's origin. The caption says so. film_reach passed in the
same run: the last stock was reached by the wheel, a drag, the scrollbar
and the keys.

Looked at as contact sheets of each GIF's middle and last frames and
each PNG. launch, launch-folder, library-nesting.png and
library-collection-menu came out byte-identical after optipng and are
unchanged. develop-zoom's deepest frames are smooth on Xvfb as before;
its caption does not claim blocks. Settings shows version 0.15.0,
the build's own, until the release commit bumps it.
 2026-09-26 08:02:39 -04:00
dtourolle 35d0696b66 Show upgrade_endpoint and the https-only client in the storage design 
storage.md's BackendProvider listing and its notes predated #65: the
trait gained upgrade_endpoint (core/dr-sync/src/provider.rs:95), run at
launch by AccountStore::upgrade_endpoints to move an http:// account to
https:// with its keyring entry, and the Nextcloud client refuses plain
http below every URL it sends (adade27, ea31791, 5569a06). The listing
gains the method and a note says why it is not normalise_endpoint again.
 2026-09-26 07:49:39 -04:00
dtourolle 9e099a07ab Record in the catalog design how duplicate originals are proved 
catalog.md said content_hash is computed only for import duplicate
detection and reconnection, and left "the same image catalogued twice"
as unspecified. FR-CAT-11a now handles the within-root case: it proves a
group by content_hash where every copy has one, otherwise by first and
last megabyte digests kept in dedup_probes, a table created on first use
rather than by migration (core/dr-catalog/src/duplicates.rs:296). The
cross-root case stays open, and the bullet now says which half is done.
 2026-09-26 07:49:19 -04:00
dtourolle a76e3bdd02 Describe flagging, scrollbars, Help and long steps in the manual 
The manual described what the pictures show and missed what this round
changed around them:

- Rating and flagging gave stars only. The keys (0-5, P, X, U), Flag on
  the selection bar, judging in develop without moving on, and the flag
  and stars on the roll's cells had no sentence.
- Nothing said the desktop draws scrollbars on the grid, the sidebar,
  the develop column and Settings, or how the grid's bar is used.
- Nothing said how to open Help: the header's Help or F1, and in
  develop the "?" beside Settings (d9f2596, 380cfda), with its See it
  links and its Manual button.
- Stepping along the roll now goes on past what the roll has loaded
  (a030bfd); Settings gained the duplicate originals line and the
  Manual row.

index.html is regenerated from the README.
 2026-09-26 07:48:46 -04:00
dtourolle 0103200fc5 Point the docs index at the help sheet, the bundled manual and its page 
The index described the manual's contents as they were before colour
labels and the duplicates review, and did not say the application
carries the manual or where the in-app copy of the gesture book is
(Help or F1 in the grid, "?" or F1 in develop since d9f2596 and
380cfda). Its conventions named two generated files; manual/index.html
is a third, with its own CI check.
 2026-09-26 07:47:00 -04:00
dtourolle 7c838691e9 Bring the README's feature list and "Where it stands" up to the tree 
The requirement count is 191, as traceability.md's summary now reads;
the 84% it quotes still rounds from 83.8%.

"Not built" left out two things the matrix lists as untagged and the
outstanding register describes: importing a Lightroom or darktable
catalog (FR-CAT-14) and translations past the launch screen
(NFR-A11Y-1).

The feature paragraphs predated this round: the library now filters and
sets colour labels and consolidates duplicate originals, develop corrects
converging verticals and intersects mask parts, and the keyboard
vocabulary, the help sheet (F1, or "?" in develop) and the bundled
manual it links into had no mention. The version line is left for the
release commit.
 2026-09-26 07:46:46 -04:00
dtourolle 6979b1a2b8 Tell contributors about the key and manual gates 
CONTRIBUTING named four CI commands and the traceability tag, and
nothing about the three checks added since 0.14.1 that a first UI change
is most likely to meet: gestures-check, which fails on a key bound
without a GESTURE block or a block naming an unbound key (9d1e31f);
manual-check, which fails when index.html is not the render of the
manual (1021635); and record.sh --check, which fails when the manual
shows a picture no scene makes (70583b9). Two short paragraphs say what
each holds and where the recording tools are.
 2026-09-26 07:46:17 -04:00
dtourolle 9253a16fed Say which packages carry the manual, and that NFR-R8 is decided 
distribution.md's list of what every channel must get right had the
face models as the one LFS trap. Since 0.15.0 (d8f26fb) the Arch
package, the Windows installer and the APK also carry the rendered
manual and its pictures, and refuse LFS pointers for the same reason;
the Flatpak manifest does not install it.

The Vulkan bullet still called NFR-R8 open. It was decided on
2026-09-19: no CPU pipeline, and a viewer on embedded previews with
develop and export withheld.
 2026-09-26 07:45:57 -04:00
dtourolle 9b1f74e6d5 Say in the architecture what the decoder trait became 
§3.2 sketches a RawDecoder over a seekable reader, and the crate map
named it. FR-RAW-2 was built in 0.15.0 as dr_decode::Decoder over bytes
(core/dr-decode/src/decoder.rs:33): the decoder states how much it needs
and where its preview is, and the storage layer fetches it. The sketch
stays as the argument for four entry points; a note under it says what
shipped, and the crate map uses the built name.
 2026-09-26 07:45:42 -04:00
dtourolle e31990550f Record progressive refinement as built, and the fit view's two speed-ups 
display-and-extension.md still called FR-DSP-4 absent, and
frame-budget.md ended its reading at "satisfied vacuously". 0.15.0 built
it (refine.rs, the provisional histogram, the fade), so the table row
says so, frame-budget.md gains a note under its FR-DSP-4 section, and the
register gains a status note like FR-RAW-2's and FR-UI-5's.

frame-budget.md also gains a short section with the figures 1dc7b45
(the fit view's gather cached per framing) and d430ec9 (an identity
detail pass dropped) measured, since its fit rows no longer describe the
fused path. They are quoted from those commits, on the machine they name,
and are marked as not re-run here.
 2026-09-26 07:45:26 -04:00
dtourolle 79c051e8a6 Bring the outstanding register up to 0.16.0 
Five things in it were no longer true of the tree:

- FR-DSP-4 said "Unbuilt" and that nothing tracked a provisional frame.
  0.15.0 finished it: the settle debounce in ui/dr-ui/src/refine.rs,
  the draft flag reaching the histogram (canvas-draft,
  Levels.provisional), and the 150 ms fade of the last draft
  (app.slint's canvas-previous).
- The note that dedup.rs is re-import detection stood alone. FR-CAT-11a
  is now built beside it: dr_catalog::duplicates, dr_ui::duplicates and
  the Duplicate originals review.
- Culling counted three unbuilt clauses and named two.
- Section 6 still counted zero @tr( and five accessible-* lines, figures
  from before 2026-08-30. The launch screen is converted (build.rs holds
  the mechanism, no .po exists), 127 accessible-* lines sit across eleven
  files, and ui_controls_are_accessible.rs holds the structure.
- Section 11 said nothing of the panorama was built. It was, the same
  week; FR-MRG-9, NFR-MRG-1 and NFR-MRG-2 are what carry no tag.

A new section 4a lists what the develop, mask and keyboard work left
open, so that its closed clauses are not looked for: FR-UI-5's wheel on
sliders, FR-RAW-2's second decoder, mask-editing M2's remainder, and
FR-DEV-17's deliberate blind spot for range and region parts. The
Android paragraph now says develop is zero-copy there since TD-1 was
paid off, and that the APK carries the manual.
 2026-09-26 07:45:00 -04:00
dtourolle ea44aba110 Put a scrollbar on the library grid 
The timeline beside the grid says *when* the view is. It does not say
how far through the library the view is, and scrubbing it jumps by
date. The bar gives the plain desktop answer: a thumb in proportion to
one screen of the whole grid, dragged or paged by clicking the track.
It reads the viewport that the rows are counted into, so it spans every
photograph and not only the loaded window of cells.

The Flickable keeps its id, its `interactive` arbitration with the
hold-to-pick-up and its pinch and zoom catchers. It now fills a
Rectangle that takes its place, and its stretch, in the grid's layout.
 2026-09-26 07:25:46 -04:00
dtourolle cc905fa2ed Put a scrollbar on the controls and shortcuts sheet 
The gesture book runs to several screens inside a card that otherwise
looks complete. Same wrapping as the develop column. The bar is drawn
over the list's right edge, so it overlaps the last few pixels of each
"See it" button.
 2026-09-26 07:25:45 -04:00
dtourolle f7df276295 Put a scrollbar on the settings page 
The page is several screens long, and nothing said so until something
had been scrolled. Same wrapping as the develop column; the bar sits at
the window's right edge, clear of the capped 680px form.
 2026-09-26 07:25:45 -04:00
dtourolle 005dc3a835 Put a scrollbar on the collections sidebar 
A tree longer than the panel looked, to a mouse, like the whole tree.
Same shape as the develop column: the Flickable fills a Rectangle that
takes its stretch, and a ScrollBar is drawn over its right edge. It is
drawn only when the tree overflows.
 2026-09-26 07:25:07 -04:00
dtourolle 825015a58e Put a scrollbar on the develop column 
The column (histogram, compose, every adjustment group, history) runs
several screens past the window. A mouse without a wheel could only drag
the column's contents, and a drag that starts on a slider moves the
slider.

The Flickable now sits in a plain Rectangle with a ScrollBar beside it,
bound to its viewport. Nothing about how the column sizes itself
changes: the Flickable fills the Rectangle, and the Rectangle takes the
stretch the Flickable had in the layout under the group strip. The bar
is drawn over the column's right-hand 10px, so the mandated panel width
is not reduced. When the pointer is on the bar, the lit track covers
the value labels of the compose rows, which already sit flush against
the column's edge.
 2026-09-26 07:25:06 -04:00
dtourolle 2ef971bf37 Fail when the last film stock cannot be reached 
The film list bug gave no failure anywhere: the data was right, the
markup compiled, and the list rendered. Two guards now check that the
list can be walked to its end, both by driving input rather than by
reading markup.

tests/film_list_reaches_every_stock.rs runs in CI and needs no display.
It builds the real AppWindow on Slint's testing backend and gives it 28
stocks. It dispatches window events through the same routing a window
uses: popup, Flickables, arbitration. It then checks that the last stock
is on screen, that is, not clipped away:
- after Down past the end, and that Enter chooses it;
- after drags on the list;
- after a run of wheel events with a still pointer;
- after dragging the scrollbar thumb.

Element queries need the Slint compiler's debug tables, which build.rs
emitted only for the `automation` feature. It now emits them for every
debug build too. Release builds, the ones that ship, are unchanged. The
testing backend is a dev-dependency at the same pinned version the
automation feature already uses, so no new crate enters the lockfile.
The test is compiled out of release test runs.

film_reach in tools/manual/scenes.py is the same check on the recording
rig: a real X pointer from xdotool, the release build, and the demo
library. It makes no picture, so it adds nothing to the manual. It runs
with every recording, or alone with `record.sh LIBRARY film_reach`, and
fails the run if the last stock (Ilford HP5 Plus) is out of reach by
the wheel, a drag, the scrollbar or the keys.

Both have to add the popup's position back. The testing backend reports
anything inside a popup relative to the popup, and so does the
automation hook built on it. They take the popup's position from the
Film row and Slint's clamp into the window.
 2026-09-26 07:24:46 -04:00
dtourolle e957d483fc Give the film list a scrollbar on the desktop 
A list cut off at its edge looks, to a mouse, like a list that ends
there. The open film list shows ten rows of twenty-eight, and nothing
said there were more. The user asked for visible scrollbars on every
platform but Android.

ScrollBar (widgets.slint) is a vertical bar drawn over a Flickable's
right-hand edge. It shows the share on screen and the position, it can
be dragged by the thumb from wherever it was grabbed, a click on the
track moves a page toward the click, and the wheel over it scrolls. It
is the Flickable's sibling rather than a wrapper, bound to
`viewport-y <=> flick.viewport-y` and the two heights, so a scroller
keeps its own sizing. It is drawn over the content rather than beside
it, so the mandated column widths are not reduced. With nothing to
scroll it is not drawn and takes no input.

Whether to draw it is Scrolling.bars, which Rust sets from
dr_plat::is_touch_first(). That is the same function that puts the
develop groups in the rail, and it answers the same question: what is
the user pointing with? On Android, lists still scroll by flick only.
Placing the bar inside another scroller would put it back in that
scroller's arbitration. The film list can have one because it is now a
popup.
 2026-09-26 07:24:45 -04:00
dtourolle 80938b0527 Open the film list as a popup, so every stock can be scrolled to 
The open stock list showed ten rows, None to Kodak Kodachrome 64, and
the other eighteen - all seven black-and-white stocks among them - could
not be reached. The data was whole; the list could not be scrolled.

Reproduced on the manual rig (Xvfb, xdotool, the automation hook):

- a drag on the list scrolled the develop column, never the list;
- a wheel run over the list scrolled the column past it, whenever the
  column had scrolled under that pointer in the last 800 ms - which is
  how the list is reached, by wheeling the column down to it. After a
  pause and a pointer move the wheel did reach the list;
- no key did anything.

The cause is Slint's routing, not the list. Since 2d878c2 the list was a
Flickable inside the develop column's Flickable, and Slint offers every
pointer event to the outermost Flickable first
(input_event_filter_before_children, i-slint-core 1.17.1 flickable.rs).
The column holds a press back (DelayForwarding) and intercepts the first
move past 8 px on the axis it can scroll, so the list never saw a drag.
For the wheel it intercepts while its own last wheel event is under
800 ms old and within 2 px, and always for a touchpad gesture that opens
with TouchPhase::Started - so on a touchpad the list could get no wheel
at all.

The list is now a PopupWindow under the Film row. A popup is its own
item tree: while it is open, events go to it and to nothing beneath it,
so the list scrolls by wheel, drag and flick however the panel is nested
and whatever the column did last. The alternative, standing the column
down while the pointer is over the list (the sliders' hover trick), fixes
the drag but not the wheel - `interactive: false` does not gate wheel
interception - so it would have left the bug for touchpad users.

The reason 2d878c2 bounded the list still holds: it is at most 320 px
and never lengthens the column, and now it covers the sliders instead of
pushing them down. The column cannot be scrolled while it is open, which
suits a one-click question; it closes on choosing, on Escape or Back, or
on a press outside it.

Keys, with the list open: Up and Down move along it from the chosen
stock and scroll it into view, Enter chooses, Escape or Back closes it
unchanged. A popup is its own focus tree, so the keys are taken when it
opens and Slint returns focus to the develop view's scope when it closes.
The Film row gains a button role, so a screen reader and the automation
hook can name it.
 2026-09-26 07:24:28 -04:00
dtourolle 6640ce0ca8 Regenerate the matrix, the gesture book and the manual page for the duplicates review 2026-09-26 07:20:19 -04:00
dtourolle 7c9a4ee358 Describe consolidating duplicate originals in the manual and the register 
FR-CAT-11a records what #67 built: groups of the same file listed on
demand, proved the same before anything moves, merged onto one survivor
and the rest trashed a group at a time. The manual's new section under
The library says how to reach the page, how the copy that stays is
chosen, what the check reads and what merges.
 2026-09-26 07:19:14 -04:00
dtourolle 54aee50539 Count duplicate originals once the sweep has dated them 
A copy becomes a duplicate only once its capture time is read, and on a
fresh library that is the sweep, not the scan: the sidebar row stayed
hidden until the next launch. The count is refreshed when a sweep that
dated anything finishes.

Also says on a group left out of the plan that nothing will move, drops
an unused method, and names the review's completion callback type for
clippy.
 2026-09-26 07:19:13 -04:00
dtourolle 220e9af222 Add the duplicate originals review, from the sidebar and from Settings 
"Duplicate originals" appears under the trash in the collections
sidebar while the catalog holds any, and Settings says how many there
are beside the other whole-library passes. Both open one page: every
group with its picture and paths, the copy that stays (tap another path
to change it), a per-group Include box, what the survivor will gain and
any flag, label or face conflict, and why a group was skipped.

The summary is the dry run -- "N groups, M files to trash, K skipped" --
and nothing moves until "Check" has read the copies and "Move M copies
to trash" is pressed. Both run on workers with progress on the page, in
the activity register and, for the move, on the library status line;
Stop ends a job between groups. When it ends the grid, the sidebar and
the trash are refreshed and the survivors' judgements are written to
their sidecars and XMP the way a rating keystroke writes them.

The page is paginated at 30 groups, so a redraw decodes 30 thumbnails
and previews 30 merges whatever the size of the library. Back and
Escape leave it like its own Back button.
 2026-09-26 07:19:13 -04:00
dtourolle 8d4ecb75c1 Prove duplicate originals the same and consolidate them on workers 
dr_ui::duplicates is the half of #67 that touches files. The check reads
each copy's first and last megabyte by range through the backend and
hashes them (or compares stored content hashes where every copy has one),
keeps the probes in the catalog, and reads each copy's sidecar: a group
whose bytes differ, whose copies cannot be read, or whose develop edits
differ is left out and the review says why.

Consolidating a group carries the one edit onto the survivor's sidecar
where it has none, moves the other copies into the trash, and then
commits dr_catalog::duplicates::consolidate. A failure after the first
move puts the files and the sidecar back; a run that died between the
moves and the commit is finished by the next one, which finds each moved
file at its trash path.

Tested end to end on a folder library of real files: the copies land in
.darkroom-trash, the skipped groups are untouched, the edit reaches the
survivor, a catalog failure moves everything back, and restore returns
the copies byte for byte.
 2026-09-26 07:19:13 -04:00
dtourolle 3c2eacbf3f Find catalog duplicates and fold a group onto one copy in one transaction 
The library holds the same RAW in several folders: a dated folder, a
bck/ beside it, a renamed Darktable export tree. dr_catalog::duplicates
is the catalog half of consolidating them (#67).

candidates() is one grouped query over root, camera, capture instant and
size, joined back for the rows; count() is the same grouping under
COUNT. On a copy of the reference catalog (23,582 images) both take
10-35 ms and find 1,836 groups holding 3,379 spare copies.

survivor() prefers a copy outside a backup-looking folder, then one
still named the way the camera named it, then the oldest, then the
lowest id.

consolidate() re-checks the plan against the catalog, merges the copies'
judgements onto the survivor (highest rating, keywords unioned,
collections unioned with the survivor keeping its place, a flag or label
the copies agree on, faces via faces::carry_onto_copy) and records the
copies as trashed, all in one transaction, so a failure part way leaves
the group untouched. preview() runs the same code and rolls it back.

Sameness probes are kept in dedup_probes, created on first use rather
than by a migration: a schema bump would make older builds refuse this
catalog's snapshot at sync. trash::record_trashed_within lets the trash
write share the merge's transaction.
 2026-09-26 07:19:13 -04:00
dtourolle d430ec9528 Drop a detail pass that changes nothing when it sits between two others 
Capture sharpening at a scale too coarse to draw its radius emits one pass
with an empty body (`nothing_to_sharpen`), so that a chain still ends in
something that performs the output transform. At fit on any modern sensor
that is most of the time. When another neighbourhood operation follows it
- dehaze, clarity, texture - the pass is not last and does nothing: it
reads the rgba16float intermediate and writes the same texels to the other
one. It still cost a full render-sized read and write every frame:

  scene                           before     after
  sharpen+clarity 2560x1600 fit   18.66 ms   14.16 ms
  sharpen+clarity 3840x2160 fit   37.93 ms   27.88 ms
  every operation 2560x1600 fit   71.80 ms   67.24 ms
  clarity alone   2560x1600 fit   14.23 ms   14.20 ms  (control)
  sharpen+clarity 2560x1600 1:1   23.01 ms   22.97 ms  (control: resolves)

(Laptop RTX 3050 held at 420/810 MHz by its power cap, synthetic 60 MP
source, median of five alternated runs of forty frames each.)

`compose_detail_with` now drops such a pass where dropping it is exact:
not the last pass, whose output transform would otherwise move onto the
previous pass's f32 result and round differently; and not a pass right
after a reduced one, because a full-resolution pass is what closes the
reduced chain for the operation after it. `DetailPass::is_identity` says
what "changes nothing" means: full size, nothing bound at binding 3, and a
body with no code in it.

The rgba8 output is bit-identical: every scene above hashed the same
before and after, and the sharpen+clarity frame hashes the same as clarity
on its own, which is the claim in one line. A new dr-pipeline test pins
the three cases - dropped ahead of another operation, kept when last, kept
when alone.
 2026-09-26 07:10:41 -04:00
dtourolle 1dc7b45cfe Read the fit view's source gather once per framing, not once per frame 
At fit, every output pixel of the fused pass loads one texel from a source
three or four times its width, on a stride. The memory system fetches the
texels it skips along with the one it wanted, so on a 60 MP rgba16float
source that gather was most of what the fused pass cost: 10.6 ms of a
2560x1600 frame against 3.8 ms for the same shader reading a contiguous
window (the 1:1 view). At 3840x2160 it was 21.1 ms. Those are the laptop
RTX 3050 with its clocks held at 420/810 MHz by the power cap; unthrottled
the same frames were about 2.0 and 3.2 ms, and the gather is the same
share of them.

Which texel an output pixel reads depends only on the framing prologue,
the framing and warp uniforms, the source and the render size. None of
those move during a slider drag, so the gather is the same work every
frame. The fused shader now takes a render-sized rgba16float cache of it
(bindings 6 and 7, declared in every generated shader like the masks) and
a pair of uniform flags: write what was gathered, or read it back at the
pixel's own coordinate. AdjustPass keeps the cache and decides per
dispatch. The composer supplies `ComposedShader::sample_key`, a hash of
the prologue and those uniforms, and AdjustPass adds the image and the
size; an image gets a process-unique id for this rather than being held
alive by the key.

The picture is bit-for-bit the same. The source is rgba16float and so is
the cache, so the stored texel is the texel, and only the path that reads
a texel whole takes part: an interpolated sample (straightening, lens
warps, CA) is a blend that f16 could not hold exactly, so the composer
gives it no key and it reads directly as before.

The cache is written on the second frame with a given key, not the first:
a crop or zoom drag changes the key every frame, and writing then would
add a render-sized write to exactly the gestures that can afford it least.
It is kept only up to 3840x2400, so an export never parks a full-frame
copy on the device, and `release_caches` drops it.

Measured with a scratch probe rendering the synthetic 60 MP frame from
examples/frame_budget.rs, forty frames per run after six warm-up, five
runs of each binary alternated, median of the per-run p50 (GPU idle apart
from the power cap):

  scene                    before     after
  neutral   2560x1600 fit  10.62 ms   3.88 ms
  exposure  2560x1600 fit  10.83 ms   3.87 ms
  nr chroma 2560x1600 fit  19.84 ms  12.69 ms
  neutral   3840x2160 fit  21.05 ms   7.11 ms
  exposure  3840x2160 fit  21.08 ms   6.94 ms
  clarity   3840x2160 fit  42.20 ms  27.88 ms
  neutral   2560x1600 1:1   3.83 ms   3.84 ms  (control: nothing to gain)

The rgba8 output of every scene hashed identically before and after, in
isolated runs and across all 38 scene/size/view combinations of the
probe. New tests walk a pass through direct, write and read frames, a
slider move, a neighbourhood operation and a framing change, and compare
every frame with a fresh pass that can only have read directly.
 2026-09-26 07:10:35 -04:00
dtourolle 284fc4a456 Regenerate the traceability matrix after the rebase 2026-09-25 23:25:08 -04:00
dtourolle 9de37bed81 Hold develop's judgement keys to the open photograph, and to staying on it 
Rating and flagging in develop (FR-UI-5, amended 2026-09-19) landed with
the keyboard audit: 0-5, P, X and U in develop's key scope, stars and
Pick/Reject in its top bar, and flag and stars on the roll's cells. Two
of the amendment's rules were held by nothing. The keys must judge the
photograph on screen and never a selection left behind in the grid, and
judging must not move on to the next frame, which is culling's
auto-advance and not develop's.

The rating, flag and label callbacks that take a row each spelled the
row-to-image lookup themselves. It is now one function, `image_at_row`,
which answers None for a negative row as well as one past the end: the
roll passes -1 when the open photograph is outside the loaded window,
and the right answer then is to judge nothing. A test says so. The key
bindings live in Slint, where no test can press them, so a second test
reads develop's handler as the gestures gate and the canvas-order test
do, and checks that each judgement key calls the row callback on
`library-roll-current` and that none of them steps the roll or the
cursor.

The writes themselves go through `apply_judgement`, the grid's own path:
one catalog statement, then the sidecar and XMP writes behind it.
 2026-09-25 23:24:52 -04:00
dtourolle 380cfda695 Make develop's Help a "?" beside Settings, so Settings fits at 1600 
Adding a labelled Help button to develop's top bar made the strip about
100 pixels wider than a 1600-pixel window. The strip scrolls, so nothing
became unreachable, but Settings was off the right-hand end until the
bar was dragged.

Help is now a square IconButton with a drawn question mark (a new "help"
icon, drawn rather than typed for the reason icons.slint gives about
Android's fonts), and screen readers still hear it as "Help". That saves
60 pixels, which was not enough alone: the strip had fit with 3 to spare
before. The rest comes from the spacing. Controls that belong together
now sit in groups at gap-sm with gap between groups: Pick and Reject,
Undo and Redo, Copy, Paste and Presets, and "?" and Settings. The empty
export-status caption no longer takes a slot and two spacings while it
has nothing to say. At 1600 wide with the panel open the whole bar now
shows, Settings included.

The grid's header keeps its worded Help button; it has room for it.
The develop "Open this list" gesture now names the "?" button, and the
gesture book is regenerated from it.
 2026-09-25 23:24:52 -04:00
dtourolle d9f259656a Open Help from develop as well as from the grid 
The "Controls and shortcuts" sheet was drawn by LibraryGrid, so only the
grid's Help button and its F1 could open it. Develop, where most of the
keys it lists are bound (Ctrl+E, Ctrl+Shift+C, A/D, Z, R, H, [ ]), had no
way to it: a photographer who wanted to look a shortcut up had to leave
the photograph they wanted it for.

The sheet now hangs off the shell beside the export and copy sheets, on a
`help-open` property both views set. The grid's Help button and F1 raise
it through a callback, and its keys stand down through the `sheet-open`
they already honour for the export sheet, so Escape falls through to the
shell, which closes it. Develop gains a Help button beside Settings in
its top bar, as in the library header, and F1 in its key scope; its keys
decline while the sheet is up, as they do for the other two sheets, so
nothing behind it is rated or stepped. The book already begins with the
Develop section, so from develop it opens where the reader wants it.

Tests hold the shape: the sheet is drawn by the shell and not the grid,
and develop's opening guard names all three sheets and its F1 opens this
one. The new GESTURE: block puts the develop route in the book.
 2026-09-25 23:24:52 -04:00
dtourolle 19dd3257e3 Release the offset borrow before bring_window_to reloads the window 
Holding D in develop across the edge of the loaded window panicked with
"RefCell already borrowed" at the first step that had to move it. The
`*ctl.offset.borrow()` written inside the `if let` condition is a
temporary that lives to the end of the `if let` block (edition 2021),
and the block calls `load_window`, which borrows the offset mutably.
The unit tests drive the placement arithmetic, not the RefCells, so
they could not see it; stepping 400 frames in the app did.

The offset is copied out before the test. `follow_open` gets the same
treatment for `image_ids`: the lookup's result is bound first, so no
borrow is held while it writes properties back to the window.
 2026-09-25 23:06:24 -04:00
dtourolle a030bfd239 Step along the roll by library ordinal, and keep its mark on the open photograph 
In a library's develop view the arrows, space, A and D opened
`library-roll-pick(library-roll-current ± 1)`. `roll-current` is a row
of the loaded window, set when a photograph was opened and never again.
Two things followed on any library larger than one window:

- Holding D stopped dead at the end of the loaded window: the pick of a
  row past `ctl.paths` found no path and did nothing, a few screenfuls
  into a library of thousands. A stopped at its start the same way.
- Any reload of the window — a background sync, a judgement that drops
  the open frame out of a filter — left `roll-current` on a row that now
  held another photograph. The roll marked it, develop's rating keys
  judged it, and the next step walked on from it.

The keys now call `library-roll-step(delta)`, and Rust steps as
`move_cursor` does in the grid: from the open photograph's ordinal
(`index`, which `report_position` already keeps), clamped to the
library, loading the window around the target only when it lies
outside. The window-bringing half of `place_cursor` is shared for this
as `bring_window_to`, so the grid cursor and the roll move the window
the same way. Catalog reads stay proportional to the window: one
`load_window` per window crossed, none per step within it.

The open photograph is also remembered by image id. Every
`load_window` finds it again in the rows it just read (a scan of one
window, no query), puts `roll-current` and `index` back on it, or takes
the mark off when it is not there. When it is missing but its ordinal
is still inside the window, it has left the grid and its successor
moved up into its place, so the next step forward lands on that ordinal
instead of skipping the successor.

A click on the roll still reports a row; it is right because the cells
it is drawn from and `ctl.paths` are replaced together, and it now goes
through the same open path as a step.

Fixes #64.
 2026-09-25 23:00:55 -04:00
dtourolle ce72fe49a0 Record the catalog lessons of the 2026-09-25 performance pass 
Three habits the pass found broken, in the style of the 2026-09-19 notes:
SQL text passed to `execute`/`query_row` in a loop is a prepare per row
(the merge, `persist` and the shard sync all paid it), a case-insensitive
`LIKE` cannot use the `source_ref` key where a range can, and the backfill
inside `Catalog::open` is paid by every worker thread, the develop view's
fetches included. And where the two new benches are and how to run them.
 2026-09-25 22:06:58 -04:00
dtourolle 4583048595 Find a sidecar's photographs by an index range, not a case-insensitive LIKE 
When a scan pull takes in sidecars another device wrote, `apply_judgement`
finds the photographs each one describes with
`source_ref LIKE '{stem}.%'`. SQLite's LIKE folds ASCII case, and nothing
indexes `source_ref` case-insensitively, so every lookup read all 24,000
names of the root through the `(root_id, source_ref)` index: 1.5-2 ms per
sidecar, 520-690 ms for the 342 `.drsc` files the reference catalog has
read. Another device culling a shoot is several hundred of them.

Every name beginning `{stem}.` lies in the half-open range
`[{stem}., {stem}/)` -- `/` is the byte after `.` -- which the unique key
serves as a seek. The rows LIKE matched beyond these differed only in
case, and the check that decides, `sidecar_path(source) == sidecar`, has
always compared exactly and refused them; the escaping of `%` and `_`
goes too, since a range has no wildcards.

persist_bench: 342 lookups 634-691 ms -> 9 ms CPU. Checked against the
reference catalog directly as well: for all 18,430 distinct sidecar
names its images imply, the range and the old LIKE, each filtered by
`sidecar_path`, pick the same photographs. A test pins the neighbours of
the range: a case variant, a longer stem, a subfolder named like the
stem, and a folder whose name holds `%` and `_`.

The XMP reader's LIKE (`xmp_sync::images_for`) is left alone: its check
is case-insensitive, so the range would not be a superset there, and it
only runs when the exact darktable-style name is not found.
 2026-09-25 22:06:58 -04:00
dtourolle 32b2a5e317 Write a scan's findings with prepared statements, and its jobs in the same commit 
`persist` runs after every scan, for every photograph the scan listed. On
a settled library that is the folders whose ETag changed -- a sidecar
written there by a rating is enough -- so one relisted folder of 1,600
images is an ordinary pass, and a first scan is all 24,000.

Per photograph it prepared four statements from their SQL (a folder
lookup, the image upsert, the id read-back, the remote upsert) and then,
after the commit, found the image again by path and enqueued its
thumbnail job as an autocommitting statement of its own -- a commit per
photograph, for rows that were almost all already queued.

Now the statements are prepared once per pass, a folder's id is looked up
once per folder rather than once per photograph in it, and the job is
enqueued inside the transaction with the id already in hand. That also
makes the job atomic with the row it points at, which is what the old
ordering after the commit was trying to guarantee. `jobs::enqueue` uses a
cached statement for the same reason.

persist_bench on a copy of the reference catalog, CPU, best of runs:

  largest folder (1,589 images)   102-118 ms ->  10-13 ms
  whole library (23,582 images)   1.55-2.19 s -> 188-192 ms

The fingerprint of images, remote, jobs and folders after the run is the
same for both builds.
 2026-09-25 22:06:58 -04:00
dtourolle 9e786532a1 Look for orphaned keywords once per word, not once per assignment 
`adopt_orphan_terms` runs in the backfill on every catalog open. Its
check -- is there a vocabulary row for this word, tombstones included --
cannot use `keyword_terms_name`, which is partial on `deleted = 0`, so the
correlated subquery scanned the vocabulary once for each of the 10,800
assignment rows before `DISTINCT` threw the repeats away: 3.5 ms per open
on the reference library.

The distinct words are taken first and the check runs once per word -- a
few dozen scans of a few dozen rows. Same rows out, since `DISTINCT` over
the assignments is exactly the set of words.

catalog_bench, best of 20: 3.45 ms -> 0.30 ms.
 2026-09-25 22:06:58 -04:00
dtourolle 1f26e1e627 Pair RAW and JPEG from the unpaired JPEGs, not from every RAW 
`Catalog::open` runs the backfill every time, and every worker thread
opens its own catalog: the develop view does it to fetch each original and
again for each neighbour it prefetches, and the sync, sweep, burst and
thumbnail workers each do it too. On the reference library (24k images)
an open cost 26 ms of CPU, and most of it was `pair_raw_and_jpeg` reading
all 17,000 RAWs into a map of lowercased stems to find partners for the
1,900 JPEGs that have none -- the same 1,900 on every open.

It now starts from the small side. The unpaired JPEGs are read first, and
it stops there if there are none; otherwise it reads the RAWs in the
folders those JPEGs sit in (plus the unfiled ones when an unfiled JPEG is
waiting), which is 142 on the reference library. A pair is same-folder by
definition, so no pairing is lost; the RAWs are read in id order, so where
two share a stem the later one still wins as it did in the table scan; and
a pass with nothing to pair no longer opens and commits an empty write
transaction.

catalog_bench, best of 20, CPU: `Catalog::open` 26 ms -> 12 ms together
with the next commit (the backfill 24 ms -> 11 ms; this step is ~10 ms of
that). A test covers pairs found among other folders and unfiled images.
 2026-09-25 22:06:58 -04:00
dtourolle 2fd1b0b8ce Read the face shard index once per sync pass, not once per image 
Every sync pass exports this device's faces to the shard store and imports
what peers sent, and both walked the whole library asking the store's index
about one image at a time: the export 19,000 `indexed_at` lookups (one per
face marker), the import 23,000 `held_model` lookups (one per image with a
server id), each a statement prepared and run against the index. With
nothing new either way -- the usual pass -- that was all they did.

Measured with catalog_bench against copies of the reference catalog and
face store, best of 5, CPU:

  export_to_shards (steady)   119 ms ->  27 ms
  import_from_shards (steady) 250 ms ->  87 ms

Each now reads the index in one statement into a map. The import's query
is `held_model`'s, ordered the same way, keeping the first row per file,
and nothing in the loop changes which pipeline a file is held under
(`set_indexed_at` touches only a file already decided; candidates are
distinct files). The export's `put_image_at` does rewrite entries -- but
only its own file's, its generation and the siblings it supersedes -- so a
file already written in this pass is asked of the store again, and every
other answer is the one the lookup would have given. An index that cannot
be read gives an empty map, which is what each failed lookup returned.

The store index and the catalog are identical after the old and new
builds' runs.
 2026-09-25 22:06:58 -04:00
dtourolle 9cff677392 Merge a synced catalog's face assignments without re-preparing per face 
A sync pass that brought nothing new cost 450-540 ms of CPU in
`merge_remote_catalog` on the reference library (24k images, 19k faces),
measured by catalog_bench merging a copy of the catalog with itself.

Most of it was the loop over the other device's confirmed faces and the
faces under its ignored groups -- 13,000 rows. For each one it prepared
three statements from scratch (`query_row`/`execute` with a SQL string
compile the statement every call) and then rewrote the `face_person` row
with the values it already held, dirtying a page per face on every pass.
The rejection loop prepared three more per row.

The statements are now `prepare_cached`, the local assignment is read once
per face (whether it is confirmed, and what it holds, come from the same
row), and the upsert is skipped when the row already says exactly that.
`faces_assigned` is still counted for those rows, so the report is the one
the old code gave, and nothing else reads the difference: the row is
byte-for-byte what the upsert would have written.

After: 279 ms (best of 5, CPU), with every catalog table identical after
the run to the old build's.
 2026-09-25 22:06:58 -04:00
dtourolle 454375243c Measure what opening the catalog, a sync pass and a scan cost on a real library 
Two benches for reading side by side before and after a change, against a
copy of a real catalog, in the manner of identity_bench:

- `dr-catalog --example catalog_bench CATALOG [FACES_DIR]` times
  `Catalog::open` and the backfill inside it step by step, the upload
  snapshot, a merge of the catalog with a copy of itself, and the face
  shard export and import in the steady state where nothing is new.

- `persist_bench`, an ignored test in dr-ui's scan module because
  `persist` and `apply_judgement` are private to it, replays the
  catalog's own rows through `persist` (the largest folder, and the whole
  library) and looks up every `.drsc` sidecar the catalog has read. It
  works on a scratch copy and prints a fingerprint of what `persist` left,
  so two builds can be shown to agree.

Both print best, median and CPU time; the CPU figure is the one to compare
while other builds share the machine.
 2026-09-25 22:06:58 -04:00
dtourolle c559d31dad Size TextRow's field box from the field's height, so its label shows 
Every TextRow drew a field and a hint but no name: "Filename template"
and "Destination" on the settings page and in the export sheet, the two
storage budgets, the export size fields. The label was there, painted
behind the entry, with a clipped "ws" of "Thumbnails and previews"
poking out beside the thumbnail field.

The field sits in a Rectangle so the row can dim it and watch it lose
focus, and that Rectangle took its height from `field.preferred-height`.
`Field` sets its own `height` outright and has no layout inside it, so
its preferred height is zero. The box was zero tall, the row around it
took its height from the unit label beside it, and the field, centred
on an empty box, sat half its own height above the row, on top of the
FieldRow. Segmented never showed it because its chips live in a layout
that reports a real height.

Reading `field.height` instead gives the box the height the field
actually draws at, so the row reserves it and the label sits above. The
note on `Field.label` that recorded the fault now records the trap.
 2026-09-25 20:43:00 -04:00
dtourolle 058286750b Say the develop view skips the readback on Android too
Benchmarks / CPU and I/O (per commit) (push) Successful in 2m4s
Details
Benchmarks / Frame budget (on demand) (push) Skipped
Details
Build and test / Desktop (Linux) (push) Failing after 12h40m50s
Details
Build and test / Layer separation (push) Successful in 42s
Details
Traceability / Requirement traces (push) Successful in 1m28s
Details
🐳 Android image / Build and push (push) Successful in 5s
Details
Build and test / android-image (push) Successful in 5s
Details
🐳 Windows image / Build and push (push) Successful in 2s
Details
Build and test / windows-image (push) Successful in 2s
Details
Build and test / Android (aarch64) (push) Successful in 31m32s
Details
Build and test / Windows (x86_64, cross) (push) Successful in 36m3s
Details
Build and test / Publish the release (push) Skipped
Details 
"On desktop the develop view draws the compute pass's texture directly"
was the README's way of marking Android as the exception. Since TD-1 was
paid off in 0.15.0 the tablet draws the same texture, so the sentence
now names both.
 2026-09-25 19:41:14 -04:00
dtourolle 548caa733c Drop the README's note that Android reads its frame back through the CPU
Benchmarks / CPU and I/O (per commit) (push) Successful in 2m10s
Details
Benchmarks / Frame budget (on demand) (push) Skipped
Details
Build and test / Desktop (Linux) (push) Successful in 46m51s
Details
Build and test / Layer separation (push) Successful in 50s
Details
Traceability / Requirement traces (push) Successful in 31s
Details
🐳 Android image / Build and push (push) Successful in 3s
Details
Build and test / android-image (push) Successful in 3s
Details
🐳 Windows image / Build and push (push) Successful in 1s
Details
Build and test / windows-image (push) Successful in 1s
Details
Build and test / Android (aarch64) (push) Successful in 31m39s
Details
Build and test / Windows (x86_64, cross) (push) Successful in 36m12s
Details
Build and test / Publish the release (push) Skipped
Details 
"Where it stands" still named TD-1 as the one deliberate compromise worth
knowing about: the Android develop view reading its frame back through the
CPU because wgpu's swapchain tore in portrait. 0.15.0 paid that off — the
swapchain is pre-rotated and the frame reaches the compositor as a texture,
as on desktop, and architecture.md §6.1 now reads "No exceptions since
0.15.0". The release commit updated the version line above and left this
paragraph describing a state the release had just ended.
 2026-09-25 19:40:33 -04:00
162 changed files with 19892 additions and 4250 deletions
Expand all files Collapse all files
CLAUDE.md
+33
View File
@@ -31,6 +31,30 @@ over a result set — including `deep_count` per sidebar row, which is fine
at sidebar scale and would not be at grid scale. Aggregate in one at sidebar scale and would not be at grid scale. Aggregate in one
statement and look up in memory. statement and look up in memory.
**SQL text in a loop is a prepare in a loop.** rusqlite's `execute` and
`query_row` compile their statement on every call. A loop that calls them
per row pays a prepare per row even when each query is a primary-key seek:
the merge of a synced catalog prepared four statements for each of 13,000
incoming faces (450 ms of a pass that changed nothing), `persist` four per
photograph a scan listed (1.5 s for a first scan), the shard sync one per
image each way. Hoist the statement, use `prepare_cached`, or — better, when
the loop asks the same table about every row — read that table once into a
map. And do not rewrite a row with what it already holds: an upsert of
identical values still dirties the page.
**A `LIKE` is case-insensitive, and no index here serves that.**
`source_ref LIKE 'stem.%'` read every name of the root per sidecar a pull
took in. When the check that decides is exact, spell the prefix as a range
(`>= 'stem.' AND < 'stem/'`, `/` being the byte after `.`), which the
`(root_id, source_ref)` key answers with a seek.
**`Catalog::open` is not free, and every worker thread calls it.** The
backfill runs on every open, and the develop view opens a catalog to fetch
each original and again for each neighbour it prefetches. Keep each
backfill step's no-op case to a read of the small side — the unpaired
JPEGs, not every RAW; the distinct keywords, not every assignment — and
measure an open with `catalog_bench` after adding one.
**Filter and aggregate in SQL, and aggregate the small side first.** **Filter and aggregate in SQL, and aggregate the small side first.**
`faces::people` read 19,000 rows, grouped, sorted them by name, and the `faces::people` read 19,000 rows, grouped, sorted them by name, and the
screen threw 17,000 away (empty unnamed groups). `people_in_use` filters in screen threw 17,000 away (empty unnamed groups). `people_in_use` filters in
@@ -124,6 +148,15 @@ other builds are running on the machine — the wall clock doubles under
load, the CPU figure does not. Keep the binary from before the change and load, the CPU figure does not. Keep the binary from before the change and
run both back to back rather than trusting numbers taken an hour apart. run both back to back rather than trusting numbers taken an hour apart.
`cargo run --release -p dr-catalog --example catalog_bench -- CATALOG
[FACES_DIR]` does the same for opening the catalog (the backfill step by
step), the upload snapshot, a merge, and the face shard export and import;
`persist_bench`, an ignored test in `dr-ui`'s scan module, replays a scan's
`persist` and a sidecar pull (`DR_BENCH_CATALOG=copy.sqlite cargo test
--release -p dr-ui --lib persist_bench -- --ignored --nocapture
--test-threads=1`). Both take copies; hand `catalog_bench` a copy of the
face store directory too.
Reference figures from the 2026-09-19 fixes, largest person (754 faces), Reference figures from the 2026-09-19 fixes, largest person (754 faces),
24k images, 19k faces, before → after. What one click read: `load_people` 24k images, 19k faces, before → after. What one click read: `load_people`
22 ms → 12 ms, `load_faces` 316 ms → 2.4 ms, `audit` 190 ms → not run 22 ms → 12 ms, `load_faces` 316 ms → 2.4 ms, `audit` 190 ms → not run
CONTRIBUTING.md
+20 -2
View File
@@ -1,6 +1,6 @@
# Contributing to DarkRoom # Contributing to DarkRoom
There is a lot of documentation here — 14 documents and 177 numbered There is a lot of documentation here — twenty-odd documents and 192 numbered
requirements — and almost all of it is written for someone who has already requirements — and almost all of it is written for someone who has already
decided to work on this. This file is the other thing: how to get a first decided to work on this. This file is the other thing: how to get a first
change landed without reading any of it. change landed without reading any of it.
@@ -62,7 +62,7 @@ sudo apt-get install pkg-config libfontconfig1-dev libxkbcommon-dev
cargo run -p darkroom-desktop cargo run -p darkroom-desktop
``` ```
The first build resolves 826 crates and takes a while — on a laptop, long The first build resolves some 850 crates and takes a while — on a laptop, long
enough to look like a hang. It is not one. enough to look like a hang. It is not one.
Android is a containerised toolchain and is not needed for most work; see Android is a containerised toolchain and is not needed for most work; see
@@ -130,6 +130,24 @@ strength of plumbing a future feature would use. So: **close a requirement
with a test that would fail if the behaviour were removed.** Coverage that with a test that would fail if the behaviour were removed.** Coverage that
moves slowly and means something beats coverage that moves quickly. moves slowly and means something beats coverage that moves quickly.
**Keys and gestures are held the same way.** A key handler in Slint compares
one canonical chord, `Keys.chord(event) == "Ctrl+Z"`, under a `// KEYMAP:`
comment naming its section of the gesture book, and every key it binds must be
named by a `GESTURE:` block beside it. `cargo run -p traceability -- gestures`
regenerates [`docs/gestures.md`](docs/gestures.md) and the in-app help sheet
from those blocks; `-- gestures-check` fails when a handler binds a key no tag
names, or a tag names a key no handler binds. A `manual:` field in a block
links the gesture to a section of the manual, and a heading that is not there
fails the scan.
**The manual is checked too.** `docs/manual/index.html` is rendered from
`docs/manual/README.md` by `-- manual` and `-- manual-check` fails when they
differ; `tools/manual/record.sh --check` fails when the manual shows a picture
no scene in `tools/manual/scenes.py` makes. If you change what a pictured
screen looks like, [`tools/manual`](tools/manual/README.md) says how to record
it again. The pre-commit hook regenerates the matrix, the gesture book and the
page; CI runs all three checks.
## Two invariants the build defends ## Two invariants the build defends
Worth knowing before you trip one, because both failures name a requirement Worth knowing before you trip one, because both failures name a requirement
Cargo.lock
Generated
+182 -26
View File
@@ -347,6 +347,28 @@ dependencies = [
"libloading", "libloading",
] ]
[[package]]
name = "ashpd"
version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d2f3f79755c74fd155000314eb349864caa787c6592eace6c6882dad873d9c39"
dependencies = [
"async-fs",
"async-net",
"enumflags2",
"futures-channel",
"futures-util",
"rand 0.9.5",
"raw-window-handle",
"serde",
"serde_repr",
"url",
"wayland-backend",
"wayland-client",
"wayland-protocols",
"zbus",
]
[[package]] [[package]]
name = "async-broadcast" name = "async-broadcast"
version = "0.7.2" version = "0.7.2"
@@ -385,6 +407,17 @@ dependencies = [
"slab", "slab",
] ]
[[package]]
name = "async-fs"
version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8034a681df4aed8b8edbd7fbe472401ecf009251c8b40556b304567052e294c5"
dependencies = [
"async-lock",
"blocking",
"futures-lite",
]
[[package]] [[package]]
name = "async-io" name = "async-io"
version = "2.6.0" version = "2.6.0"
@@ -414,6 +447,17 @@ dependencies = [
"pin-project-lite", "pin-project-lite",
] ]
[[package]]
name = "async-net"
version = "2.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b948000fad4873c1c9339d60f2623323a0cfd3816e5181033c6a5cb68b2accf7"
dependencies = [
"async-io",
"blocking",
"futures-lite",
]
[[package]] [[package]]
name = "async-process" name = "async-process"
version = "2.5.0" version = "2.5.0"
@@ -1221,7 +1265,7 @@ checksum = "f27ae1dd37df86211c42e150270f82743308803d90a6f6e6651cd730d5e1732f"
[[package]] [[package]]
name = "darkroom-android" name = "darkroom-android"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"android_logger", "android_logger",
"dr-plat", "dr-plat",
@@ -1234,7 +1278,7 @@ dependencies = [
[[package]] [[package]]
name = "darkroom-desktop" name = "darkroom-desktop"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"dr-plat", "dr-plat",
@@ -1356,6 +1400,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38" checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38"
dependencies = [ dependencies = [
"bitflags 2.13.1", "bitflags 2.13.1",
"block2 0.6.2",
"libc",
"objc2 0.6.4", "objc2 0.6.4",
] ]
@@ -1408,7 +1454,7 @@ checksum = "d8b14ccef22fc6f5a8f4d7d768562a182c04ce9a3b3157b91390b52ddfdf1a76"
[[package]] [[package]]
name = "dr-bench" name = "dr-bench"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"dr-catalog", "dr-catalog",
@@ -1425,7 +1471,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-catalog" name = "dr-catalog"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-face", "dr-face",
"dr-plat", "dr-plat",
@@ -1440,7 +1486,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-decode" name = "dr-decode"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-types", "dr-types",
"env_logger", "env_logger",
@@ -1454,7 +1500,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-export" name = "dr-export"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-decode", "dr-decode",
"dr-gpu", "dr-gpu",
@@ -1473,7 +1519,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-face" name = "dr-face"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-inference-engine", "dr-inference-engine",
"env_logger", "env_logger",
@@ -1486,7 +1532,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-film" name = "dr-film"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"log", "log",
"serde", "serde",
@@ -1495,7 +1541,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-gpu" name = "dr-gpu"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"bytemuck", "bytemuck",
"dr-decode", "dr-decode",
@@ -1513,7 +1559,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-inference-engine" name = "dr-inference-engine"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"env_logger", "env_logger",
"libloading", "libloading",
@@ -1528,7 +1574,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-ingest" name = "dr-ingest"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-plat", "dr-plat",
"dr-types", "dr-types",
@@ -1540,7 +1586,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-lens" name = "dr-lens"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"lensfun", "lensfun",
"log", "log",
@@ -1548,7 +1594,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-pano" name = "dr-pano"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-decode", "dr-decode",
"dr-inference-engine", "dr-inference-engine",
@@ -1562,7 +1608,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-pipeline" name = "dr-pipeline"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-types", "dr-types",
"log", "log",
@@ -1571,7 +1617,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-plat" name = "dr-plat"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"android-native-keyring-store", "android-native-keyring-store",
"dr-types", "dr-types",
@@ -1587,7 +1633,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-preset-xmp" name = "dr-preset-xmp"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-pipeline", "dr-pipeline",
"log", "log",
@@ -1597,7 +1643,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-segment" name = "dr-segment"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-inference-engine", "dr-inference-engine",
"env_logger", "env_logger",
@@ -1610,7 +1656,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-sync" name = "dr-sync"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"dr-plat", "dr-plat",
@@ -1624,7 +1670,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-sync-folder" name = "dr-sync-folder"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"dr-sync", "dr-sync",
@@ -1636,7 +1682,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-sync-nextcloud" name = "dr-sync-nextcloud"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"dr-decode", "dr-decode",
@@ -1658,7 +1704,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-thumbs" name = "dr-thumbs"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-types", "dr-types",
"jpeg-encoder", "jpeg-encoder",
@@ -1670,7 +1716,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-types" name = "dr-types"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"serde", "serde",
"serde_json", "serde_json",
@@ -1679,7 +1725,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-ui" name = "dr-ui"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"async-trait", "async-trait",
@@ -1710,10 +1756,13 @@ dependencies = [
"ndk-context", "ndk-context",
"png", "png",
"pollster", "pollster",
"raw-window-handle",
"reqwest", "reqwest",
"rfd",
"rusqlite", "rusqlite",
"serde_json", "serde_json",
"serde_norway", "serde_norway",
"sha2",
"slint", "slint",
"slint-build", "slint-build",
"thiserror 2.0.20", "thiserror 2.0.20",
@@ -1724,7 +1773,7 @@ dependencies = [
[[package]] [[package]]
name = "dr-xmp" name = "dr-xmp"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"dr-types", "dr-types",
"log", "log",
@@ -5667,6 +5716,30 @@ dependencies = [
"zune-jpeg 0.5.15", "zune-jpeg 0.5.15",
] ]
[[package]]
name = "rfd"
version = "0.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a15ad77d9e70a92437d8f74c35d99b4e4691128df018833e99f90bcd36152672"
dependencies = [
"ashpd",
"block2 0.6.2",
"dispatch2",
"js-sys",
"log",
"objc2 0.6.4",
"objc2-app-kit 0.3.2",
"objc2-core-foundation",
"objc2-foundation 0.3.2",
"pollster",
"raw-window-handle",
"urlencoding",
"wasm-bindgen",
"wasm-bindgen-futures",
"web-sys",
"windows-sys 0.60.2",
]
[[package]] [[package]]
name = "rgb" name = "rgb"
version = "0.8.53" version = "0.8.53"
@@ -6285,6 +6358,7 @@ dependencies = [
"num-traits", "num-traits",
"once_cell", "once_cell",
"pin-weak", "pin-weak",
"raw-window-handle",
"slint-macros", "slint-macros",
"unicode-segmentation", "unicode-segmentation",
"vtable", "vtable",
@@ -7035,7 +7109,7 @@ checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3"
[[package]] [[package]]
name = "traceability" name = "traceability"
version = "0.15.0" version = "0.18.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"pulldown-cmark", "pulldown-cmark",
@@ -7446,8 +7520,15 @@ dependencies = [
"idna", "idna",
"percent-encoding", "percent-encoding",
"serde", "serde",
"serde_derive",
] ]
[[package]]
name = "urlencoding"
version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da"
[[package]] [[package]]
name = "usvg" name = "usvg"
version = "0.47.0" version = "0.47.0"
@@ -8168,6 +8249,15 @@ dependencies = [
"windows-targets 0.52.6", "windows-targets 0.52.6",
] ]
[[package]]
name = "windows-sys"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb"
dependencies = [
"windows-targets 0.53.5",
]
[[package]] [[package]]
name = "windows-sys" name = "windows-sys"
version = "0.61.2" version = "0.61.2"
@@ -8216,13 +8306,30 @@ dependencies = [
"windows_aarch64_gnullvm 0.52.6", "windows_aarch64_gnullvm 0.52.6",
"windows_aarch64_msvc 0.52.6", "windows_aarch64_msvc 0.52.6",
"windows_i686_gnu 0.52.6", "windows_i686_gnu 0.52.6",
"windows_i686_gnullvm", "windows_i686_gnullvm 0.52.6",
"windows_i686_msvc 0.52.6", "windows_i686_msvc 0.52.6",
"windows_x86_64_gnu 0.52.6", "windows_x86_64_gnu 0.52.6",
"windows_x86_64_gnullvm 0.52.6", "windows_x86_64_gnullvm 0.52.6",
"windows_x86_64_msvc 0.52.6", "windows_x86_64_msvc 0.52.6",
] ]
[[package]]
name = "windows-targets"
version = "0.53.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3"
dependencies = [
"windows-link",
"windows_aarch64_gnullvm 0.53.1",
"windows_aarch64_msvc 0.53.1",
"windows_i686_gnu 0.53.1",
"windows_i686_gnullvm 0.53.1",
"windows_i686_msvc 0.53.1",
"windows_x86_64_gnu 0.53.1",
"windows_x86_64_gnullvm 0.53.1",
"windows_x86_64_msvc 0.53.1",
]
[[package]] [[package]]
name = "windows-threading" name = "windows-threading"
version = "0.2.1" version = "0.2.1"
@@ -8250,6 +8357,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53"
[[package]] [[package]]
name = "windows_aarch64_msvc" name = "windows_aarch64_msvc"
version = "0.42.2" version = "0.42.2"
@@ -8268,6 +8381,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_aarch64_msvc"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006"
[[package]] [[package]]
name = "windows_i686_gnu" name = "windows_i686_gnu"
version = "0.42.2" version = "0.42.2"
@@ -8286,12 +8405,24 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
[[package]]
name = "windows_i686_gnu"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3"
[[package]] [[package]]
name = "windows_i686_gnullvm" name = "windows_i686_gnullvm"
version = "0.52.6" version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_gnullvm"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c"
[[package]] [[package]]
name = "windows_i686_msvc" name = "windows_i686_msvc"
version = "0.42.2" version = "0.42.2"
@@ -8310,6 +8441,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_i686_msvc"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2"
[[package]] [[package]]
name = "windows_x86_64_gnu" name = "windows_x86_64_gnu"
version = "0.42.2" version = "0.42.2"
@@ -8328,6 +8465,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnu"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499"
[[package]] [[package]]
name = "windows_x86_64_gnullvm" name = "windows_x86_64_gnullvm"
version = "0.42.2" version = "0.42.2"
@@ -8346,6 +8489,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1"
[[package]] [[package]]
name = "windows_x86_64_msvc" name = "windows_x86_64_msvc"
version = "0.42.2" version = "0.42.2"
@@ -8364,6 +8513,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "windows_x86_64_msvc"
version = "0.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650"
[[package]] [[package]]
name = "winit" name = "winit"
version = "0.30.13" version = "0.30.13"
@@ -8829,6 +8984,7 @@ dependencies = [
"endi", "endi",
"enumflags2", "enumflags2",
"serde", "serde",
"url",
"winnow 1.0.4", "winnow 1.0.4",
"zvariant_derive", "zvariant_derive",
"zvariant_utils", "zvariant_utils",
Cargo.toml
+1 -1
View File
@@ -32,7 +32,7 @@ members = [
exclude = ["third_party"] exclude = ["third_party"]
[workspace.package] [workspace.package]
version = "0.15.0" version = "0.18.0"
edition = "2021" edition = "2021"
rust-version = "1.92" rust-version = "1.92"
license = "GPL-3.0-or-later" license = "GPL-3.0-or-later"
README.md
+38 -26
View File
@@ -15,19 +15,27 @@ is still missing.
**A library.** Point it at a folder — on this machine, on a network mount, **A library.** Point it at a folder — on this machine, on a network mount,
or one a Nextcloud client keeps in virtual-files mode, where a placeholder or one a Nextcloud client keeps in virtual-files mode, where a placeholder
is treated as the photograph rather than as a one-byte file — or at a is treated as the photograph rather than as a one-byte file — or at a
Nextcloud account directly. The grid is virtualised, ordered by capture Nextcloud account directly; a photograph that is only on the server opens on
time with a timeline beside it, and filtered by rating, flag, person and its thumbnail with the download's progress over it. The grid is virtualised,
whether the file is here. Ratings, keywords, collections and a ordered by capture time with a timeline beside it, and filtered by rating,
trash that survives a crash mid-operation. Card ingest. Bursts fold. Face flag, colour label, person and whether the file is here. Ratings, colour
detection and identity, with the index syncing between devices. labels, keywords, collections and a trash that survives a crash
mid-operation. Card ingest. Bursts fold. The same RAW catalogued twice — a
dated folder and a backup beside it — is found, proved the same, and folded
onto one copy with the spares in the trash. Face detection and identity,
with the index syncing between devices.
**Developing.** Eighteen declared operations fused into one compute **Developing.** Eighteen declared operations fused into one compute
dispatch, plus the neighbourhood work that cannot be: clarity, texture, dispatch, plus the neighbourhood work that cannot be: clarity, texture,
capture sharpening, noise reduction, lens correction, spectral film capture sharpening, noise reduction, lens correction, spectral film
simulation. Crop and straighten, spot repair, and local adjustments over simulation. Crop, straighten and correct converging verticals, spot repair,
masks the model draws — click a subject or a category, then paint, subtract and local adjustments over masks the model draws — click a subject or a
a gradient, grow or shrink the edge. Focus peaking and a raw histogram for category, then paint, subtract a gradient or keep only where two selections
judging what is recoverable. Named presets; XMP sidecars other editors read. agree, grow or shrink the edge. Focus peaking and a raw histogram for judging
what is recoverable. Presets, with a collection shipped in the application —
everyday corrections, and a look for each measured colour, cinema and
black-and-white stock — and Lightroom presets imported as looks that leave a
photograph's own corrections alone. XMP sidecars other editors read.
[![Segmenting an urban scene and choosing the sky as a mask](docs/manual/media/local-segment.png)](docs/manual/README.md#local-adjustments) [![Segmenting an urban scene and choosing the sky as a mask](docs/manual/media/local-segment.png)](docs/manual/README.md#local-adjustments)
@@ -37,13 +45,22 @@ sources as a DNG, with a sidecar recording what it was merged from.
[![Twelve hand-held frames aligned on a cylinder](docs/manual/media/panorama-aligned.png)](docs/manual/README.md#merging-a-panorama) [![Twelve hand-held frames aligned on a cylinder](docs/manual/media/panorama-aligned.png)](docs/manual/README.md#merging-a-panorama)
**From the keyboard, and with its manual.** Rating, flagging and labelling
have keys in the grid and in develop, as do zoom, undo and stepping through a
shoot in develop, and none of them is keyboard-only. The help sheet (`F1`, or
`?` in develop) lists every key and gesture, generated from the code that
binds it, and links them to the sections of the manual that show them — the
manual ships with the application and opens offline.
**Export.** JPEG, PNG, AVIF, JPEG XL, 8- and 16-bit TIFF, with resize, output **Export.** JPEG, PNG, AVIF, JPEG XL, 8- and 16-bit TIFF, with resize, output
sharpening, a naming template and a colour space — to a folder here or back sharpening, a naming template and a colour space — into albums: named export
into the library. folders on this machine or on the server, never inside the library, which
remember the photograph behind each file and sync between devices as
collections do.
**On both platforms.** The same core runs on a desktop and a 12-inch **On both platforms.** The same core runs on a desktop and a 12-inch
tablet; the interface is one layout, tuned for a wide viewport with touch tablet; the interface is one layout, tuned for a wide viewport with touch
targets throughout. On desktop the develop view draws the compute pass's targets throughout. On both, the develop view draws the compute pass's
texture directly — no readback between the GPU and the screen. texture directly — no readback between the GPU and the screen.
## Getting it ## Getting it
@@ -53,7 +70,7 @@ texture directly — no readback between the GPU and the screen.
| Arch Linux | [`packaging/PKGBUILD`](packaging/PKGBUILD) — `makepkg -si` | Built from every release | | Arch Linux | [`packaging/PKGBUILD`](packaging/PKGBUILD) — `makepkg -si` | Built from every release |
| Android | The APK from each CI run, or `./docker/android/package.sh --install` | Runs on a tablet; F-Droid not yet submitted | | Android | The APK from each CI run, or `./docker/android/package.sh --install` | Runs on a tablet; F-Droid not yet submitted |
| Windows | `DarkRoom-<version>-x86_64-setup.exe`, cross-built by CI ([windows.md](docs/dev/windows.md)) | Verified under Wine only; unsigned | | Windows | `DarkRoom-<version>-x86_64-setup.exe`, cross-built by CI ([windows.md](docs/dev/windows.md)) | Verified under Wine only; unsigned |
| Flatpak | [`packaging/flatpak/`](packaging/flatpak/) | Manifest in tree; choosing a library does not yet work in the sandbox | | Flatpak | [`packaging/flatpak/`](packaging/flatpak/) | Manifest in tree; folders are chosen through the portal, but no Flatpak has been built to prove it |
Or build it. Git LFS is required for the model weights, and the toolchain Or build it. Git LFS is required for the model weights, and the toolchain
pins itself to 1.92.0: pins itself to 1.92.0:
@@ -76,27 +93,22 @@ controls, its place in the chain and its tests.
## Where it stands ## Where it stands
**0.15.0**, twenty-three tagged releases in. 190 numbered requirements in **0.18.0**, twenty-six tagged releases in. 192 numbered requirements in
scope, 84% of them claimed by code and [traced to it](docs/dev/traceability.md); scope, 84% of them claimed by code and [traced to it](docs/dev/traceability.md);
the rest are written down rather than merely absent. the rest are written down rather than merely absent.
**Not built:** plugins (post-v1, [D12](docs/dev/requirements.md)), compare and **Not built:** plugins (post-v1, [D12](docs/dev/requirements.md)), compare and
survey culling, AI denoise, tiled rendering, HDR merge and survey culling, AI denoise, tiled rendering, HDR merge and
focus stacking, most of the Android platform integration beyond running, focus stacking, importing a Lightroom or darktable catalog, translations
and the Flatpak's library chooser. The performance targets are half beyond the launch screen, most of the Android platform integration beyond
verified: the per-commit benchmark suite §8 requires exists for everything running, and a Flatpak actually built and run in its sandbox. The
that does not need a frame — the catalog, the scan, the thumbnails — and performance targets are half verified: the per-commit benchmark suite §8
not yet for the render path, so a regression there fails nothing. requires exists for everything that does not need a frame — the catalog,
the scan, the thumbnails — and not yet for the render path, so a regression
there fails nothing.
[outstanding.md](docs/dev/outstanding.md) is the list, with the reasoning for [outstanding.md](docs/dev/outstanding.md) is the list, with the reasoning for
each. each.
**The one deliberate compromise worth knowing about before reading
anything else:** the Android develop view reads its frame back through the
CPU, because zero-copy there needs wgpu's Vulkan swapchain and that tears a
portrait window on a tablet whose panel is mounted landscape. It is debt,
not a revision of the rule — [technical-debt.md TD-1](docs/dev/technical-debt.md)
has the measurements and the three things any one of which would remove it.
## Documentation ## Documentation
[docs/README.md](docs/README.md) is the index. The short version, for someone using it: [docs/README.md](docs/README.md) is the index. The short version, for someone using it:
apps/darkroom-android/android/AndroidManifest.xml
+12
View File
@@ -158,6 +158,18 @@
android:theme="@style/ManualTheme" android:theme="@style/ManualTheme"
android:configChanges="orientation|keyboardHidden|screenSize|screenLayout|uiMode" /> android:configChanges="orientation|keyboardHidden|screenSize|screenLayout|uiMode" />
<!-- FR-EXP-10: the system's folder picker, for an album's folder on
this device. NativeActivity's onActivityResult is not ours, so
this activity exists only to ask and hand the answer back (see
FolderPicker.java). Translucent and without a title so nothing
of it shows but the system chooser; not exported, and started by
class name from dr_ui::saf. -->
<activity
android:name="paris.tourolle.darkroom.FolderPicker"
android:exported="false"
android:theme="@android:style/Theme.Translucent.NoTitleBar"
android:configChanges="orientation|keyboardHidden|screenSize|screenLayout|uiMode" />
<!-- FR-PLAT-AND-6, outbound. Android has refused file:// URIs <!-- FR-PLAT-AND-6, outbound. Android has refused file:// URIs
between apps since API 24 — handing one out raises between apps since API 24 — handing one out raises
FileUriExposedException in *this* process — so an exported JPEG FileUriExposedException in *this* process — so an exported JPEG
apps/darkroom-android/android/java/paris/tourolle/darkroom/FolderPicker.java
+117
View File
@@ -0,0 +1,117 @@
package paris.tourolle.darkroom;
import android.app.Activity;
import android.content.ActivityNotFoundException;
import android.content.Context;
import android.content.Intent;
import android.net.Uri;
import android.os.Bundle;
import android.util.Log;
/**
* The system's folder picker, for an album's folder on this device (FR-EXP-10).
*
* <h2>Why an activity of its own</h2>
*
* <p>{@code ACTION_OPEN_DOCUMENT_TREE} answers through
* {@code onActivityResult}, and the main activity is {@code NativeActivity},
* whose result callback is not ours to override. So this one exists only to
* ask: it starts the picker, takes the answer, and finishes — no layout, a
* translucent theme, nothing on screen but the system's own chooser, which has
* its own "New folder".
*
* <p>The answer is left in a static for Rust to poll ({@link #poll}), rather
* than called back into native code: a callback would need a registered
* native method and a thread to deliver on, and a poll from the Slint timer
* that is already running is one static call.
*
* <h2>The grant</h2>
*
* <p>A tree URI is usable only while its permission is held, and a plain
* result grants it until the process dies. {@code takePersistableUriPermission}
* keeps it across restarts — an album's folder is chosen once and exported to
* for months.
*/
public final class FolderPicker extends Activity {
private static final String TAG = "DarkRoom";
private static final int REQUEST = 0x5AF;
/** The last answer: a tree URI, "" for a cancel, null while none has come. */
private static volatile String answer = null;
/**
* Start asking. Clears any answer left from before.
*
* <p>Takes a {@code Context} rather than an {@code Activity}, because what
* native code holds (ndk_context's handle) is the application context,
* and starting an activity from one that is not an activity needs
* {@code FLAG_ACTIVITY_NEW_TASK} — without it the call throws. The picker
* shares the app's task affinity, so it still opens over the app and Back
* still returns to it.
*/
public static void start(Context from) {
answer = null;
Intent intent = new Intent(from, FolderPicker.class);
if (!(from instanceof Activity)) {
intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
}
from.startActivity(intent);
}
/**
* The answer, once: a tree URI, "" if the user backed out, or null while
* the picker is still open. Reading it clears it, so a second poll after a
* cancel does not see the cancel again.
*/
public static String poll() {
String a = answer;
if (a != null) {
answer = null;
}
return a;
}
@Override
protected void onCreate(Bundle state) {
super.onCreate(state);
// Recreated after a rotation with the picker already up: asking again
// would stack a second chooser over the first.
if (state != null) {
return;
}
Intent pick = new Intent(Intent.ACTION_OPEN_DOCUMENT_TREE);
pick.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION
| Intent.FLAG_GRANT_WRITE_URI_PERMISSION
| Intent.FLAG_GRANT_PERSISTABLE_URI_PERMISSION);
try {
startActivityForResult(pick, REQUEST);
} catch (ActivityNotFoundException e) {
Log.w(TAG, "no folder picker on this device", e);
answer = "";
finish();
}
}
@Override
protected void onActivityResult(int request, int result, Intent data) {
if (request != REQUEST) {
return;
}
Uri tree = (result == RESULT_OK && data != null) ? data.getData() : null;
if (tree == null) {
answer = "";
} else {
try {
getContentResolver().takePersistableUriPermission(tree,
Intent.FLAG_GRANT_READ_URI_PERMISSION
| Intent.FLAG_GRANT_WRITE_URI_PERMISSION);
} catch (SecurityException e) {
// Still usable this session; said in the log so a folder that
// stops working after a restart has an explanation.
Log.w(TAG, "the folder grant could not be kept: " + tree, e);
}
answer = tree.toString();
}
finish();
}
}
apps/darkroom-android/android/java/paris/tourolle/darkroom/Saf.java
+114
View File
@@ -0,0 +1,114 @@
package paris.tourolle.darkroom;
import android.content.ContentResolver;
import android.content.Context;
import android.database.Cursor;
import android.net.Uri;
import android.provider.DocumentsContract;
import android.util.Log;
import java.io.IOException;
import java.io.OutputStream;
/**
* Writing an export into a folder the user granted through
* {@link FolderPicker} — the Storage Access Framework, which is the only way
* this app reaches a folder on the device (FR-PLAT-AND-1).
*
* <p>A tree URI is not a path: a child is found by listing the folder and
* matching its display name, and created through the provider, which may
* rename it on a collision. So the name that was actually written is handed
* back, and the album records that one.
*
* <p>Two static calls, strings and a byte array in, a string out, for the
* reason {@link Intents} gives: every call here would be a signature typed as
* a string on the Rust side, and the fewer of those the better.
*/
public final class Saf {
private static final String TAG = "DarkRoom";
private Saf() {
}
/** Whether {@code name} already exists in the folder. False on any error. */
public static boolean exists(Context context, String tree, String name) {
try {
return find(context.getContentResolver(), Uri.parse(tree), name) != null;
} catch (RuntimeException e) {
Log.w(TAG, "checking " + name + " in " + tree, e);
return false;
}
}
/**
* Write {@code bytes} as {@code name} in the folder, replacing a file of
* that name when {@code replace} is set.
*
* @return the name the file has in the folder — the provider may have
* added " (1)" — or null on failure, with the reason in the log.
*/
public static String write(Context context, String tree, String name, String mime,
byte[] bytes, boolean replace) {
ContentResolver resolver = context.getContentResolver();
Uri treeUri = Uri.parse(tree);
try {
Uri target = replace ? find(resolver, treeUri, name) : null;
if (target == null) {
Uri folder = DocumentsContract.buildDocumentUriUsingTree(treeUri,
DocumentsContract.getTreeDocumentId(treeUri));
target = DocumentsContract.createDocument(resolver, folder, mime, name);
}
if (target == null) {
Log.w(TAG, "the folder refused to create " + name + " in " + tree);
return null;
}
// "wt": truncate. A replacement shorter than what it replaces
// must not keep the old file's tail.
try (OutputStream out = resolver.openOutputStream(target, "wt")) {
if (out == null) {
Log.w(TAG, "no stream for " + target);
return null;
}
out.write(bytes);
}
String written = displayName(resolver, target);
return written != null ? written : name;
} catch (IOException | RuntimeException e) {
Log.w(TAG, "writing " + name + " to " + tree, e);
return null;
}
}
/** The document for {@code name} directly in the tree's folder, or null. */
private static Uri find(ContentResolver resolver, Uri tree, String name) {
String folderId = DocumentsContract.getTreeDocumentId(tree);
Uri children = DocumentsContract.buildChildDocumentsUriUsingTree(tree, folderId);
String[] columns = {
DocumentsContract.Document.COLUMN_DOCUMENT_ID,
DocumentsContract.Document.COLUMN_DISPLAY_NAME,
};
try (Cursor c = resolver.query(children, columns, null, null, null)) {
if (c == null) {
return null;
}
while (c.moveToNext()) {
if (name.equals(c.getString(1))) {
return DocumentsContract.buildDocumentUriUsingTree(tree, c.getString(0));
}
}
}
return null;
}
private static String displayName(ContentResolver resolver, Uri document) {
String[] columns = {DocumentsContract.Document.COLUMN_DISPLAY_NAME};
try (Cursor c = resolver.query(document, columns, null, null, null)) {
if (c != null && c.moveToFirst()) {
return c.getString(0);
}
} catch (RuntimeException e) {
Log.w(TAG, "reading the name of " + document, e);
}
return null;
}
}
core/dr-catalog/examples/catalog_bench.rs
+294
View File
@@ -0,0 +1,294 @@
//! What the catalog's routine reads cost on a real library, off the GUI.
//!
//! cargo run --release -p dr-catalog --example catalog_bench -- CATALOG.sqlite [FACES_DIR] [--remote PEER.sqlite]
//!
//! Times `Catalog::open` — which every worker thread pays, including the
//! develop view's fetch of each original and each neighbour it prefetches —
//! and the backfill that runs inside it, step by step. Run it against a
//! *copy* of a real catalog: opening migrates and backfills, which write.
//!
//! Then what the library screen reads on every keystroke and scroll: the
//! filter chips' counts, the keyword panel, and the grid's total. Two of
//! those live in `dr-ui` (`library::local_original_count` and the grid
//! count), whose `library` module is private; their SQL is spelled here as
//! it is spelled there, and has to be kept in step by hand.
//!
//! `--remote` also times a merge with another device's catalog — the
//! server snapshot — which is the pass where the two disagree: faces one
//! side found and the other did not, boxes that moved. The merge with a copy
//! of itself matches every face by its box and never reaches that work. The
//! first of its runs writes what the peer brought; the rest are the steady
//! state, so compare two builds from two fresh copies of one catalog.
//!
//! The figures are for reading side by side before and after a change; they
//! are not a gate. Compare the `cpu` column when the machine is busy. The
//! answers are printed too, so two builds can be checked for agreeing.
use std::path::PathBuf;
use std::time::{Duration, Instant};
use dr_catalog::{keywords, rating, schema, Catalog};
fn main() {
let mut args: Vec<String> = std::env::args().skip(1).collect();
let peer = args.iter().position(|a| a == "--remote").map(|at| {
let path = args.get(at + 1).map(PathBuf::from).unwrap_or_else(|| {
eprintln!("--remote needs a catalog");
std::process::exit(2);
});
args.drain(at..at + 2);
path
});
let Some(path) = args.first().map(PathBuf::from) else {
eprintln!("usage: catalog_bench CATALOG.sqlite");
std::process::exit(2);
};
// Once untimed, so a migration or a first backfill is not in the figures.
drop(Catalog::open(&path).expect("catalog"));
time("Catalog::open", 20, || {
drop(Catalog::open(&path).unwrap());
});
// One develop landing, in the two shapes the app has had. Five opens is
// what `fetch_original` and a `holds_original` per prefetched neighbour
// cost when each asked on a connection of its own; two is the fetch plus
// one connection the prefetch worker keeps for its batch's row checks.
// Five images from the library, against an empty cache: the question is
// asked the same way whatever the answer.
let images: Vec<dr_types::ImageId> = {
let c = Catalog::open(&path).unwrap();
let mut stmt = c
.connection()
.prepare("SELECT id FROM images ORDER BY id LIMIT 5 OFFSET 1000")
.unwrap();
let ids = stmt
.query_map([], |r| r.get::<_, i64>(0))
.unwrap()
.map(|id| dr_types::ImageId(id.unwrap() as u64))
.collect();
ids
};
let cache_dir = path.with_extension("bench-cache");
let budget = dr_catalog::Budget::default();
time("landing: 5 opens (fetch + 4 row checks)", 20, || {
let store = dr_catalog::Cache::open(&cache_dir, budget).unwrap();
let c = Catalog::open(&path).unwrap();
let _ = store.load(c.connection(), images[0], 0).unwrap();
for &image in &images[1..] {
let store = dr_catalog::Cache::open(&cache_dir, budget).unwrap();
let c = Catalog::open(&path).unwrap();
let _ = store.holds_original(c.connection(), image);
}
});
time("landing: 2 opens (fetch + held row checks)", 20, || {
let store = dr_catalog::Cache::open(&cache_dir, budget).unwrap();
let c = Catalog::open(&path).unwrap();
let _ = store.load(c.connection(), images[0], 0).unwrap();
let held = Catalog::open(&path).unwrap();
for &image in &images[1..] {
let store = dr_catalog::Cache::open(&cache_dir, budget).unwrap();
let _ = store.holds_original(held.connection(), image);
}
});
let _ = std::fs::remove_dir_all(&cache_dir);
let catalog = Catalog::open(&path).unwrap();
let conn = catalog.connection();
time("schema::backfill (all steps)", 20, || {
schema::backfill(conn).unwrap();
});
time(" rating::ensure_default_versions", 20, || {
rating::ensure_default_versions(conn).unwrap();
});
time(" rating::align_default_version_uuids", 20, || {
rating::align_default_version_uuids(conn).unwrap();
});
time(" keywords::adopt_orphan_terms", 20, || {
keywords::adopt_orphan_terms(conn).unwrap();
});
interactive(conn);
// A sync pass: the upload snapshot, then a merge of the catalog with a
// copy of itself — every row a match, which is the steady state.
let scratch = path.with_extension("bench-snapshot");
time("snapshot_for_upload", 3, || {
let _ = std::fs::remove_file(&scratch);
catalog.snapshot_for_upload(&scratch).unwrap();
});
println!(
" snapshot size {:.1} MB",
std::fs::metadata(&scratch).map(|m| m.len()).unwrap_or(0) as f64 / 1e6
);
let remote = path.with_extension("bench-remote");
let _ = std::fs::remove_file(&remote);
conn.execute("VACUUM INTO ?1", [remote.to_string_lossy().as_ref()])
.unwrap();
time("merge_remote_catalog (self)", 5, || {
catalog.merge_remote_catalog(&remote).unwrap();
});
let _ = std::fs::remove_file(&scratch);
let _ = std::fs::remove_file(&remote);
if let Some(peer) = &peer {
// A copy, so nothing the merge does to its input reaches the file
// the caller named.
std::fs::copy(peer, &remote).unwrap();
let mut first = None;
time("merge_remote_catalog (--remote)", 5, || {
let report = catalog.merge_remote_catalog(&remote).unwrap();
first.get_or_insert(report);
});
println!(" first pass: {first:?}");
let _ = std::fs::remove_file(&remote);
}
// The face half of a sync pass, against a copy of the face store: both
// directions in the steady state, where nothing is new either way.
if let Some(faces) = args.get(1).map(PathBuf::from) {
let model = "scrfd_10g+w600k_mbf";
let mut store = dr_catalog::FaceShardStore::open(&faces).unwrap();
println!(
" first export sent {}, first import adopted {}",
dr_catalog::face_shard::export_to_shards(conn, &mut store, model).unwrap(),
dr_catalog::face_shard::import_from_shards(conn, &store, model).unwrap()
);
time("face_shard::export_to_shards (steady)", 5, || {
dr_catalog::face_shard::export_to_shards(conn, &mut store, model).unwrap();
});
time("face_shard::import_from_shards (steady)", 5, || {
dr_catalog::face_shard::import_from_shards(conn, &store, model).unwrap();
});
}
}
/// What one click in the library reads: a rating or label keystroke
/// refreshes the chips, a selection change redraws the keyword panel, and
/// every scroll reload counts the grid.
fn interactive(conn: &rusqlite::Connection) {
println!(
" rating_histogram {:?}, label_histogram {:?}, local originals {}, grid {} / rated {}",
rating::rating_histogram(conn).unwrap(),
rating::label_histogram(conn).unwrap(),
local_original_count(conn),
grid_count(conn, ""),
grid_count(conn, RATED_AT_LEAST_ONE),
);
let words = keywords::list(conn).unwrap();
println!(
" keywords::list {} terms, digest {:016x}",
words.len(),
digest(&format!("{words:?}"))
);
// A selection the size of a grid window, from the start of the library.
let selection: Vec<dr_types::ImageId> = conn
.prepare("SELECT id FROM images ORDER BY id LIMIT 120")
.unwrap()
.query_map([], |r| Ok(dr_types::ImageId(r.get::<_, i64>(0)? as u64)))
.unwrap()
.collect::<Result<_, _>>()
.unwrap();
println!(
" keywords::for_images digest {:016x}",
digest(&format!(
"{:?}",
keywords::for_images(conn, &selection).unwrap()
))
);
time("rating::rating_histogram", 50, || {
rating::rating_histogram(conn).unwrap();
});
time("library::local_original_count", 50, || {
local_original_count(conn);
});
time("rating::label_histogram", 50, || {
rating::label_histogram(conn).unwrap();
});
time("keywords::list", 50, || {
keywords::list(conn).unwrap();
});
time("keywords::for_images (120)", 50, || {
keywords::for_images(conn, &selection).unwrap();
});
time("grid count", 50, || {
grid_count(conn, "");
});
time("grid count, rated >= 1", 50, || {
grid_count(conn, RATED_AT_LEAST_ONE);
});
}
/// `dr_ui::library::local_original_count`, spelled as it is there.
fn local_original_count(conn: &rusqlite::Connection) -> i64 {
conn.query_row(
"SELECT count(*) FROM images i
WHERE i.shadowed_by IS NULL AND i.trashed_at IS NULL
AND i.id IN (SELECT ic.image_id FROM image_cache ic
WHERE ic.tier_actual >= 2)",
[],
|r| r.get(0),
)
.unwrap()
}
/// `RatingFilter::sql` for one star and up.
const RATED_AT_LEAST_ONE: &str = " AND coalesce((SELECT dv.rating FROM versions dv
WHERE dv.image_id = i.id AND dv.is_default = 1
LIMIT 1), 0) >= 1";
/// `dr_ui::library::total_images_filtered`, spelled as it is there.
fn grid_count(conn: &rusqlite::Connection, rated: &str) -> i64 {
let visible = "i.shadowed_by IS NULL AND i.trashed_at IS NULL";
let hidden = dr_catalog::bursts::collapsed_away_frames("i");
conn.query_row(
&format!(
"SELECT (SELECT count(*) FROM images i WHERE {visible}{rated})
- (SELECT count(*) FROM {hidden} AND {visible}{rated})"
),
[],
|r| r.get(0),
)
.unwrap()
}
/// FNV-1a, to print a long answer as something two runs can compare.
fn digest(s: &str) -> u64 {
s.bytes().fold(0xcbf29ce484222325, |h, b| {
(h ^ u64::from(b)).wrapping_mul(0x100000001b3)
})
}
/// Run `f` a few times and print the best wall-clock, the median, and the
/// best CPU time — the figure to compare across runs on a busy machine.
fn time(label: &str, runs: usize, mut f: impl FnMut()) {
let mut wall: Vec<Duration> = Vec::with_capacity(runs);
let mut cpu: Vec<Duration> = Vec::with_capacity(runs);
for _ in 0..runs {
let c = cpu_now();
let t = Instant::now();
f();
wall.push(t.elapsed());
cpu.push(cpu_now().saturating_sub(c));
}
wall.sort();
cpu.sort();
println!(
"{label:42} best {:8.2} ms median {:8.2} ms cpu {:8.2} ms",
wall[0].as_secs_f64() * 1e3,
wall[runs / 2].as_secs_f64() * 1e3,
cpu[0].as_secs_f64() * 1e3
);
}
/// This thread's time on a CPU so far, from `/proc/self/schedstat`; zero where
/// the file is missing, which only makes the CPU column useless.
fn cpu_now() -> Duration {
std::fs::read_to_string("/proc/self/schedstat")
.ok()
.and_then(|s| s.split_whitespace().next()?.parse::<u64>().ok())
.map(Duration::from_nanos)
.unwrap_or_default()
}
core/dr-catalog/examples/dedup_people.rs
+141
View File
@@ -0,0 +1,141 @@
//! Run the people and face deduplication (#78) on a copy of a real catalog.
//!
//! cargo run --release -p dr-catalog --example dedup_people -- COPY.sqlite [--peer PEER_COPY.sqlite]
//!
//! It writes: run it against a *copy* (`sqlite3 catalog.sqlite ".backup
//! copy.sqlite"`), never the library's own file. Prints the live people and
//! faces before and after, what the first run merged and kept apart, and
//! how long the first and a second run took -- the second is the cost the
//! job adds to every sync once a catalog is clean.
//!
//! `--peer` then plays a sync round trip with another device's catalog (a
//! copy of the server snapshot, which it also writes): the peer merges this
//! one as the previous release would, with no job after it, then this one
//! merges the peer back through `sync::merge_remote`, twice. The named
//! people each side lists are printed after each step; they should agree.
use std::path::PathBuf;
use std::time::Instant;
use dr_catalog::{dedup_people, merge, schema, sync};
use rusqlite::Connection;
fn open(path: &std::path::Path) -> Connection {
let conn = Connection::open(path).expect("open the catalog copy");
schema::configure(&conn).expect("configure");
schema::migrate(&conn).expect("migrate");
conn
}
/// The named people a device lists, as `name (uuid prefix)`, sorted.
fn named(conn: &Connection) -> Vec<String> {
let mut v: Vec<String> = conn
.prepare(
"SELECT name, substr(uuid, 1, 8) FROM people
WHERE merged_into IS NULL AND trim(name) <> ''",
)
.unwrap()
.query_map([], |r| {
Ok(format!(
"{} ({})",
r.get::<_, String>(0)?,
r.get::<_, String>(1)?
))
})
.unwrap()
.collect::<Result<_, _>>()
.unwrap();
v.sort();
v
}
fn main() {
let mut args: Vec<String> = std::env::args().skip(1).collect();
let peer = args.iter().position(|a| a == "--peer").map(|at| {
let p = PathBuf::from(&args[at + 1]);
args.drain(at..at + 2);
p
});
let Some(path) = args.first().map(PathBuf::from) else {
eprintln!("usage: dedup_people COPY.sqlite [--peer PEER_COPY.sqlite]");
std::process::exit(2);
};
let conn = open(&path);
let counts = |label: &str| {
let q = |sql: &str| -> i64 { conn.query_row(sql, [], |r| r.get(0)).unwrap() };
println!(
"{label}: {} people listed ({} named), {} redirects, {} faces, {} confirmed",
q("SELECT COUNT(*) FROM people WHERE merged_into IS NULL"),
q("SELECT COUNT(*) FROM people WHERE merged_into IS NULL AND trim(name) <> ''"),
q("SELECT COUNT(*) FROM people WHERE merged_into IS NOT NULL"),
q("SELECT COUNT(*) FROM faces"),
q("SELECT COUNT(*) FROM face_person WHERE confirmed = 1"),
);
};
counts("before");
for pass in ["first", "second", "third"] {
let started = Instant::now();
let report = dedup_people::run(&conn).expect("dedup");
let took = started.elapsed();
println!("{pass} run: {took:?}, changed: {}", report.changed());
if pass == "first" {
println!(" merged: {:?}", report.merged);
for k in &report.kept_apart {
println!(
" kept apart: {:?} ({}) from {}: {:?}",
k.name, k.uuid, k.survivor, k.why
);
}
println!(
" redirects followed {}, cycles broken {}, faces fused {}, faces confirmed apart {}",
report.redirects_followed,
report.cycles_broken,
report.faces_fused,
report.faces_confirmed_apart
);
}
}
counts("after");
let Some(peer_path) = peer else { return };
let peer = open(&peer_path);
let show = |step: &str| {
let (ours, theirs) = (named(&conn), named(&peer));
println!(
"{step}: this device lists {} named, the peer {}; {}",
ours.len(),
theirs.len(),
if ours == theirs {
"the same".to_string()
} else {
format!("differ:\n here {ours:?}\n peer {theirs:?}")
}
);
};
show("before the round trip");
for round in 1..=2 {
peer.execute(
"ATTACH DATABASE ?1 AS remote_cat",
[path.to_string_lossy().as_ref()],
)
.unwrap();
let theirs = merge::merge_all(&peer).expect("the peer's merge");
peer.execute("DETACH DATABASE remote_cat", []).unwrap();
println!(
"round {round}: the peer took {} people updated, {} inserted",
theirs.people_updated, theirs.people_inserted
);
show(&format!("round {round}, after the peer's merge"));
let started = Instant::now();
let ours = sync::merge_remote(&conn, &peer_path).expect("our merge");
println!(
"round {round}: merge_remote with the job took {:?}; {} people updated, {} inserted",
started.elapsed(),
ours.people_updated,
ours.people_inserted
);
show(&format!("round {round}, after ours"));
}
}
core/dr-catalog/src/albums.rs
+516
View File
@@ -0,0 +1,516 @@
//! TRACES: FR-EXP-10 | FR-EXP-6 | FR-CAT-7
//! Albums: named export folders, and which photographs went into each.
//!
//! An album is where finished pictures go — a folder of JPEGs somebody else
//! looks at — as opposed to a collection, which is a set of originals the
//! photographer works on. The folder holds only the exported files. What the
//! catalog adds is the link back: each export is recorded against the image
//! it was rendered from, so opening an album in the library shows the RAWs
//! behind its JPEGs, and re-exporting after an edit is one selection away.
//!
//! # Where the folder is, and why that is two tables
//!
//! An album's folder is either on the library's server or on this device.
//!
//! A **server folder** is one path on the account, the same from every device
//! signed in to it, so it lives on the album row and syncs with it.
//!
//! A **local folder** — a filesystem path on a desktop, a Storage Access
//! Framework tree on Android — means nothing on any other device. It lives in
//! `album_folders`, which the merge never reads and the upload snapshot drops
//! ([`crate::sync::snapshot_for_upload`]). An album made on the desktop with a
//! local folder therefore reaches the tablet as an album with no folder there
//! yet, which is true, and which the tablet can fix by choosing one.
//!
//! # Created on first use, not by a migration
//!
//! A new schema version makes every older build refuse this catalog's
//! snapshot at sync (`crate::sync::remote_is_mergeable`), so the tablet would
//! stop merging collections, keywords and people until it was updated — for
//! a feature it does not have. The tables are created by [`ensure_tables`]
//! instead, the way `dedup_probes` is; an older build that meets them ignores
//! them, and its merge keeps working.
//!
//! # Sync
//!
//! Albums merge by uuid and revision with tombstones, and their exports as a
//! set union keyed on the image's server file id — the rules
//! [`crate::merge`] applies to collections, for the same reasons.
use rusqlite::{Connection, OptionalExtension};
use dr_types::ImageId;
use crate::error::CatalogError;
/// Identifies an album within one catalog. Local, like every integer id here;
/// the uuid is what crosses devices.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct AlbumId(pub u64);
/// Where an album's files go.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Place {
/// A folder on the library's server, relative to the account root, with no
/// leading slash. The same on every device.
Server(String),
/// A folder on this device: a filesystem path, or on Android a SAF tree
/// URI. Never synced.
Local(String),
}
/// One album, as the sidebar and the export sheet show it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Album {
pub id: AlbumId,
pub uuid: String,
pub name: String,
/// Where exports go from this device, or `None` for an album whose folder
/// is local to another device and has not been chosen here.
pub place: Option<Place>,
/// Distinct photographs exported into it — what the grid shows when the
/// album is opened.
pub sources: usize,
}
/// Create the album tables if this catalog does not have them yet.
///
/// Cheap when they exist: `IF NOT EXISTS` is answered from the schema, and
/// every function below calls this first so no caller has to remember to.
pub fn ensure_tables(conn: &Connection) -> Result<(), CatalogError> {
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS albums (
id INTEGER PRIMARY KEY,
-- The merge identity; the integer id is local.
uuid TEXT NOT NULL UNIQUE,
name TEXT NOT NULL,
-- A folder on the server, relative to the account root. NULL for
-- an album whose folder is local to some device.
server_path TEXT,
created INTEGER NOT NULL,
revision INTEGER NOT NULL DEFAULT 1,
modified INTEGER NOT NULL,
deleted INTEGER NOT NULL DEFAULT 0
);
-- One row per file written into an album. Keyed on the file, not the
-- image: a photograph exported twice — two crops, or once before an
-- edit and once after — is two files in the folder and two rows here.
CREATE TABLE IF NOT EXISTS album_exports (
album_id INTEGER NOT NULL REFERENCES albums(id) ON DELETE CASCADE,
file_name TEXT NOT NULL,
image_id INTEGER NOT NULL REFERENCES images(id) ON DELETE CASCADE,
exported_at INTEGER NOT NULL,
PRIMARY KEY (album_id, file_name)
);
CREATE INDEX IF NOT EXISTS album_exports_image ON album_exports(image_id);
-- This device's folder for an album. Never merged, never uploaded.
CREATE TABLE IF NOT EXISTS album_folders (
album_id INTEGER PRIMARY KEY REFERENCES albums(id) ON DELETE CASCADE,
folder TEXT NOT NULL
);",
)?;
Ok(())
}
/// Make an album.
///
/// The name is trimmed and must not be empty; two albums may share one, as
/// two collections may, because the uuid is the identity and refusing a
/// duplicate name here would refuse it on one device and not another.
pub fn create(conn: &Connection, name: &str, place: &Place) -> Result<AlbumId, CatalogError> {
ensure_tables(conn)?;
let name = name.trim();
if name.is_empty() {
return Err(CatalogError::EmptyName);
}
let now = now_secs();
let tx = conn.unchecked_transaction()?;
tx.execute(
"INSERT INTO albums(uuid, name, server_path, created, revision, modified)
VALUES (?1, ?2, ?3, ?4, 1, ?4)",
rusqlite::params![
crate::collections::new_uuid(),
name,
server_path(place),
now
],
)?;
let id = AlbumId(tx.last_insert_rowid() as u64);
if let Place::Local(folder) = place {
tx.execute(
"INSERT INTO album_folders(album_id, folder) VALUES (?1, ?2)",
rusqlite::params![id.0 as i64, folder],
)?;
}
tx.commit()?;
Ok(id)
}
/// Rename an album. The folder keeps its name: the album is what the
/// photographer calls it, the folder is what is already out there.
pub fn rename(conn: &Connection, id: AlbumId, name: &str) -> Result<(), CatalogError> {
ensure_tables(conn)?;
let name = name.trim();
if name.is_empty() {
return Err(CatalogError::EmptyName);
}
let n = conn.execute(
"UPDATE albums SET name = ?2, revision = revision + 1, modified = ?3
WHERE id = ?1 AND deleted = 0",
rusqlite::params![id.0 as i64, name, now_secs()],
)?;
if n == 0 {
return Err(CatalogError::NoSuchAlbum(id.0));
}
Ok(())
}
/// Point an album at a different folder, from this device.
///
/// A server folder replaces the synced path, and bumps the revision so the
/// move reaches every device. A local folder is recorded for this device
/// only; it also clears a server path, because an album goes to one place and
/// the photographer has just said which.
pub fn set_place(conn: &Connection, id: AlbumId, place: &Place) -> Result<(), CatalogError> {
ensure_tables(conn)?;
let tx = conn.unchecked_transaction()?;
let n = tx.execute(
"UPDATE albums SET server_path = ?2, revision = revision + 1, modified = ?3
WHERE id = ?1 AND deleted = 0",
rusqlite::params![id.0 as i64, server_path(place), now_secs()],
)?;
if n == 0 {
return Err(CatalogError::NoSuchAlbum(id.0));
}
match place {
Place::Local(folder) => tx.execute(
"INSERT INTO album_folders(album_id, folder) VALUES (?1, ?2)
ON CONFLICT(album_id) DO UPDATE SET folder = excluded.folder",
rusqlite::params![id.0 as i64, folder],
)?,
Place::Server(_) => tx.execute(
"DELETE FROM album_folders WHERE album_id = ?1",
[id.0 as i64],
)?,
};
tx.commit()?;
Ok(())
}
/// Delete an album, leaving a tombstone. The files in its folder are not
/// touched: they are finished work somebody may already have been sent a
/// link to, and the album was only ever this catalog's note of them.
pub fn delete(conn: &Connection, id: AlbumId) -> Result<(), CatalogError> {
ensure_tables(conn)?;
let tx = conn.unchecked_transaction()?;
let n = tx.execute(
"UPDATE albums SET deleted = 1, revision = revision + 1, modified = ?2
WHERE id = ?1 AND deleted = 0",
rusqlite::params![id.0 as i64, now_secs()],
)?;
if n == 0 {
return Err(CatalogError::NoSuchAlbum(id.0));
}
tx.execute(
"DELETE FROM album_exports WHERE album_id = ?1",
[id.0 as i64],
)?;
tx.execute(
"DELETE FROM album_folders WHERE album_id = ?1",
[id.0 as i64],
)?;
tx.commit()?;
Ok(())
}
/// Every live album, by name, with how many photographs each holds.
///
/// One statement: the counts are aggregated from `album_exports` first and
/// joined to the (few) albums, not counted per row.
pub fn list(conn: &Connection) -> Result<Vec<Album>, CatalogError> {
ensure_tables(conn)?;
let mut stmt = conn.prepare(
"SELECT a.id, a.uuid, a.name, a.server_path, f.folder, coalesce(e.n, 0)
FROM albums a
LEFT JOIN album_folders f ON f.album_id = a.id
LEFT JOIN (SELECT album_id, count(DISTINCT image_id) AS n
FROM album_exports GROUP BY album_id) e
ON e.album_id = a.id
WHERE a.deleted = 0
ORDER BY a.name COLLATE NOCASE, a.id",
)?;
let rows = stmt
.query_map([], album_from_row)?
.collect::<Result<Vec<_>, _>>()?;
Ok(rows)
}
/// One album, or `None` if it is gone.
pub fn get(conn: &Connection, id: AlbumId) -> Result<Option<Album>, CatalogError> {
ensure_tables(conn)?;
Ok(conn
.query_row(
"SELECT a.id, a.uuid, a.name, a.server_path, f.folder,
(SELECT count(DISTINCT image_id) FROM album_exports WHERE album_id = a.id)
FROM albums a
LEFT JOIN album_folders f ON f.album_id = a.id
WHERE a.id = ?1 AND a.deleted = 0",
[id.0 as i64],
album_from_row,
)
.optional()?)
}
/// The album with this uuid, if this catalog holds it live.
pub fn id_for_uuid(conn: &Connection, uuid: &str) -> Result<Option<AlbumId>, CatalogError> {
ensure_tables(conn)?;
Ok(conn
.query_row(
"SELECT id FROM albums WHERE uuid = ?1 AND deleted = 0",
[uuid],
|r| r.get::<_, i64>(0),
)
.optional()?
.map(|id| AlbumId(id as u64)))
}
/// Record the files one export wrote into an album, and which image each
/// came from. One transaction for the batch, however many files it placed.
///
/// A file name already recorded is re-pointed at the image that wrote it
/// last: an export that overwrote `IMG_0001.jpg` replaced the picture in the
/// folder, and the link must say what is there now.
pub fn record_exports(
conn: &Connection,
id: AlbumId,
files: &[(ImageId, String)],
) -> Result<(), CatalogError> {
ensure_tables(conn)?;
if files.is_empty() {
return Ok(());
}
let tx = conn.unchecked_transaction()?;
let now = now_secs();
{
let mut insert = tx.prepare(
"INSERT INTO album_exports(album_id, file_name, image_id, exported_at)
VALUES (?1, ?2, ?3, ?4)
ON CONFLICT(album_id, file_name) DO UPDATE SET
image_id = excluded.image_id, exported_at = excluded.exported_at",
)?;
for (image, name) in files {
insert.execute(rusqlite::params![id.0 as i64, name, image.0 as i64, now])?;
}
}
tx.commit()?;
Ok(())
}
/// The photographs behind an album's files, most recently exported first —
/// what the grid shows when the album is opened.
pub fn sources(conn: &Connection, id: AlbumId) -> Result<Vec<ImageId>, CatalogError> {
ensure_tables(conn)?;
let mut stmt = conn.prepare(
"SELECT image_id FROM album_exports
WHERE album_id = ?1
GROUP BY image_id
ORDER BY max(exported_at) DESC, image_id",
)?;
let rows = stmt
.query_map([id.0 as i64], |r| r.get::<_, i64>(0))?
.map(|r| r.map(|i| ImageId(i as u64)))
.collect::<Result<Vec<_>, _>>()?;
Ok(rows)
}
/// The names of the files an image left in an album — the "which JPEG is
/// this" half of the link.
pub fn files_of(
conn: &Connection,
id: AlbumId,
image: ImageId,
) -> Result<Vec<String>, CatalogError> {
ensure_tables(conn)?;
let mut stmt = conn.prepare(
"SELECT file_name FROM album_exports
WHERE album_id = ?1 AND image_id = ?2
ORDER BY exported_at DESC, file_name",
)?;
let rows = stmt
.query_map(rusqlite::params![id.0 as i64, image.0 as i64], |r| r.get(0))?
.collect::<Result<Vec<_>, _>>()?;
Ok(rows)
}
fn album_from_row(r: &rusqlite::Row<'_>) -> rusqlite::Result<Album> {
let server: Option<String> = r.get(3)?;
let local: Option<String> = r.get(4)?;
Ok(Album {
id: AlbumId(r.get::<_, i64>(0)? as u64),
uuid: r.get(1)?,
name: r.get(2)?,
// A server path wins: `set_place` clears the local folder when it
// sets one, so both being present means a merge brought a server
// path in over a local choice — and the newer revision decided that.
place: server.map(Place::Server).or(local.map(Place::Local)),
sources: r.get::<_, i64>(5)? as usize,
})
}
fn server_path(place: &Place) -> Option<&str> {
match place {
Place::Server(p) => Some(p.trim_matches('/')),
Place::Local(_) => None,
}
}
fn now_secs() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
.unwrap_or(0)
}
#[cfg(test)]
mod tests {
use super::*;
/// A catalog, and the connection to it. The `Catalog` has to outlive the
/// connection it hands out, so tests hold both.
fn catalog() -> crate::Catalog {
let cat = crate::Catalog::in_memory().unwrap();
cat.connection()
.execute(
"INSERT INTO roots(id, kind, label) VALUES (1, 'local', 'lib')",
[],
)
.unwrap();
cat
}
fn image(conn: &Connection, path: &str) -> ImageId {
conn.execute(
"INSERT INTO images(root_id, source_ref, added_at) VALUES (1, ?1, 0)",
[path],
)
.unwrap();
ImageId(conn.last_insert_rowid() as u64)
}
#[test]
fn an_album_lists_with_its_place_and_no_photographs() {
let cat = catalog();
let conn = cat.connection();
let web = create(conn, " Web ", &Place::Server("Shared/Web/".into())).unwrap();
let print = create(conn, "Print", &Place::Local("/mnt/print".into())).unwrap();
let all = list(conn).unwrap();
assert_eq!(all.len(), 2);
assert_eq!(all[0].id, print, "sorted by name");
assert_eq!(all[0].place, Some(Place::Local("/mnt/print".into())));
assert_eq!(all[1].id, web);
assert_eq!(all[1].name, "Web", "trimmed");
assert_eq!(all[1].place, Some(Place::Server("Shared/Web".into())));
assert_eq!(all[1].sources, 0);
}
#[test]
fn an_empty_name_is_refused() {
let cat = catalog();
let conn = cat.connection();
assert!(matches!(
create(conn, " ", &Place::Local("/x".into())),
Err(CatalogError::EmptyName)
));
}
#[test]
fn exports_link_files_back_to_their_images() {
let cat = catalog();
let conn = cat.connection();
let album = create(conn, "Web", &Place::Local("/out".into())).unwrap();
let a = image(conn, "a.cr3");
let b = image(conn, "b.cr3");
record_exports(
conn,
album,
&[
(a, "a.jpg".into()),
(a, "a (1).jpg".into()),
(b, "b.jpg".into()),
],
)
.unwrap();
let got = get(conn, album).unwrap().unwrap();
assert_eq!(got.sources, 2, "two photographs, three files");
let mut s = sources(conn, album).unwrap();
s.sort();
assert_eq!(s, vec![a, b]);
assert_eq!(files_of(conn, album, a).unwrap().len(), 2);
}
#[test]
fn an_overwritten_file_points_at_what_wrote_it_last() {
let cat = catalog();
let conn = cat.connection();
let album = create(conn, "Web", &Place::Local("/out".into())).unwrap();
let a = image(conn, "a.cr3");
let b = image(conn, "b.cr3");
record_exports(conn, album, &[(a, "x.jpg".into())]).unwrap();
record_exports(conn, album, &[(b, "x.jpg".into())]).unwrap();
assert_eq!(sources(conn, album).unwrap(), vec![b]);
}
#[test]
fn moving_to_the_server_forgets_the_local_folder() {
let cat = catalog();
let conn = cat.connection();
let album = create(conn, "Web", &Place::Local("/out".into())).unwrap();
set_place(conn, album, &Place::Server("Web".into())).unwrap();
assert_eq!(
get(conn, album).unwrap().unwrap().place,
Some(Place::Server("Web".into()))
);
set_place(conn, album, &Place::Local("/again".into())).unwrap();
assert_eq!(
get(conn, album).unwrap().unwrap().place,
Some(Place::Local("/again".into()))
);
}
#[test]
fn a_deleted_album_is_gone_and_its_uuid_no_longer_resolves() {
let cat = catalog();
let conn = cat.connection();
let album = create(conn, "Web", &Place::Local("/out".into())).unwrap();
let uuid = get(conn, album).unwrap().unwrap().uuid;
let a = image(conn, "a.cr3");
record_exports(conn, album, &[(a, "a.jpg".into())]).unwrap();
delete(conn, album).unwrap();
assert!(list(conn).unwrap().is_empty());
assert_eq!(id_for_uuid(conn, &uuid).unwrap(), None);
assert!(matches!(
rename(conn, album, "Again"),
Err(CatalogError::NoSuchAlbum(_))
));
}
#[test]
fn a_rename_bumps_the_revision_the_merge_compares() {
let cat = catalog();
let conn = cat.connection();
let album = create(conn, "Web", &Place::Local("/out".into())).unwrap();
rename(conn, album, "Website").unwrap();
let rev: i64 = conn
.query_row(
"SELECT revision FROM albums WHERE id = ?1",
[album.0 as i64],
|r| r.get(0),
)
.unwrap();
assert_eq!(rev, 2);
}
}
core/dr-catalog/src/backfilled.rs
+417
View File
@@ -0,0 +1,417 @@
//! TRACES: NFR-P9
//! Which catalog files this process has already backfilled, and as of what.
//!
//! # Why this exists
//!
//! [`crate::schema::backfill`] used to run inside every [`crate::Catalog::open`],
//! and every worker thread opens its own connection. Landing on a photograph
//! in develop opened the catalog five times — the fetch of the original and a
//! cache check per prefetched neighbour — and each open paid the whole
//! backfill: an anti-join of every image against its versions, a pass over
//! every default version's uuid, the unpaired JPEGs and the keyword
//! vocabulary. On the reference library that was ~12 ms an open and ~60 ms of
//! CPU a landing, spent confirming that nothing had changed since the open
//! before.
//!
//! # What makes skipping it safe
//!
//! Everything the backfill repairs is a row some write *added*: an image
//! inserted by a scan or an import has no default version and may be the RAW
//! beside an unpaired JPEG; a version merged or restored from an older build
//! may carry a minted uuid; a keyword assignment merged from a remote may name
//! a word with no term. So the question "is any work owed?" is answered by
//! whether those tables have gained rows since the last backfill, and that is
//! a read of each table's last row — the last page of its b-tree — rather than
//! a scan.
//!
//! # Why the last row, and not only its id
//!
//! None of these tables is `AUTOINCREMENT`, so SQLite hands out the largest
//! rowid plus one, and an id freed by deleting the newest row is handed out
//! again. That is an ordinary sequence, not a contrived one: emptying the
//! trash of the newest photograph and then scanning a new one, or a local
//! folder's walk removing a renamed file's row and inserting the new name in
//! the same pass. `max(id)` does not move, and neither does `count(*)`. And
//! the row that took the id is exactly one that needs the backfill, because
//! neither scan creates default versions — `persist` and the walk insert the
//! image and leave the version, the pairing and the keyword terms to the next
//! open. Skipped, it would go without them until the app restarted: a rating
//! or a keyword with nowhere to land, a JPEG beside its RAW shown twice.
//!
//! So the stamp carries the last row's content as well as its id: the newest
//! image's path, when it was added, and **whether it has a version**; the
//! newest version's image; the newest assignment's word and version. Whether
//! the newest image has a version is the part that cannot be fooled: once the
//! backfill has run, every image has one, and a row that has just taken a
//! freed id has none, so the two stamps differ whatever the path and the time
//! say. The others make the newest version or assignment a different row
//! whenever a different one took its id; one that is the same content at the
//! same id is the same row as far as the backfill is concerned.
//!
//! The [`Stamp`] is those, the schema version, and the file's identity.
//! An open whose stamp matches the one recorded at the last backfill of the
//! same path skips it; anything else runs it. That covers the cases that must
//! run it:
//!
//! - **The first open in a process.** Nothing is recorded yet.
//! - **A migration.** `user_version` is in the stamp, and [`crate::Catalog::open`]
//! also runs the backfill unconditionally whenever `migrate` moved the
//! schema, because that is what the backfill was written for.
//! - **A pulled catalog.** The merge inserts assignments, which moves the
//! stamp; and [`crate::sync::merge_remote`] [`forget`]s the path as well, so
//! the next open backfills even when every incoming row collided.
//! - **A file replaced underneath the path** — a restore from backup, a
//! rebuild, a catalog copied in. On unix the device and inode are in the
//! stamp, and a replacement is a new inode; [`crate::recovery::set_aside`],
//! the first step of both a restore and a rebuild, forgets the path too.
//! - **Another process writing.** The stamp is read from the file, not from
//! anything this process did, so a scan in a second instance moves it just
//! the same.
//!
//! # Why the stamp is taken before the backfill
//!
//! The backfill adds versions and terms itself, so a stamp read afterwards
//! would describe its own writes. Read afterwards it could also describe an
//! image another connection inserted between the backfill's read and the
//! stamp's — and record that image as covered when it was not. Read before,
//! the worst case is the reverse: the backfill's own inserts move the stamp,
//! and the next open runs one more backfill that finds nothing. That costs one
//! redundant pass after a backfill that did real work, and never misses a row.
//!
//! # What it does not see
//!
//! An `UPDATE` that creates work without adding a row. None of this build's
//! writers does: a scan's move of a file is a new `source_ref` and so a new
//! image, and uuids are only rewritten by the backfill itself. Should one
//! appear, the cost is that its repair waits for the next insert or the next
//! start of the app — which is exactly where the backfill ran before it ran on
//! every open.
//!
//! Kept in memory rather than in the catalog on purpose: a row in the file
//! would travel in the sync snapshot and would need a table an older build
//! does not have, and a flag that another device's catalog carried in would
//! say nothing about this one.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::{Mutex, OnceLock};
use rusqlite::Connection;
use crate::error::CatalogError;
/// What a catalog looked like, as far as the backfill cares.
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct Stamp {
/// Device and inode, so a file swapped in under the same name is a new
/// catalog. `None` where the platform has no such thing.
file: Option<(u64, u64)>,
user_version: i64,
/// The newest image: id, path, when added, and whether it has a version.
last_image: Option<String>,
/// The newest version: id and the image it belongs to.
last_version: Option<String>,
/// The newest keyword assignment: rowid, version and word.
last_keyword: Option<String>,
}
/// The stamp recorded at the last backfill, per catalog file.
fn done() -> &'static Mutex<HashMap<PathBuf, Stamp>> {
static DONE: OnceLock<Mutex<HashMap<PathBuf, Stamp>>> = OnceLock::new();
DONE.get_or_init(Default::default)
}
/// One name per file, whichever spelling of its path the caller used.
fn key(path: &Path) -> PathBuf {
std::fs::canonicalize(path).unwrap_or_else(|_| path.to_path_buf())
}
/// Read the stamp of the catalog behind `conn`, which was opened from `path`.
///
/// One statement: the last row of each of three tables, each found by
/// descending its rowid b-tree to the last page, plus one probe of
/// `versions_image` for the newest image — and a `stat` of the file.
pub(crate) fn stamp(conn: &Connection, path: &Path) -> Result<Stamp, CatalogError> {
let (user_version, last_image, last_version, last_keyword) = conn.query_row(
"SELECT (SELECT user_version FROM pragma_user_version),
(SELECT printf('%d|%d|%d|%s', i.id, i.added_at,
EXISTS (SELECT 1 FROM versions v WHERE v.image_id = i.id),
i.source_ref)
FROM images i ORDER BY i.id DESC LIMIT 1),
(SELECT printf('%d|%d', id, image_id)
FROM versions ORDER BY id DESC LIMIT 1),
(SELECT printf('%d|%d|%s', rowid, version_id, keyword)
FROM keywords ORDER BY rowid DESC LIMIT 1)",
[],
|r| Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?)),
)?;
Ok(Stamp {
file: file_identity(path),
user_version,
last_image,
last_version,
last_keyword,
})
}
#[cfg(unix)]
fn file_identity(path: &Path) -> Option<(u64, u64)> {
use std::os::unix::fs::MetadataExt;
std::fs::metadata(path).ok().map(|m| (m.dev(), m.ino()))
}
#[cfg(not(unix))]
fn file_identity(_path: &Path) -> Option<(u64, u64)> {
None
}
/// Whether the catalog at `path` was last backfilled at exactly `stamp`.
pub(crate) fn is_current(path: &Path, stamp: &Stamp) -> bool {
done()
.lock()
.unwrap_or_else(|e| e.into_inner())
.get(&key(path))
== Some(stamp)
}
/// Record that the catalog at `path` has been backfilled as of `stamp`.
pub(crate) fn record(path: &Path, stamp: Stamp) {
done()
.lock()
.unwrap_or_else(|e| e.into_inner())
.insert(key(path), stamp);
}
/// Make the next open of `path` backfill, whatever its stamp says.
///
/// For the writers that know they have changed the catalog wholesale — a
/// merge of a pulled catalog, a restore from backup — so their correctness
/// does not rest on the stamp happening to move.
pub(crate) fn forget(path: &Path) {
done()
.lock()
.unwrap_or_else(|e| e.into_inner())
.remove(&key(path));
}
#[cfg(test)]
mod tests {
use crate::rating::derived_version_uuid;
use crate::Catalog;
use std::path::PathBuf;
/// A catalog file of its own, holding one image the server has named,
/// backfilled and settled.
///
/// Opened three times on the way: to create it; after the image went in,
/// which gives the image its default version; and once more, because that
/// version moved the stamp and the next open runs the one redundant pass
/// the module header describes. After that the stamp stands still.
fn catalog(tag: &str) -> PathBuf {
let dir = std::env::temp_dir().join(format!(
"dr-backfilled-{tag}-{}-{:?}",
std::process::id(),
std::thread::current().id()
));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("catalog.sqlite");
{
let cat = Catalog::open(&path).unwrap();
let c = cat.connection();
c.execute_batch(
"INSERT INTO roots(id, kind, label) VALUES (1, 'remote', 'Photos');
INSERT INTO images(id, root_id, source_ref, added_at)
VALUES (1, 1, 'Photos/a.CR3', 0);
INSERT INTO remote(image_id, file_id) VALUES (1, 77);",
)
.unwrap();
}
assert_eq!(uuid(&path, 1), Some(derived_version_uuid(77)));
assert_eq!(uuid(&path, 1), Some(derived_version_uuid(77)));
path
}
/// The default version's uuid for `image`, read through an ordinary open.
fn uuid(path: &std::path::Path, image: i64) -> Option<String> {
let cat = Catalog::open(path).unwrap();
cat.connection()
.query_row(
"SELECT uuid FROM versions WHERE image_id = ?1 AND is_default = 1",
[image],
|r| r.get(0),
)
.ok()
}
/// Put the one row back into the state the backfill repairs, with an
/// `UPDATE` — which moves none of the stamp's maxima, so only the stamp's
/// other parts or an explicit `forget` can bring the backfill back.
fn unalign(path: &std::path::Path) {
rusqlite::Connection::open(path)
.unwrap()
.execute("UPDATE versions SET uuid = 'minted' WHERE image_id = 1", [])
.unwrap();
}
#[test]
fn an_unchanged_catalog_is_not_backfilled_again() {
let path = catalog("unchanged");
unalign(&path);
assert_eq!(
uuid(&path, 1).as_deref(),
Some("minted"),
"nothing was added since the last backfill, so the open skipped it"
);
}
#[test]
fn an_image_a_scan_added_is_backfilled_on_the_next_open() {
let path = catalog("scanned");
rusqlite::Connection::open(&path)
.unwrap()
.execute(
"INSERT INTO images(id, root_id, source_ref, added_at)
VALUES (2, 1, 'Photos/b.CR3', 0)",
[],
)
.unwrap();
assert!(uuid(&path, 2).is_some(), "the new image got its version");
}
/// The id of a deleted newest row is handed out again, so `max(id)` is
/// the same before and after — the sequence emptying the trash and then
/// scanning makes. The image that took the id still needs its version.
///
/// A virtual copy on the older image holds the newest version id, so
/// the deletion does not move `max(versions.id)` either: nothing the old
/// stamp read changes, which is the case that went unrepaired.
#[test]
fn an_image_that_reuses_a_deleted_id_is_backfilled_on_the_next_open() {
let path = catalog("reused");
rusqlite::Connection::open(&path)
.unwrap()
.execute(
"INSERT INTO images(id, root_id, source_ref, added_at)
VALUES (2, 1, 'Photos/b.CR3', 0)",
[],
)
.unwrap();
assert!(uuid(&path, 2).is_some());
rusqlite::Connection::open(&path)
.unwrap()
.execute(
"INSERT INTO versions(image_id, uuid, name, is_default)
VALUES (1, 'copy', 'Crop', 0)",
[],
)
.unwrap();
// Settle: one open backfills after the new version, one more runs
// the redundant pass and records the stamp that stands.
assert!(uuid(&path, 2).is_some());
assert!(uuid(&path, 2).is_some());
let c = rusqlite::Connection::open(&path).unwrap();
let before: (i64, i64) = c
.query_row(
"SELECT (SELECT max(id) FROM images), (SELECT max(id) FROM versions)",
[],
|r| Ok((r.get(0)?, r.get(1)?)),
)
.unwrap();
c.execute("DELETE FROM images WHERE id = 2", []).unwrap();
c.execute(
"INSERT INTO images(root_id, source_ref, added_at)
VALUES (1, 'Photos/c.CR3', 0)",
[],
)
.unwrap();
let after: (i64, i64) = c
.query_row(
"SELECT (SELECT max(id) FROM images), (SELECT max(id) FROM versions)",
[],
|r| Ok((r.get(0)?, r.get(1)?)),
)
.unwrap();
assert_eq!(before, after, "SQLite handed the freed id out again");
drop(c);
assert!(uuid(&path, 2).is_some(), "the new image got its version");
}
/// The same, with the same file coming back at the same id in the same
/// second: path and time match, and only the missing version tells.
#[test]
fn the_same_file_back_at_the_same_id_is_backfilled_on_the_next_open() {
let path = catalog("returned");
let c = rusqlite::Connection::open(&path).unwrap();
// As above: a newer version on another image keeps the deletion
// from moving `max(versions.id)`.
c.execute_batch(
"INSERT INTO images(id, root_id, source_ref, added_at)
VALUES (0, 1, 'Photos/0.CR3', 0);
INSERT INTO versions(image_id, uuid, name, is_default)
VALUES (0, 'copy', 'Crop', 0);",
)
.unwrap();
drop(c);
assert!(uuid(&path, 1).is_some());
assert!(uuid(&path, 1).is_some());
let c = rusqlite::Connection::open(&path).unwrap();
c.execute_batch(
"DELETE FROM images WHERE id = 1;
INSERT INTO images(id, root_id, source_ref, added_at)
VALUES (1, 1, 'Photos/a.CR3', 0);
INSERT INTO remote(image_id, file_id) VALUES (1, 77);",
)
.unwrap();
drop(c);
assert_eq!(uuid(&path, 1), Some(derived_version_uuid(77)));
}
#[test]
fn the_first_open_after_a_migration_backfills() {
let path = catalog("migrated");
unalign(&path);
rusqlite::Connection::open(&path)
.unwrap()
.pragma_update(None, "user_version", crate::schema::SCHEMA_VERSION - 1)
.unwrap();
assert_eq!(uuid(&path, 1), Some(derived_version_uuid(77)));
}
#[test]
fn the_first_open_after_a_pulled_catalog_backfills() {
let path = catalog("pulled");
// The remote is this catalog as it stands, so every row the merge
// offers collides and nothing in the stamp moves: only the merge
// saying so can make the next open backfill.
let remote = path.with_file_name("remote.sqlite");
rusqlite::Connection::open(&path)
.unwrap()
.execute("VACUUM INTO ?1", [remote.to_string_lossy().as_ref()])
.unwrap();
unalign(&path);
assert_eq!(uuid(&path, 1).as_deref(), Some("minted"));
Catalog::open(&path)
.unwrap()
.merge_remote_catalog(&remote)
.unwrap();
assert_eq!(uuid(&path, 1), Some(derived_version_uuid(77)));
}
#[cfg(unix)]
#[test]
fn a_catalog_replaced_under_the_same_name_backfills() {
let path = catalog("replaced");
unalign(&path);
assert_eq!(uuid(&path, 1).as_deref(), Some("minted"));
// Every connection is closed, so the WAL is folded in and the main
// file is the whole catalog. A copy renamed over it is the same rows
// in a new file — which is what a restore or a copied-in catalog is.
let copy = path.with_file_name("copy.sqlite");
std::fs::copy(&path, &copy).unwrap();
std::fs::rename(&copy, &path).unwrap();
assert_eq!(uuid(&path, 1), Some(derived_version_uuid(77)));
}
}
core/dr-catalog/src/bursts.rs
+65
View File
@@ -722,6 +722,31 @@ pub fn not_collapsed_away(image: &str) -> String {
) )
} }
/// SQL for the rows [`not_collapsed_away`] drops, as a `FROM ... WHERE`
/// joining each such frame to its image under the alias `image`.
///
/// For counting. A count that applies [`not_collapsed_away`] to every row
/// pays two primary-key probes per image to find the handful a collapsed
/// burst hides; counting everything and subtracting what this lists walks
/// only `burst_members`, which is empty on a library without bursts. The
/// caller appends its own conditions on `image` with `AND`, the same ones
/// it counted the whole with, so the subtraction takes away only rows the
/// whole included. `image_id` is `burst_members`' key, so no image is
/// listed twice.
///
/// The two must describe the same rows: change one, change both, and
/// `the_collapsed_frames_are_what_the_predicate_drops` will say if they drift.
///
/// Never interpolate anything user-supplied as `image`.
pub fn collapsed_away_frames(image: &str) -> String {
format!(
"burst_members bm CROSS JOIN images {image} ON {image}.id = bm.image_id
WHERE bm.representative = 0
AND NOT EXISTS (SELECT 1 FROM burst_expanded be
WHERE be.burst_id = bm.burst_id)"
)
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@@ -1235,6 +1260,46 @@ mod tests {
assert_eq!(visible(cat.connection()), vec![1, 2, 3, 4]); assert_eq!(visible(cat.connection()), vec![1, 2, 3, 4]);
} }
#[test]
fn the_collapsed_frames_are_what_the_predicate_drops() {
// `collapsed_away_frames` is `not_collapsed_away` turned inside out
// for counting; the two must name the same rows, open or closed.
let cat = seeded(&[
(1, 1000, Some(0xFF00)),
(2, 1001, Some(0xFF00)),
(3, 1002, Some(0xFF00)),
(4, 9000, Some(0xAA00)),
(5, 9001, Some(0xAA00)),
(6, 20000, Some(0xFF00)),
]);
regroup(cat.connection(), Rules::default()).unwrap();
let ids = |sql: String| -> Vec<i64> {
let c = cat.connection();
let mut stmt = c.prepare(&sql).unwrap();
let rows = stmt.query_map([], |r| r.get::<_, i64>(0)).unwrap();
rows.collect::<Result<Vec<_>, _>>().unwrap()
};
let dropped = || {
ids(format!(
"SELECT id FROM images i WHERE NOT {} ORDER BY id",
not_collapsed_away("i")
))
};
let listed = || {
ids(format!(
"SELECT i.id FROM {} ORDER BY i.id",
collapsed_away_frames("i")
))
};
assert_eq!(listed(), dropped());
set_expanded(cat.connection(), ImageId(1), false).unwrap();
assert_eq!(dropped(), vec![2, 3]);
assert_eq!(listed(), dropped());
set_expanded(cat.connection(), ImageId(4), false).unwrap();
assert_eq!(listed(), dropped());
}
#[test] #[test]
fn a_library_with_no_bursts_hides_nothing() { fn a_library_with_no_bursts_hides_nothing() {
// The predicate is in every grid query, so its cost and its effect on a // The predicate is in every grid query, so its cost and its effect on a
core/dr-catalog/src/dedup_people.rs
+1227
View File
File diff suppressed because it is too large Load Diff
core/dr-catalog/src/duplicates.rs
+1100
View File
File diff suppressed because it is too large Load Diff
core/dr-catalog/src/error.rs
+17
View File
@@ -65,6 +65,16 @@ pub enum CatalogError {
#[error("no such collection: {0}")] #[error("no such collection: {0}")]
NoSuchCollection(u64), NoSuchCollection(u64),
/// An album the caller named is gone — deleted here, or by a merge while
/// its id sat in a UI model.
#[error("no such album: {0}")]
NoSuchAlbum(u64),
/// A name that is empty once trimmed. Refused rather than stored, because
/// a row with no name is one the sidebar cannot draw and nobody can pick.
#[error("a name is required")]
EmptyName,
/// A keyword the caller named is gone — deleted, or fused into another by a /// A keyword the caller named is gone — deleted, or fused into another by a
/// merge while its id sat in a UI model. /// merge while its id sat in a UI model.
/// ///
@@ -96,6 +106,13 @@ pub enum CatalogError {
#[error("io: {0}")] #[error("io: {0}")]
Io(String), Io(String),
/// TRACES: FR-CAT-11a
/// A duplicate group planned earlier no longer holds: a copy was trashed,
/// rescanned or changed since the review was drawn. The group is left
/// untouched rather than consolidated on a stale plan.
#[error("no longer a duplicate: {0}")]
StaleDuplicate(String),
} }
impl From<rusqlite::Error> for CatalogError { impl From<rusqlite::Error> for CatalogError {
core/dr-catalog/src/face_shard.rs
+74 -5
View File
@@ -30,6 +30,7 @@
//! identity every client agrees on (FR-NC-5), and it survives a server-side //! identity every client agrees on (FR-NC-5), and it survives a server-side
//! move, so a shard written before a reorganisation still applies after it. //! move, so a shard written before a reorganisation still applies after it.
use std::collections::{HashMap, HashSet};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use rusqlite::{Connection, OptionalExtension}; use rusqlite::{Connection, OptionalExtension};
@@ -155,6 +156,45 @@ impl FaceShardStore {
.flatten() .flatten()
} }
/// [`indexed_at`](Self::indexed_at) for every entry at once.
///
/// What a pass over the whole library asks instead of one lookup per image:
/// the export compares every marker in the catalog with this, and a lookup
/// each was 19,000 statements prepared and run on every sync pass that had
/// nothing to send.
fn all_indexed_at(&self) -> Result<HashMap<(u64, String), Option<i64>>, CatalogError> {
let mut q = self
.index
.prepare("SELECT file_id, model_id, indexed_at FROM entries")?;
let rows = q.query_map([], |r| {
Ok((
(r.get::<_, i64>(0)? as u64, r.get::<_, String>(1)?),
r.get::<_, Option<i64>>(2)?,
))
})?;
Ok(rows.collect::<Result<_, _>>()?)
}
/// [`held_model`](Self::held_model) for every file at once: one statement,
/// ordered exactly as that one is, keeping the first row per file.
fn all_held_models(&self, model_id: &str) -> Result<HashMap<u64, String>, CatalogError> {
let mut q = self.index.prepare(&format!(
"SELECT file_id, model_id FROM entries
WHERE {} = ?1
ORDER BY file_id, indexed_at DESC NULLS LAST, model_id",
crate::faces::embedder_sql("model_id")
))?;
let rows = q.query_map([crate::faces::embedder_of(model_id)], |r| {
Ok((r.get::<_, i64>(0)? as u64, r.get::<_, String>(1)?))
})?;
let mut out = HashMap::new();
for row in rows {
let (file, model) = row?;
out.entry(file).or_insert(model);
}
Ok(out)
}
/// The pipeline this store holds an image under, among those sharing /// The pipeline this store holds an image under, among those sharing
/// `model_id`'s embedder — the most recently indexed where a peer has /// `model_id`'s embedder — the most recently indexed where a peer has
/// sent more than one. /// sent more than one.
@@ -784,6 +824,18 @@ pub fn export_to_shards_reporting(
/// enough that the reporting is lost in the write it accompanies. /// enough that the reporting is lost in the write it accompanies.
const REPORT_EVERY: usize = 25; const REPORT_EVERY: usize = 25;
// What the store holds, read once. A put below rewrites only its own
// file's entries -- its generation, and siblings it supersedes -- so a
// file already written in this pass is asked of the store again and every
// other answer is the one a lookup would have given.
//
// An index that cannot be read answers as each lookup did: nothing held.
let held = store.all_indexed_at().unwrap_or_else(|e| {
log::debug!("reading the shard index: {e}");
HashMap::new()
});
let mut written: HashSet<u64> = HashSet::new();
let total = rows.len(); let total = rows.len();
let mut exported = 0; let mut exported = 0;
for (seen, (file_id, image_id, edge, indexed_at, model_id)) in rows.into_iter().enumerate() { for (seen, (file_id, image_id, edge, indexed_at, model_id)) in rows.into_iter().enumerate() {
@@ -800,10 +852,14 @@ pub fn export_to_shards_reporting(
// //
// The comparison is against when the *catalog* indexed it, so a // The comparison is against when the *catalog* indexed it, so a
// re-index is visible and an unchanged image still costs nothing. // re-index is visible and an unchanged image still costs nothing.
if store let was = if written.contains(&(file_id as u64)) {
.indexed_at(file_id as u64, model_id) store.indexed_at(file_id as u64, model_id)
.is_some_and(|was| was >= indexed_at) } else {
{ held.get(&(file_id as u64, model_id.to_string()))
.copied()
.flatten()
};
if was.is_some_and(|was| was >= indexed_at) {
continue; continue;
} }
let mut fq = conn.prepare( let mut fq = conn.prepare(
@@ -840,6 +896,7 @@ pub fn export_to_shards_reporting(
&faces, &faces,
Some(indexed_at), Some(indexed_at),
)?; )?;
written.insert(file_id as u64);
exported += 1; exported += 1;
} }
progress(total, total); progress(total, total);
@@ -902,6 +959,18 @@ pub fn import_from_shards(
/// the lock, waits a fraction of a second and not the whole import. /// the lock, waits a fraction of a second and not the whole import.
const CHUNK: usize = 100; const CHUNK: usize = 100;
// Every file's held pipeline, read once rather than asked per candidate --
// 23,000 prepared lookups on every pass, nearly all of them for images
// this device already holds. Nothing below changes which pipeline the
// store holds a file under (`set_indexed_at` touches only a file already
// decided), and each candidate is a different file, so these are the
// answers the lookups gave.
// An index that cannot be read answers as each lookup did: nothing held.
let held_models = store.all_held_models(model_id).unwrap_or_else(|e| {
log::debug!("reading the shard index: {e}");
HashMap::new()
});
let mut adopted = 0; let mut adopted = 0;
let mut tx = conn.unchecked_transaction()?; let mut tx = conn.unchecked_transaction()?;
let mut in_chunk = 0; let mut in_chunk = 0;
@@ -911,7 +980,7 @@ pub fn import_from_shards(
tx = conn.unchecked_transaction()?; tx = conn.unchecked_transaction()?;
in_chunk = 0; in_chunk = 0;
} }
let Some(held) = store.held_model(file_id as u64, model_id) else { let Some(held) = held_models.get(&(file_id as u64)).cloned() else {
continue; continue;
}; };
if let Some(local) = local { if let Some(local) = local {
core/dr-catalog/src/faces.rs
+273 -7
View File
@@ -1029,7 +1029,8 @@ fn people_where(conn: &Connection, in_use: bool) -> Result<Vec<Person>, CatalogE
/// ///
/// Confirmations survive the move: a face the user confirmed as the source /// Confirmations survive the move: a face the user confirmed as the source
/// person is now a confirmed face of the target, which is what the user meant /// person is now a confirmed face of the target, which is what the user meant
/// by saying they are the same person. /// by saying they are the same person. So do rejections — see
/// [`merge_people_within`].
pub fn merge_people( pub fn merge_people(
conn: &Connection, conn: &Connection,
target: PersonId, target: PersonId,
@@ -1039,7 +1040,53 @@ pub fn merge_people(
return Ok(0); return Ok(0);
} }
let tx = conn.unchecked_transaction()?; let tx = conn.unchecked_transaction()?;
let moved = merge_people_within(&tx, target, source)?;
tx.commit()?;
Ok(moved)
}
/// [`merge_people`] inside a transaction the caller holds, so a job that
/// merges several pairs commits once (`crate::dedup_people`).
///
/// **Rejections move with the faces.** "This face is not Annie" is a
/// judgement about the person, and once Annie is Anna it is one about Anna.
/// Left on the redirect it binds nothing, and the next grouping pass
/// suggests the face the user pushed away to the person it now belongs to. Where the
/// two halves disagree about one face — confirmed as one, rejected as the
/// other — the confirmation stands, which is the rule [`confirm`] applies
/// to one face; and a moved rejection takes a suggestion of the same face
/// with it, the rule [`reject`] applies.
pub(crate) fn merge_people_within(
tx: &Connection,
target: PersonId,
source: PersonId,
) -> Result<u64, CatalogError> {
if target == source {
return Ok(0);
}
let moved = move_judgements(tx, target, source)?;
tx.execute(
"UPDATE people SET merged_into = ?1, revision = revision + 1, modified = ?3
WHERE id = ?2",
rusqlite::params![target.0 as i64, source.0 as i64, now_secs()],
)?;
Ok(moved)
}
/// The half of [`merge_people_within`] that moves faces and rejections,
/// without touching either person's row.
///
/// Also what follows a redirect that arrived by sync
/// (`crate::dedup_people`): the other device merged the people, and this
/// one still holds judgements on the person merged away. Bumping the
/// person's revision there would be an edit of this device's own, sent back
/// on every pass, so the row is left as the merge wrote it.
pub(crate) fn move_judgements(
tx: &Connection,
target: PersonId,
source: PersonId,
) -> Result<u64, CatalogError> {
let (t, s) = (target.0 as i64, source.0 as i64);
// A face already assigned to the target must not gain a second row — // A face already assigned to the target must not gain a second row —
// `face_person` is keyed by face. Where both hold the same face, the // `face_person` is keyed by face. Where both hold the same face, the
// target's row wins and the source's is dropped. // target's row wins and the source's is dropped.
@@ -1047,18 +1094,31 @@ pub fn merge_people(
"DELETE FROM face_person "DELETE FROM face_person
WHERE person_id = ?2 WHERE person_id = ?2
AND face_id IN (SELECT face_id FROM face_person WHERE person_id = ?1)", AND face_id IN (SELECT face_id FROM face_person WHERE person_id = ?1)",
rusqlite::params![target.0 as i64, source.0 as i64], rusqlite::params![t, s],
)?; )?;
let moved = tx.execute( let moved = tx.execute(
"UPDATE face_person SET person_id = ?1 WHERE person_id = ?2", "UPDATE face_person SET person_id = ?1 WHERE person_id = ?2",
rusqlite::params![target.0 as i64, source.0 as i64], rusqlite::params![t, s],
)?; )?;
tx.execute( tx.execute(
"UPDATE people SET merged_into = ?1, revision = revision + 1, modified = ?3 "INSERT OR IGNORE INTO face_person_rejected (face_id, person_id)
WHERE id = ?2", SELECT face_id, ?1 FROM face_person_rejected WHERE person_id = ?2",
rusqlite::params![target.0 as i64, source.0 as i64, now_secs()], rusqlite::params![t, s],
)?;
tx.execute("DELETE FROM face_person_rejected WHERE person_id = ?1", [s])?;
tx.execute(
"DELETE FROM face_person_rejected
WHERE person_id = ?1
AND face_id IN (SELECT face_id FROM face_person
WHERE person_id = ?1 AND confirmed = 1)",
[t],
)?;
tx.execute(
"DELETE FROM face_person
WHERE person_id = ?1 AND confirmed = 0
AND face_id IN (SELECT face_id FROM face_person_rejected WHERE person_id = ?1)",
[t],
)?; )?;
tx.commit()?;
Ok(moved as u64) Ok(moved as u64)
} }
@@ -1599,6 +1659,170 @@ fn iou(a: (f32, f32, f32, f32), b: (f32, f32, f32, f32)) -> f32 {
} }
} }
/// TRACES: FR-CAT-11a | FR-CULL-10
/// What [`carry_onto_copy`] did with one byte-identical copy's faces.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct FaceCarry {
/// Faces moved onto the survivor outright, because it had none.
pub moved: usize,
/// Names and suggestions put on a survivor's face that lacked one.
pub named: usize,
/// Rejections added to a survivor's face.
pub rejections: usize,
/// Faces named one person on the copy and another on the survivor. The
/// survivor's name is kept; the copy keeps its own, in the trash.
pub conflicts: usize,
/// Named faces on the copy that match nothing on the survivor. Left with
/// the copy rather than mixed into another pipeline's faces.
pub unmatched_named: usize,
}
/// TRACES: FR-CAT-11a | FR-CULL-10
/// Bring what one copy of a photograph knows about its faces onto another
/// copy of the same bytes, inside the caller's transaction.
///
/// Faces are per image, so two copies of one file indexed separately hold
/// two sets of the same boxes, and a name confirmed on one is invisible on
/// the other. The rule is the one [`record_detections_within`] keeps: an
/// image holds one pipeline's faces at a time, and the user's judgements
/// are what must survive.
///
/// - **The survivor has no faces at all.** The copy's faces and its run
/// markers move over wholesale — nothing is duplicated, and the survivor
/// is spared a detection pass it would only repeat. Its own markers go
/// first, because a marker saying "examined, nothing found" over an image
/// that now holds faces is the V12 state.
/// - **The survivor has faces.** Each of the copy's faces is paired with the
/// survivor face its box overlaps most (IoU above one half — the bytes are
/// the same, so the boxes coincide). A name or suggestion is carried onto
/// a survivor face that has none, a confirmation outranks a suggestion,
/// and two different confirmed names are a conflict the survivor wins.
/// Rejections are unioned. The copy's faces stay where they are, with the
/// copy.
pub fn carry_onto_copy(
tx: &Connection,
from: ImageId,
to: ImageId,
) -> Result<FaceCarry, CatalogError> {
let mut out = FaceCarry::default();
let boxes = |image: ImageId| -> Result<Vec<CopyFace>, CatalogError> {
let mut q = tx.prepare(
"SELECT f.id, f.x, f.y, f.w, f.h, fp.person_id, fp.probability, fp.confirmed
FROM faces f
LEFT JOIN face_person fp ON fp.face_id = f.id
WHERE f.image_id = ?1
ORDER BY f.id",
)?;
let rows = q.query_map([image.0 as i64], |r| {
let person: Option<i64> = r.get(5)?;
Ok(CopyFace {
id: r.get(0)?,
rect: (
r.get::<_, f64>(1)? as f32,
r.get::<_, f64>(2)? as f32,
r.get::<_, f64>(3)? as f32,
r.get::<_, f64>(4)? as f32,
),
assignment: match person {
Some(p) => Some((p, r.get::<_, f64>(6)?, r.get::<_, i64>(7)? != 0)),
None => None,
},
})
})?;
Ok(rows.collect::<Result<Vec<_>, _>>()?)
};
let theirs = boxes(from)?;
if theirs.is_empty() {
return Ok(out);
}
let ours = boxes(to)?;
if ours.is_empty() {
tx.execute("DELETE FROM face_index WHERE image_id = ?1", [to.0 as i64])?;
tx.execute(
"UPDATE face_index SET image_id = ?2 WHERE image_id = ?1",
rusqlite::params![from.0 as i64, to.0 as i64],
)?;
out.moved = tx.execute(
"UPDATE faces SET image_id = ?2 WHERE image_id = ?1",
rusqlite::params![from.0 as i64, to.0 as i64],
)?;
return Ok(out);
}
let mut taken = vec![false; ours.len()];
for face in &theirs {
let best = ours
.iter()
.enumerate()
.filter(|(i, _)| !taken[*i])
.map(|(i, o)| (i, iou(face.rect, o.rect)))
.filter(|(_, overlap)| *overlap > 0.5)
.max_by(|a, b| a.1.total_cmp(&b.1));
let Some((at, _)) = best else {
if face.assignment.is_some_and(|(_, _, confirmed)| confirmed) {
out.unmatched_named += 1;
}
continue;
};
taken[at] = true;
let target = &ours[at];
out.rejections += tx.execute(
"INSERT OR IGNORE INTO face_person_rejected (face_id, person_id)
SELECT ?2, person_id FROM face_person_rejected WHERE face_id = ?1",
rusqlite::params![face.id, target.id],
)?;
let Some((person, probability, confirmed)) = face.assignment else {
continue;
};
let carry = match target.assignment {
None => true,
// The same person: only a confirmation upgrades a suggestion.
Some((p, _, theirs_confirmed)) if p == person => confirmed && !theirs_confirmed,
// Another person, only suggested there: the user's word wins.
Some((_, _, false)) => confirmed,
// Another person, confirmed there: the survivor keeps its name.
Some((_, _, true)) => {
if confirmed {
out.conflicts += 1;
}
false
}
};
if carry {
tx.execute(
"INSERT INTO face_person (face_id, person_id, probability, confirmed)
VALUES (?1, ?2, ?3, ?4)
ON CONFLICT(face_id) DO UPDATE SET
person_id = excluded.person_id,
probability = excluded.probability,
confirmed = excluded.confirmed",
rusqlite::params![target.id, person, probability, confirmed],
)?;
// A name the user gave outranks a rejection of the same pair made
// on the survivor — the same order `confirm` applies.
if confirmed {
tx.execute(
"DELETE FROM face_person_rejected WHERE face_id = ?1 AND person_id = ?2",
rusqlite::params![target.id, person],
)?;
}
out.named += 1;
}
}
Ok(out)
}
/// One face as [`carry_onto_copy`] pairs it.
struct CopyFace {
id: i64,
rect: (f32, f32, f32, f32),
assignment: Option<(i64, f64, bool)>,
}
pub(crate) fn now_secs() -> i64 { pub(crate) fn now_secs() -> i64 {
std::time::SystemTime::now() std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH) .duration_since(std::time::UNIX_EPOCH)
@@ -2354,6 +2578,48 @@ mod tests {
assert_eq!(people(&c).unwrap().len(), 1); assert_eq!(people(&c).unwrap().len(), 1);
} }
/// A rejection left on the redirect bound nothing: the next grouping
/// pass suggested the face to the merged person, whom the user had told
/// it was somebody else.
#[test]
fn merging_moves_the_rejections_too() {
let c = db();
let ids: Vec<FaceId> = (1..=3)
.map(|n| {
let img = image(&c, n);
record_detections(&c, img, "w600k_mbf", 1024, &[face(n as u8)]).unwrap()[0]
})
.collect();
let anna = create_person(&c, "Anna").unwrap();
let annie = create_person(&c, "Annie").unwrap();
// Rejected as Annie, and nothing said about Anna.
reject(&c, ids[0], annie).unwrap();
// Rejected as Annie, suggested as Anna: the rejection now covers it.
suggest(&c, ids[1], anna, 0.8).unwrap();
reject(&c, ids[1], annie).unwrap();
// Rejected as Annie, confirmed as Anna: the confirmation stands.
confirm(&c, ids[2], anna).unwrap();
reject(&c, ids[2], annie).unwrap();
merge_people(&c, anna, annie).unwrap();
let rejected: Vec<(i64, i64)> = c
.prepare("SELECT face_id, person_id FROM face_person_rejected ORDER BY face_id")
.unwrap()
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
.unwrap()
.collect::<Result<_, _>>()
.unwrap();
let anna_id = anna.0 as i64;
assert_eq!(
rejected,
[(ids[0].0 as i64, anna_id), (ids[1].0 as i64, anna_id)]
);
assert_eq!(for_image(&c, ImageId(2)).unwrap()[0].person, None);
let kept = &for_image(&c, ImageId(3)).unwrap()[0];
assert_eq!((kept.person, kept.confirmed), (Some(anna), true));
}
#[test] #[test]
fn merging_does_not_duplicate_a_face_both_people_hold() { fn merging_does_not_duplicate_a_face_both_people_hold() {
let c = db(); let c = db();
core/dr-catalog/src/jobs.rs
+53 -5
View File
@@ -29,7 +29,11 @@ pub enum JobKind {
ScanFolder = 0, ScanFolder = 0,
/// Promote an image from stat-only to full EXIF. /// Promote an image from stat-only to full EXIF.
ExtractMetadata = 1, ExtractMetadata = 1,
/// Build or rebuild a thumbnail. /// Build or rebuild a thumbnail. **Retired** — see [`JobKind::RETIRED`].
///
/// Kept so the number stays taken: a catalog written by 0.16.0 or earlier
/// holds rows of kind 2, and reusing it would hand them to whatever took
/// its place.
Thumbnail = 2, Thumbnail = 2,
/// A sidecar on disk is newer than what the catalog read. /// A sidecar on disk is newer than what the catalog read.
ReadSidecar = 3, ReadSidecar = 3,
@@ -69,6 +73,18 @@ impl JobKind {
JobKind::DetectFaces, JobKind::DetectFaces,
]; ];
/// Kinds that are no longer queued by anything, whose rows are deleted on
/// sight by [`drop_retired`].
///
/// `Thumbnail` is here because thumbnails are owed by the store, not by
/// the queue. The grid's worker and the thumbnail sweep both find their
/// work by asking `ThumbStore` what it lacks, and the store is shared
/// between devices, so it is the only thing that can say another device
/// already made one. Up to 0.16.0 every scan enqueued a job per
/// photograph anyway and no handler ever claimed one: the reference
/// catalog held 23,582 of them (#73; catalog.md §6.1).
pub const RETIRED: [JobKind; 1] = [JobKind::Thumbnail];
fn from_i64(v: i64) -> Option<Self> { fn from_i64(v: i64) -> Option<Self> {
Some(match v { Some(match v {
0 => JobKind::ScanFolder, 0 => JobKind::ScanFolder,
@@ -185,7 +201,8 @@ pub fn enqueue(
priority: Priority, priority: Priority,
payload: Option<&str>, payload: Option<&str>,
) -> Result<(), CatalogError> { ) -> Result<(), CatalogError> {
conn.execute( // Cached: a scan enqueues one per photograph it lists.
conn.prepare_cached(
"INSERT INTO jobs(kind, subject_id, priority, state, payload) "INSERT INTO jobs(kind, subject_id, priority, state, payload)
VALUES (?1, ?2, ?3, 0, ?4) VALUES (?1, ?2, ?3, 0, ?4)
ON CONFLICT(kind, subject_id) DO UPDATE SET ON CONFLICT(kind, subject_id) DO UPDATE SET
@@ -195,8 +212,13 @@ pub fn enqueue(
state = CASE WHEN jobs.state = 2 THEN 0 ELSE jobs.state END, state = CASE WHEN jobs.state = 2 THEN 0 ELSE jobs.state END,
attempts = CASE WHEN jobs.state = 2 THEN 0 ELSE jobs.attempts END, attempts = CASE WHEN jobs.state = 2 THEN 0 ELSE jobs.attempts END,
not_before = CASE WHEN jobs.state = 2 THEN 0 ELSE jobs.not_before END", not_before = CASE WHEN jobs.state = 2 THEN 0 ELSE jobs.not_before END",
rusqlite::params![kind as i64, subject_id, priority as i64, payload], )?
)?; .execute(rusqlite::params![
kind as i64,
subject_id,
priority as i64,
payload
])?;
Ok(()) Ok(())
} }
@@ -393,7 +415,7 @@ pub fn recover_orphaned(conn: &Connection) -> Result<usize, CatalogError> {
/// ///
/// Coalescing keeps the table one row per unit of work, but nothing shrinks it /// Coalescing keeps the table one row per unit of work, but nothing shrinks it
/// when the work stops existing: a library that has been culled carries a /// when the work stops existing: a library that has been culled carries a
/// thumbnail job for every photograph deleted since the last time anything /// job for every photograph deleted since the last time anything
/// looked. Each one would be claimed, run, and failed five times. /// looked. Each one would be claimed, run, and failed five times.
/// ///
/// Only kinds whose subject really is an image ([`JobKind::subject_is_image`]) /// Only kinds whose subject really is an image ([`JobKind::subject_is_image`])
@@ -425,6 +447,32 @@ pub fn reap_orphan_subjects(conn: &Connection) -> Result<usize, CatalogError> {
Ok(n) Ok(n)
} }
/// Delete every row of a [`JobKind::RETIRED`] kind.
///
/// Not a migration, deliberately. A schema bump makes an older build refuse
/// the synced catalog snapshot, and a device still on 0.16.0 would lose the
/// catalog to save a megabyte. So this runs where the queue is readied —
/// [`crate::runner::recover`], at every open — and has to be cheap when there
/// is nothing to do: `kind` leads the `UNIQUE(kind, subject_id)` index, so an
/// empty answer is one index probe, not a table scan.
///
/// Every open rather than once, because once is not enough: an older build
/// opening the same catalog enqueues them again on its next scan.
///
/// Rows in any state go. Nothing claims these kinds, so none can be running,
/// and a failed one would be a report about work nobody was going to do.
pub fn drop_retired(conn: &Connection) -> Result<usize, CatalogError> {
let kinds: Vec<i64> = JobKind::RETIRED.iter().map(|k| *k as i64).collect();
let placeholders = std::iter::repeat_n("?", kinds.len())
.collect::<Vec<_>>()
.join(",");
let n = conn.execute(
&format!("DELETE FROM jobs WHERE kind IN ({placeholders})"),
rusqlite::params_from_iter(kinds.iter()),
)?;
Ok(n)
}
/// How much is left, by state. /// How much is left, by state.
/// ///
/// One query rather than a listing, because the caller is a progress line: a /// One query rather than a listing, because the caller is a progress line: a
core/dr-catalog/src/keywords.rs
+29 -2
View File
@@ -244,7 +244,8 @@ pub fn delete(conn: &Connection, id: KeywordId) -> Result<usize, CatalogError> {
/// every assignment, and a query per keyword would be one statement per word /// every assignment, and a query per keyword would be one statement per word
/// in the library. /// in the library.
pub fn list(conn: &Connection) -> Result<Vec<Keyword>, CatalogError> { pub fn list(conn: &Connection) -> Result<Vec<Keyword>, CatalogError> {
let mut stmt = conn.prepare( ensure_term_index(conn);
let mut stmt = conn.prepare_cached(
// DISTINCT image, not row: a word on two versions of one frame is one // DISTINCT image, not row: a word on two versions of one frame is one
// photograph, and reporting two is the kind of small lie that makes a // photograph, and reporting two is the kind of small lie that makes a
// user stop trusting the counts. // user stop trusting the counts.
@@ -269,6 +270,27 @@ pub fn list(conn: &Connection) -> Result<Vec<Keyword>, CatalogError> {
Ok(rows) Ok(rows)
} }
/// The index [`list`]'s per-word count is served from: `keywords_term`
/// with the version beside the word, so the count reads no `keywords` row.
///
/// `keywords_term` alone gave the row id, and each of the 10,800 assignments
/// on the reference library cost a probe of the table for its version --
/// 4 ms of the keyword panel's redraw, on every selection change.
///
/// Created on first use rather than by a migration, for the reason
/// `duplicates::ensure_probe_table` gives: a new schema version makes every
/// older build refuse this catalog's snapshot at sync, and an older build
/// that meets an extra index ignores it. Once it exists, the statement is a
/// lookup in the schema (microseconds). A failure to create it is logged and
/// the list read without it: the index is a speed-up, never an answer.
fn ensure_term_index(conn: &Connection) {
if let Err(e) = conn.execute_batch(
"CREATE INDEX IF NOT EXISTS keywords_term_version ON keywords(keyword, version_id);",
) {
log::warn!("keywords: could not create keywords_term_version: {e}");
}
}
/// Assign a keyword to images, creating the keyword if it is new. /// Assign a keyword to images, creating the keyword if it is new.
/// ///
/// The bulk form is the *only* form, because keywording a selection is the /// The bulk form is the *only* form, because keywording a selection is the
@@ -491,7 +513,12 @@ pub fn adopt_orphan_terms(conn: &Connection) -> Result<usize, CatalogError> {
// quietly readmitted to the vocabulary; it stays visible as an // quietly readmitted to the vocabulary; it stays visible as an
// orphan in [`for_images`] instead, which is a state someone can // orphan in [`for_images`] instead, which is a state someone can
// see and act on rather than one that silently undoes a deletion. // see and act on rather than one that silently undoes a deletion.
"SELECT DISTINCT k.keyword FROM keywords k //
// The distinct words first, then the check: the vocabulary has no
// index a tombstone-inclusive lookup can use, so checking once per
// assignment scanned it 10,000 times on every open. Once per word
// is a few dozen scans of a few dozen rows.
"SELECT k.keyword FROM (SELECT DISTINCT keyword FROM keywords) k
WHERE NOT EXISTS (SELECT 1 FROM keyword_terms t WHERE NOT EXISTS (SELECT 1 FROM keyword_terms t
WHERE t.name = k.keyword)", WHERE t.name = k.keyword)",
)?; )?;
core/dr-catalog/src/lib.rs
+17 -2
View File
@@ -36,10 +36,14 @@ use std::path::Path;
use dr_types::{Availability, ImageId}; use dr_types::{Availability, ImageId};
use rusqlite::Connection; use rusqlite::Connection;
pub mod albums;
mod backfilled;
pub mod bursts; pub mod bursts;
pub mod cache; pub mod cache;
pub mod collections; pub mod collections;
pub mod dedup; pub mod dedup;
pub mod dedup_people;
pub mod duplicates;
pub mod error; pub mod error;
pub mod face_shard; pub mod face_shard;
pub mod faces; pub mod faces;
@@ -56,6 +60,7 @@ pub mod sync;
pub mod trash; pub mod trash;
pub mod walk; pub mod walk;
pub use albums::{Album, AlbumId, Place};
pub use cache::{Budget, Cache, DEFAULT_BUDGET_BYTES}; pub use cache::{Budget, Cache, DEFAULT_BUDGET_BYTES};
pub use collections::{Collection, CollectionKind, TreeRow}; pub use collections::{Collection, CollectionKind, TreeRow};
pub use dedup::{seen_by_content, seen_by_metadata, set_content_hash}; pub use dedup::{seen_by_content, seen_by_metadata, set_content_hash};
@@ -246,8 +251,18 @@ impl Catalog {
// A migration adds a column; it cannot know what the value should be // A migration adds a column; it cannot know what the value should be
// for rows that already existed. Backfilling on open is what stops // for rows that already existed. Backfilling on open is what stops
// those rows being silently partial. // those rows being silently partial.
for (what, n) in schema::backfill(&conn)? { //
log::info!("backfilled {what} for {n} row(s) (schema was v{from})"); // Once per catalog state rather than once per open (NFR-P9): every
// worker thread opens its own connection, and a develop landing made
// five, each paying the whole backfill to confirm nothing had changed.
// [`backfilled`] says what "changed" means and why it is enough. A
// migration always backfills, stamp or no stamp.
let stamp = backfilled::stamp(&conn, path)?;
if from < schema::SCHEMA_VERSION || !backfilled::is_current(path, &stamp) {
for (what, n) in schema::backfill(&conn)? {
log::info!("backfilled {what} for {n} row(s) (schema was v{from})");
}
backfilled::record(path, stamp);
} }
Ok(Catalog { conn }) Ok(Catalog { conn })
} }
core/dr-catalog/src/merge.rs
+792 -105
View File
File diff suppressed because it is too large Load Diff
core/dr-catalog/src/rating.rs
+156 -17
View File
@@ -449,19 +449,26 @@ pub fn toggled_label(
/// TRACES: FR-CAT-5 | FR-CAT-6 /// TRACES: FR-CAT-5 | FR-CAT-6
/// How the library divides by colour label, for the filter chips' counts. /// How the library divides by colour label, for the filter chips' counts.
/// ///
/// Index 0 is unlabelled and index `n` the label whose code is `n`. One /// Index 0 is unlabelled and index `n` the label whose code is `n`. The
/// grouped statement — the same shape as [`rating_histogram`], and for the /// same shape as [`rating_histogram`], and for the same reason the
/// same reason it LEFT JOINs: an image without a version row is unlabelled, /// unlabelled slot is what is left of [`judged_rows`]: an image without a
/// not missing. /// version row is unlabelled, not missing.
///
/// Only labelled rows are grouped. The join this replaced (2026-09-26)
/// probed `versions_judgement` per image and then read each version's row
/// for `label`, which the index does not carry -- 10 ms on the reference
/// library, on every label keystroke, to find that none of 23,500 images
/// had one. This walks the default versions in the index's order, which
/// is close to the table's, and groups the few that are labelled.
pub fn label_histogram(conn: &Connection) -> Result<[usize; 6], CatalogError> { pub fn label_histogram(conn: &Connection) -> Result<[usize; 6], CatalogError> {
let mut out = [0usize; 6]; let mut out = [0usize; 6];
let mut stmt = conn.prepare( let mut stmt = conn.prepare_cached(
"SELECT coalesce(v.label, 0) AS l, count(*) "SELECT label, count(*) FROM versions
FROM images i WHERE is_default = 1 AND label IS NOT NULL
LEFT JOIN versions v ON v.image_id = i.id AND v.is_default = 1 GROUP BY label",
GROUP BY l",
)?; )?;
let rows = stmt.query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?)))?; let rows = stmt.query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?)))?;
let mut counted = 0usize;
for (code, count) in rows.flatten() { for (code, count) in rows.flatten() {
// A code this build does not know counts as unlabelled, which is how // A code this build does not know counts as unlabelled, which is how
// `label_from_code` reads it everywhere else. // `label_from_code` reads it everywhere else.
@@ -471,7 +478,9 @@ pub fn label_histogram(conn: &Connection) -> Result<[usize; 6], CatalogError> {
0 0
}; };
out[slot] += count as usize; out[slot] += count as usize;
counted += count as usize;
} }
out[0] += judged_rows(conn)?.saturating_sub(counted);
Ok(out) Ok(out)
} }
@@ -582,25 +591,61 @@ pub fn judgements(
pub fn rating_histogram(conn: &Connection) -> Result<[usize; 6], CatalogError> { pub fn rating_histogram(conn: &Connection) -> Result<[usize; 6], CatalogError> {
let mut out = [0usize; 6]; let mut out = [0usize; 6];
// LEFT JOIN, so an image whose version row is missing still counts as // Only the rated rows are grouped; the unrated slot is what is left of
// unrated rather than vanishing from the totals. The histogram has to sum // [`judged_rows`]. So an image whose version row is missing still counts
// to the library size or it is not believable. // as unrated rather than vanishing from the totals -- the histogram has
let mut stmt = conn.prepare( // to sum to the library size or it is not believable.
"SELECT coalesce(v.rating, 0) AS r, count(*) //
FROM images i // It was one `images LEFT JOIN versions ... GROUP BY` until 2026-09-26:
LEFT JOIN versions v ON v.image_id = i.id AND v.is_default = 1 // a probe of `versions_judgement` per image and a sort of every row, to
GROUP BY r", // put 22,000 of 23,500 in slot zero. 9 ms on every star keystroke on the
// reference library; this is a pass over the index that sorts only the
// rated few, and [`judged_rows`] is three index-only counts.
let mut stmt = conn.prepare_cached(
"SELECT rating, count(*) FROM versions
WHERE is_default = 1 AND rating != 0
GROUP BY rating",
)?; )?;
let rows = stmt.query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?)))?; let rows = stmt.query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?)))?;
let mut counted = 0usize;
for (rating, count) in rows.flatten() { for (rating, count) in rows.flatten() {
if let Some(slot) = out.get_mut(rating.clamp(0, MAX_RATING as i64) as usize) { if let Some(slot) = out.get_mut(rating.clamp(0, MAX_RATING as i64) as usize) {
*slot += count as usize; *slot += count as usize;
counted += count as usize;
} }
} }
out[0] += judged_rows(conn)?.saturating_sub(counted);
Ok(out) Ok(out)
} }
/// How many rows `images LEFT JOIN versions ON ... AND is_default = 1` has:
/// one per image with no default version, and one per default version for
/// the rest. The total both histograms divide up, and the unjudged slot is
/// what is left of it once the judged rows are counted.
///
/// Spelled as three counts rather than as that join because the join probes
/// `versions_judgement` once per image, where each count here is one pass
/// over an index without reading a row: the library size, the default
/// versions, and the images holding one. An image with two default versions
/// -- nothing prevents it -- is two rows of the join and one image of the
/// third count, so it adds one here exactly as it did there. A version
/// always belongs to an image; `foreign_keys` is on and deletes cascade.
///
/// `count(DISTINCT image_id)` alone in its statement: that is what lets
/// SQLite read the distinct values off the index's order instead of
/// building a temporary b-tree of them.
fn judged_rows(conn: &Connection) -> Result<usize, CatalogError> {
let n: i64 = conn
.prepare_cached(
"SELECT (SELECT count(*) FROM images)
+ (SELECT count(*) FROM versions WHERE is_default = 1)
- (SELECT count(DISTINCT image_id) FROM versions WHERE is_default = 1)",
)?
.query_row([], |r| r.get(0))?;
Ok(n.max(0) as usize)
}
/// How many images carry each flag: `(picks, rejects)`. /// How many images carry each flag: `(picks, rejects)`.
pub fn flag_counts(conn: &Connection) -> Result<(usize, usize), CatalogError> { pub fn flag_counts(conn: &Connection) -> Result<(usize, usize), CatalogError> {
let picks: i64 = conn.query_row( let picks: i64 = conn.query_row(
@@ -987,6 +1032,100 @@ mod tests {
assert_eq!(h.iter().sum::<usize>(), 4); assert_eq!(h.iter().sum::<usize>(), 4);
} }
/// The rows of the join the histograms used to be spelled as, grouped the
/// way `rating_histogram` groups them. What the counts must still agree
/// with, in the states nothing in the schema prevents.
fn by_join(cat: &Catalog, column: &str) -> Vec<(i64, i64)> {
cat.connection()
.prepare(&format!(
"SELECT coalesce(v.{column}, 0) AS c, count(*)
FROM images i
LEFT JOIN versions v ON v.image_id = i.id AND v.is_default = 1
GROUP BY c ORDER BY c"
))
.unwrap()
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
.unwrap()
.map(Result::unwrap)
.collect()
}
/// A library in every awkward state at once: an image with no version,
/// one with only a virtual copy, one with two default versions, and
/// values out of range on both axes.
fn awkward() -> Catalog {
let cat = with_images(8);
ensure_default_versions(cat.connection()).unwrap();
let all = ids(&cat);
let c = cat.connection();
set_rating(c, all[0], 5).unwrap();
set_rating(c, all[1], 2).unwrap();
set_label(c, all[1], Some(ColourLabel::Blue)).unwrap();
c.execute(
"DELETE FROM versions WHERE image_id = ?1",
[all[2].0 as i64],
)
.unwrap();
c.execute(
"UPDATE versions SET is_default = 0 WHERE image_id = ?1",
[all[3].0 as i64],
)
.unwrap();
c.execute(
"INSERT INTO versions(image_id, uuid, name, is_default, rating, label)
VALUES (?1, 'second-default', 'Copy', 1, 4, 3)",
[all[4].0 as i64],
)
.unwrap();
c.execute(
"UPDATE versions SET rating = -1, label = 9 WHERE image_id = ?1",
[all[5].0 as i64],
)
.unwrap();
c.execute(
"UPDATE versions SET rating = 7, label = 0 WHERE image_id = ?1",
[all[6].0 as i64],
)
.unwrap();
cat
}
/// Fold the join's rows into slots the way the old code did.
fn folded(rows: &[(i64, i64)], slot: impl Fn(i64) -> usize) -> [usize; 6] {
let mut out = [0usize; 6];
for &(code, n) in rows {
out[slot(code)] += n as usize;
}
out
}
#[test]
fn the_rating_histogram_agrees_with_the_join_it_replaced() {
let cat = awkward();
let expected = folded(&by_join(&cat, "rating"), |r| {
r.clamp(0, MAX_RATING as i64) as usize
});
assert_eq!(rating_histogram(cat.connection()).unwrap(), expected);
// Nine rows for eight images: the doubled default counts twice, as
// it always has.
assert_eq!(expected.iter().sum::<usize>(), 9);
}
#[test]
fn the_label_histogram_agrees_with_the_join_it_replaced() {
let cat = awkward();
let expected = folded(&by_join(&cat, "label"), |code| {
if label_from_code(Some(code)).is_some() {
code as usize
} else {
0
}
});
assert_eq!(label_histogram(cat.connection()).unwrap(), expected);
assert_eq!(expected[3], 1, "the second default's label is counted");
assert_eq!(expected.iter().sum::<usize>(), 9);
}
#[test] #[test]
fn flag_counts_separate_picks_from_rejects() { fn flag_counts_separate_picks_from_rejects() {
let cat = with_images(5); let cat = with_images(5);
core/dr-catalog/src/recovery.rs
+4
View File
@@ -322,6 +322,10 @@ pub fn restore(catalog: &Path, backup: &Path) -> Result<(), CatalogError> {
/// to move — a caller may be recovering from a file SQLite could not open /// to move — a caller may be recovering from a file SQLite could not open
/// because it was never created. /// because it was never created.
pub fn set_aside(catalog: &Path) -> Result<Option<PathBuf>, CatalogError> { pub fn set_aside(catalog: &Path) -> Result<Option<PathBuf>, CatalogError> {
// Whatever takes this name next — a rebuild or a restored backup — is not
// the file this process last backfilled. Forgotten while the path still
// resolves, so it is the same key the open recorded.
crate::backfilled::forget(catalog);
let moved = if catalog.exists() { let moved = if catalog.exists() {
let dest = with_suffix(catalog, DAMAGED_SUFFIX); let dest = with_suffix(catalog, DAMAGED_SUFFIX);
// An earlier damaged copy is replaced rather than accumulating: two of // An earlier damaged copy is replaced rather than accumulating: two of
core/dr-catalog/src/runner.rs
+57 -11
View File
@@ -77,7 +77,7 @@ pub enum Outcome {
/// Something that can actually do the work a job describes. /// Something that can actually do the work a job describes.
/// ///
/// The catalog knows what needs doing and nothing about how — a thumbnail /// The catalog knows what needs doing and nothing about how — face detection
/// needs a decoder, a fetch needs a network stack, and neither belongs under /// needs a decoder, a fetch needs a network stack, and neither belongs under
/// `core/dr-catalog` (ARCH §4.1: calls go downward). So the queue lives here /// `core/dr-catalog` (ARCH §4.1: calls go downward). So the queue lives here
/// and the handlers are supplied from above. /// and the handlers are supplied from above.
@@ -187,17 +187,19 @@ pub struct Recovered {
pub reclaimed: usize, pub reclaimed: usize,
/// Jobs deleted because the photograph they name no longer exists. /// Jobs deleted because the photograph they name no longer exists.
pub reaped: usize, pub reaped: usize,
/// Jobs deleted because their kind is retired ([`JobKind::RETIRED`]).
pub retired: usize,
} }
impl Recovered { impl Recovered {
pub fn did_anything(&self) -> bool { pub fn did_anything(&self) -> bool {
self.reclaimed > 0 || self.reaped > 0 self.reclaimed > 0 || self.reaped > 0 || self.retired > 0
} }
} }
/// Ready the queue for a fresh run, before any worker touches it. /// Ready the queue for a fresh run, before any worker touches it.
/// ///
/// Two distinct cleanups, and both are startup-only: /// Three distinct cleanups, and all are startup-only:
/// ///
/// - **Reclaim.** A `Running` row has no owner; the process that claimed it is /// - **Reclaim.** A `Running` row has no owner; the process that claimed it is
/// gone. On Android that is a routine morning, not a crash (FR-PLAT-AND-3). /// gone. On Android that is a routine morning, not a crash (FR-PLAT-AND-3).
@@ -205,19 +207,27 @@ impl Recovered {
/// process down with it three times running should not be retried forever, /// process down with it three times running should not be retried forever,
/// and the attempt counter is the only evidence of that we have. /// and the attempt counter is the only evidence of that we have.
/// - **Reap.** Jobs naming an image the catalog no longer has. A library that /// - **Reap.** Jobs naming an image the catalog no longer has. A library that
/// has been culled leaves thumbnail jobs for photographs that were deleted /// has been culled leaves jobs for photographs that were deleted
/// months ago, and every one of them would be claimed, run and failed. /// months ago, and every one of them would be claimed, run and failed.
/// - **Retire.** Rows of a kind nothing enqueues or claims any more
/// ([`jobs::drop_retired`]). Here rather than in a migration so that no
/// schema bump locks an older device out of the synced catalog, and every
/// time rather than once because an older build sharing the catalog will
/// queue them again.
/// ///
/// Reclaim runs first so its count is the honest number of interrupted jobs, /// Retiring runs first, so the other two never touch rows about to go.
/// Reclaim runs next so its count is the honest number of interrupted jobs,
/// before reaping removes whichever of them pointed at nothing. /// before reaping removes whichever of them pointed at nothing.
/// ///
/// **Call this exactly once per catalog, at startup.** It cannot distinguish a /// **Call this exactly once per catalog, at startup.** It cannot distinguish a
/// job a dead process was holding from one a live runner is holding right now, /// job a dead process was holding from one a live runner is holding right now,
/// because there is no owner column — the queue is durable, not distributed. /// because there is no owner column — the queue is durable, not distributed.
pub fn recover(conn: &Connection) -> Result<Recovered, CatalogError> { pub fn recover(conn: &Connection) -> Result<Recovered, CatalogError> {
let retired = jobs::drop_retired(conn)?;
Ok(Recovered { Ok(Recovered {
reclaimed: jobs::recover_orphaned(conn)?, reclaimed: jobs::recover_orphaned(conn)?,
reaped: jobs::reap_orphan_subjects(conn)?, reaped: jobs::reap_orphan_subjects(conn)?,
retired,
}) })
} }
@@ -729,7 +739,7 @@ mod tests {
// which is the window a durable queue exists to survive: no `complete`, // which is the window a durable queue exists to survive: no `complete`,
// no `fail`, just a row marked `Running` with nobody holding it. // no `fail`, just a row marked `Running` with nobody holding it.
let c = db(); let c = db();
queued(&c, JobKind::Thumbnail, 1); queued(&c, JobKind::ContentHash, 1);
// The dead process. It claimed the job and never came back. // The dead process. It claimed the job and never came back.
let claimed = jobs::claim_next(&c, 0).unwrap().expect("claimable"); let claimed = jobs::claim_next(&c, 0).unwrap().expect("claimable");
@@ -738,7 +748,7 @@ mod tests {
// A fresh runner, before it starts, finds the queue empty — the row is // A fresh runner, before it starts, finds the queue empty — the row is
// `Running` and no claim will touch it. // `Running` and no claim will touch it.
let seen = Arc::new(Mutex::new(Vec::new())); let seen = Arc::new(Mutex::new(Vec::new()));
let mut runner = Runner::new(&c).with(recording(vec![JobKind::Thumbnail], seen.clone())); let mut runner = Runner::new(&c).with(recording(vec![JobKind::ContentHash], seen.clone()));
assert_eq!( assert_eq!(
runner.drain_all(0).unwrap().ran(), runner.drain_all(0).unwrap().ran(),
0, 0,
@@ -766,7 +776,7 @@ mod tests {
// the only evidence we keep across a death. Without this a poison-pill // the only evidence we keep across a death. Without this a poison-pill
// job would be reclaimed and re-run forever. // job would be reclaimed and re-run forever.
let c = db(); let c = db();
queued(&c, JobKind::Thumbnail, 1); queued(&c, JobKind::ContentHash, 1);
for _ in 0..MAX_ATTEMPTS { for _ in 0..MAX_ATTEMPTS {
jobs::claim_next(&c, 0).unwrap().expect("claimable"); jobs::claim_next(&c, 0).unwrap().expect("claimable");
@@ -782,13 +792,49 @@ mod tests {
assert_eq!(state, JobState::Failed as i64); assert_eq!(state, JobState::Failed as i64);
} }
#[test]
fn recovery_drops_retired_kinds_every_time_and_nothing_else() {
// What 0.16.0 left behind: a thumbnail job per photograph that nothing
// would ever claim, beside live work that must survive.
let c = db();
queued(&c, JobKind::Thumbnail, 1);
queued(&c, JobKind::Thumbnail, 2);
enqueue(
&c,
JobKind::DetectFaces,
Some(1),
Priority::Background,
None,
)
.unwrap();
let first = recover(&c).unwrap();
assert_eq!(first.retired, 2);
assert!(first.did_anything());
let kinds: Vec<i64> = c
.prepare("SELECT kind FROM jobs")
.unwrap()
.query_map([], |r| r.get(0))
.unwrap()
.map(Result::unwrap)
.collect();
assert_eq!(kinds, vec![JobKind::DetectFaces as i64]);
// An older build opening the same catalog queues them again on its
// next scan. The next open by this one clears them again.
enqueue(&c, JobKind::Thumbnail, Some(1), Priority::Background, None).unwrap();
assert_eq!(recover(&c).unwrap().retired, 1);
assert_eq!(recover(&c).unwrap(), Recovered::default());
}
#[test] #[test]
fn recovery_drops_jobs_whose_photograph_is_gone() { fn recovery_drops_jobs_whose_photograph_is_gone() {
// A culled library leaves thumbnail jobs for images deleted months // A culled library leaves thumbnail jobs for images deleted months
// ago. Every one would be claimed, run and failed. // ago. Every one would be claimed, run and failed.
let c = db(); let c = db();
queued(&c, JobKind::Thumbnail, 1); queued(&c, JobKind::ContentHash, 1);
queued(&c, JobKind::Thumbnail, 2); queued(&c, JobKind::ContentHash, 2);
c.execute("DELETE FROM images WHERE id = 2", []).unwrap(); c.execute("DELETE FROM images WHERE id = 2", []).unwrap();
let recovered = recover(&c).unwrap(); let recovered = recover(&c).unwrap();
@@ -804,7 +850,7 @@ mod tests {
#[test] #[test]
fn a_quiet_startup_recovers_nothing() { fn a_quiet_startup_recovers_nothing() {
let c = db(); let c = db();
queued(&c, JobKind::Thumbnail, 1); queued(&c, JobKind::ContentHash, 1);
assert_eq!(recover(&c).unwrap(), Recovered::default()); assert_eq!(recover(&c).unwrap(), Recovered::default());
assert!(!recover(&c).unwrap().did_anything()); assert!(!recover(&c).unwrap().did_anything());
} }
core/dr-catalog/src/schema.rs
+83 -23
View File
@@ -301,8 +301,11 @@ pub const EYE_COLUMNS: [&str; 7] = [
/// silently partial — present, queryable, and wrong — which is worse than /// silently partial — present, queryable, and wrong — which is worse than
/// missing, because nothing signals that they need attention. /// missing, because nothing signals that they need attention.
/// ///
/// Cheap enough to run on every open: each pass is one indexed UPDATE, and /// Idempotent: re-running it is a no-op once the values are already right.
/// re-running it is a no-op once the values are already right. /// [`crate::Catalog::open`] runs it once per catalog state rather than on
/// every open — each pass scans a whole table, and together they were most of
/// what an open cost — and `backfilled` in this crate says what counts as a
/// new state.
/// ///
/// Returns how many rows each backfill touched, for logging. /// Returns how many rows each backfill touched, for logging.
pub fn backfill(conn: &Connection) -> Result<Vec<(&'static str, usize)>, CatalogError> { pub fn backfill(conn: &Connection) -> Result<Vec<(&'static str, usize)>, CatalogError> {
@@ -494,15 +497,51 @@ fn rewrite_for_attached(sql: &str, schema_name: &str) -> String {
fn pair_raw_and_jpeg(conn: &Connection) -> Result<usize, CatalogError> { fn pair_raw_and_jpeg(conn: &Connection) -> Result<usize, CatalogError> {
use std::collections::HashMap; use std::collections::HashMap;
// (folder, lowercase stem) -> RAW id, built in one pass over the RAWs. // The small side first: the JPEGs not yet paired. On a settled library
// these are the ones with no RAW beside them -- 1,900 of 24,000 on the
// reference library -- and this runs on every open, including the ones
// the develop view makes for each photograph it fetches. Reading every
// RAW to find the handful that share a folder with one of them was most
// of what opening the catalog cost.
let jpegs: Vec<(i64, Option<i64>, String)> = {
let mut stmt = conn.prepare(
"SELECT id, folder_id, source_ref FROM images
WHERE lower(format) IN ('jpg','jpeg') AND shadowed_by IS NULL",
)?;
let rows = stmt.query_map([], |r| {
Ok((
r.get::<_, i64>(0)?,
r.get::<_, Option<i64>>(1)?,
r.get::<_, String>(2)?,
))
})?;
rows.filter_map(Result::ok).collect()
};
if jpegs.is_empty() {
return Ok(0);
}
// (folder, lowercase stem) -> RAW id, over the folders those JPEGs are in
// and no others: a pair is same-folder by definition. Ordered by id so
// that where two RAWs share a stem the later one wins, as it did when this
// read every RAW in table order.
let mut folders: Vec<i64> = jpegs.iter().filter_map(|(_, f, _)| *f).collect();
folders.sort_unstable();
folders.dedup();
let unfiled = jpegs.iter().any(|(_, f, _)| f.is_none());
let folders = serde_json::to_string(&folders).unwrap_or_else(|_| "[]".to_string());
let mut raws: HashMap<(Option<i64>, String), i64> = HashMap::new(); let mut raws: HashMap<(Option<i64>, String), i64> = HashMap::new();
{ {
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT id, folder_id, source_ref FROM images "SELECT id, folder_id, source_ref FROM images
WHERE lower(format) IN WHERE lower(format) IN
('cr2','cr3','nef','arw','raf','rw2','orf','dng')", ('cr2','cr3','nef','arw','raf','rw2','orf','dng')
AND (folder_id IN (SELECT value FROM json_each(?1))
OR (?2 AND folder_id IS NULL))
ORDER BY id",
)?; )?;
let rows = stmt.query_map([], |r| { let rows = stmt.query_map(rusqlite::params![folders, unfiled], |r| {
Ok(( Ok((
r.get::<_, i64>(0)?, r.get::<_, i64>(0)?,
r.get::<_, Option<i64>>(1)?, r.get::<_, Option<i64>>(1)?,
@@ -518,25 +557,16 @@ fn pair_raw_and_jpeg(conn: &Connection) -> Result<usize, CatalogError> {
return Ok(0); return Ok(0);
} }
let pairs: Vec<(i64, i64)> = { let pairs: Vec<(i64, i64)> = jpegs
let mut stmt = conn.prepare( .iter()
"SELECT id, folder_id, source_ref FROM images .filter_map(|(id, folder, path)| {
WHERE lower(format) IN ('jpg','jpeg') AND shadowed_by IS NULL", let raw = raws.get(&(*folder, stem_of(path).to_ascii_lowercase()))?;
)?; Some((*id, *raw))
let rows = stmt.query_map([], |r| {
Ok((
r.get::<_, i64>(0)?,
r.get::<_, Option<i64>>(1)?,
r.get::<_, String>(2)?,
))
})?;
rows.filter_map(|row| {
let (id, folder, path) = row.ok()?;
let raw = raws.get(&(folder, stem_of(&path).to_ascii_lowercase()))?;
Some((id, *raw))
}) })
.collect() .collect();
}; if pairs.is_empty() {
return Ok(0);
}
let tx = conn.unchecked_transaction()?; let tx = conn.unchecked_transaction()?;
for (jpeg, raw) in &pairs { for (jpeg, raw) in &pairs {
@@ -1582,6 +1612,36 @@ mod tests {
assert_eq!(backfilled(&c, "shadowed_by"), 0); assert_eq!(backfilled(&c, "shadowed_by"), 0);
} }
#[test]
fn a_pair_is_found_among_other_folders_and_unfiled_images() {
// The RAWs are read only from the folders an unpaired JPEG is in,
// plus the unfiled ones when an unfiled JPEG is waiting: each JPEG
// must still find its own sibling, and only its own.
let c = with_root();
let raw_a = image(&c, Some(1), "a/IMG_7.CR2", "cr2");
image(&c, Some(2), "b/IMG_7.CR2", "cr2");
image(&c, Some(2), "b/IMG_8.CR2", "cr2");
let raw_unfiled = image(&c, None, "IMG_9.DNG", "dng");
let jpeg_a = image(&c, Some(1), "a/IMG_7.JPG", "jpg");
let jpeg_unfiled = image(&c, None, "IMG_9.jpg", "jpg");
image(&c, Some(1), "a/IMG_9.jpg", "jpg");
assert_eq!(backfilled(&c, "shadowed_by"), 2);
let of = |id: i64| -> Option<i64> {
c.query_row("SELECT shadowed_by FROM images WHERE id = ?1", [id], |r| {
r.get(0)
})
.unwrap()
};
assert_eq!(of(jpeg_a), Some(raw_a));
assert_eq!(of(jpeg_unfiled), Some(raw_unfiled));
assert_eq!(
backfilled(&c, "shadowed_by"),
0,
"settled on the second pass"
);
}
#[test] #[test]
fn a_raw_is_never_shadowed_by_a_jpeg() { fn a_raw_is_never_shadowed_by_a_jpeg() {
// The relationship is one-way: the RAW is the photograph. // The relationship is one-way: the RAW is the photograph.
core/dr-catalog/src/sync.rs
+493 -45
View File
@@ -46,18 +46,210 @@ pub fn checkpoint(conn: &Connection) -> Result<(), CatalogError> {
Ok(()) Ok(())
} }
/// Write a consistent snapshot of the catalog to `dest`, ready to upload. /// Write a consistent snapshot of the catalog to `dest`, ready to upload,
/// without the face crops.
/// ///
/// Uses the backup API rather than a filesystem copy so the snapshot is /// Built rather than copied. The snapshot is the *whole catalog* bar the
/// coherent even with writers active. Callers should still prefer a quiet /// crops, uploaded on every sync and downloaded by every device. The crops are
/// moment — this competes with background jobs for the write lock. /// most of the file (96 MB of a 158 MB reference catalog), and copying them in
/// only to delete them was most of the cost. A backup-API copy followed by
/// `UPDATE faces SET crop = NULL` and `VACUUM` wrote the file roughly three
/// times over to produce 50 MB (#71). So this creates the schema in an empty
/// file and copies every table into it with `crop` left NULL. That is one
/// pass, with nothing written that is not uploaded.
///
/// Consistency comes from doing the whole copy inside one transaction on the
/// snapshot's connection, which holds a single read snapshot of the source for
/// its duration. A writer committing meanwhile lands in the source's WAL and is
/// simply not seen, the same serialisation the backup API gave.
///
/// Crops are not lost by this: they travel in the face shards
/// ([`crate::face_shard::export_to_shards`]), which are written once and
/// downloaded once. Nothing reads a crop out of a merged remote catalog. The
/// merge reads a remote face's box and model to match it to a local one, and
/// no more. So leaving them out costs a receiving device nothing it would
/// otherwise have had. A device never adopts a downloaded catalog as its own,
/// so a fresh one gets its crops from the shards too.
///
/// The result must stay what every earlier build already merges: same schema,
/// same `user_version`, same page size and the same WAL flag in the header.
/// The `the_snapshot_*` tests pin those.
pub fn snapshot_for_upload(conn: &Connection, dest: &Path) -> Result<(), CatalogError> { pub fn snapshot_for_upload(conn: &Connection, dest: &Path) -> Result<(), CatalogError> {
let out = copy_to(conn, dest)?; // The source is read through a second connection, attached to the
strip_face_crops(&out)?; // snapshot's, so it needs to be a file. Every catalog is one.
let source = conn
.path()
.filter(|p| !p.is_empty())
.map(PathBuf::from)
.ok_or_else(|| CatalogError::Io("the catalog to snapshot has no file".into()))?;
// Not needed for consistency, since the read transaction below sees the
// WAL, but it keeps the live WAL from growing across syncs, as before.
checkpoint(conn)?;
// A leftover from a pass that died mid-build would otherwise be built on.
for stale in [
dest.to_path_buf(),
sidecar_of(dest, "-wal"),
sidecar_of(dest, "-journal"),
sidecar_of(dest, "-shm"),
] {
match std::fs::remove_file(&stale) {
Ok(()) => {}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => return Err(CatalogError::Io(format!("{}: {e}", stale.display()))),
}
}
let out = Connection::open(dest)?;
build_snapshot(conn, &source, &out)?;
// This device's local album folders are paths and SAF grants nobody
// else can use; the merge never reads them, and the snapshot is what
// a fresh device would otherwise adopt whole.
out.execute_batch("DROP TABLE IF EXISTS album_folders")?;
verify_snapshot(&out)?; verify_snapshot(&out)?;
Ok(()) Ok(())
} }
/// `catalog.sqlite` + `-wal` → `catalog.sqlite-wal`.
fn sidecar_of(path: &Path, suffix: &str) -> PathBuf {
let mut s = path.as_os_str().to_owned();
s.push(suffix);
PathBuf::from(s)
}
/// Schema name the source catalog is attached under while a snapshot is built.
const SOURCE_SCHEMA: &str = "snap_src";
/// The body of [`snapshot_for_upload`]: fill the empty database `out` from
/// the catalog at `source`.
fn build_snapshot(conn: &Connection, source: &Path, out: &Connection) -> Result<(), CatalogError> {
// Settings that only take on an empty file, copied from the source so the
// result is the file a backup would have been.
let page_size: i64 = conn.query_row("PRAGMA main.page_size", [], |r| r.get(0))?;
let auto_vacuum: i64 = conn.query_row("PRAGMA main.auto_vacuum", [], |r| r.get(0))?;
out.pragma_update(None, "page_size", page_size)?;
out.pragma_update(None, "auto_vacuum", auto_vacuum)?;
// A scratch file, rebuilt whole on every pass and checked before upload:
// durability during the build buys nothing. MEMORY rather than OFF keeps
// ROLLBACK defined on the failure path.
out.pragma_update(None, "journal_mode", "MEMORY")?;
out.pragma_update(None, "synchronous", "OFF")?;
// The rows were checked when they were written, and the copy has them
// all by the end. The bundled SQLite turns foreign keys on by default,
// and with them a multi-row INSERT into `images` scans `images` for
// children of every row it adds (`shadowed_by` refers to the same table
// and has no index): 1.2 s of a 1.7 s snapshot on 24k images.
out.pragma_update(None, "foreign_keys", false)?;
// Bound as a parameter, so a path containing a quote cannot break out.
out.execute(
&format!("ATTACH DATABASE ?1 AS {SOURCE_SCHEMA}"),
[source.to_string_lossy().as_ref()],
)?;
let result = copy_schema_and_rows(out);
if let Err(e) = out.execute(&format!("DETACH DATABASE {SOURCE_SCHEMA}"), []) {
log::warn!("failed to detach the catalog from its snapshot: {e}");
}
result?;
// Last, and outside any transaction, which is the only place it can be
// set: the header says WAL, as every snapshot uploaded so far has.
out.pragma_update(None, "journal_mode", "WAL")?;
Ok(())
}
fn copy_schema_and_rows(out: &Connection) -> Result<(), CatalogError> {
let tx = out.unchecked_transaction()?;
// The first read of the source opens its read snapshot. Everything from
// here, schema included, is as of that one moment.
let objects: Vec<(String, String, String)> = {
let mut stmt = tx.prepare(&format!(
"SELECT type, name, sql FROM {SOURCE_SCHEMA}.sqlite_master
WHERE sql IS NOT NULL AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
ORDER BY rowid"
))?;
let rows = stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))?;
rows.collect::<Result<_, _>>()?
};
let user_version: i64 =
tx.query_row(&format!("PRAGMA {SOURCE_SCHEMA}.user_version"), [], |r| {
r.get(0)
})?;
let application_id: i64 =
tx.query_row(&format!("PRAGMA {SOURCE_SCHEMA}.application_id"), [], |r| {
r.get(0)
})?;
// Tables and their rows first, then indexes, triggers and views, so that
// an index is built once over the data rather than maintained per row,
// and no trigger fires on the copy. Foreign keys are off on this
// connection, so the order tables are filled in does not matter.
for (_, name, sql) in objects.iter().filter(|(k, _, _)| k == "table") {
// Verbatim: an unqualified CREATE lands in `main`, the snapshot.
tx.execute_batch(sql)?;
let columns: Vec<String> = {
let mut stmt = tx.prepare("SELECT name FROM pragma_table_info(?1, 'main')")?;
let rows = stmt.query_map([name], |r| r.get::<_, String>(0))?;
rows.collect::<Result<_, _>>()?
};
let select = columns
.iter()
.map(|c| {
if name == "faces" && c == "crop" {
"NULL".to_string()
} else {
quote_ident(c)
}
})
.collect::<Vec<_>>()
.join(", ");
let insert = columns
.iter()
.map(|c| quote_ident(c))
.collect::<Vec<_>>()
.join(", ");
let table = quote_ident(name);
tx.execute(
&format!(
"INSERT INTO main.{table} ({insert})
SELECT {select} FROM {SOURCE_SCHEMA}.{table}"
),
[],
)?;
}
// AUTOINCREMENT's counters live in a table the filter above skips; the
// CREATE of such a table makes an empty one here.
let has_sequence: bool = tx.query_row(
&format!(
"SELECT EXISTS(SELECT 1 FROM {SOURCE_SCHEMA}.sqlite_master
WHERE name = 'sqlite_sequence')"
),
[],
|r| r.get(0),
)?;
if has_sequence {
tx.execute_batch(&format!(
"DELETE FROM main.sqlite_sequence;
INSERT INTO main.sqlite_sequence SELECT * FROM {SOURCE_SCHEMA}.sqlite_sequence;"
))?;
}
for (_, _, sql) in objects.iter().filter(|(k, _, _)| k != "table") {
tx.execute_batch(sql)?;
}
tx.pragma_update(None, "user_version", user_version)?;
tx.pragma_update(None, "application_id", application_id)?;
tx.commit()?;
Ok(())
}
/// `name` as an SQL identifier, whatever it contains.
fn quote_ident(name: &str) -> String {
format!("\"{}\"", name.replace('"', "\"\""))
}
/// TRACES: NFR-R2 /// TRACES: NFR-R2
/// Refuse to hand over a snapshot that will not pass `quick_check`. /// Refuse to hand over a snapshot that will not pass `quick_check`.
/// ///
@@ -82,12 +274,12 @@ fn verify_snapshot(snapshot: &Connection) -> Result<(), CatalogError> {
/// Checkpoint, then copy the whole database to `dest`, and hand back the /// Checkpoint, then copy the whole database to `dest`, and hand back the
/// connection to the copy. /// connection to the copy.
/// ///
/// Split out from [`snapshot_for_upload`] because [`crate::recovery`] wants /// What [`crate::recovery`] takes its NFR-R2 backups with. A backup is the
/// exactly this and none of what follows it there: an NFR-R2 backup is the /// file the user may have to *live on*, so it keeps the face crops that
/// file the user may have to *live on*, so it keeps the face crops that an /// [`snapshot_for_upload`] leaves out, and a byte-for-byte page copy is the
/// upload strips. Sharing the copy rather than reimplementing it is what keeps /// right tool. Keeping it here beside the upload keeps the WAL discipline in
/// the WAL discipline in one place — a backup taken with `fs::copy` would be /// one place: a backup taken with `fs::copy` would be the torn snapshot this
/// the torn snapshot this module's header exists to warn about. /// module's header exists to warn about.
pub(crate) fn copy_to(conn: &Connection, dest: &Path) -> Result<Connection, CatalogError> { pub(crate) fn copy_to(conn: &Connection, dest: &Path) -> Result<Connection, CatalogError> {
checkpoint(conn)?; checkpoint(conn)?;
@@ -103,39 +295,6 @@ pub(crate) fn copy_to(conn: &Connection, dest: &Path) -> Result<Connection, Cata
Ok(out) Ok(out)
} }
/// Drop the stored face crops from a snapshot before it is uploaded.
///
/// The snapshot is the *whole catalog*, uploaded on every sync and downloaded
/// by every device. Face crops are a few KB each and a fully indexed library
/// holds tens of thousands of them, so leaving them in would put tens of MB on
/// every round trip — the exact cost `face_shard`'s 25 MB cap exists to bound,
/// and the reason the bulk per-face data lives in shards in the first place.
///
/// Crops are not lost by this: they travel in the face shards
/// ([`crate::face_shard::export_to_shards`]), which are written once and
/// downloaded once. Nothing reads a crop out of a merged remote catalog —
/// [`merge_all`] touches collections and keywords only — so removing them here
/// costs a receiving device nothing it would otherwise have had.
///
/// `VACUUM` afterwards because SQLite does not return freed pages to the file
/// on its own, and an upload sized by the file rather than by its contents
/// would keep paying for bytes that are no longer there.
fn strip_face_crops(snapshot: &Connection) -> Result<(), CatalogError> {
// A catalog older than the crop column is a legitimate input here — a
// snapshot taken mid-migration, or a test fixture built from an earlier
// schema — so an absent column is nothing to fail over.
let has_crop = snapshot
.prepare("SELECT crop FROM faces LIMIT 1")
.map(|_| true)
.unwrap_or(false);
if !has_crop {
return Ok(());
}
snapshot.execute("UPDATE faces SET crop = NULL WHERE crop IS NOT NULL", [])?;
snapshot.execute_batch("VACUUM")?;
Ok(())
}
/// Whether a downloaded remote catalog is worth merging. /// Whether a downloaded remote catalog is worth merging.
/// ///
/// Cheap guard before attaching: a remote written by a newer build may contain /// Cheap guard before attaching: a remote written by a newer build may contain
@@ -171,6 +330,13 @@ pub fn merge_remote(conn: &Connection, remote: &Path) -> Result<MergeReport, Cat
let result = merge::merge_all(conn); let result = merge::merge_all(conn);
// The merge brings in rows the backfill exists for — assignments whose
// word this device has no term for, from a remote older than v6 — so the
// next open must run it, whether or not the stamp happened to move.
if let Some(path) = conn.path().filter(|p| !p.is_empty()) {
crate::backfilled::forget(Path::new(path));
}
// Detach even if the merge failed, or the next attempt errors with // Detach even if the merge failed, or the next attempt errors with
// "database remote_cat is already in use". // "database remote_cat is already in use".
let detach = conn.execute(&format!("DETACH DATABASE {REMOTE_SCHEMA}"), []); let detach = conn.execute(&format!("DETACH DATABASE {REMOTE_SCHEMA}"), []);
@@ -178,6 +344,18 @@ pub fn merge_remote(conn: &Connection, remote: &Path) -> Result<MergeReport, Cat
log::warn!("failed to detach remote catalog: {e}"); log::warn!("failed to detach remote catalog: {e}");
} }
// After every merge, because a merge is where two devices' people meet:
// the same name typed on each, or a redirect one of them made. Its own
// transaction, and a failure is logged rather than returned -- what the
// merge took is committed and valid whether or not the duplicates were
// folded, and the next pass tries again. Runs on the sync worker, never
// the UI thread, and costs ~10 ms when there is nothing to do.
if result.is_ok() {
if let Err(e) = crate::dedup_people::run(conn) {
log::warn!("dedup after the catalog merge: {e}");
}
}
result result
} }
@@ -288,6 +466,92 @@ mod tests {
assert_eq!(n, 2); assert_eq!(n, 2);
} }
/// One image, known to the server by `file_id`, in a catalog.
fn with_image(c: &Connection, file_id: i64) -> dr_types::ImageId {
c.execute(
"INSERT OR IGNORE INTO roots(id, kind, label) VALUES (1, 'remote', 'lib')",
[],
)
.unwrap();
c.execute(
"INSERT INTO images(root_id, source_ref, added_at) VALUES (1, ?1, 0)",
[format!("IMG_{file_id}.CR3")],
)
.unwrap();
let id = c.last_insert_rowid();
c.execute(
"INSERT INTO remote(image_id, file_id) VALUES (?1, ?2)",
[id, file_id],
)
.unwrap();
dr_types::ImageId(id as u64)
}
#[test]
fn an_album_and_its_exports_reach_another_device_but_its_folder_does_not() {
use crate::albums::{self, Place};
let dir = tempdir();
let remote_path = dir.join("remote.sqlite");
let snap = dir.join("snap.sqlite");
{
// The desktop: two albums, one on the server and one on its own
// disk, each with an export of the same photograph.
let r = seeded(&dir.join("desktop.sqlite"));
let img = with_image(&r, 4242);
let web = albums::create(&r, "Web", &Place::Server("Shared/Web".into())).unwrap();
let print = albums::create(&r, "Print", &Place::Local("/mnt/print".into())).unwrap();
albums::record_exports(&r, web, &[(img, "IMG_4242.jpg".into())]).unwrap();
albums::record_exports(&r, print, &[(img, "IMG_4242.tif".into())]).unwrap();
snapshot_for_upload(&r, &snap).unwrap();
std::fs::rename(&snap, &remote_path).unwrap();
}
// The tablet knows the same file under its own image id.
let local = seeded(&dir.join("tablet.sqlite"));
with_image(&local, 1);
let img = with_image(&local, 4242);
let report = merge_remote(&local, &remote_path).unwrap();
assert_eq!(report.albums_taken, 2);
assert_eq!(report.album_exports_added, 2);
assert!(report.local_changed());
let all = albums::list(&local).unwrap();
let print = all.iter().find(|a| a.name == "Print").unwrap();
let web = all.iter().find(|a| a.name == "Web").unwrap();
assert_eq!(print.place, None, "the desktop's disk is not the tablet's");
assert_eq!(web.place, Some(Place::Server("Shared/Web".into())));
assert_eq!(albums::sources(&local, web.id).unwrap(), vec![img]);
// Nothing changed on either side, so a second pass takes nothing.
let again = merge_remote(&local, &remote_path).unwrap();
assert_eq!(again.albums_taken, 0);
assert_eq!(again.album_exports_added, 0);
}
#[test]
fn a_device_that_never_made_an_album_still_uploads_this_ones() {
use crate::albums::{self, Place};
let dir = tempdir();
let remote_path = dir.join("remote.sqlite");
{
// A snapshot from a build that predates albums altogether.
let r = seeded(&remote_path);
checkpoint(&r).unwrap();
}
let local = seeded(&dir.join("local.sqlite"));
albums::create(&local, "Web", &Place::Server("Web".into())).unwrap();
let report = merge_remote(&local, &remote_path).unwrap();
assert_eq!(report.albums_taken, 0);
// Its albums table is absent, so nothing was compared — and the local
// album has still to reach the server.
assert!(
albums::list(&local).unwrap().len() == 1,
"the local album survives a merge with a catalog that has none"
);
}
#[test] #[test]
fn the_remote_can_be_merged_twice_without_attach_conflict() { fn the_remote_can_be_merged_twice_without_attach_conflict() {
// Detach must happen even on the failure path, or the second attempt // Detach must happen even on the failure path, or the second attempt
@@ -380,4 +644,188 @@ mod tests {
.unwrap(); .unwrap();
assert_eq!(kept, 1, "stripping the snapshot damaged the live catalog"); assert_eq!(kept, 1, "stripping the snapshot damaged the live catalog");
} }
/// Device-side setup for the snapshot tests: an image both devices know by
/// its cross-device file id, and one face on it carrying `crop`.
fn with_a_face(c: &Connection, face_id: i64, x: f64, crop: &[u8]) {
c.execute(
"INSERT OR IGNORE INTO roots(id, kind, label) VALUES (1, 'local', 'lib')",
[],
)
.unwrap();
c.execute(
"INSERT INTO images(id, root_id, source_ref, added_at) VALUES (1, 1, 'a.CR3', 0)",
[],
)
.unwrap();
c.execute("INSERT INTO remote(image_id, file_id) VALUES (1, 5000)", [])
.unwrap();
c.execute(
"INSERT INTO faces
(id, image_id, x, y, w, h, landmarks, detector_confidence, embedding,
crop_px, model_id, detected_at, crop)
VALUES (?1, 1, ?2, 0.2, 0.2, 0.2, X'00', 0.9, X'00', 150.0, 'w600k_mbf', 0, ?3)",
rusqlite::params![face_id, x, crop],
)
.unwrap();
}
fn crop_of(c: &Connection, face_id: i64) -> Option<Vec<u8>> {
c.query_row("SELECT crop FROM faces WHERE id = ?1", [face_id], |r| {
r.get(0)
})
.unwrap()
}
/// The snapshot is built table by table rather than copied, so what has to
/// hold is that it is still the same database bar the crops: every table,
/// index and row, and the header fields an older build checks before it
/// will merge (`user_version`) or open it the way it always has (the WAL
/// flag and page size a backup-API copy carried).
#[test]
fn the_snapshot_is_the_catalog_bar_the_crops() {
let dir = tempdir();
let live = dir.join("catalog.sqlite");
let snap = dir.join("snap.sqlite");
let c = seeded(&live);
with_a_face(&c, 7, 0.3, &[7u8; 4096]);
c.execute(
"INSERT INTO collections(uuid, name, kind, created, revision, modified)
VALUES ('u1', 'Iceland', 0, 0, 1, 1)",
[],
)
.unwrap();
// Created on first use rather than by a migration: the copy must not
// depend on the migrations knowing every table.
crate::duplicates::ensure_probe_table(&c).unwrap();
snapshot_for_upload(&c, &snap).unwrap();
let out = Connection::open(&snap).unwrap();
let objects = |conn: &Connection| -> Vec<(String, String)> {
let mut stmt = conn
.prepare("SELECT type, name FROM sqlite_master ORDER BY type, name")
.unwrap();
let rows = stmt
.query_map([], |r| Ok((r.get(0).unwrap(), r.get(1).unwrap())))
.unwrap();
rows.map(Result::unwrap).collect()
};
assert_eq!(objects(&out), objects(&c), "the snapshot's schema differs");
for (kind, table) in objects(&c) {
if kind != "table" {
continue;
}
let count = |conn: &Connection| -> i64 {
conn.query_row(&format!("SELECT COUNT(*) FROM \"{table}\""), [], |r| {
r.get(0)
})
.unwrap()
};
assert_eq!(count(&out), count(&c), "rows differ in {table}");
}
for pragma in [
"user_version",
"application_id",
"page_size",
"journal_mode",
] {
let read = |conn: &Connection| -> String {
conn.query_row(&format!("PRAGMA {pragma}"), [], |r| {
r.get::<_, rusqlite::types::Value>(0)
})
.map(|v| format!("{v:?}"))
.unwrap()
};
assert_eq!(read(&out), read(&c), "{pragma} differs");
}
drop(out);
// Bytes 18 and 19 of the header are 2 for a WAL database, which is
// what every snapshot uploaded before this one said.
let header = std::fs::read(&snap).unwrap();
assert_eq!(&header[18..20], &[2, 2], "the snapshot is not WAL-flagged");
// The face is there; its pixels are not.
let out = Connection::open(&snap).unwrap();
assert_eq!(crop_of(&out, 7), None);
}
/// A pass that died mid-build leaves a file behind; the next one must
/// build afresh rather than on top of it.
#[test]
fn a_leftover_snapshot_is_replaced_not_built_on() {
let dir = tempdir();
let live = dir.join("catalog.sqlite");
let snap = dir.join("snap.sqlite");
let c = seeded(&live);
std::fs::write(&snap, b"not a database").unwrap();
snapshot_for_upload(&c, &snap).unwrap();
// And twice over a good one, which is the steady state.
snapshot_for_upload(&c, &snap).unwrap();
let out = Connection::open(&snap).unwrap();
let v: i64 = out
.query_row("PRAGMA user_version", [], |r| r.get(0))
.unwrap();
assert_eq!(v, schema::SCHEMA_VERSION);
}
/// The merge side of a crop-less snapshot: the other device's names still
/// cross over — the match is by box, not by pixels — and this device's
/// own crop is left exactly as it was, never replaced by the snapshot's
/// NULL.
#[test]
fn merging_a_crop_less_snapshot_keeps_local_crops_and_takes_the_names() {
let dir = tempdir();
let snap = dir.join("snap.sqlite");
let desktop = seeded(&dir.join("desktop.sqlite"));
with_a_face(&desktop, 42, 0.31, &[1u8; 3000]);
desktop
.execute(
"INSERT INTO people(id, uuid, name, ignored, created, revision, modified)
VALUES (3, 'u-anna', 'Anna', 0, 0, 1, 1)",
[],
)
.unwrap();
desktop
.execute(
"INSERT INTO face_person(face_id, person_id, probability, confirmed)
VALUES (42, 3, 0.9, 1)",
[],
)
.unwrap();
snapshot_for_upload(&desktop, &snap).unwrap();
// The tablet found the same face itself, under its own row id, and has
// its own crop of it — from its own detection or from the shards.
let tablet = seeded(&dir.join("tablet.sqlite"));
let mine = vec![9u8; 2500];
with_a_face(&tablet, 7, 0.30, &mine);
let report = merge_remote(&tablet, &snap).unwrap();
assert_eq!(report.people_inserted, 1);
assert_eq!(report.faces_assigned, 1);
let named: (String, bool) = tablet
.query_row(
"SELECT p.name, fp.confirmed
FROM face_person fp JOIN people p ON p.id = fp.person_id
WHERE fp.face_id = 7",
[],
|r| Ok((r.get(0)?, r.get(1)?)),
)
.unwrap();
assert_eq!(named, ("Anna".to_string(), true));
assert_eq!(
crop_of(&tablet, 7),
Some(mine),
"the merge touched a local crop"
);
// Idempotent over a crop-less remote too.
let again = merge_remote(&tablet, &snap).unwrap();
assert!(!again.local_changed());
}
} }
core/dr-catalog/src/trash.rs
+30 -18
View File
@@ -129,28 +129,40 @@ pub fn record_trashed(
return Ok(0); return Ok(0);
} }
let tx = conn.unchecked_transaction()?; let tx = conn.unchecked_transaction()?;
let mut n = 0; let n = record_trashed_within(&tx, moved, now)?;
{
let mut stmt = tx.prepare(
"UPDATE images
SET trashed_from = CASE
WHEN trashed_at IS NULL THEN source_ref
ELSE trashed_from
END,
source_ref = ?2,
trashed_at = coalesce(trashed_at, ?3)
WHERE id = ?1",
)?;
for (image, path) in moved {
n += stmt.execute(rusqlite::params![image.0 as i64, path, now])?;
}
}
tx.commit()?; tx.commit()?;
Ok(n) Ok(n)
} }
/// [`record_trashed`] inside a transaction the caller owns.
///
/// For a caller whose trash is one half of a larger write that must land
/// whole or not at all — consolidating duplicates (`crate::duplicates`)
/// merges a copy's judgements onto the survivor and trashes the copy in one
/// commit. `unchecked_transaction` cannot nest, so this is offered here
/// rather than wrapped from above.
pub fn record_trashed_within(
tx: &Connection,
moved: &[(ImageId, String)],
now: i64,
) -> Result<usize, CatalogError> {
let mut n = 0;
let mut stmt = tx.prepare(
"UPDATE images
SET trashed_from = CASE
WHEN trashed_at IS NULL THEN source_ref
ELSE trashed_from
END,
source_ref = ?2,
trashed_at = coalesce(trashed_at, ?3)
WHERE id = ?1",
)?;
for (image, path) in moved {
n += stmt.execute(rusqlite::params![image.0 as i64, path, now])?;
}
Ok(n)
}
/// Record that images have been moved back out of the trash. /// Record that images have been moved back out of the trash.
/// ///
/// Call after the move succeeds, for the same reason as [`record_trashed`]. /// Call after the move succeeds, for the same reason as [`record_trashed`].
core/dr-catalog/src/walk.rs
+21 -53
View File
@@ -48,7 +48,6 @@ use dr_types::{Availability, FormatFilter, RootId, SourceRef};
use rusqlite::{Connection, OptionalExtension}; use rusqlite::{Connection, OptionalExtension};
use crate::error::CatalogError; use crate::error::CatalogError;
use crate::jobs::{self, JobKind, Priority};
use crate::query::availability_code; use crate::query::availability_code;
use crate::scan::{ use crate::scan::{
classify_dir, classify_entry, DirAction, DirState, EntryAction, KnownFile, ScanOutcome, classify_dir, classify_entry, DirAction, DirState, EntryAction, KnownFile, ScanOutcome,
@@ -336,7 +335,7 @@ pub fn scan_root(
} }
} }
EntryAction::Insert => { EntryAction::Insert => {
let id = insert_image( insert_image(
&tx, &tx,
root, root,
folder_id, folder_id,
@@ -345,11 +344,10 @@ pub fn scan_root(
entry.meta.mtime, entry.meta.mtime,
now, now,
)?; )?;
queue_reading_it(&tx, id)?;
report.inserted += 1; report.inserted += 1;
} }
EntryAction::Changed => { EntryAction::Changed => {
let id = update_image( update_image(
&tx, &tx,
root, root,
folder_id, folder_id,
@@ -357,7 +355,6 @@ pub fn scan_root(
entry.meta.size, entry.meta.size,
entry.meta.mtime, entry.meta.mtime,
)?; )?;
queue_reading_it(&tx, id)?;
report.updated += 1; report.updated += 1;
} }
EntryAction::Ignored => unreachable!("returned above"), EntryAction::Ignored => unreachable!("returned above"),
@@ -640,7 +637,7 @@ fn insert_image(
size: u64, size: u64,
mtime: i64, mtime: i64,
now: i64, now: i64,
) -> Result<i64, CatalogError> { ) -> Result<(), CatalogError> {
// `metadata_state = 1`: the scan knows the name, the size and the mtime, // `metadata_state = 1`: the scan knows the name, the size and the mtime,
// and has read no EXIF. Claiming otherwise would make a date filter // and has read no EXIF. Claiming otherwise would make a date filter
// silently wrong on a freshly scanned library. // silently wrong on a freshly scanned library.
@@ -664,7 +661,7 @@ fn insert_image(
now, now,
], ],
)?; )?;
image_id(conn, root, src) Ok(())
} }
fn update_image( fn update_image(
@@ -674,7 +671,7 @@ fn update_image(
src: &SourceRef, src: &SourceRef,
size: u64, size: u64,
mtime: i64, mtime: i64,
) -> Result<i64, CatalogError> { ) -> Result<(), CatalogError> {
// The content hash is dropped, not recomputed: it described bytes that no // The content hash is dropped, not recomputed: it described bytes that no
// longer exist, and leaving it would let reconnect-by-hash match this image // longer exist, and leaving it would let reconnect-by-hash match this image
// to a file it is no longer a copy of. `metadata_state` goes back to 1 for // to a file it is no longer a copy of. `metadata_state` goes back to 1 for
@@ -692,37 +689,7 @@ fn update_image(
src.key(), src.key(),
], ],
)?; )?;
image_id(conn, root, src) Ok(())
}
fn image_id(conn: &Connection, root: RootId, src: &SourceRef) -> Result<i64, CatalogError> {
Ok(conn.query_row(
"SELECT id FROM images WHERE root_id = ?1 AND source_ref = ?2",
rusqlite::params![root.0 as i64, src.key()],
|r| r.get(0),
)?)
}
/// Queue the work that turns a stat-only row into a usable grid cell.
///
/// Enqueued inside the scan's transaction, so a folder's rows and the jobs that
/// finish them land together — a crash between the two would otherwise leave
/// images no worker was ever told about.
fn queue_reading_it(conn: &Connection, image_id: i64) -> Result<(), CatalogError> {
jobs::enqueue(
conn,
JobKind::ExtractMetadata,
Some(image_id),
Priority::Background,
None,
)?;
jobs::enqueue(
conn,
JobKind::Thumbnail,
Some(image_id),
Priority::Background,
None,
)
} }
/// TRACES: FR-CAT-9 /// TRACES: FR-CAT-9
@@ -1096,7 +1063,7 @@ mod tests {
} }
#[test] #[test]
fn a_resaved_file_is_queued_for_rereading_and_loses_its_stale_hash() { fn a_resaved_file_owes_a_reread_and_loses_its_stale_hash() {
let lib = Library::new("resaved"); let lib = Library::new("resaved");
lib.file("IMG.CR3", b"raw"); lib.file("IMG.CR3", b"raw");
lib.scan(); lib.scan();
@@ -1121,9 +1088,8 @@ mod tests {
"a hash of bytes that no longer exist would match this image to the \ "a hash of bytes that no longer exist would match this image to the \
wrong file on reconnect" wrong file on reconnect"
); );
assert_eq!(lib.count("SELECT metadata_state FROM images"), 1);
assert_eq!( assert_eq!(
lib.count("SELECT count(*) FROM jobs WHERE kind = 1"), lib.count("SELECT metadata_state FROM images"),
1, 1,
"EXIF must be re-read" "EXIF must be re-read"
); );
@@ -1137,7 +1103,11 @@ mod tests {
let lib = Library::new("no-requeue"); let lib = Library::new("no-requeue");
lib.file("a.CR3", b"raw"); lib.file("a.CR3", b"raw");
lib.scan(); lib.scan();
lib.conn().execute("DELETE FROM jobs", []).unwrap(); // As if the metadata sweep had read it: what is owed is recorded in
// `metadata_state`, and the thumbnail store answers for itself.
lib.conn()
.execute("UPDATE images SET metadata_state = 2", [])
.unwrap();
lib.file("b.CR3", b"raw"); lib.file("b.CR3", b"raw");
let r = lib.scan(); let r = lib.scan();
@@ -1145,9 +1115,9 @@ mod tests {
assert_eq!(r.inserted, 1); assert_eq!(r.inserted, 1);
assert_eq!(r.unchanged, 1); assert_eq!(r.unchanged, 1);
assert_eq!( assert_eq!(
lib.count("SELECT count(*) FROM jobs"), lib.count("SELECT count(*) FROM images WHERE metadata_state < 2"),
2, 1,
"EXIF and a thumbnail for the new image, and nothing for the old one" "EXIF owed for the new image, and nothing for the old one"
); );
} }
@@ -1171,17 +1141,15 @@ mod tests {
} }
#[test] #[test]
fn a_new_image_is_queued_for_a_thumbnail_and_for_exif() { fn a_new_image_owes_its_exif_and_queues_nothing() {
// The sweeps find their work from `metadata_state` and the thumbnail
// store. A queued job would be a second record of the same debt, and
// no handler claims one (#73).
let lib = Library::new("queued"); let lib = Library::new("queued");
lib.file("IMG.CR3", b"raw"); lib.file("IMG.CR3", b"raw");
lib.scan(); lib.scan();
assert_eq!( assert_eq!(lib.count("SELECT count(*) FROM jobs"), 0);
lib.count("SELECT count(*) FROM jobs WHERE kind = 2"),
1,
"no thumbnail job means an empty grid cell forever"
);
assert_eq!(lib.count("SELECT count(*) FROM jobs WHERE kind = 1"), 1);
assert_eq!( assert_eq!(
lib.count("SELECT metadata_state FROM images"), lib.count("SELECT metadata_state FROM images"),
1, 1,
core/dr-catalog/tests/every_queued_kind_has_a_consumer.rs
+341
View File
@@ -0,0 +1,341 @@
// TRACES: FR-PLAT-AND-3
//! No job kind is enqueued without something that claims it.
//!
//! The queue coalesces, so a producer with no consumer does not fail — it
//! just leaves a row per subject for ever. That is how the reference catalog
//! came to hold 23,582 `Thumbnail` jobs, one per photograph, re-coalesced on
//! every scan, with no handler for the kind anywhere in the tree (#73). Nothing
//! at runtime notices: the rows are cheap one at a time and invisible in the
//! interface. So the pairing is checked here, over the source, instead.
//!
//! ## What counts
//!
//! In shipping code under `core/`, `ui/`, `apps/` and `platform/` — every
//! `src/` tree, with `#[cfg(test)]` items dropped:
//!
//! - **Enqueued**: the `JobKind::X` named in the arguments of a call to
//! `enqueue(`. An enqueue whose kind is not spelled there — passed in a
//! variable — is refused outright, because this scan could not say what it
//! queues.
//! - **Claimed**: the `JobKind::X` in the body of a `fn kinds(` (what a
//! `JobHandler` declares, and all a `Runner` claims), or named in a call to
//! `claim_next_matching(`. A call to `claim_next(` claims every kind.
//! `JobKind::ALL` in either place means every kind.
//!
//! Tests and examples are left out on purpose: a unit test of the queue's
//! mechanics enqueues and claims whatever it likes, and proves nothing about
//! the app.
use std::collections::BTreeSet;
use std::fs;
use std::path::{Path, PathBuf};
/// Every `.rs` file under each `src/` of each crate in `group`.
fn crate_sources(group: &Path, out: &mut Vec<PathBuf>) {
let Ok(crates) = fs::read_dir(group) else {
return;
};
for krate in crates {
let src = krate.expect("read dir entry").path().join("src");
if src.is_dir() {
rust_files(&src, out);
}
}
}
fn rust_files(dir: &Path, out: &mut Vec<PathBuf>) {
for entry in fs::read_dir(dir).unwrap_or_else(|e| panic!("cannot read {}: {e}", dir.display()))
{
let path = entry.expect("read dir entry").path();
if path.is_dir() {
rust_files(&path, out);
} else if path.extension().and_then(|e| e.to_str()) == Some("rs") {
out.push(path);
}
}
}
/// Blank out string literals and line comments, so neither a brace nor a
/// `JobKind::` inside prose is read as code.
fn strip_literals_and_comments(line: &str) -> String {
let mut out = String::with_capacity(line.len());
let mut chars = line.chars().peekable();
let mut in_string = false;
while let Some(c) = chars.next() {
if in_string {
match c {
'\\' => {
chars.next();
}
'"' => in_string = false,
_ => {}
}
continue;
}
match c {
'"' => in_string = true,
'/' if chars.peek() == Some(&'/') => break,
_ => out.push(c),
}
}
out
}
/// The shipping code of a file, comments and strings blanked, with every
/// `#[cfg(test)]` item dropped. The attribute must be the whole line, so a
/// doc comment mentioning it is not mistaken for one.
fn shipping_code(text: &str) -> String {
let lines: Vec<String> = text.lines().map(strip_literals_and_comments).collect();
let mut out = String::new();
let mut i = 0;
while i < lines.len() {
if lines[i].trim() != "#[cfg(test)]" {
out.push_str(&lines[i]);
out.push('\n');
i += 1;
continue;
}
let (mut j, mut depth, mut opened) = (i + 1, 0i32, false);
while j < lines.len() {
depth += lines[j].matches('{').count() as i32;
depth -= lines[j].matches('}').count() as i32;
opened |= lines[j].contains('{');
if (opened && depth <= 0) || (!opened && lines[j].contains(';')) {
break;
}
j += 1;
}
i = j + 1;
}
out
}
/// The text from `start` (just past an opening delimiter) to its matching
/// close.
fn balanced(code: &str, start: usize, open: char, close: char) -> &str {
let mut depth = 1;
for (i, c) in code[start..].char_indices() {
if c == open {
depth += 1;
} else if c == close {
depth -= 1;
if depth == 0 {
return &code[start..start + i];
}
}
}
&code[start..]
}
/// Each call of `name(` in `code` that is a call rather than the function's
/// own definition or a longer name ending in it, as its argument text.
fn calls<'a>(code: &'a str, name: &str) -> Vec<&'a str> {
let needle = format!("{name}(");
let mut found = Vec::new();
for (at, _) in code.match_indices(&needle) {
let before = &code[..at];
let prev = before.chars().next_back();
if prev.is_some_and(|c| c.is_alphanumeric() || c == '_') {
continue;
}
if before.trim_end().ends_with("fn") {
continue;
}
found.push(balanced(code, at + needle.len(), '(', ')'));
}
found
}
/// Bodies of every `fn kinds(` that has one — a trait declaration ending in
/// `;` has none.
fn kinds_bodies(code: &str) -> Vec<&str> {
let mut found = Vec::new();
for (at, _) in code.match_indices("fn kinds(") {
let rest = &code[at..];
let (Some(brace), semi) = (rest.find('{'), rest.find(';')) else {
continue;
};
if semi.is_some_and(|s| s < brace) {
continue;
}
found.push(balanced(code, at + brace + 1, '{', '}'));
}
found
}
/// The `X` of each `JobKind::X` in `text`.
fn kinds_named(text: &str) -> Vec<String> {
text.match_indices("JobKind::")
.map(|(at, m)| {
text[at + m.len()..]
.chars()
.take_while(|c| c.is_alphanumeric() || *c == '_')
.collect()
})
.collect()
}
#[derive(Default, Debug)]
struct Ledger {
/// Kind → where it is enqueued.
enqueued: Vec<(String, String)>,
/// Enqueue calls whose kind could not be read.
unreadable: Vec<String>,
claimed: BTreeSet<String>,
claims_everything: bool,
}
fn read(files: &[(String, String)]) -> Ledger {
let mut ledger = Ledger::default();
for (name, text) in files {
let code = shipping_code(text);
for args in calls(&code, "enqueue") {
let kinds = kinds_named(args);
if kinds.is_empty() {
ledger
.unreadable
.push(format!("{name}: enqueue({})", args.trim()));
}
for k in kinds {
ledger.enqueued.push((k, name.clone()));
}
}
let claimed = kinds_bodies(&code)
.into_iter()
.chain(calls(&code, "claim_next_matching"));
for text in claimed {
for k in kinds_named(text) {
if k == "ALL" {
ledger.claims_everything = true;
} else {
ledger.claimed.insert(k);
}
}
}
if !calls(&code, "claim_next").is_empty() {
ledger.claims_everything = true;
}
}
ledger
}
#[test]
fn every_kind_enqueued_is_claimed_by_something() {
let repo = Path::new(env!("CARGO_MANIFEST_DIR"))
.parent()
.and_then(Path::parent)
.expect("core/dr-catalog has a grandparent");
let mut paths = Vec::new();
for group in ["core", "ui", "apps", "platform"] {
crate_sources(&repo.join(group), &mut paths);
}
let files: Vec<(String, String)> = paths
.iter()
.map(|p| {
let text = fs::read_to_string(p)
.unwrap_or_else(|e| panic!("cannot read {}: {e}", p.display()));
let name = p.strip_prefix(repo).unwrap_or(p).display().to_string();
(name, text)
})
.collect();
// A scan over nothing passes for the wrong reason. The queue's own file
// and the scan that used to feed it must both have been read, and the
// queue's definitions found in them.
for must in [
"core/dr-catalog/src/jobs.rs",
"core/dr-catalog/src/runner.rs",
"ui/dr-ui/src/library/scan.rs",
] {
assert!(
files.iter().any(|(n, _)| n == must),
"{must} was not scanned — the source walk is wrong, not the code"
);
}
let jobs = &files
.iter()
.find(|(n, _)| n == "core/dr-catalog/src/jobs.rs")
.unwrap()
.1;
assert!(shipping_code(jobs).contains("pub fn enqueue("));
let ledger = read(&files);
assert!(
ledger.unreadable.is_empty(),
"\n\nThese enqueue calls do not name their JobKind, so this test cannot \
check that anything claims it. Spell the kind at the call:\n {}\n",
ledger.unreadable.join("\n ")
);
if ledger.claims_everything {
return;
}
let orphans: Vec<String> = ledger
.enqueued
.iter()
.filter(|(k, _)| !ledger.claimed.contains(k))
.map(|(k, at)| format!("JobKind::{k}, enqueued in {at}"))
.collect();
assert!(
orphans.is_empty(),
"\n\nEnqueued, and claimed by nothing (claimed: {:?}):\n {}\n\n\
A kind nobody claims is a row per subject that stays for ever — the \
queue coalesces, so it never fails, it only grows (#73). Register a \
JobHandler for the kind, or stop enqueueing it and add it to \
JobKind::RETIRED so the rows already queued are dropped.\n",
ledger.claimed,
orphans.join("\n ")
);
}
/// The reader itself, on code whose answer is known — so a parsing bug shows
/// up as this failing, rather than the real check quietly finding nothing.
#[test]
fn the_reader_sees_producers_and_consumers() {
let producer = r#"
use dr_catalog::jobs;
fn persist(tx: &Connection, id: i64) {
// jobs::enqueue(tx, JobKind::ContentHash, ...) in a comment is not a call
let _ = dr_catalog::jobs::enqueue(
tx,
JobKind::Thumbnail,
Some(id),
Priority::Background,
None,
);
jobs::enqueue(tx, kind, Some(id), Priority::Background, None)?;
}
pub fn enqueue(conn: &Connection, kind: JobKind) {}
#[cfg(test)]
mod tests {
fn t() { enqueue(&c, JobKind::FetchOriginal, None, P, None); }
}
"#;
let consumer = r#"
impl JobHandler for Faces {
fn kinds(&self) -> &[JobKind] {
&[JobKind::DetectFaces]
}
fn run(&mut self) {}
}
trait JobHandler { fn kinds(&self) -> &[JobKind]; }
fn pull(c: &Connection) { claim_next_matching(c, 0, &[JobKind::FetchPreview]); }
"#;
let ledger = read(&[
("producer.rs".into(), producer.into()),
("consumer.rs".into(), consumer.into()),
]);
let enqueued: Vec<&str> = ledger.enqueued.iter().map(|(k, _)| k.as_str()).collect();
assert_eq!(enqueued, vec!["Thumbnail"]);
assert_eq!(ledger.unreadable.len(), 1, "{:?}", ledger.unreadable);
assert_eq!(
ledger.claimed,
BTreeSet::from(["DetectFaces".to_string(), "FetchPreview".to_string()])
);
assert!(!ledger.claims_everything);
}
core/dr-gpu/src/adjust.rs
+406 -5
View File
@@ -124,6 +124,9 @@ pub struct AdjustPass {
/// Cleared by any render that does not write it, so a stale intermediate /// Cleared by any render that does not write it, so a stale intermediate
/// cannot survive a change of image and be handed to a later detail chain. /// cannot survive a change of image and be handed to a later detail chain.
colour_key: Option<(u64, u32, u32)>, colour_key: Option<(u64, u32, u32)>,
/// The source texels the fused pass read on an earlier frame, kept so a
/// slider drag at fit reads them contiguously. See [`SampleCache`].
sample: SampleCache,
/// Fused dispatches actually encoded. Exposed so a test can see the reuse /// Fused dispatches actually encoded. Exposed so a test can see the reuse
/// above happening rather than take it on trust. /// above happening rather than take it on trust.
colour_dispatches: usize, colour_dispatches: usize,
@@ -147,6 +150,205 @@ struct Target {
const FILM_FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba32Float; const FILM_FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba32Float;
/// TRACES: FR-DEV-3f /// TRACES: FR-DEV-3f
/// What a cached sample is valid for: the composer's
/// [`ComposedShader::sample_key`], the source image, and the render size.
type SampleKey = (u64, u64, u32, u32);
/// The largest render the sample cache is kept for, in pixels.
///
/// 4K and a little over, which is every develop view there is. An export
/// renders a whole sensor once and gains nothing from a cache it will not
/// read again; without a ceiling, two exports of the same frame in a row
/// would park a full-resolution copy of it on the device.
const SAMPLE_CACHE_MAX_PIXELS: u64 = 3840 * 2400;
/// How the fused dispatch gets its source colour this frame.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
enum SampleUse {
/// From the source, as it always did.
Direct,
/// From the source, and stored in the cache for the frames after.
Write,
/// From the cache.
Read,
}
/// TRACES: NFR-P5
/// The fused pass's gather from the source, remembered across frames.
///
/// At fit, each output pixel of the fused pass reads one texel of a source
/// three or four times its width, on a stride, and the memory system fetches
/// the neighbours it skips along with it. On a 60 MP source that gather was
/// most of the fused pass: 10.9 ms of a 2560 x 1600 frame against 3.8 ms for
/// the same work reading contiguously (RTX 3050, clocks held down). But which
/// texel a pixel reads depends on the framing and nothing else, and a slider
/// drag does not move the framing. So the shader writes what it gathered to a
/// render-sized texture on one frame and reads it back contiguously on every
/// frame after, until the framing, the image or the size changes.
///
/// Bit-for-bit the same picture: the source is `rgba16float` and so is the
/// cache, so the stored texel is the texel. Only the whole-texel sampling path
/// takes part — [`ComposedShader::sample_key`] is `None` when the sample is
/// interpolated.
///
/// **Written on the second frame with a key, not the first.** A drag of the
/// crop or of a zoom changes the key on every frame, and a cache written then
/// is never read — it would add a write per frame to exactly the gestures that
/// can least afford one. Waiting for the key to repeat once costs a slider drag
/// one uncached frame and costs a crop drag nothing.
struct SampleCache {
/// The cache itself, `rgba16float`, sampled and written as storage.
target: Option<Target>,
/// What `target` holds, once a dispatch has written it.
holds: Option<SampleKey>,
/// The key the previous fused dispatch had, cached or not.
last: Option<SampleKey>,
/// What the most recent plan decided. Read by the tests, which have no
/// other way to tell a cached frame from an uncached one — the point being
/// that the pictures are identical.
last_use: SampleUse,
/// Bound at `@binding(6)` when the cache is not being read.
no_sampled: wgpu::TextureView,
/// Bound at `@binding(7)` when the cache is not being written.
no_sample_out: wgpu::TextureView,
}
impl SampleCache {
const FORMAT: wgpu::TextureFormat = DemosaicedImage::FORMAT;
fn new(ctx: &GpuContext) -> Self {
let placeholder = |label, usage| {
ctx.device
.create_texture(&wgpu::TextureDescriptor {
label: Some(label),
size: wgpu::Extent3d {
width: 1,
height: 1,
depth_or_array_layers: 1,
},
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: Self::FORMAT,
usage,
view_formats: &[],
})
.create_view(&Default::default())
};
Self {
target: None,
holds: None,
last: None,
last_use: SampleUse::Direct,
no_sampled: placeholder("adjust-no-sampled", wgpu::TextureUsages::TEXTURE_BINDING),
no_sample_out: placeholder(
"adjust-no-sample-out",
wgpu::TextureUsages::STORAGE_BINDING,
),
}
}
/// Decide how this dispatch samples, on the assumption that it will be
/// submitted — call it after anything that can still fail.
fn plan(
&mut self,
ctx: &GpuContext,
source: &DemosaicedImage,
shader: &ComposedShader,
width: u32,
height: u32,
) -> SampleUse {
self.last_use = self.decide(ctx, source, shader, width, height);
self.last_use
}
fn decide(
&mut self,
ctx: &GpuContext,
source: &DemosaicedImage,
shader: &ComposedShader,
width: u32,
height: u32,
) -> SampleUse {
let key = shader
.sample_key
.filter(|_| u64::from(width) * u64::from(height) <= SAMPLE_CACHE_MAX_PIXELS)
.map(|k| (k, source.id(), width, height));
let last = std::mem::replace(&mut self.last, key);
let Some(key) = key else {
return SampleUse::Direct;
};
if self.holds == Some(key) {
return SampleUse::Read;
}
if last != Some(key) {
return SampleUse::Direct;
}
if !self
.target
.as_ref()
.is_some_and(|t| t.width == width && t.height == height)
{
let texture = ctx.device.create_texture(&wgpu::TextureDescriptor {
label: Some("adjust-sample-cache"),
size: wgpu::Extent3d {
width,
height,
depth_or_array_layers: 1,
},
mip_level_count: 1,
sample_count: 1,
dimension: wgpu::TextureDimension::D2,
format: Self::FORMAT,
usage: wgpu::TextureUsages::STORAGE_BINDING | wgpu::TextureUsages::TEXTURE_BINDING,
view_formats: &[],
});
let view = texture.create_view(&Default::default());
self.target = Some(Target {
texture,
view,
width,
height,
});
}
// Marked as held now: the dispatch that writes it is submitted before
// any that could read it, and one queue orders the two.
self.holds = Some(key);
SampleUse::Write
}
/// Write the flags for `usage` into a fused uniform block.
fn flag(usage: SampleUse, uniforms: &mut [f32]) {
let o = dr_pipeline::SAMPLE_CACHE_UNIFORM_OFFSET;
uniforms[o] = if usage == SampleUse::Read { 1.0 } else { 0.0 };
uniforms[o + 1] = if usage == SampleUse::Write { 1.0 } else { 0.0 };
}
/// The views for `@binding(6)` and `@binding(7)`.
fn views(&self, usage: SampleUse) -> (wgpu::TextureView, wgpu::TextureView) {
let cache = || {
self.target
.as_ref()
.expect("planned with a target")
.view
.clone()
};
match usage {
SampleUse::Direct => (self.no_sampled.clone(), self.no_sample_out.clone()),
SampleUse::Write => (self.no_sampled.clone(), cache()),
SampleUse::Read => (cache(), self.no_sample_out.clone()),
}
}
/// Forget everything; the next render starts over.
fn release(&mut self) {
self.target = None;
self.holds = None;
self.last = None;
}
}
/// A baked film stock, resident on the GPU. /// A baked film stock, resident on the GPU.
struct FilmTextures { struct FilmTextures {
curves: wgpu::TextureView, curves: wgpu::TextureView,
@@ -440,6 +642,7 @@ impl AdjustPass {
camera_pipeline_layout, camera_pipeline_layout,
camera_target: None, camera_target: None,
colour_key: None, colour_key: None,
sample: SampleCache::new(ctx),
colour_dispatches: 0, colour_dispatches: 0,
detail_dispatches: 0, detail_dispatches: 0,
} }
@@ -537,6 +740,30 @@ impl AdjustPass {
}, },
count: None, count: None,
}, },
// The sample cache, read and written. Present in every
// layout for the reason the masks are, and bound to
// placeholders whenever the flags leave it alone. See
// `SampleCache`.
wgpu::BindGroupLayoutEntry {
binding: 6,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::Texture {
sample_type: wgpu::TextureSampleType::Float { filterable: false },
view_dimension: wgpu::TextureViewDimension::D2,
multisampled: false,
},
count: None,
},
wgpu::BindGroupLayoutEntry {
binding: 7,
visibility: wgpu::ShaderStages::COMPUTE,
ty: wgpu::BindingType::StorageTexture {
access: wgpu::StorageTextureAccess::WriteOnly,
format: SampleCache::FORMAT,
view_dimension: wgpu::TextureViewDimension::D2,
},
count: None,
},
], ],
}) })
} }
@@ -716,7 +943,15 @@ impl AdjustPass {
let (width, height) = (width.max(1), height.max(1)); let (width, height) = (width.max(1), height.max(1));
self.ensure_target(width, height); self.ensure_target(width, height);
let uniforms = Self::fused_uniforms(source, shader); // Compile first: `pipeline` takes &mut self, and the sample cache's
// plan below assumes the dispatch it plans for is submitted, so nothing
// after it may fail.
let _ = self.pipeline(shader)?;
let mut uniforms = Self::fused_uniforms(source, shader);
let sampling = self.sample.plan(&self.ctx, source, shader, width, height);
SampleCache::flag(sampling, &mut uniforms);
let (sampled, sample_out) = self.sample.views(sampling);
let params_buf = self let params_buf = self
.ctx .ctx
@@ -727,8 +962,6 @@ impl AdjustPass {
usage: wgpu::BufferUsages::UNIFORM, usage: wgpu::BufferUsages::UNIFORM,
}); });
// Borrow order: compile first, since `pipeline` takes &mut self.
let _ = self.pipeline(shader)?;
let pipeline = self let pipeline = self
.cache .cache
.get(&shader.structure_hash) .get(&shader.structure_hash)
@@ -768,6 +1001,14 @@ impl AdjustPass {
binding: 5, binding: 5,
resource: wgpu::BindingResource::TextureView(self.film_lut_view()), resource: wgpu::BindingResource::TextureView(self.film_lut_view()),
}, },
wgpu::BindGroupEntry {
binding: 6,
resource: wgpu::BindingResource::TextureView(&sampled),
},
wgpu::BindGroupEntry {
binding: 7,
resource: wgpu::BindingResource::TextureView(&sample_out),
},
], ],
}); });
@@ -885,7 +1126,12 @@ impl AdjustPass {
label: Some("adjust-detail-encoder"), label: Some("adjust-detail-encoder"),
}); });
let mut sampling = SampleUse::Direct;
if !reuse { if !reuse {
let mut uniforms = uniforms;
sampling = self.sample.plan(&self.ctx, source, shader, width, height);
SampleCache::flag(sampling, &mut uniforms);
let (sampled, sample_out) = self.sample.views(sampling);
let params_buf = let params_buf =
self.ctx self.ctx
.device .device
@@ -927,6 +1173,14 @@ impl AdjustPass {
binding: 5, binding: 5,
resource: wgpu::BindingResource::TextureView(&film_lut), resource: wgpu::BindingResource::TextureView(&film_lut),
}, },
wgpu::BindGroupEntry {
binding: 6,
resource: wgpu::BindingResource::TextureView(&sampled),
},
wgpu::BindGroupEntry {
binding: 7,
resource: wgpu::BindingResource::TextureView(&sample_out),
},
], ],
}); });
let pipeline = self let pipeline = self
@@ -954,9 +1208,20 @@ impl AdjustPass {
.expect("ensured above") .expect("ensured above")
.view .view
.clone(); .clone();
let ran = self let ran = match self
.detail .detail
.encode(&mut enc, detail, &target_view, width, height)?; .encode(&mut enc, detail, &target_view, width, height)
{
Ok(ran) => ran,
Err(e) => {
// Nothing is submitted, so a cache this frame was to write
// holds nothing, and must not be read as though it did.
if sampling == SampleUse::Write {
self.sample.release();
}
return Err(e);
}
};
self.ctx.queue.submit(Some(enc.finish())); self.ctx.queue.submit(Some(enc.finish()));
self.detail_dispatches += ran; self.detail_dispatches += ran;
self.colour_key = Some((key, width, height)); self.colour_key = Some((key, width, height));
@@ -1090,6 +1355,7 @@ impl AdjustPass {
self.detail.release_caches(); self.detail.release_caches();
self.targets = [None, None]; self.targets = [None, None];
self.colour_key = None; self.colour_key = None;
self.sample.release();
} }
/// How many distinct pipelines are compiled. Exposed for tests asserting /// How many distinct pipelines are compiled. Exposed for tests asserting
@@ -1244,6 +1510,16 @@ impl AdjustPass {
binding: 5, binding: 5,
resource: wgpu::BindingResource::TextureView(self.film_lut_view()), resource: wgpu::BindingResource::TextureView(self.film_lut_view()),
}, },
// The sample cache is the display's; a camera-space tap
// reads its source directly (its flags are zero).
wgpu::BindGroupEntry {
binding: 6,
resource: wgpu::BindingResource::TextureView(&self.sample.no_sampled),
},
wgpu::BindGroupEntry {
binding: 7,
resource: wgpu::BindingResource::TextureView(&self.sample.no_sample_out),
},
], ],
}); });
@@ -2624,6 +2900,131 @@ mod tests {
/// A flat RGBA8 image on the JPEG path — already gamma-encoded, as a /// A flat RGBA8 image on the JPEG path — already gamma-encoded, as a
/// decoded JPEG is. /// decoded JPEG is.
/// A frame with a different value at every pixel, several times the size
/// of the renders below, so that a fit view reads it on a stride and a
/// texel read from the wrong place cannot go unnoticed.
fn busy_image(ctx: &GpuContext) -> DemosaicedImage {
let (w, h) = (97u32, 61u32);
let mut data = Vec::with_capacity((w * h * 4) as usize);
for y in 0..h {
for x in 0..w {
let n = (x.wrapping_mul(2_654_435_761) ^ y.wrapping_mul(1_640_531_527)) >> 7;
data.extend_from_slice(&[n as u8, (n >> 8) as u8, (x * 2 + y) as u8, 255]);
}
}
DemosaicedImage::from_rgba8(ctx, &data, w, h).expect("upload")
}
/// Render `g` with a pass that has never seen it, which reads the source
/// directly by construction: the reference a cached frame must equal.
fn fresh(ctx: &GpuContext, img: &DemosaicedImage, g: &EditGraph) -> Vec<u8> {
let mut pass = AdjustPass::new(ctx);
let detail = g.compose_detail(img.size(), (23, 15));
pass.render_detailed(img, &g.compose(), 23, 15, None, &detail, 1)
.expect("render");
assert_eq!(pass.sample.last_use, SampleUse::Direct);
pass.export_pixels().expect("read").0
}
#[test]
fn a_cached_sample_is_the_same_picture() {
// TRACES: NFR-P5
// The sample cache's whole claim: a frame that reads the source through
// it is bit-for-bit the frame that reads the source directly. Walked
// through each state — direct, writing, reading, reading after a
// slider moved, and direct again once the framing moves — against a
// fresh pass each time.
let Some(ctx) = ctx() else { return };
let img = busy_image(&ctx);
let mut pass = AdjustPass::new(&ctx);
let mut g = EditGraph::default_chain();
let frame = |pass: &mut AdjustPass, g: &EditGraph, key: u64| {
let detail = g.compose_detail(img.size(), (23, 15));
pass.render_detailed(&img, &g.compose(), 23, 15, None, &detail, key)
.expect("render");
(pass.sample.last_use, pass.export_pixels().expect("read").0)
};
for (i, (expected, value)) in [
(SampleUse::Direct, 0.3),
(SampleUse::Write, 0.4),
(SampleUse::Read, 0.5),
(SampleUse::Read, -0.7),
]
.into_iter()
.enumerate()
{
g.set_param(exposure::ID, exposure::EXPOSURE, value);
let (used, pixels) = frame(&mut pass, &g, i as u64);
assert_eq!(used, expected, "frame {i}");
assert_eq!(pixels, fresh(&ctx, &img, &g), "frame {i} ({used:?})");
}
// A neighbourhood operation: the fused pass writes the linear
// intermediate instead, through the same sampling.
g.set_param(
dr_pipeline::ops::noise_reduction::ID,
dr_pipeline::ops::noise_reduction::CHROMA,
60.0,
);
for i in 10..13 {
g.set_param(exposure::ID, exposure::EXPOSURE, i as f32 * 0.01);
let (used, pixels) = frame(&mut pass, &g, i);
assert_eq!(used, SampleUse::Read, "detail frame {i}");
assert_eq!(pixels, fresh(&ctx, &img, &g), "detail frame {i}");
}
// The framing moves: what was cached is for the old framing.
g.set_param(dr_pipeline::framing::ID, dr_pipeline::framing::CROP_W, 0.6);
let (used, pixels) = frame(&mut pass, &g, 20);
assert_eq!(used, SampleUse::Direct, "a new framing reads directly");
assert_eq!(pixels, fresh(&ctx, &img, &g));
let (used, _) = frame(&mut pass, &g, 21);
assert_eq!(used, SampleUse::Write, "and caches once it holds still");
let (used, pixels) = frame(&mut pass, &g, 22);
assert_eq!(used, SampleUse::Read);
assert_eq!(pixels, fresh(&ctx, &img, &g));
}
#[test]
fn a_cache_is_not_read_for_another_image_or_size() {
// The key is the composer's half plus the two things only this side
// knows. A second photograph with the same edit and the same framing
// must not be shown the first one's texels.
let Some(ctx) = ctx() else { return };
let (a, b) = (busy_image(&ctx), grey_image(&ctx, 8000));
let mut pass = AdjustPass::new(&ctx);
let shader = EditGraph::default_chain().compose();
for _ in 0..3 {
pass.render(&a, &shader, 23, 15).expect("render");
}
assert_eq!(pass.sample.last_use, SampleUse::Read);
pass.render(&b, &shader, 23, 15).expect("render");
assert_eq!(pass.sample.last_use, SampleUse::Direct, "another image");
pass.render(&b, &shader, 23, 15).expect("render");
pass.render(&b, &shader, 24, 15).expect("render");
assert_eq!(pass.sample.last_use, SampleUse::Direct, "another size");
}
#[test]
fn a_straightened_frame_samples_directly() {
// Interpolated: the sample is a blend of four texels, which the cache's
// format could not hold exactly, so the composer offers no key.
let Some(ctx) = ctx() else { return };
let img = busy_image(&ctx);
let mut pass = AdjustPass::new(&ctx);
let mut g = EditGraph::default_chain();
g.set_param(dr_pipeline::framing::ID, dr_pipeline::framing::ANGLE, 3.0);
let shader = g.compose();
assert!(shader.sample_key.is_none());
for _ in 0..3 {
pass.render(&img, &shader, 23, 15).expect("render");
assert_eq!(pass.sample.last_use, SampleUse::Direct);
}
}
fn jpeg_image(ctx: &GpuContext, rgb: [u8; 3]) -> DemosaicedImage { fn jpeg_image(ctx: &GpuContext, rgb: [u8; 3]) -> DemosaicedImage {
let size = 16u32; let size = 16u32;
let mut data = Vec::with_capacity((size * size) as usize * 4); let mut data = Vec::with_capacity((size * size) as usize * 4);
core/dr-gpu/src/demosaic.rs
+24
View File
@@ -95,6 +95,15 @@ pub struct DemosaicedImage {
base_curve: BaseCurve, base_curve: BaseCurve,
/// Whether the texture holds gamma-encoded rather than linear values. /// Whether the texture holds gamma-encoded rather than linear values.
non_linear: bool, non_linear: bool,
/// Which upload this is, unique for the life of the process. See
/// [`Self::id`].
id: u64,
}
/// The next [`DemosaicedImage::id`].
fn next_image_id() -> u64 {
static NEXT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(1);
NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
} }
impl DemosaicedImage { impl DemosaicedImage {
@@ -112,6 +121,18 @@ impl DemosaicedImage {
(self.width, self.height) (self.width, self.height)
} }
/// Which texture this is, as a number that is never reused.
///
/// For a cache that has to know it is still looking at the same pixels
/// (`AdjustPass`'s sample cache) without holding the texture alive to find
/// out: keeping a handle would keep half a gigabyte of a closed photograph
/// on the device, and comparing addresses would mistake a new upload for an
/// old one the moment the allocator reused the slot. A texture here is
/// never written after it is built, so the same id is the same pixels.
pub(crate) fn id(&self) -> u64 {
self.id
}
/// Camera RGB → linear sRGB, row-major. Identity where the body is /// Camera RGB → linear sRGB, row-major. Identity where the body is
/// uncalibrated, so the image renders uncalibrated rather than black. /// uncalibrated, so the image renders uncalibrated rather than black.
pub fn color_matrix(&self) -> [f32; 9] { pub fn color_matrix(&self) -> [f32; 9] {
@@ -246,6 +267,7 @@ impl DemosaicedImage {
// the highlights of an image that was already finished. // the highlights of an image that was already finished.
base_curve: BaseCurve::IDENTITY, base_curve: BaseCurve::IDENTITY,
non_linear: true, non_linear: true,
id: next_image_id(),
}) })
} }
} }
@@ -322,6 +344,7 @@ impl DemosaicedImage {
as_shot_wb: [raw.wb_coeffs[0], raw.wb_coeffs[1], raw.wb_coeffs[2]], as_shot_wb: [raw.wb_coeffs[0], raw.wb_coeffs[1], raw.wb_coeffs[2]],
base_curve: raw.base_curve, base_curve: raw.base_curve,
non_linear: false, non_linear: false,
id: next_image_id(),
}) })
} }
} }
@@ -660,6 +683,7 @@ impl Demosaicer {
// normalises against black and white levels and applies no // normalises against black and white levels and applies no
// transfer function. // transfer function.
non_linear: false, non_linear: false,
id: next_image_id(),
}) })
} }
} }
core/dr-pipeline/presets/bw_film.drpl
+37
View File
@@ -0,0 +1,37 @@
drpl 1
# Black and white film: each preset is one measured stock, developed and — for a
# negative — printed on the paper its profile names, with every other
# control left where it was. The stock is the look; a grade on top is the
# photographer's to add.
#
# The measurements are spektrafilm's (Andrea Volpato, CC BY-SA 4.0), as
# converted in core/dr-film/profiles. A preset names a stock by id, so
# these lines are that attribution's reach: nothing of the data is here.
[preset Ilford Delta 100]
film = ilford_delta_100
film_print = kodak_2302
[preset Ilford Delta 400]
film = ilford_delta_400
film_print = kodak_2302
[preset Ilford FP4 Plus]
film = ilford_fp4_plus
film_print = kodak_2302
[preset Ilford HP5 Plus]
film = ilford_hp5_plus
film_print = kodak_2302
[preset Ilford Pan F Plus]
film = ilford_pan_f_plus
film_print = kodak_2302
[preset Kodak Double-X 5222]
film = kodak_doublex
film_print = kodak_2302
[preset Kodak Tri-X Reversal 7266]
film = kodak_trix
core/dr-pipeline/presets/cinema_film.drpl
+30
View File
@@ -0,0 +1,30 @@
drpl 1
# Cinema film: each preset is one measured stock, developed and — for a
# negative — printed on the paper its profile names, with every other
# control left where it was. The stock is the look; a grade on top is the
# photographer's to add.
#
# The measurements are spektrafilm's (Andrea Volpato, CC BY-SA 4.0), as
# converted in core/dr-film/profiles. A preset names a stock by id, so
# these lines are that attribution's reach: nothing of the data is here.
[preset Kodak Verita 200D]
film = kodak_verita_200d
film_print = kodak_2383
[preset Kodak Vision3 200T]
film = kodak_vision3_200t
film_print = kodak_2383
[preset Kodak Vision3 250D]
film = kodak_vision3_250d
film_print = kodak_2383
[preset Kodak Vision3 500T]
film = kodak_vision3_500t
film_print = kodak_2383
[preset Kodak Vision3 50D]
film = kodak_vision3_50d
film_print = kodak_2383
core/dr-pipeline/presets/colour_film.drpl
+66
View File
@@ -0,0 +1,66 @@
drpl 1
# Colour film: each preset is one measured stock, developed and — for a
# negative — printed on the paper its profile names, with every other
# control left where it was. The stock is the look; a grade on top is the
# photographer's to add.
#
# The measurements are spektrafilm's (Andrea Volpato, CC BY-SA 4.0), as
# converted in core/dr-film/profiles. A preset names a stock by id, so
# these lines are that attribution's reach: nothing of the data is here.
[preset Fujifilm C200]
film = fujifilm_c200
film_print = fujifilm_crystal_archive_typeii
[preset Fujifilm Pro 400H]
film = fujifilm_pro_400h
film_print = fujifilm_crystal_archive_typeii
[preset Fujifilm Provia 100F]
film = fujifilm_provia_100f
[preset Fujifilm Velvia 100]
film = fujifilm_velvia_100
[preset Fujifilm X-Tra 400]
film = fujifilm_xtra_400
film_print = fujifilm_crystal_archive_typeii
[preset Kodak Ektachrome 100]
film = kodak_ektachrome_100
[preset Kodak Ektar 100]
film = kodak_ektar_100
film_print = kodak_portra_endura
[preset Kodak Gold 200]
film = kodak_gold_200
film_print = kodak_portra_endura
[preset Kodak Kodachrome 64]
film = kodak_kodachrome_64
[preset Kodak Portra 160]
film = kodak_portra_160
film_print = kodak_portra_endura
[preset Kodak Portra 400]
film = kodak_portra_400
film_print = kodak_portra_endura
[preset Kodak Portra 800]
film = kodak_portra_800
film_print = kodak_portra_endura
[preset Kodak Portra 800 pushed one stop]
film = kodak_portra_800_push1
film_print = kodak_portra_endura
[preset Kodak Portra 800 pushed two stops]
film = kodak_portra_800_push2
film_print = kodak_portra_endura
[preset Kodak Ultramax 400]
film = kodak_ultramax_400
film_print = kodak_portra_endura
core/dr-pipeline/presets/essentials.drpl
+41
View File
@@ -0,0 +1,41 @@
drpl 1
# Essentials: small, general corrections written against this pipeline.
# These were the first-run starter set; they ship here now so that a
# new release can improve them without rewriting anybody's own presets.
# Deliberately mild — see bundled.rs.
[preset Crisp detail]
capture_sharpen.amount = 35
clarity.amount = 10
texture.amount = 20
[preset Lift the shadows]
blacks_whites.blacks = 12
contrast.contrast = -5
highlights_shadows.shadows = 40
[preset Muted]
contrast.contrast = -10
highlights_shadows.shadows = 12
saturation.saturation = -30
vibrance.vibrance = 10
[preset Punch]
blacks_whites.blacks = -8
clarity.amount = 12
contrast.contrast = 18
vibrance.vibrance = 18
[preset Recover the sky]
blacks_whites.whites = -10
highlights_shadows.highlights = -55
highlights_shadows.shadows = 35
[preset Soft portrait]
clarity.amount = -10
contrast.contrast = -8
highlights_shadows.highlights = -20
highlights_shadows.shadows = 15
saturation.saturation = -5
vibrance.vibrance = 10
core/dr-pipeline/presets/skies.drpl
+52
View File
@@ -0,0 +1,52 @@
drpl 1
# Skies: a bluer, deeper sky without touching the rest of the picture.
#
# Written against this pipeline, not derived from anybody's preset. Each
# works the colour mixer's azure and blue bands — the hues a clear sky
# occupies, 210° and 240° — darkening them and adding chroma, which is what
# a polarising filter does to a sky and why it reads as "more blue" rather
# than "more saturated". A grey sky has no hue for the bands to find, so on
# an overcast frame these do little, by construction: they cannot invent a
# sky, and a preset that tinted grey clouds blue would be one nobody trusted.
#
# Highlights come down with the sky in the stronger ones, because a darker
# blue beside a clipped white cloud looks like a mask edge.
[preset Blue sky]
colour_mixer.azure_lum = -20
colour_mixer.azure_sat = 25
colour_mixer.blue_lum = -15
colour_mixer.blue_sat = 20
highlights_shadows.highlights = -15
[preset Deep blue sky]
colour_mixer.azure_hue = 10
colour_mixer.azure_lum = -30
colour_mixer.azure_sat = 35
colour_mixer.blue_lum = -25
colour_mixer.blue_sat = 30
colour_mixer.cyan_sat = 10
highlights_shadows.highlights = -30
[preset Polariser]
colour_mixer.azure_hue = 10
colour_mixer.azure_lum = -35
colour_mixer.azure_sat = 40
colour_mixer.blue_lum = -30
colour_mixer.blue_sat = 35
colour_mixer.cyan_lum = -10
colour_mixer.cyan_sat = 15
dehaze.amount = 20
highlights_shadows.highlights = -35
vibrance.vibrance = 10
[preset Blue sky, golden land]
colour_mixer.azure_lum = -20
colour_mixer.azure_sat = 25
colour_mixer.blue_lum = -15
colour_mixer.blue_sat = 20
colour_mixer.orange_sat = 12
colour_mixer.yellow_hue = -10
colour_mixer.yellow_sat = 15
highlights_shadows.highlights = -20
core/dr-pipeline/src/bundled.rs
+445
View File
@@ -0,0 +1,445 @@
//! TRACES: FR-DEV-6
//! The presets that ship with the application.
//!
//! # Shipped, not seeded
//!
//! The first six used to be *copied* into the photographer's own library on
//! the first run and were theirs from then on. That was the right answer for
//! six, and it cannot grow: a copy is frozen at the version that made it, so a
//! better "Portra" in the next release would reach nobody who already had the
//! old one, and re-seeding would overwrite a preset someone had tuned. So the
//! shipped set is now read from the binary every time, never written to the
//! user's file, and changes when the application does.
//!
//! # Your copy wins, as long as it keeps the name
//!
//! Saving over a shipped preset's name makes the photographer's version the
//! one that name means, here and in every apply. It is still *that* preset —
//! listed where the shipped one was, marked as changed — and deleting it
//! reveals the shipped one again, which is what "revert" means to the person
//! pressing it. Renaming it cuts the link: it becomes one of their own, and
//! the shipped preset reappears beside it. The lookup is by name because the
//! name is what the photographer sees and chooses by; an id they never see
//! would link two presets they believe are different.
//!
//! # Looks, not whole edits
//!
//! Every shipped preset reaches only the operations it names
//! ([`Reach::Named`]): a look applied to a corrected photograph must keep the
//! correction. A photographer's own saved edits keep [`Reach::Whole`], which
//! is what saving an edit has always meant.
//!
//! # Why the data is text files
//!
//! The same format the user's library is written in, so a shipped preset can
//! be read, diffed and copied into one's own library by hand, and each file
//! can say in a comment where its looks came from — the attribution a licence
//! may require travels with the data it covers.
//!
//! # Why this lives in the core
//!
//! It names operations — "Punch" is a statement about contrast and clarity —
//! and nothing in `ui/` may (`ui_names_no_operation.rs`, ARCH §4.3a). The
//! frontend asks for the listing and applies what it is handed.
use crate::preset::{Preset, PresetLibrary, Reach};
/// One group of shipped presets, as the sheet lists it.
pub struct Section {
/// A stable identifier, for a frontend that remembers which sections a
/// photographer folded away. Never shown.
pub id: &'static str,
/// What the section is called on screen.
pub title: &'static str,
/// The presets in it, every one reaching only what it names.
pub presets: PresetLibrary,
}
/// The files, in the order the sheet lists them.
const SECTIONS: &[(&str, &str, &str)] = &[
(
"essentials",
"Essentials",
include_str!("../presets/essentials.drpl"),
),
("skies", "Skies", include_str!("../presets/skies.drpl")),
(
"colour_film",
"Colour film",
include_str!("../presets/colour_film.drpl"),
),
(
"cinema_film",
"Cinema film",
include_str!("../presets/cinema_film.drpl"),
),
(
"bw_film",
"Black and white film",
include_str!("../presets/bw_film.drpl"),
),
];
/// Every shipped section, parsed.
///
/// Parsed on each call rather than held: it is a few kilobytes read when the
/// preset sheet is drawn, and a static would be one more thing to keep
/// consistent with the files in a test. A file that fails to parse costs its
/// section, with a warning, rather than the sheet — and the tests below make
/// sure none does.
pub fn sections() -> Vec<Section> {
SECTIONS
.iter()
.filter_map(|(id, title, text)| match PresetLibrary::parse(text) {
Ok(library) => Some(Section {
id,
title,
presets: as_looks(library),
}),
Err(e) => {
log::warn!("shipped preset section {id} is unreadable ({e}); skipping");
None
}
})
.collect()
}
/// Mark every preset in `library` as a look.
///
/// Here rather than as a `reach = named` line in every block of every file:
/// the rule is about where a preset came from, and a file that forgot the
/// line would ship a preset that wiped a photographer's corrections.
fn as_looks(library: PresetLibrary) -> PresetLibrary {
let mut looks = PresetLibrary::default();
for (name, preset) in library.iter() {
let _ = looks.insert(name, preset.clone().with_reach(Reach::Named));
}
looks
}
/// Where a listed preset comes from.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Origin {
/// The photographer's own, with no shipped preset of that name.
Yours,
/// Shipped, and not overridden.
Shipped,
/// Shipped, and overridden by the photographer's copy under the same
/// name. The copy is what applies; deleting it reverts to the shipped one.
Changed,
}
/// One row of the preset sheet.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Listed {
pub name: String,
pub origin: Origin,
}
/// One group of rows: the photographer's own, then each shipped section.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ListedSection {
/// `None` for the photographer's own presets.
pub id: Option<&'static str>,
pub title: &'static str,
pub rows: Vec<Listed>,
}
/// Everything the sheet lists, in order, with the photographer's copies
/// standing in for the shipped presets they override.
///
/// Their own presets come first, because a photographer reaches for their own
/// work more than for anybody's defaults, and a copy of a shipped preset is
/// listed in the shipped section rather than among their own — it is still
/// that preset, changed, and belongs where they would look for it.
pub fn listing(yours: &PresetLibrary) -> Vec<ListedSection> {
let shipped = sections();
let is_shipped = |name: &str| shipped.iter().any(|section| section.presets.contains(name));
let mut out = vec![ListedSection {
id: None,
title: "Yours",
rows: yours
.names()
.filter(|name| !is_shipped(name))
.map(|name| Listed {
name: name.to_string(),
origin: Origin::Yours,
})
.collect(),
}];
out.extend(shipped.iter().map(|section| {
ListedSection {
id: Some(section.id),
title: section.title,
rows: section
.presets
.names()
.map(|name| Listed {
name: name.to_string(),
origin: if yours.contains(name) {
Origin::Changed
} else {
Origin::Shipped
},
})
.collect(),
}
}));
out
}
/// The preset a name means: the photographer's if they have one, the shipped
/// one otherwise.
pub fn lookup(yours: &PresetLibrary, name: &str) -> Option<Preset> {
yours.get(name).cloned().or_else(|| {
sections()
.into_iter()
.find_map(|section| section.presets.get(name).cloned())
})
}
/// Whether `name` is a shipped preset's.
pub fn is_shipped(name: &str) -> bool {
sections()
.iter()
.any(|section| section.presets.contains(name))
}
/// Remove the copies a first run used to seed, where they are still exactly
/// as seeded. Returns how many went.
///
/// Those copies would otherwise all list as changed — overriding a shipped
/// preset with an identical one — and would freeze the six at their old
/// values forever. One that differs in any way was tuned by somebody and is
/// kept: it is theirs, and it now overrides the shipped one, which is the
/// rule above doing what it is for.
///
/// Compared on the parameters and the film and not on [`Reach`]: the seeded
/// copies were whole edits, the shipped ones are looks, and that difference
/// is the thing this migration exists to deliver.
pub fn forget_unchanged_copies(yours: &mut PresetLibrary) -> usize {
let shipped = sections();
let stale: Vec<String> = yours
.iter()
.filter(|(name, preset)| {
shipped.iter().any(|section| {
section
.presets
.get(name)
.is_some_and(|s| s.params() == preset.params() && s.film() == preset.film())
})
})
.map(|(name, _)| name.to_string())
.collect();
for name in &stale {
yours.remove(name);
}
stale.len()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{EditGraph, Scope};
fn all() -> Vec<(&'static str, String, Preset)> {
sections()
.into_iter()
.flat_map(|s| {
let id = s.id;
s.presets
.iter()
.map(|(n, p)| (id, n.to_string(), p.clone()))
.collect::<Vec<_>>()
})
.collect()
}
#[test]
fn every_file_parses_and_every_line_is_understood() {
// A misspelt key would otherwise be kept as a line this build does
// not understand — preserved faithfully, and doing nothing.
assert_eq!(
sections().len(),
SECTIONS.len(),
"a section failed to parse"
);
for (id, _, text) in SECTIONS {
let library = PresetLibrary::parse(text).unwrap();
assert_eq!(library.unread_lines(), 0, "{id} has lines nobody reads");
assert!(!library.is_empty(), "{id} is empty");
}
}
#[test]
fn every_shipped_name_is_unique_across_sections() {
// A name is what the lookup and the override key on. Two shipped
// presets sharing one would make one of them unreachable.
let mut names: Vec<String> = all().into_iter().map(|(_, n, _)| n).collect();
let before = names.len();
names.sort();
names.dedup();
assert_eq!(names.len(), before);
}
#[test]
fn every_shipped_preset_is_a_look() {
for (id, name, preset) in all() {
assert_eq!(preset.reach(), Reach::Named, "{id}/{name}");
}
}
#[test]
fn every_shipped_preset_names_parameters_this_build_actually_has() {
// A renamed parameter must break the build rather than ship a preset
// that quietly does nothing.
let graph = EditGraph::default_chain();
let capabilities = graph.capabilities();
for (id, name, preset) in all() {
for (op, param) in preset.params().keys() {
let capability = capabilities
.iter()
.find(|c| c.id.0 == op)
.unwrap_or_else(|| panic!("{id}/{name}: no operation {op:?}"));
assert!(
capability.params.iter().any(|p| p.id.0 == param),
"{id}/{name}: operation {op:?} has no parameter {param:?}"
);
}
}
}
#[test]
fn every_shipped_preset_changes_something() {
// A preset that applies to nothing teaches the photographer that the
// list does not work.
for (id, name, preset) in all() {
let mut graph = EditGraph::default_chain();
let rebake = preset.apply(&mut graph, Scope::adjustments());
assert!(
rebake.wanted().is_some() || Preset::capture(&graph) != Preset::default(),
"{id}/{name} left the graph at its defaults"
);
}
}
#[test]
fn no_shipped_preset_carries_a_crop() {
for (id, name, preset) in all() {
assert!(!preset.touches_framing(), "{id}/{name} carries framing");
}
}
#[test]
fn the_values_stay_inside_what_the_controls_accept() {
// Clamping happens on apply, so an out-of-range literal would be
// silently trimmed and the preset would not be the one written.
for (id, name, preset) in all() {
let mut graph = EditGraph::default_chain();
preset
.clone()
.with_film(None)
.apply(&mut graph, Scope::everything())
.expect_no_film();
for ((op, param), value) in preset.params() {
let (op_id, param_id) = crate::preset::resolve(&graph, op, param).unwrap();
assert_eq!(
graph.param(op_id, param_id),
Some(*value),
"{id}/{name}: {op}.{param} = {value} was clamped"
);
}
}
}
// --- the listing and the override ------------------------------------
fn yours_with(names: &[(&str, Preset)]) -> PresetLibrary {
let mut lib = PresetLibrary::default();
for (n, p) in names {
lib.insert(n, p.clone()).unwrap();
}
lib
}
fn first_shipped() -> (String, Preset) {
let (_, name, preset) = all().into_iter().next().unwrap();
(name, preset)
}
#[test]
fn your_copy_under_a_shipped_name_is_what_that_name_applies() {
let (name, shipped) = first_shipped();
let mine = Preset::default();
let yours = yours_with(&[(&name, mine.clone())]);
assert_eq!(lookup(&yours, &name), Some(mine));
assert_eq!(lookup(&PresetLibrary::default(), &name), Some(shipped));
}
#[test]
fn your_copy_is_listed_in_the_shipped_section_as_changed() {
let (name, _) = first_shipped();
let yours = yours_with(&[(&name, Preset::default()), ("Mine", Preset::default())]);
let listing = listing(&yours);
assert_eq!(listing[0].id, None);
assert_eq!(
listing[0].rows,
vec![Listed {
name: "Mine".into(),
origin: Origin::Yours
}],
"an override must not be listed twice"
);
let row = listing[1..]
.iter()
.flat_map(|s| &s.rows)
.find(|r| r.name == name)
.unwrap();
assert_eq!(row.origin, Origin::Changed);
}
#[test]
fn deleting_your_copy_reverts_to_the_shipped_one() {
let (name, shipped) = first_shipped();
let mut yours = yours_with(&[(&name, Preset::default())]);
yours.remove(&name);
assert_eq!(lookup(&yours, &name), Some(shipped));
}
#[test]
fn renaming_your_copy_cuts_the_link() {
let (name, shipped) = first_shipped();
let mut yours = yours_with(&[(&name, Preset::default())]);
yours.rename(&name, "My version").unwrap();
assert_eq!(lookup(&yours, &name), Some(shipped));
assert_eq!(lookup(&yours, "My version"), Some(Preset::default()));
assert_eq!(listing(&yours)[0].rows[0].name, "My version");
}
#[test]
fn the_old_seeded_copies_are_forgotten_and_tuned_ones_kept() {
// The six as a first run wrote them: the same parameters, as whole
// edits, because that is what a seeded copy was.
let essentials = sections().into_iter().next().unwrap().presets;
let mut yours = PresetLibrary::default();
for (name, preset) in essentials.iter() {
yours
.insert(name, preset.clone().with_reach(Reach::Whole))
.unwrap();
}
let tuned = essentials.names().next().unwrap().to_string();
yours.insert(&tuned, Preset::default()).unwrap();
yours.insert("Mine", Preset::default()).unwrap();
let forgotten = forget_unchanged_copies(&mut yours);
assert_eq!(forgotten, essentials.len() - 1);
assert!(yours.contains(&tuned), "a tuned copy was thrown away");
assert!(yours.contains("Mine"));
assert_eq!(yours.len(), 2);
}
}
core/dr-pipeline/src/detail.rs
+102
View File
@@ -431,6 +431,24 @@ pub struct DetailPass {
pub storage: Vec<[f32; 4]>, pub storage: Vec<[f32; 4]>,
} }
impl DetailPass {
/// Whether this pass hands on exactly what it was given: a full-size pass
/// with nothing to bind and a body with no code in it, only comments.
///
/// The composer drops such a pass where that is exact (see
/// [`compose_detail_with`]); the operation still emits it, because whether
/// dropping it is exact depends on what is around it in the chain.
pub fn is_identity(&self) -> bool {
self.output_scale <= 1
&& self.storage.is_empty()
&& self
.wgsl
.lines()
.map(|l| l.split("//").next().unwrap_or("").trim())
.all(str::is_empty)
}
}
/// TRACES: FR-DEV-3 | FR-DEV-8 /// TRACES: FR-DEV-3 | FR-DEV-8
/// An operation that reads pixels other than the one it is writing. /// An operation that reads pixels other than the one it is writing.
/// ///
@@ -646,6 +664,40 @@ pub fn compose_detail_with(
}; };
} }
// TRACES: NFR-P5
// A pass whose body is empty changes nothing but where the pixels are: it
// reads the intermediate and writes the same values to the other one.
// Capture sharpening emits exactly that at a scale too coarse to draw its
// radius (`nothing_to_sharpen`), and at fit on any modern sensor that is
// most of the time — so an edit with sharpening *and* another kernel paid
// a full render-sized read and write for it on every frame, 4.4 ms of a
// 2560 x 1600 frame on the reference laptop with its clocks held down.
//
// Dropped here, where the chain is still a list, and only where dropping
// it is exact:
//
// - **Not the last pass.** The last pass performs the output transform on
// what it read from an `rgba16float` intermediate. Moving that transform
// onto the pass before would apply it to that pass's `f32` result
// instead, which is a different rounding of the same picture.
// - **Not after a reduced pass.** A full-resolution pass ends the reduced
// chain (see `DetailRunner::encode`), so one that follows a scaled pass
// is what stops the next operation reading the last one's base. None of
// today's operations leave a reduced chain open, but a declared one may.
//
// Everywhere else the pass before and the pass after exchange the same
// `rgba16float` texels either way, `aux` included.
let mut kept: Vec<(&str, &[Helper], DetailPass, usize)> = Vec::with_capacity(planned.len());
let total = planned.len();
for (position, entry) in planned.into_iter().enumerate() {
let after_full = kept.last().is_none_or(|(_, _, p, _)| p.output_scale <= 1);
let droppable = position + 1 < total && after_full && entry.2.is_identity();
if !droppable {
kept.push(entry);
}
}
let planned = kept;
let last = planned.len().saturating_sub(1); let last = planned.len().saturating_sub(1);
let passes = planned let passes = planned
.into_iter() .into_iter()
@@ -1211,6 +1263,56 @@ mod tests {
assert_eq!(fused(&ops).output_mode, OutputMode::LinearWorking); assert_eq!(fused(&ops).output_mode, OutputMode::LinearWorking);
} }
#[test]
fn a_pass_that_changes_nothing_is_dropped_where_that_is_exact() {
// TRACES: NFR-P5
// Capture sharpening at a scale too coarse to draw its radius emits a
// pass with an empty body. Between two other passes it costs a
// render-sized read and write and changes no texel, so it goes; as the
// last pass it performs the output transform on the intermediate, and
// moving that onto the pass before would round differently, so it
// stays.
use crate::ops::{capture_sharpen, CaptureSharpen, NoiseReduction};
let sharpen = || -> Box<dyn Operation> {
let mut op = CaptureSharpen::new();
op.set_param(capture_sharpen::AMOUNT, 60.0);
Box::new(op)
};
let chroma = || -> Box<dyn Operation> { Box::new(NoiseReduction::with_amounts(0.0, 60.0)) };
// A 24 MP frame fitted to a panel: a one-source-pixel radius is a
// quarter of a render pixel.
let scale = RenderScale::new((1500, 1000), (6000, 4000));
let unresolved = sharpen().detail().expect("a detail stage").passes(scale);
assert!(
unresolved.len() == 1 && unresolved[0].is_identity(),
"the premise: sharpening at this scale is one pass that does nothing"
);
let labels = |ops: &[Box<dyn Operation>]| -> Vec<String> {
compose_detail(ops, scale, dr_types::ColourSpace::Srgb)
.passes
.iter()
.map(|p| p.label.clone())
.collect()
};
// First, ahead of the chroma passes: dropped.
let first = labels(&[sharpen(), chroma()]);
assert_eq!(
first,
[
"noise_reduction/chroma-horizontal",
"noise_reduction/chroma-vertical"
]
);
// Last, after them: kept, and it is the pass that encodes.
let last = labels(&[chroma(), sharpen()]);
assert_eq!(last.len(), 3);
assert_eq!(last[2], "capture_sharpen/unresolved");
// Alone: kept, because the fused pass stopped short and something has
// to finish the frame.
assert_eq!(labels(&[sharpen()]), ["capture_sharpen/unresolved"]);
}
#[test] #[test]
fn a_pass_can_hand_a_scalar_to_the_next_one_alongside_the_colour() { fn a_pass_can_hand_a_scalar_to_the_next_one_alongside_the_colour() {
// What makes an unsharp mask — sharpening, clarity, texture, dehaze — // What makes an unsharp mask — sharpening, clarity, texture, dehaze —
core/dr-pipeline/src/graph.rs
+8 -2
View File
@@ -623,7 +623,9 @@ impl EditGraph {
} = self; } = self;
EditState { EditState {
params: Preset::capture(self), // Without the film: it has a field of its own below, and one
// edit must not have two places to disagree about its stock.
params: Preset::capture_params(self),
// A refcount bump. See `EditState::masks` for why that matters on // A refcount bump. See `EditState::masks` for why that matters on
// a path called once a frame. // a path called once a frame.
masks: Arc::clone(masks), masks: Arc::clone(masks),
@@ -663,7 +665,11 @@ impl EditGraph {
// At full scope. `Scope` is a question about what a paste carries // At full scope. `Scope` is a question about what a paste carries
// *between* photographs; this is one photograph's own edit being put // *between* photographs; this is one photograph's own edit being put
// back, so there is nothing to leave behind. // back, so there is nothing to leave behind.
params.apply(self, Scope::everything()); //
// What the parameters say about the film is discarded: the stock is
// `film`'s to decide, below, and clearing it is what happens there
// either way.
let _ = params.apply(self, Scope::everything());
self.masks = Arc::clone(masks); self.masks = Arc::clone(masks);
self.spots = spots.clone(); self.spots = spots.clone();
core/dr-pipeline/src/lib.rs
+3 -3
View File
@@ -32,6 +32,7 @@
//! single multiply and white balance a per-channel scale; on gamma-encoded //! single multiply and white balance a per-channel scale; on gamma-encoded
//! data neither would be physically meaningful (ARCH §5.2). //! data neither would be physically meaningful (ARCH §5.2).
pub mod bundled;
pub mod coverage; pub mod coverage;
pub mod declared; pub mod declared;
pub mod descriptor; pub mod descriptor;
@@ -48,7 +49,6 @@ pub mod orphan;
pub mod preset; pub mod preset;
pub mod sidecar; pub mod sidecar;
pub mod spot; pub mod spot;
pub mod starter;
pub mod state; pub mod state;
pub use coverage::Coverage; pub use coverage::Coverage;
@@ -67,9 +67,9 @@ pub use lens::{compose_warps, ComposedWarp, LensProfile, Tca, Warp};
pub use operation::{ pub use operation::{
compose, compose_with_framing, Affects, ComposedShader, Helper, Invalidation, Operation, compose, compose_with_framing, Affects, ComposedShader, Helper, Invalidation, Operation,
OutputMode, Uniform, BASE_CURVE_POINTS, BASE_CURVE_UNIFORM_OFFSET, CLIP_ONSET, OutputMode, Uniform, BASE_CURVE_POINTS, BASE_CURVE_UNIFORM_OFFSET, CLIP_ONSET,
RESERVED_UNIFORM_FIELDS, RESERVED_UNIFORM_FIELDS, SAMPLE_CACHE_UNIFORM_OFFSET,
}; };
pub use preset::{LibraryParseError, NameError, Preset, PresetLibrary, Scope}; pub use preset::{LibraryParseError, NameError, Preset, PresetLibrary, Reach, Scope};
pub use sidecar::{Sidecar, Version}; pub use sidecar::{Sidecar, Version};
pub use spot::{Spot, SpotMode, SpotSet}; pub use spot::{Spot, SpotMode, SpotSet};
pub use state::{EditState, FilmRebake, FilmRef}; pub use state::{EditState, FilmRebake, FilmRef};
core/dr-pipeline/src/operation.rs
+71 -3
View File
@@ -456,6 +456,25 @@ pub struct ComposedShader {
pub structure_hash: u64, pub structure_hash: u64,
/// What this shader writes. See [`OutputMode`]. /// What this shader writes. See [`OutputMode`].
pub output_mode: OutputMode, pub output_mode: OutputMode,
/// What decides which source texel each output pixel reads, when that
/// texel is read whole — `None` when it is interpolated.
///
/// A fit view reads one texel in every three or four of a 60 MP source,
/// on a stride, and that gather is most of what the fused pass costs
/// there: the texel it wants shares a cache line with neighbours nobody
/// reads. But the gather depends on the framing and nothing else, so it
/// is the same on every frame of a slider drag. The shader can therefore
/// write what it gathered to a viewport-sized texture once and read it
/// back contiguously thereafter; the flags in the uniform block at
/// [`SAMPLE_CACHE_UNIFORM_OFFSET`] say which, and `dr-gpu` decides.
///
/// This key is the half of that decision only the composer can make: a
/// hash of the generated prologue and the framing and warp uniforms, which
/// together are everything that maps an output pixel to a source texel.
/// The caller mixes in the source image and the render size. `None` for
/// the interpolating paths, whose sample is a blend of four texels and
/// not representable exactly in the source's own format.
pub sample_key: Option<u64>,
} }
/// Fields the generated uniform struct always carries, before op uniforms. /// Fields the generated uniform struct always carries, before op uniforms.
@@ -466,7 +485,19 @@ pub struct ComposedShader {
/// Twelve of the twenty-eight are the camera profile's base curve /// Twelve of the twenty-eight are the camera profile's base curve
/// ([`BASE_CURVE_UNIFORM_FIELDS`]); the rest are the matrix, the as-shot /// ([`BASE_CURVE_UNIFORM_FIELDS`]); the rest are the matrix, the as-shot
/// balance and framing's own block. /// balance and framing's own block.
const BASE_UNIFORM_FIELDS: usize = 16 + BASE_CURVE_UNIFORM_FIELDS; const BASE_UNIFORM_FIELDS: usize = 16 + SAMPLE_CACHE_UNIFORM_FIELDS + BASE_CURVE_UNIFORM_FIELDS;
/// Slots the sample cache's two flags occupy: read, write, and two spare to
/// keep the block a whole `vec4`. See [`ComposedShader::sample_key`].
const SAMPLE_CACHE_UNIFORM_FIELDS: usize = 4;
/// Where the sample cache's flags sit in the generated uniform block: `x` says
/// read the source colour from the cache, `y` says write it there.
///
/// Exported for the reason [`BASE_CURVE_UNIFORM_OFFSET`] is — `dr-gpu` writes
/// these by index — and zero in every block the composer hands out, so a
/// caller that never heard of the cache gets the direct read it always had.
pub const SAMPLE_CACHE_UNIFORM_OFFSET: usize = 16;
/// TRACES: FR-DEV-3e /// TRACES: FR-DEV-3e
/// Slots the base curve occupies: five `(x, y)` points and an active flag. /// Slots the base curve occupies: five `(x, y)` points and an active flag.
@@ -484,7 +515,8 @@ const BASE_CURVE_UNIFORM_FIELDS: usize = 12;
/// Exported for the same reason [`RESERVED_UNIFORM_FIELDS`] is: `dr-gpu` /// Exported for the same reason [`RESERVED_UNIFORM_FIELDS`] is: `dr-gpu`
/// writes these by index, and an offset computed independently at both ends is /// writes these by index, and an offset computed independently at both ends is
/// an offset that will eventually disagree with itself. /// an offset that will eventually disagree with itself.
pub const BASE_CURVE_UNIFORM_OFFSET: usize = 16; pub const BASE_CURVE_UNIFORM_OFFSET: usize =
SAMPLE_CACHE_UNIFORM_OFFSET + SAMPLE_CACHE_UNIFORM_FIELDS;
/// How many control points a base curve carries. /// How many control points a base curve carries.
/// ///
@@ -756,6 +788,10 @@ fn compose_inner(
\x20 // gamma-encoded JPEG, 0.0 for demosaiced sensor data), which the\n\ \x20 // gamma-encoded JPEG, 0.0 for demosaiced sensor data), which the\n\
\x20 // prologue reads to decide whether to linearise.\n\ \x20 // prologue reads to decide whether to linearise.\n\
\x20 as_shot_wb: vec4<f32>,\n\ \x20 as_shot_wb: vec4<f32>,\n\
\x20 // The sample cache (see `ComposedShader::sample_key`): `.x` reads\n\
\x20 // the source colour from `sampled`, `.y` writes it to\n\
\x20 // `sample_out`. Zero for both is the direct read.\n\
\x20 sample_cache: vec4<f32>,\n\
\x20 // The camera profile's base curve (FR-DEV-3e): five points on a\n\ \x20 // The camera profile's base curve (FR-DEV-3e): five points on a\n\
\x20 // monotone spline, packed as x0..x3, y0..y3, then (x4, y4, on).\n\ \x20 // monotone spline, packed as x0..x3, y0..y3, then (x4, y4, on).\n\
\x20 // `.z` of the last is the flag, not padding — it is 0 for a\n\ \x20 // `.z` of the last is the flag, not padding — it is 0 for a\n\
@@ -936,6 +972,19 @@ fn compose_inner(
); );
let sampler_helper = if interpolate { BILINEAR_HELPER } else { "" }; let sampler_helper = if interpolate { BILINEAR_HELPER } else { "" };
// Everything that decides which texel an output pixel reads: the code that
// computes `coord`, and the uniforms that code reads. Only on the path that
// reads a texel whole — see `ComposedShader::sample_key`.
let sample_key = (!interpolate && !warp.splits_channels).then(|| {
framing
.uniforms()
.iter()
.chain(&warp.uniforms)
.fold(hash_source(&prologue), |h, v| {
mix(h, u64::from(v.to_bits()))
})
});
// The tail, and it is the whole of the difference between the two output // The tail, and it is the whole of the difference between the two output
// modes. Everything above — the prologue, the fragments, the mask layers, // modes. Everything above — the prologue, the fragments, the mask layers,
// the camera matrix — is emitted identically either way, so an operation // the camera matrix — is emitted identically either way, so an operation
@@ -1087,6 +1136,12 @@ struct Params {{
// stock is loaded, which costs eight bytes and no branch. // stock is loaded, which costs eight bytes and no branch.
@group(0) @binding(4) var film_curves: texture_2d<f32>; @group(0) @binding(4) var film_curves: texture_2d<f32>;
@group(0) @binding(5) var film_lut_texture: texture_3d<f32>; @group(0) @binding(5) var film_lut_texture: texture_3d<f32>;
// The sample cache: the source texel each output pixel read on an earlier
// frame with this framing, and where this frame writes it when asked. See
// `ComposedShader::sample_key`. Declared unconditionally, like the masks, and
// bound to 1x1 placeholders whenever the flags say not to touch them.
@group(0) @binding(6) var sampled: texture_2d<f32>;
@group(0) @binding(7) var sample_out: texture_storage_2d<rgba16float, write>;
{sampler_helper}{helper_src}{encode_output} {sampler_helper}{helper_src}{encode_output}
// Display-encoded sRGB back to linear, for sources that arrive that way. // Display-encoded sRGB back to linear, for sources that arrive that way.
@@ -1196,6 +1251,7 @@ fn main(@builtin(global_invocation_id) gid: vec3<u32>) {{
uniforms: uniform_values, uniforms: uniform_values,
structure_hash, structure_hash,
output_mode, output_mode,
sample_key,
} }
} }
@@ -1415,7 +1471,19 @@ pub(crate) fn sample_source(interpolate: bool, splits_channels: bool) -> &'stati
// amount that changes with the aspect ratio. It reads as a correction that // amount that changes with the aspect ratio. It reads as a correction that
// is simply too weak, which is indistinguishable from a bad profile. // is simply too weak, which is indistinguishable from a bad profile.
let radius = length(p) / (0.5 * length(aspect)); let radius = length(p) / (0.5 * length(aspect));
var c = textureLoad(source, coord, 0).rgb; // The texel itself, from the source or from the cache of it an earlier
// frame wrote (see `ComposedShader::sample_key`). Both branches yield the
// same bits: the source is `rgba16float` and so is the cache. The flags
// are uniforms, so the whole dispatch takes one branch.
var c: vec3<f32>;
if (u.sample_cache.x > 0.5) {
c = textureLoad(sampled, vec2<i32>(gid.xy), 0).rgb;
} else {
c = textureLoad(source, coord, 0).rgb;
if (u.sample_cache.y > 0.5) {
textureStore(sample_out, vec2<i32>(gid.xy), vec4<f32>(c, 1.0));
}
}
" "
} }
} }
core/dr-pipeline/src/ops/dehaze.rs
+118 -144
View File
@@ -116,12 +116,34 @@
//! mottling across smooth gradients. This decomposition samples nothing — it //! mottling across smooth gradients. This decomposition samples nothing — it
//! evaluates the exact minimum over every pixel of the window, in two steps. //! evaluates the exact minimum over every pixel of the window, in two steps.
//! //!
//! # Why it is now two passes and not five
//!
//! The decomposition was run as four erosion passes — run and span along x,
//! then along y — and a fifth for the recovery. On the reference laptop the
//! taps turned out not to be what a pass costs: with the memory clock held at
//! 810 MHz by the power cap, a pass that only reads the render-sized
//! `rgba16float` intermediate and writes the other one costs about 4 ms at
//! 2560 x 1600, and dehaze's five came to 22 ms, of which the taps were about
//! 2. So each axis is now one pass that takes the minimum over the whole
//! window directly — 36 texture reads per pixel at that size instead of 12,
//! nearly all of them served by the cache — and the recovery rides in the y
//! pass, which already has the veil and this pixel's colour in hand. Two
//! passes: 9.2 ms.
//!
//! It is the same picture, bit for bit. A minimum is exact in any order, so
//! the minimum over the window's pixels is one value however it is grouped,
//! and the window is the one [`Split`] always covered, surplus pixel
//! included. The veil reaching the recovery was always exactly representable
//! in the `rgba16float` lane it crossed — a minimum of channel values that
//! were themselves read from `rgba16float` — so no rounding was lost by not
//! storing it between passes.
//!
//! It is also why this operation does not use the reduced chain that TD-4 gave //! It is also why this operation does not use the reduced chain that TD-4 gave
//! clarity. The runner holds one reduced buffer, so every scaled pass in a //! clarity. The runner holds one reduced buffer, so every scaled pass in a
//! chain must declare the same `output_scale`; clarity's steps down with the //! chain must declare the same `output_scale`; clarity's steps down with the
//! viewport, so a second operation choosing its own would disagree with it at //! viewport, so a second operation choosing its own would disagree with it at
//! some window sizes and not others. Four cheap full-resolution passes cost //! some window sizes and not others. Two full-resolution passes cost less than
//! less than that coupling, and the decomposition is what makes them cheap. //! that coupling.
//! //!
//! # The artefact this does not fix //! # The artefact this does not fix
//! //!
@@ -275,22 +297,23 @@ impl Dehaze {
/// ///
/// See the module documentation: eroding by a contiguous run and then by a set /// See the module documentation: eroding by a contiguous run and then by a set
/// of points spaced one run apart erodes by the sum of the two, which is the /// of points spaced one run apart erodes by the sum of the two, which is the
/// whole window. This is the arithmetic of that split, in one place, because /// whole window. The passes no longer run the two stages apart (see "Why it is
/// both axes need it and a second copy is a second chance to get the centring /// now two passes"), but the window they read is still the one this split
/// wrong. /// covers — [`Self::first`] and [`Self::width`] — surplus pixel included,
/// because that is the window every edit made so far was tuned against.
#[derive(Debug, Clone, Copy, PartialEq)] #[derive(Debug, Clone, Copy, PartialEq)]
pub struct Split { pub struct Split {
/// Length of the contiguous run the first pass takes the minimum over. /// Length of the contiguous run the first pass takes the minimum over.
pub run: u32, pub run: u32,
/// How many runs the second pass chains together, spaced `run` apart. /// How many runs the second pass chains together, spaced `run` apart.
pub span: u32, pub span: u32,
/// What the second pass subtracts from its offsets to centre the window. /// How far before the pixel being written the window starts.
/// ///
/// The composite covers `run * span` pixels, which is at least the window /// The composite covers `run * span` pixels, which is at least the window
/// asked for and can be one or two more; the surplus falls on the far side /// asked for and can be one or two more; the surplus falls on the far side
/// rather than being trimmed, because trimming it would need a third pass /// rather than being trimmed, because trimming it would have needed a
/// and a patch a pixel wider on one side is not a visible difference in a /// third pass and a patch a pixel wider on one side is not a visible
/// field this smooth. /// difference in a field this smooth.
pub shift: i32, pub shift: i32,
} }
@@ -311,14 +334,25 @@ impl Split {
} }
} }
/// The furthest the second pass reads, in pixels. /// The window's first offset from the pixel being written: `-shift`.
pub fn first(&self) -> i32 {
-self.shift
}
/// How many pixels the window covers, `run * span` — the patch asked for
/// and the one or two surplus pixels on the far side the split leaves.
pub fn width(&self) -> u32 {
self.run * self.span
}
/// The furthest the window reads from the pixel being written, in pixels.
/// ///
/// Stated rather than assumed symmetric: the composite window is centred /// Stated rather than assumed symmetric: the window is centred to within a
/// to within a pixel and not exactly, so the two directions can differ by /// pixel and not exactly, so the two directions can differ by one. An
/// one. An understated radius is a seam at every tile boundary (ARCH /// understated radius is a seam at every tile boundary (ARCH §5.3), which
/// §5.3), which is the kind of artefact that looks like a driver bug. /// is the kind of artefact that looks like a driver bug.
pub fn reach(&self) -> u32 { pub fn extent(&self) -> u32 {
let far = (self.span.saturating_sub(1) * self.run) as i32 - self.shift; let far = self.width() as i32 - 1 - self.shift;
self.shift.max(far).max(0) as u32 self.shift.max(far).max(0) as u32
} }
} }
@@ -367,83 +401,50 @@ impl DetailStage for Dehaze {
fn passes(&self, scale: RenderScale) -> Vec<DetailPass> { fn passes(&self, scale: RenderScale) -> Vec<DetailPass> {
let split = Split::of(self.patch(scale)); let split = Split::of(self.patch(scale));
// The run stage's uniforms are the same on both axes, and so are the // Both axes erode over the same window; only the offset expression
// span stage's. Only the offset expression differs, which is what // differs, which is what `erode` takes as an argument — one filter
// `erode_run` and `erode_span` take as an argument — each filter
// written once, so the two axes cannot drift into being different // written once, so the two axes cannot drift into being different
// filters. // filters.
let run = vec![Uniform { let window = vec![
name: "run",
value: split.run as f32,
}];
let span = vec![
Uniform { Uniform {
name: "span", name: "first",
value: split.span as f32, value: split.first() as f32,
}, },
Uniform { Uniform {
name: "stride", name: "width",
value: split.run as f32, value: split.width() as f32,
},
Uniform {
name: "shift",
value: split.shift as f32,
}, },
]; ];
let mut recovery = window.clone();
recovery.extend([
Uniform {
name: "omega",
value: self.omega(),
},
Uniform {
name: "min_transmission",
value: MIN_TRANSMISSION,
},
]);
vec![ vec![
DetailPass { DetailPass {
output_scale: 1, output_scale: 1,
label: "veil-run-x", label: "veil-x",
// The run starts at this pixel and walks forward, so it reads radius: split.extent(),
// `run - 1` beyond itself and nothing behind.
radius: split.run.saturating_sub(1),
storage: Vec::new(), storage: Vec::new(),
uniforms: run.clone(), uniforms: window,
wgsl: erode_run(Axis::X), wgsl: erode(Axis::X),
}, },
DetailPass { DetailPass {
output_scale: 1, output_scale: 1,
label: "veil-span-x", label: "veil-y-clear",
radius: split.reach(), radius: split.extent(),
storage: Vec::new(), storage: Vec::new(),
uniforms: span.clone(), uniforms: recovery,
wgsl: erode_span(Axis::X), // The erosion in a block of its own, so its locals do not
}, // collide with the recovery's.
DetailPass { wgsl: format!("{{\n{}\n}}\n\n{CLEAR}", erode(Axis::Y)),
output_scale: 1,
label: "veil-run-y",
radius: split.run.saturating_sub(1),
storage: Vec::new(),
uniforms: run,
wgsl: erode_run(Axis::Y),
},
DetailPass {
output_scale: 1,
label: "veil-span-y",
radius: split.reach(),
storage: Vec::new(),
uniforms: span,
wgsl: erode_span(Axis::Y),
},
DetailPass {
output_scale: 1,
label: "clear",
// Reads only the pixel it writes: the veil arrived in the
// scratch lane four passes ago.
radius: 0,
storage: Vec::new(),
uniforms: vec![
Uniform {
name: "omega",
value: self.omega(),
},
Uniform {
name: "min_transmission",
value: MIN_TRANSMISSION,
},
],
wgsl: CLEAR.to_string(),
}, },
] ]
} }
@@ -466,33 +467,33 @@ impl Axis {
} }
} }
/// The first stage: the minimum over a contiguous run. /// The minimum over the window along one axis.
/// ///
/// Along x it reads the colour and reduces it to the dark channel; along y the /// Along x it reads the colour and reduces it to the dark channel; along y the
/// dark channel is already in the scratch lane, so it reads that instead. /// dark channel's x minimum is already in the scratch lane, so it reads that
/// Doing the channel minimum again on the second axis would be reducing a /// instead. Doing the channel minimum again on the second axis would be
/// scalar and would quietly discard the x erosion. /// reducing a scalar and would quietly discard the x erosion.
/// ///
/// Neither stage touches `c`. The recovery needs the original colour *and* the /// The x pass does not touch `c`. The recovery needs the original colour *and*
/// veil in the same place at the same time, and the ping-pong hands each pass /// the veil in the same place at the same time, and the ping-pong hands each
/// only what the pass before it wrote — so the veil travels in `aux` and the /// pass only what the pass before it wrote — so the veil travels in `aux` and
/// colour rides through untouched. See [`DetailPass::wgsl`]. /// the colour rides through untouched. See [`DetailPass::wgsl`].
fn erode_run(axis: Axis) -> String { fn erode(axis: Axis) -> String {
let source = match axis { let source = match axis {
Axis::X => "dark_channel(tap(coord, OFFSET))", Axis::X => "dark_channel(tap(coord, OFFSET))",
Axis::Y => "tap_aux(coord, OFFSET)", Axis::Y => "tap_aux(coord, OFFSET)",
}; };
let first = source.replace("OFFSET", &axis.offset("0")); let head = source.replace("OFFSET", &axis.offset("o"));
let rest = source.replace("OFFSET", &axis.offset("i")); let rest = source.replace("OFFSET", &axis.offset("o + i"));
format!( format!(
"\ "\
// Half of the erosion's first stage: the minimum over `run` contiguous pixels, // The minimum over every pixel of the window along this axis, from `first`
// walking forward from this one. The second stage chains these together, and // for `width` pixels. A minimum is exact in any order, so this is the same
// the two structuring elements add up to the whole patch — which is why this // value, bit for bit, as chaining a run and a span over the same pixels.
// one is not centred and does not need to be. let o = i32(first);
let n = i32(run); let n = i32(width);
var veil = {first}; var veil = {head};
for (var i = 1; i < n; i = i + 1) {{ for (var i = 1; i < n; i = i + 1) {{
veil = min(veil, {rest}); veil = min(veil, {rest});
}} }}
@@ -500,38 +501,10 @@ aux = veil;"
) )
} }
/// The second stage: the minimum over `span` points spaced `stride` apart.
///
/// Each of those points already holds the minimum over the run that starts
/// there, so this reads the whole window while touching `span` pixels of it.
/// `shift` is what centres the composite on the pixel being written; without
/// it the veil would be measured from a patch lying entirely to one side, and
/// the correction would appear to lag the picture by half a patch.
fn erode_span(axis: Axis) -> String {
let offset = axis.offset("j * s - o");
let first = axis.offset("-o");
format!(
"\
// The erosion's second stage. `span` taps, spaced a whole run apart, each
// standing for the run that begins at it — so the minimum over the patch costs
// `run + span` taps rather than the `run * span` pixels it covers, and it is
// the exact minimum over all of them rather than a sample of them.
let n = i32(span);
let s = i32(stride);
let o = i32(shift);
var veil = tap_aux(coord, {first});
for (var j = 1; j < n; j = j + 1) {{
veil = min(veil, tap_aux(coord, {offset}));
}}
aux = veil;"
)
}
/// The recovery: invert the scattering model with the transmission the erosion /// The recovery: invert the scattering model with the transmission the erosion
/// implies. /// implies.
const CLEAR: &str = "\ const CLEAR: &str = "\
// The veil the four erosion passes measured: the smallest channel anywhere in // The veil the two erosions measured: the smallest channel anywhere in
// the patch around this pixel, which the dark-channel prior reads as the // the patch around this pixel, which the dark-channel prior reads as the
// airlight that has been composited over the scene here. // airlight that has been composited over the scene here.
// //
@@ -581,7 +554,7 @@ mod tests {
#[test] #[test]
fn dehaze_starts_neutral_and_costs_nothing() { fn dehaze_starts_neutral_and_costs_nothing() {
// The rule the whole pipeline rests on. An unedited photograph must not // The rule the whole pipeline rests on. An unedited photograph must not
// pay for a slider nobody has touched — and this one is five dispatches // pay for a slider nobody has touched — and this one is two dispatches
// when it is on, so "nothing" here is a worthwhile amount of nothing. // when it is on, so "nothing" here is a worthwhile amount of nothing.
assert!(!Dehaze::new().is_active()); assert!(!Dehaze::new().is_active());
assert!(composed(0.0, RenderScale::full((2000, 1500))).is_empty()); assert!(composed(0.0, RenderScale::full((2000, 1500))).is_empty());
@@ -623,7 +596,7 @@ mod tests {
// develop view about it would read as a bug. // develop view about it would read as a bug.
let tiny = RenderScale::full((48, 32)); let tiny = RenderScale::full((48, 32));
assert_eq!(Dehaze::with_amount(50.0).patch(tiny), 1); assert_eq!(Dehaze::with_amount(50.0).patch(tiny), 1);
assert_eq!(composed(50.0, tiny).len(), 5); assert_eq!(composed(50.0, tiny).len(), 2);
} }
#[test] #[test]
@@ -648,7 +621,8 @@ mod tests {
// Centred to within the pixel the odd surplus leaves over: the window // Centred to within the pixel the odd surplus leaves over: the window
// covers [-31, 32] around the pixel being written. // covers [-31, 32] around the pixel being written.
assert_eq!(split.shift, 31); assert_eq!(split.shift, 31);
assert_eq!(split.reach(), 31); assert_eq!((split.first(), split.width()), (-31, 64));
assert_eq!(split.extent(), 32);
} }
#[test] #[test]
@@ -662,33 +636,32 @@ mod tests {
assert_eq!(split.span, 2); assert_eq!(split.span, 2);
assert!(split.run * split.span >= 3, "the window is not covered"); assert!(split.run * split.span >= 3, "the window is not covered");
assert_eq!(split.shift, 1); assert_eq!(split.shift, 1);
assert_eq!(split.reach(), 1); // [-1, 2]: the surplus pixel is on the far side.
assert_eq!((split.first(), split.width()), (-1, 4));
assert_eq!(split.extent(), 2);
} }
#[test] #[test]
fn the_chain_is_four_erosions_and_a_recovery() { fn the_chain_is_an_erosion_per_axis_the_second_carrying_the_recovery() {
// The shape of the operation, asserted where it is cheap to assert. // The shape of the operation, asserted where it is cheap to assert.
// The erosions leave the colour alone and hand the veil forward in the // The x erosion leaves the colour alone and hands the veil forward in
// scratch lane; only the last pass touches `c`, which is what makes an // the scratch lane; only the y pass touches `c`, which is what makes an
// unsharp-mask-shaped operation expressible in a chain that hands each // unsharp-mask-shaped operation expressible in a chain that hands each
// pass exactly one texture. // pass exactly one texture. Two passes and not five: each extra pass
// is a render-sized read and write, which is what a pass costs (see
// "Why it is now two passes").
let composed = composed(60.0, RenderScale::full((2000, 1500))); let composed = composed(60.0, RenderScale::full((2000, 1500)));
let labels: Vec<&str> = composed.passes.iter().map(|p| p.label.as_str()).collect(); let labels: Vec<&str> = composed.passes.iter().map(|p| p.label.as_str()).collect();
assert_eq!( assert_eq!(labels, ["dehaze/veil-x", "dehaze/veil-y-clear"]);
labels,
[ // Both declare the whole window they read.
"dehaze/veil-run-x", let split = Split::of(Dehaze::with_amount(60.0).patch(RenderScale::full((2000, 1500))));
"dehaze/veil-span-x", assert!(composed.passes.iter().all(|p| p.radius == split.extent()));
"dehaze/veil-run-y",
"dehaze/veil-span-y",
"dehaze/clear",
]
);
// Only the last writes the display texture, so the output transform // Only the last writes the display texture, so the output transform
// happens exactly once (FR-DEV-2). // happens exactly once (FR-DEV-2).
assert!(composed.passes[..4].iter().all(|p| !p.writes_output)); assert!(!composed.passes[0].writes_output);
assert!(composed.passes[4].writes_output); assert!(composed.passes[1].writes_output);
// Nothing here uses the reduced chain — see the module documentation // Nothing here uses the reduced chain — see the module documentation
// for why a second operation cannot pick its own `output_scale` while // for why a second operation cannot pick its own `output_scale` while
@@ -730,7 +703,8 @@ mod tests {
// nothing to say so. // nothing to say so.
let composed = composed(60.0, RenderScale::full((2000, 1500))); let composed = composed(60.0, RenderScale::full((2000, 1500)));
assert!(composed.passes[0].source.contains("dark_channel(tap(coord")); assert!(composed.passes[0].source.contains("dark_channel(tap(coord"));
assert!(!composed.passes[2].source.contains("dark_channel(tap(coord")); assert!(!composed.passes[1].source.contains("dark_channel(tap(coord"));
assert!(composed.passes[1].source.contains("tap_aux(coord"));
// The helper is still emitted for every pass of the operation, and it // The helper is still emitted for every pass of the operation, and it
// must define the function it is named for or the shader fails to // must define the function it is named for or the shader fails to
// compile a long way from here. // compile a long way from here.
core/dr-pipeline/src/preset.rs
+437 -23
View File
@@ -44,6 +44,7 @@ use std::fmt::Write as _;
use crate::descriptor::{Attribute, OpId, ParamId}; use crate::descriptor::{Attribute, OpId, ParamId};
use crate::graph::EditGraph; use crate::graph::EditGraph;
use crate::state::{FilmRebake, FilmRef};
/// Which parts of an edit a copy carries. /// Which parts of an edit a copy carries.
/// ///
@@ -173,6 +174,15 @@ impl Scope {
self.bits == 0 self.bits == 0
} }
/// Whether this scope carries the film — the stock as well as its sliders.
///
/// Asked of the film node's own classification rather than decided here,
/// so the stock travels under exactly the scopes its exposure slider
/// does. Splitting the two would put one stock's exposure on another.
pub fn carries_film(self) -> bool {
self.covers(crate::ops::film_sim::ID.0)
}
fn bit(attribute: Attribute) -> u8 { fn bit(attribute: Attribute) -> u8 {
1 << Attribute::ALL 1 << Attribute::ALL
.iter() .iter()
@@ -216,6 +226,17 @@ fn attributes_of(op: &str) -> Option<&'static [Attribute]> {
TABLE.get(op).map(Vec::as_slice) TABLE.get(op).map(Vec::as_slice)
} }
/// How much of the target an applied preset replaces. See the module note.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum Reach {
/// Everything in scope: absence means default. A copy, a saved edit.
#[default]
Whole,
/// Only the operations the preset names, and the film if it names one.
/// Everything else in the target is left as it was. A look.
Named,
}
/// A set of non-default parameter values, ready to apply elsewhere. /// A set of non-default parameter values, ready to apply elsewhere.
/// ///
/// Ordered, so two captures of the same edit compare equal and a caller can /// Ordered, so two captures of the same edit compare equal and a caller can
@@ -223,6 +244,10 @@ fn attributes_of(op: &str) -> Option<&'static [Attribute]> {
#[derive(Debug, Clone, PartialEq, Default)] #[derive(Debug, Clone, PartialEq, Default)]
pub struct Preset { pub struct Preset {
params: BTreeMap<(String, String), f32>, params: BTreeMap<(String, String), f32>,
reach: Reach,
/// TRACES: FR-DEV-3f
/// The stock this edit develops on, by id. See the module note.
film: Option<FilmRef>,
} }
impl Preset { impl Preset {
@@ -237,6 +262,21 @@ impl Preset {
/// have to be re-copied to change one's mind about framing, and a preset /// have to be re-copied to change one's mind about framing, and a preset
/// that had already discarded the crop could never grow it back. /// that had already discarded the crop could never grow it back.
pub fn capture(graph: &EditGraph) -> Self { pub fn capture(graph: &EditGraph) -> Self {
Self {
film: graph.film().map(|f| FilmRef {
stock: f.stock.clone(),
print: f.print.clone(),
}),
..Self::capture_params(graph)
}
}
/// The parameters alone, without the film.
///
/// For [`EditGraph::state`], which carries the film in a field of its own
/// ([`crate::EditState::film`]). Capturing it twice would give one edit
/// two places to disagree about its stock.
pub(crate) fn capture_params(graph: &EditGraph) -> Self {
let mut params = BTreeMap::new(); let mut params = BTreeMap::new();
for cap in graph.capabilities() { for cap in graph.capabilities() {
for p in &cap.params { for p in &cap.params {
@@ -245,12 +285,59 @@ impl Preset {
} }
} }
} }
Self { params } Self::from_params(params)
} }
/// Build from an already-captured parameter map — a sidecar's, typically. /// Build from an already-captured parameter map — a sidecar's, typically.
pub fn from_params(params: BTreeMap<(String, String), f32>) -> Self { pub fn from_params(params: BTreeMap<(String, String), f32>) -> Self {
Self { params } Self {
params,
reach: Reach::Whole,
film: None,
}
}
/// The same preset, reaching as far as `reach` says.
pub fn with_reach(self, reach: Reach) -> Self {
Self { reach, ..self }
}
/// How much of the target this preset replaces.
pub fn reach(&self) -> Reach {
self.reach
}
/// Whether applying this preset replaces `op` — before any scope is asked.
fn reaches(&self, op: &str) -> bool {
match self.reach {
Reach::Whole => true,
Reach::Named => {
self.params.keys().any(|(o, _)| o == op)
|| (self.film.is_some() && op == crate::ops::film_sim::ID.0)
}
}
}
/// The same preset, developed on `film`.
pub fn with_film(self, film: Option<FilmRef>) -> Self {
Self { film, ..self }
}
/// The stock this preset develops on, if it names one.
pub fn film(&self) -> Option<&FilmRef> {
self.film.as_ref()
}
/// What applying at `scope` does to the target's film.
///
/// `None` when the scope leaves the film alone; `Some(None)` when it
/// develops the target without one. The same two levels the sidecar
/// writer takes, which is who asks: the batch path amends files rather
/// than graphs, and has to know whether the film line is being written at
/// all before it knows what to write.
pub fn film_for(&self, scope: Scope) -> Option<Option<&FilmRef>> {
(scope.carries_film() && self.reaches(crate::ops::film_sim::ID.0))
.then_some(self.film.as_ref())
} }
/// The parameters, for a caller that stores them. /// The parameters, for a caller that stores them.
@@ -270,7 +357,7 @@ impl Preset {
/// answers is whether there is a clipboard to offer, which is why the UI /// answers is whether there is a clipboard to offer, which is why the UI
/// asks it before enabling a paste. /// asks it before enabling a paste.
pub fn is_empty(&self) -> bool { pub fn is_empty(&self) -> bool {
self.params.is_empty() self.params.is_empty() && self.film.is_none()
} }
/// How many parameters were captured. /// How many parameters were captured.
@@ -296,10 +383,14 @@ impl Preset {
/// rather than parameters because thirty-six mixer sliders is a number /// rather than parameters because thirty-six mixer sliders is a number
/// about the mixer's shape, not about how much was copied. /// about the mixer's shape, not about how much was copied.
pub fn op_count(&self, scope: Scope) -> usize { pub fn op_count(&self, scope: Scope) -> usize {
// A stock with every film slider at default is still the film node
// at work, and a preset holding nothing else is not "Neutral".
let film = self.film.is_some().then_some(crate::ops::film_sim::ID.0);
let mut ops: Vec<&str> = self let mut ops: Vec<&str> = self
.params .params
.keys() .keys()
.map(|(op, _)| op.as_str()) .map(|(op, _)| op.as_str())
.chain(film)
.filter(|op| scope.covers(op)) .filter(|op| scope.covers(op))
.collect(); .collect();
ops.sort_unstable(); ops.sort_unstable();
@@ -319,7 +410,13 @@ impl Preset {
/// to a fitted view mid-comparison, which reads as the paste having /// to a fitted view mid-comparison, which reads as the paste having
/// navigated somewhere. This mirrors `DevelopSession::reset_framing`, and /// navigated somewhere. This mirrors `DevelopSession::reset_framing`, and
/// lives here so every caller inherits it rather than each remembering. /// lives here so every caller inherits it rather than each remembering.
pub fn apply(&self, graph: &mut EditGraph, scope: Scope) { ///
/// The **film** is replaced when the scope carries it, and handed back as
/// a [`FilmRebake`] because this crate cannot bake a stock — the same debt
/// [`EditGraph::set_state`] returns, for the same reason. It is cleared
/// even when the preset names the stock already on the graph: the tables
/// were baked from the film sliders this call has just replaced.
pub fn apply(&self, graph: &mut EditGraph, scope: Scope) -> FilmRebake {
let view = graph.framing().view(); let view = graph.framing().view();
// Clear the scope first, so absence means default (see the module // Clear the scope first, so absence means default (see the module
@@ -328,7 +425,7 @@ impl Preset {
let clears: Vec<(OpId, ParamId, f32)> = graph let clears: Vec<(OpId, ParamId, f32)> = graph
.capabilities() .capabilities()
.iter() .iter()
.filter(|cap| scope.covers(cap.id.0)) .filter(|cap| scope.covers(cap.id.0) && self.reaches(cap.id.0))
.flat_map(|cap| cap.params.iter().map(|p| (cap.id, p.id, p.default))) .flat_map(|cap| cap.params.iter().map(|p| (cap.id, p.id, p.default)))
.collect(); .collect();
for (op, param, default) in clears { for (op, param, default) in clears {
@@ -347,6 +444,17 @@ impl Preset {
} }
graph.framing_mut().set_view(view); graph.framing_mut().set_view(view);
match self.film_for(scope) {
None => FilmRebake::NotNeeded,
Some(film) => {
graph.set_film(None);
match film {
None => FilmRebake::NotNeeded,
Some(film) => FilmRebake::Wanted(film.clone()),
}
}
}
} }
/// Apply to a parameter map — the sidecar of an image that is not open. /// Apply to a parameter map — the sidecar of an image that is not open.
@@ -361,8 +469,11 @@ impl Preset {
/// Same replacement rule as [`Self::apply`]: the target's in-scope keys go, /// Same replacement rule as [`Self::apply`]: the target's in-scope keys go,
/// the preset's arrive, and out-of-scope keys — the target's own crop, on /// the preset's arrive, and out-of-scope keys — the target's own crop, on
/// the default scope — are left exactly as they were. /// the default scope — are left exactly as they were.
///
/// A parameter map has no film in it, so the stock is not written here:
/// the caller asks [`Self::film_for`] and writes it beside the map.
pub fn amend(&self, target: &mut BTreeMap<(String, String), f32>, scope: Scope) { pub fn amend(&self, target: &mut BTreeMap<(String, String), f32>, scope: Scope) {
target.retain(|(op, _), _| !scope.covers(op)); target.retain(|(op, _), _| !(scope.covers(op) && self.reaches(op)));
for ((op, param), value) in &self.params { for ((op, param), value) in &self.params {
if scope.covers(op) { if scope.covers(op) {
target.insert((op.clone(), param.clone()), *value); target.insert((op.clone(), param.clone()), *value);
@@ -556,6 +667,14 @@ impl PresetLibrary {
self.presets.is_empty() self.presets.is_empty()
} }
/// How many lines were kept without being understood.
///
/// For a file this build wrote itself, or shipped, that should be none: a
/// misspelt key is preserved faithfully and does nothing.
pub fn unread_lines(&self) -> usize {
self.unknown.values().map(Vec::len).sum()
}
/// Serialise to the on-disk form. /// Serialise to the on-disk form.
/// ///
/// Deterministic, like the sidecar's: the same library always produces the /// Deterministic, like the sidecar's: the same library always produces the
@@ -567,6 +686,22 @@ impl PresetLibrary {
for ((op, param), value) in preset.params() { for ((op, param), value) in preset.params() {
let _ = writeln!(out, "{op}.{param} = {}", format_value(*value)); let _ = writeln!(out, "{op}.{param} = {}", format_value(*value));
} }
// TRACES: FR-DEV-3f
// Spelled as the sidecar spells them, so a block can still be
// pasted from one file into the other. A build that predates
// these lines reads them as lines it does not understand and
// writes them back untouched, which is the promise below.
// Only when it differs from the default, so every library written
// before looks existed still writes the same bytes.
if preset.reach() == Reach::Named {
let _ = writeln!(out, "reach = named");
}
if let Some(film) = preset.film() {
let _ = writeln!(out, "film = {}", film.stock);
if let Some(print) = &film.print {
let _ = writeln!(out, "film_print = {print}");
}
}
for line in self.unknown.get(name).into_iter().flatten() { for line in self.unknown.get(name).into_iter().flatten() {
let _ = writeln!(out, "{line}"); let _ = writeln!(out, "{line}");
} }
@@ -597,6 +732,8 @@ impl PresetLibrary {
let mut library = Self::default(); let mut library = Self::default();
let mut current: Option<String> = None; let mut current: Option<String> = None;
let mut params: BTreeMap<(String, String), f32> = BTreeMap::new(); let mut params: BTreeMap<(String, String), f32> = BTreeMap::new();
let mut film: Option<FilmRef> = None;
let mut reach = Reach::Whole;
for line in lines { for line in lines {
let line = line.trim(); let line = line.trim();
@@ -609,9 +746,16 @@ impl PresetLibrary {
.and_then(|l| l.strip_suffix(']')) .and_then(|l| l.strip_suffix(']'))
{ {
if let Some(name) = current.take() { if let Some(name) = current.take() {
library.presets.insert(name, Preset::from_params(params)); library.presets.insert(
params = BTreeMap::new(); name,
Preset::from_params(std::mem::take(&mut params))
.with_film(film.take())
.with_reach(reach),
);
} }
params.clear();
film = None;
reach = Reach::Whole;
// A name the writer should never have produced is dropped // A name the writer should never have produced is dropped
// rather than taken: accepting it would mean writing a file // rather than taken: accepting it would mean writing a file
// back out that no longer parses as this one. // back out that no longer parses as this one.
@@ -631,6 +775,23 @@ impl PresetLibrary {
}; };
match line.split_once('=') { match line.split_once('=') {
// TRACES: FR-DEV-3f
// Not checked against the installed stocks: this crate does
// not link them, and a preset naming a stock this device
// lacks must survive being stored here. Whoever bakes it
// reports the miss — the sidecar's rule, for its reason.
// `get_or_insert_with` because a hand-edited block may name
// the paper first.
Some((key, value)) if key.trim() == "reach" && value.trim() == "named" => {
reach = Reach::Named;
}
Some((key, value)) if key.trim() == "film" && !value.trim().is_empty() => {
film.get_or_insert_with(FilmRef::default).stock = value.trim().to_string();
}
Some((key, value)) if key.trim() == "film_print" && !value.trim().is_empty() => {
film.get_or_insert_with(FilmRef::default).print =
Some(value.trim().to_string());
}
Some((key, value)) => { Some((key, value)) => {
let key = key.trim(); let key = key.trim();
let value = value.trim(); let value = value.trim();
@@ -654,7 +815,21 @@ impl PresetLibrary {
} }
if let Some(name) = current { if let Some(name) = current {
library.presets.insert(name, Preset::from_params(params)); library.presets.insert(
name,
Preset::from_params(params)
.with_film(film)
.with_reach(reach),
);
}
// A paper with no film named beside it is a print of nothing. Dropped
// rather than kept, since baking it would have no stock to start from.
for preset in library.presets.values_mut() {
if preset.film.as_ref().is_some_and(|f| f.stock.is_empty()) {
log::warn!("preset library: a paper without a film; ignoring it");
preset.film = None;
}
} }
// A block whose every line was unreadable still produced a preset, and // A block whose every line was unreadable still produced a preset, and
@@ -803,7 +978,9 @@ mod tests {
}; };
target.set_crop(target_crop); target.set_crop(target_crop);
preset.apply(&mut target, Scope::adjustments()); preset
.apply(&mut target, Scope::adjustments())
.expect_no_film();
assert_eq!( assert_eq!(
target.param(exposure::ID, exposure::EXPOSURE), target.param(exposure::ID, exposure::EXPOSURE),
@@ -835,7 +1012,9 @@ mod tests {
fn pasting_everything_carries_the_composition_too() { fn pasting_everything_carries_the_composition_too() {
let preset = Preset::capture(&edited()); let preset = Preset::capture(&edited());
let mut target = EditGraph::default_chain(); let mut target = EditGraph::default_chain();
preset.apply(&mut target, Scope::everything()); preset
.apply(&mut target, Scope::everything())
.expect_no_film();
assert_eq!(target.param(framing::ID, framing::ANGLE), Some(-2.0)); assert_eq!(target.param(framing::ID, framing::ANGLE), Some(-2.0));
assert_eq!(target.param(framing::ID, framing::KEYSTONE_V), Some(40.0)); assert_eq!(target.param(framing::ID, framing::KEYSTONE_V), Some(40.0));
@@ -857,7 +1036,9 @@ mod tests {
target.set_param(exposure::ID, exposure::EXPOSURE, 2.0); target.set_param(exposure::ID, exposure::EXPOSURE, 2.0);
target.set_param(saturation::ID, saturation::SATURATION, -50.0); target.set_param(saturation::ID, saturation::SATURATION, -50.0);
neutral.apply(&mut target, Scope::adjustments()); neutral
.apply(&mut target, Scope::adjustments())
.expect_no_film();
assert_eq!(target.param(exposure::ID, exposure::EXPOSURE), Some(0.0)); assert_eq!(target.param(exposure::ID, exposure::EXPOSURE), Some(0.0));
assert_eq!( assert_eq!(
@@ -876,7 +1057,9 @@ mod tests {
let mut target = EditGraph::default_chain(); let mut target = EditGraph::default_chain();
target.set_param(framing::ID, framing::ANGLE, 3.5); target.set_param(framing::ID, framing::ANGLE, 3.5);
neutral.apply(&mut target, Scope::adjustments()); neutral
.apply(&mut target, Scope::adjustments())
.expect_no_film();
assert_eq!(target.param(framing::ID, framing::ANGLE), Some(3.5)); assert_eq!(target.param(framing::ID, framing::ANGLE), Some(3.5));
} }
@@ -894,7 +1077,9 @@ mod tests {
height: 0.25, height: 0.25,
}); });
preset.apply(&mut target, Scope::everything()); preset
.apply(&mut target, Scope::everything())
.expect_no_film();
assert!( assert!(
target.framing().is_zoomed(), target.framing().is_zoomed(),
"the paste threw away the viewport: {:?}", "the paste threw away the viewport: {:?}",
@@ -911,7 +1096,9 @@ mod tests {
let mut sideways = EditGraph::default_chain(); let mut sideways = EditGraph::default_chain();
sideways.set_orientation(dr_types::Orientation::from_exif(6)); sideways.set_orientation(dr_types::Orientation::from_exif(6));
preset.apply(&mut sideways, Scope::everything()); preset
.apply(&mut sideways, Scope::everything())
.expect_no_film();
assert_eq!( assert_eq!(
sideways.framing().baseline(), sideways.framing().baseline(),
@@ -928,7 +1115,9 @@ mod tests {
params.insert(("exposure".to_string(), "exposure".to_string()), 1.25); params.insert(("exposure".to_string(), "exposure".to_string()), 1.25);
let mut target = EditGraph::default_chain(); let mut target = EditGraph::default_chain();
Preset::from_params(params).apply(&mut target, Scope::adjustments()); Preset::from_params(params)
.apply(&mut target, Scope::adjustments())
.expect_no_film();
assert_eq!(target.param(exposure::ID, exposure::EXPOSURE), Some(1.25)); assert_eq!(target.param(exposure::ID, exposure::EXPOSURE), Some(1.25));
} }
@@ -939,7 +1128,9 @@ mod tests {
params.insert(("exposure".to_string(), "exposure".to_string()), 99.0); params.insert(("exposure".to_string(), "exposure".to_string()), 99.0);
let mut target = EditGraph::default_chain(); let mut target = EditGraph::default_chain();
Preset::from_params(params).apply(&mut target, Scope::adjustments()); Preset::from_params(params)
.apply(&mut target, Scope::adjustments())
.expect_no_film();
assert_eq!(target.param(exposure::ID, exposure::EXPOSURE), Some(5.0)); assert_eq!(target.param(exposure::ID, exposure::EXPOSURE), Some(5.0));
} }
@@ -1001,11 +1192,13 @@ mod tests {
let mut target_map = Preset::capture(&target_graph).into_params(); let mut target_map = Preset::capture(&target_graph).into_params();
preset.apply(&mut target_graph, scope); preset.apply(&mut target_graph, scope).expect_no_film();
preset.amend(&mut target_map, scope); preset.amend(&mut target_map, scope);
let mut rebuilt = EditGraph::default_chain(); let mut rebuilt = EditGraph::default_chain();
Preset::from_params(target_map).apply(&mut rebuilt, Scope::everything()); Preset::from_params(target_map)
.apply(&mut rebuilt, Scope::everything())
.expect_no_film();
for cap in target_graph.capabilities() { for cap in target_graph.capabilities() {
for p in &cap.params { for p in &cap.params {
@@ -1030,7 +1223,9 @@ mod tests {
let preset = Preset::capture(&source); let preset = Preset::capture(&source);
let mut target = EditGraph::default_chain(); let mut target = EditGraph::default_chain();
preset.apply(&mut target, Scope::everything()); preset
.apply(&mut target, Scope::everything())
.expect_no_film();
for cap in source.capabilities() { for cap in source.capabilities() {
for p in &cap.params { for p in &cap.params {
@@ -1068,6 +1263,188 @@ mod tests {
assert_eq!(excluded, vec![framing::ID.0]); assert_eq!(excluded, vec![framing::ID.0]);
} }
// --- the film ------------------------------------------------------------
/// A graph developing on `stock`. The tables are invented; what is under
/// test is whether the *choice* travels.
fn on_film(stock: &str) -> EditGraph {
let mut g = edited();
g.set_film(Some(crate::graph::Film {
stock: stock.to_string(),
print: Some("kodak_portra_endura".to_string()),
tables: crate::ops::FilmTables {
exposure_matrix: [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]],
curves: vec![[0.5, 0.5, 0.5]; crate::ops::film_sim::CURVE_SAMPLES],
curve_log_min: -3.0,
curve_log_max: 1.0,
lut: vec![[0.5, 0.5, 0.5]; 8],
density_max: 2.0,
lut_size: 2,
grain_particles: [0.0; 3],
grain_density_max: [2.0; 3],
grain_uniformity: 1.0,
},
}));
g
}
/// TRACES: FR-DEV-6 | FR-DEV-3f
#[test]
fn a_copy_carries_the_stock_it_was_developed_on() {
let preset = Preset::capture(&on_film("kodak_portra_400"));
let film = preset.film().expect("the stock was captured");
assert_eq!(film.stock, "kodak_portra_400");
assert_eq!(film.print.as_deref(), Some("kodak_portra_endura"));
assert!(Preset::capture(&edited()).film().is_none());
}
/// TRACES: FR-DEV-6 | FR-DEV-3f
/// The graph's own state keeps the film in one place only.
#[test]
fn an_edit_state_does_not_carry_the_film_twice() {
let state = on_film("kodak_portra_400").state();
assert!(state.film.is_some());
assert_eq!(state.params.film(), None);
}
/// TRACES: FR-DEV-6 | FR-DEV-3f
/// Applying a preset that names a stock asks for it to be baked, and
/// clears the tables it found — they came from the sliders it replaced.
#[test]
fn applying_a_film_preset_asks_for_its_stock() {
let preset = Preset::capture(&on_film("kodak_portra_400"));
let mut target = on_film("ilford_hp5");
let rebake = preset.apply(&mut target, Scope::adjustments());
assert_eq!(
rebake.wanted().map(|f| f.stock.as_str()),
Some("kodak_portra_400")
);
assert!(target.film().is_none(), "the old tables were left standing");
}
/// TRACES: FR-DEV-6 | FR-DEV-3f
/// Replacement, as for every parameter: a preset with no stock, at a
/// scope that carries the film, develops the target without one.
#[test]
fn a_preset_without_a_film_clears_the_targets() {
let mut target = on_film("ilford_hp5");
Preset::capture(&edited())
.apply(&mut target, Scope::adjustments())
.expect_no_film();
assert!(target.film().is_none());
}
/// TRACES: FR-DEV-6 | FR-DEV-3f
/// A scope that leaves the film node behind leaves the stock behind too,
/// so an exposure never lands on a stock it was not set for.
#[test]
fn a_scope_without_the_film_leaves_the_stock_alone() {
let preset = Preset::capture(&on_film("kodak_portra_400"));
let mut target = on_film("ilford_hp5");
let tone = Scope::of([Attribute::Tone]);
assert!(!tone.carries_film());
preset.apply(&mut target, tone).expect_no_film();
assert_eq!(target.film().map(|f| f.stock.as_str()), Some("ilford_hp5"));
assert_eq!(preset.film_for(tone), None);
assert!(preset.film_for(Scope::adjustments()).is_some());
}
/// TRACES: FR-DEV-6 | FR-DEV-3f
#[test]
fn a_stock_alone_is_not_a_neutral_preset() {
let preset = Preset::default().with_film(Some(FilmRef {
stock: "kodak_portra_400".into(),
print: None,
}));
assert!(!preset.is_empty());
assert_eq!(preset.op_count(Scope::adjustments()), 1);
assert_eq!(preset.op_count(Scope::of([Attribute::Tone])), 0);
}
// --- reach ---------------------------------------------------------------
/// A look: a stock and a contrast, and nothing else.
fn look() -> Preset {
let mut params = BTreeMap::new();
params.insert(("contrast".to_string(), "contrast".to_string()), 20.0);
Preset::from_params(params)
.with_film(Some(FilmRef {
stock: "kodak_portra_400".into(),
print: None,
}))
.with_reach(Reach::Named)
}
/// TRACES: FR-DEV-6
/// The reason `Reach` exists: a look applied over a corrected photograph
/// keeps the correction.
#[test]
fn a_look_leaves_what_it_does_not_name_alone() {
let mut target = EditGraph::default_chain();
target.set_param(exposure::ID, exposure::EXPOSURE, 1.25);
target.set_param(saturation::ID, saturation::SATURATION, -40.0);
let rebake = look().apply(&mut target, Scope::adjustments());
assert_eq!(target.param(exposure::ID, exposure::EXPOSURE), Some(1.25));
assert_eq!(
target.param(saturation::ID, saturation::SATURATION),
Some(-40.0)
);
assert_eq!(
rebake.wanted().map(|f| f.stock.as_str()),
Some("kodak_portra_400")
);
}
/// TRACES: FR-DEV-6
/// A look without a film leaves the target's film alone, where a whole
/// edit without one would clear it.
#[test]
fn a_look_without_a_film_keeps_the_targets() {
let mut target = on_film("ilford_hp5");
let only_contrast = look().with_film(None);
assert_eq!(only_contrast.film_for(Scope::adjustments()), None);
only_contrast
.apply(&mut target, Scope::adjustments())
.expect_no_film();
assert_eq!(target.film().map(|f| f.stock.as_str()), Some("ilford_hp5"));
}
/// TRACES: FR-DEV-6
/// The batch path agrees: a look amends only the operations it names.
#[test]
fn a_look_amends_only_what_it_names() {
let mut target = BTreeMap::new();
target.insert(("exposure".to_string(), "exposure".to_string()), 1.25);
target.insert(("contrast".to_string(), "contrast".to_string()), -50.0);
look().amend(&mut target, Scope::adjustments());
assert_eq!(
target.get(&("exposure".to_string(), "exposure".to_string())),
Some(&1.25)
);
assert_eq!(
target.get(&("contrast".to_string(), "contrast".to_string())),
Some(&20.0)
);
}
/// TRACES: FR-DEV-6
#[test]
fn a_look_survives_the_library_round_trip() {
let mut lib = named();
lib.insert("Look", look()).unwrap();
let text = lib.to_text();
assert!(text.contains("reach = named\n"), "{text}");
let back = PresetLibrary::parse(&text).unwrap();
assert_eq!(back, lib);
// And the default is not written, so an existing library's bytes are
// unchanged by this format ever having grown the line.
assert_eq!(named().to_text().matches("reach").count(), 0);
}
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
// Named presets // Named presets
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
@@ -1147,6 +1524,37 @@ mod tests {
assert!(out.contains("not_an_op.not_a_param = 0.25"), "{out}"); assert!(out.contains("not_an_op.not_a_param = 0.25"), "{out}");
} }
/// TRACES: FR-DEV-6 | FR-DEV-3f
#[test]
fn a_film_survives_the_library_round_trip() {
let mut lib = named();
lib.insert("Portra", Preset::capture(&on_film("kodak_portra_400")))
.unwrap();
let text = lib.to_text();
assert!(text.contains("film = kodak_portra_400\n"), "{text}");
assert!(
text.contains("film_print = kodak_portra_endura\n"),
"{text}"
);
assert_eq!(PresetLibrary::parse(&text).unwrap(), lib);
}
/// TRACES: FR-DEV-6 | FR-DEV-3f
/// The paper may come first in a hand-edited block, and a paper with no
/// film is dropped rather than baked from nothing.
#[test]
fn film_lines_read_in_either_order_and_a_lone_paper_is_dropped() {
let text = format!(
"drpl {LIBRARY_FORMAT_VERSION}\n\n[preset A]\nfilm_print = p\nfilm = s\n\n\
[preset B]\nfilm_print = p\nexposure.exposure = 1\n"
);
let lib = PresetLibrary::parse(&text).unwrap();
let a = lib.get("A").unwrap().film().unwrap();
assert_eq!((a.stock.as_str(), a.print.as_deref()), ("s", Some("p")));
assert_eq!(lib.get("B").unwrap().film(), None);
assert_eq!(lib.get("B").unwrap().len(), 1);
}
#[test] #[test]
fn a_file_from_a_newer_build_is_refused_rather_than_guessed_at() { fn a_file_from_a_newer_build_is_refused_rather_than_guessed_at() {
let text = format!("drpl {}\n", LIBRARY_FORMAT_VERSION + 1); let text = format!("drpl {}\n", LIBRARY_FORMAT_VERSION + 1);
@@ -1234,7 +1642,9 @@ mod tests {
let preset = stored.get("Warm portrait").unwrap(); let preset = stored.get("Warm portrait").unwrap();
let mut target = EditGraph::default_chain(); let mut target = EditGraph::default_chain();
preset.apply(&mut target, Scope::adjustments()); preset
.apply(&mut target, Scope::adjustments())
.expect_no_film();
assert_eq!(target.param(exposure::ID, exposure::EXPOSURE), Some(0.75)); assert_eq!(target.param(exposure::ID, exposure::EXPOSURE), Some(0.75));
// The target keeps its own framing on the default scope. // The target keeps its own framing on the default scope.
assert_eq!(target.param(framing::ID, framing::ANGLE), Some(0.0)); assert_eq!(target.param(framing::ID, framing::ANGLE), Some(0.0));
@@ -1268,7 +1678,9 @@ mod tests {
fn a_hand_picked_scope_carries_only_the_kinds_it_names() { fn a_hand_picked_scope_carries_only_the_kinds_it_names() {
let preset = Preset::capture(&edited()); let preset = Preset::capture(&edited());
let mut target = EditGraph::default_chain(); let mut target = EditGraph::default_chain();
preset.apply(&mut target, Scope::of([Attribute::Tone])); preset
.apply(&mut target, Scope::of([Attribute::Tone]))
.expect_no_film();
// Tone was picked, so the exposure travelled. // Tone was picked, so the exposure travelled.
assert_eq!( assert_eq!(
@@ -1333,7 +1745,9 @@ mod tests {
let mut target = EditGraph::default_chain(); let mut target = EditGraph::default_chain();
target.set_param(exposure::ID, exposure::EXPOSURE, -1.25); target.set_param(exposure::ID, exposure::EXPOSURE, -1.25);
Preset::capture(&edited()).apply(&mut target, empty); Preset::capture(&edited())
.apply(&mut target, empty)
.expect_no_film();
assert_eq!( assert_eq!(
target.param(exposure::ID, exposure::EXPOSURE), target.param(exposure::ID, exposure::EXPOSURE),
Some(-1.25), Some(-1.25),
core/dr-pipeline/src/starter.rs
-220
View File
@@ -1,220 +0,0 @@
//! TRACES: FR-DEV-6
//! The presets a first run starts with.
//!
//! # Why any at all
//!
//! A preset sheet that opens on "No presets yet" teaches the photographer that
//! the feature is homework. These exist so the first thing the sheet does is
//! demonstrate what a preset *is* — and so that applying one to a selection,
//! which is the action worth discovering, is available before anybody has
//! saved anything.
//!
//! # Why these are ours and not Adobe's
//!
//! Lightroom ships a large bundled set, and importing one of *those* files is
//! what [`crate::preset_import`] is for — a photographer's own library,
//! carried across. Redistributing Adobe's inside this application would be
//! shipping their creative work under a licence that does not permit it, which
//! is a reason on its own; and their numbers are calibrated against their tone
//! curve rather than ours, so the look would not survive the trip even if the
//! licence allowed it.
//!
//! So these are written against this pipeline, in its units, and they are
//! deliberately mild. A starter preset is a starting point — a photographer
//! who wanted the full effect can push the sliders, where one who is handed a
//! caricature learns to distrust the list.
//!
//! # Why they live in the core rather than in the interface
//!
//! Because they name operations, and nothing in `ui/` may
//! (`ui_names_no_operation.rs`, ARCH §4.3a). That test is right to object: a
//! preset called "Punch" *is* a statement about contrast, clarity and
//! vibrance, which makes it a statement in the pipeline's vocabulary rather
//! than a fact about any interface. The frontend asks for the set and stores
//! it; it never learns what is in it.
//!
//! # Why they are seeded rather than merged
//!
//! Written once, on the first run that finds no library at all, and never
//! again. Re-adding them on every start would resurrect one the photographer
//! deleted on purpose, and updating them in place would silently rewrite an
//! edit they had adjusted and kept under the same name. After the first run
//! these are ordinary presets: renameable, editable, deletable, and gone for
//! good when deleted.
use std::collections::BTreeMap;
use crate::{Preset, PresetLibrary};
/// One starter preset: a name and the parameters that differ from default.
struct Starter {
name: &'static str,
params: &'static [(&'static str, &'static str, f32)],
}
/// The set. Small on purpose — six a photographer might actually reach for
/// beats forty they have to scroll past.
const STARTERS: &[Starter] = &[
Starter {
name: "Punch",
params: &[
("contrast", "contrast", 18.0),
("clarity", "amount", 12.0),
("vibrance", "vibrance", 18.0),
("blacks_whites", "blacks", -8.0),
],
},
Starter {
name: "Soft portrait",
params: &[
("contrast", "contrast", -8.0),
("highlights_shadows", "highlights", -20.0),
("highlights_shadows", "shadows", 15.0),
("clarity", "amount", -10.0),
("vibrance", "vibrance", 10.0),
("saturation", "saturation", -5.0),
],
},
Starter {
name: "Recover the sky",
params: &[
// The most common single fix in landscape work: a bright sky and a
// dark foreground, both pulled back toward the middle.
("highlights_shadows", "highlights", -55.0),
("highlights_shadows", "shadows", 35.0),
("blacks_whites", "whites", -10.0),
],
},
Starter {
name: "Lift the shadows",
params: &[
("highlights_shadows", "shadows", 40.0),
("blacks_whites", "blacks", 12.0),
("contrast", "contrast", -5.0),
],
},
Starter {
name: "Crisp detail",
params: &[
("texture", "amount", 20.0),
("clarity", "amount", 10.0),
("capture_sharpen", "amount", 35.0),
],
},
Starter {
name: "Muted",
params: &[
("saturation", "saturation", -30.0),
("vibrance", "vibrance", 10.0),
("contrast", "contrast", -10.0),
("highlights_shadows", "shadows", 12.0),
],
},
];
/// The starter library, for a device that has never had one.
pub fn library() -> PresetLibrary {
let mut library = PresetLibrary::default();
for starter in STARTERS {
let params: BTreeMap<(String, String), f32> = starter
.params
.iter()
.map(|(op, param, value)| ((op.to_string(), param.to_string()), *value))
.collect();
// The name is a literal in this file, so a refusal would be a bug here
// rather than bad input — but it still must not take the whole set
// down, since the alternative to five presets is not six, it is none.
if let Err(e) = library.insert(starter.name, Preset::from_params(params)) {
log::warn!(
"starter preset {:?} is unusable ({e:?}); skipping",
starter.name
);
}
}
library
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{EditGraph, Scope};
#[test]
fn every_starter_names_parameters_this_build_actually_has() {
// The same guard the importer's table has, for the same reason: a
// renamed parameter must break the build rather than ship a preset
// that quietly does nothing.
let graph = EditGraph::default_chain();
let capabilities = graph.capabilities();
for starter in STARTERS {
for (op, param, _) in starter.params {
let capability = capabilities
.iter()
.find(|c| c.id.0 == *op)
.unwrap_or_else(|| panic!("{:?}: no operation {op:?}", starter.name));
assert!(
capability.params.iter().any(|p| p.id.0 == *param),
"{:?}: operation {op:?} has no parameter {param:?}",
starter.name
);
}
}
}
#[test]
fn every_starter_actually_changes_something() {
// A preset that applies to nothing is worse than one fewer preset: it
// teaches the photographer that the list does not work.
for (name, preset) in library().iter() {
assert!(!preset.is_empty(), "{name} carries nothing");
let mut graph = EditGraph::default_chain();
preset.apply(&mut graph, Scope::adjustments());
assert_ne!(
Preset::capture(&graph),
Preset::default(),
"{name} left the graph at its defaults"
);
}
}
#[test]
fn no_starter_carries_a_crop() {
// These are looks, not compositions. One that re-framed every image it
// was applied to would be the exact accident `Scope`'s default exists
// to prevent.
for (name, preset) in library().iter() {
assert!(!preset.touches_framing(), "{name} carries framing");
}
}
#[test]
fn the_names_are_distinct() {
assert_eq!(library().len(), STARTERS.len());
}
#[test]
fn the_values_stay_inside_what_the_controls_accept() {
// Clamping happens on apply, so an out-of-range literal here would be
// silently trimmed and the preset would not be the one written.
let graph = EditGraph::default_chain();
for starter in STARTERS {
for (op, param, value) in starter.params {
let mut applied = EditGraph::default_chain();
let capability = graph
.capabilities()
.into_iter()
.find(|c| c.id.0 == *op)
.unwrap();
let descriptor = capability.params.iter().find(|p| p.id.0 == *param).unwrap();
applied.set_param(capability.id, descriptor.id, *value);
assert_eq!(
applied.param(capability.id, descriptor.id),
Some(*value),
"{}: {op}.{param} = {value} was clamped",
starter.name
);
}
}
}
}
core/dr-preset-xmp/src/lib.rs
+30 -3
View File
@@ -44,7 +44,7 @@
use std::collections::BTreeMap; use std::collections::BTreeMap;
use dr_pipeline::Preset; use dr_pipeline::{Preset, Reach};
use quick_xml::events::Event; use quick_xml::events::Event;
use quick_xml::XmlVersion; use quick_xml::XmlVersion;
@@ -300,7 +300,11 @@ pub fn read_xmp(text: &str) -> Result<Import, ImportError> {
Ok(Import { Ok(Import {
name, name,
preset: Preset::from_params(params), // A look, because that is what a Lightroom preset is: it changes the
// settings it was saved with and leaves every other one where the
// photograph had it. Applied as a whole edit instead, a preset
// holding only a grade would reset the exposure it was put on top of.
preset: Preset::from_params(params).with_reach(Reach::Named),
skipped, skipped,
}) })
} }
@@ -472,7 +476,10 @@ mod tests {
// second kind of thing with a second apply path. // second kind of thing with a second apply path.
let import = read_xmp(ATTRIBUTE_FORM).unwrap(); let import = read_xmp(ATTRIBUTE_FORM).unwrap();
let mut graph = EditGraph::default_chain(); let mut graph = EditGraph::default_chain();
import.preset.apply(&mut graph, Scope::adjustments()); import
.preset
.apply(&mut graph, Scope::adjustments())
.expect_no_film();
assert_eq!( assert_eq!(
graph.param( graph.param(
dr_pipeline::ops::exposure::ID, dr_pipeline::ops::exposure::ID,
@@ -482,6 +489,26 @@ mod tests {
); );
} }
/// TRACES: FR-DEV-6
/// Lightroom's own rule: a preset changes what it was saved with and
/// nothing else, so a grade lands on top of the photograph's correction.
#[test]
fn an_imported_preset_leaves_what_it_does_not_set_alone() {
use dr_pipeline::ops::dehaze;
let import = read_xmp(ATTRIBUTE_FORM).unwrap();
assert_eq!(import.preset.reach(), Reach::Named);
assert!(value(&import, "dehaze", "amount").is_none());
let mut graph = EditGraph::default_chain();
graph.set_param(dehaze::ID, dehaze::AMOUNT, 30.0);
import
.preset
.apply(&mut graph, Scope::adjustments())
.expect_no_film();
assert_eq!(graph.param(dehaze::ID, dehaze::AMOUNT), Some(30.0));
}
#[test] #[test]
fn a_value_that_is_not_a_number_costs_that_setting_and_not_the_file() { fn a_value_that_is_not_a_number_costs_that_setting_and_not_the_file() {
let text = let text =
core/dr-sync-nextcloud/src/lib.rs
+27
View File
@@ -376,6 +376,33 @@ impl RemoteBackend for NextcloudBackend {
Ok(body) Ok(body)
} }
async fn get_reporting(
&self,
id: &RemoteId,
progress: &(dyn Fn(u64, Option<u64>) + Send + Sync),
) -> Result<Vec<u8>, RemoteError> {
let url = self.url_for_id(id)?;
let mut resp = self
.client
.get(&url)
.basic_auth(&self.login, Some(&self.password))
.send()
.await
.map_err(map_send_error)?;
map_status(resp.status(), &url)?;
// Read chunk by chunk rather than with `bytes()`, which is the same
// transfer with nothing to say until it ends.
let declared = resp.content_length();
let mut body = Vec::with_capacity(declared.unwrap_or(0) as usize);
progress(0, declared);
while let Some(chunk) = resp.chunk().await.map_err(map_send_error)? {
body.extend_from_slice(&chunk);
progress(body.len() as u64, declared);
}
Ok(body)
}
async fn put( async fn put(
&self, &self,
path: &RemotePath, path: &RemotePath,
core/dr-sync/src/lib.rs
+20
View File
@@ -98,6 +98,26 @@ pub trait RemoteBackend: Send + Sync {
/// either way; [`Capabilities::range_reads`] says whether it was cheap. /// either way; [`Capabilities::range_reads`] says whether it was cheap.
async fn get(&self, id: &RemoteId, range: Option<Range<u64>>) -> Result<Vec<u8>, RemoteError>; async fn get(&self, id: &RemoteId, range: Option<Range<u64>>) -> Result<Vec<u8>, RemoteError>;
/// Fetch a whole object, saying how much of it has arrived as it arrives.
///
/// `progress` is called with the bytes received so far and the length the
/// server declared, if it declared one. For the one transfer a person
/// watches: an original opened in develop is tens of megabytes, and a
/// view that can only say "downloading" for that long reads as stuck.
///
/// The default fetches with [`get`](Self::get) and reports once, at the
/// end — right for a backend whose `get` is a local read, where there is
/// no wait to report on.
async fn get_reporting(
&self,
id: &RemoteId,
progress: &(dyn Fn(u64, Option<u64>) + Send + Sync),
) -> Result<Vec<u8>, RemoteError> {
let body = self.get(id, None).await?;
progress(body.len() as u64, Some(body.len() as u64));
Ok(body)
}
/// Upload, optionally guarded by a precondition. /// Upload, optionally guarded by a precondition.
/// ///
/// Backends handle chunking internally based on body size — chunked /// Backends handle chunking internally based on body size — chunked
core/dr-types/src/settings.rs
+13
View File
@@ -727,6 +727,18 @@ pub struct ExportSettings {
/// [`Self::destination`], where empty means "ask each time" because there /// [`Self::destination`], where empty means "ask each time" because there
/// is no sensible folder to assume on a filesystem. /// is no sensible folder to assume on a filesystem.
pub remote_destination: String, pub remote_destination: String,
/// TRACES: FR-EXP-10
/// The album exports go to, by its catalog uuid. Empty until one is
/// chosen, and an export with none is refused and says so.
///
/// This replaced [`Self::target`] and the two destination fields as what
/// the export sheet chooses: a destination is now a named album in the
/// catalog, which records what was exported into it. The three older
/// fields stay so a settings file written before albums still reads, and
/// so the first album can be made from the folder they named; the batch
/// fills them from the album when it runs.
pub album: String,
} }
impl Default for ExportSettings { impl Default for ExportSettings {
@@ -748,6 +760,7 @@ impl Default for ExportSettings {
target: ExportTarget::default(), target: ExportTarget::default(),
destination: String::new(), destination: String::new(),
remote_destination: String::new(), remote_destination: String::new(),
album: String::new(),
} }
} }
} }
docs/README.md
+9 -7
View File
@@ -7,8 +7,8 @@ second.
| | | | | |
|---|---| |---|---|
| [The manual](manual/README.md) | Every feature, pictured from the application itself — opening a library, rating and filing, developing, local masks, repair, film, panoramas, export | | [The manual](manual/README.md) | Every feature, pictured from the application itself — opening a library, rating, labelling and filing, duplicate originals, developing, local masks, repair, film, presets, panoramas, export and albums. The application carries it and opens it from Help and from Settings |
| [How it is driven](gestures.md) | Every gesture and shortcut, by screen. Generated from the code, so it cannot describe one the application does not have | | [How it is driven](gestures.md) | Every gesture and shortcut, by screen. Generated from the code, so it cannot describe one the application does not have. The same list is the in-app help sheet: `Help` or `F1` in the grid, `?` or `F1` in develop |
The [top-level README](../README.md) says what DarkRoom is, how to get it on The [top-level README](../README.md) says what DarkRoom is, how to get it on
each platform, and what is still missing. each platform, and what is still missing.
@@ -74,11 +74,13 @@ recollection.
## Conventions ## Conventions
Two files here are generated and must not be edited by hand: Three files here are generated and must not be edited by hand:
`gestures.md` and `dev/traceability.md`. Both come from `gestures.md`, `dev/traceability.md` and `manual/index.html`, the page the
`cargo run -p traceability` and the pre-commit hook keeps them in step with packages install, rendered from `manual/README.md`. All three come from
the tree. The manual's pictures are recorded by `cargo run -p traceability`, the pre-commit hook keeps them in step with the
[`tools/manual`](../tools/manual/README.md) and live in LFS. tree, and CI fails when one is not what the tree generates. The manual's
pictures are recorded by [`tools/manual`](../tools/manual/README.md) and live
in LFS.
A design document links to the requirements it satisfies and to the code A design document links to the requirements it satisfies and to the code
that satisfies them. When the code moves, the link moves with it; a document that satisfies them. When the code moves, the link moves with it; a document
docs/dev/architecture.md
+10 -1
View File
@@ -52,7 +52,7 @@ darkroom/
│ ├── dr-types SourceRef, ImageId, VersionId, ParamValue — shared vocabulary │ ├── dr-types SourceRef, ImageId, VersionId, ParamValue — shared vocabulary
│ ├── dr-catalog SQLite index, scan, query, metadata │ ├── dr-catalog SQLite index, scan, query, metadata
│ ├── dr-sidecar the authoritative edit store (§6.12) │ ├── dr-sidecar the authoritative edit store (§6.12)
│ ├── dr-decode RawDecoder trait, rawler impl, embedded-preview extraction │ ├── dr-decode Decoder trait (§3.2), rawler impl, embedded-preview extraction
│ ├── dr-pipeline Operation trait, descriptors, edit graph, registry │ ├── dr-pipeline Operation trait, descriptors, edit graph, registry
│ ├── dr-gpu wgpu device, tile scheduler, WGSL shaders, mask rasteriser │ ├── dr-gpu wgpu device, tile scheduler, WGSL shaders, mask rasteriser
│ ├── dr-colour lcms2 bindings, camera profiles, working-space transforms │ ├── dr-colour lcms2 bindings, camera profiles, working-space transforms
@@ -132,6 +132,15 @@ Four separate entry points because the caller's needs differ sharply by phase. C
`RawImage` carries CFA-pattern sensor data plus black/white levels and camera colour matrices — it `RawImage` carries CFA-pattern sensor data plus black/white levels and camera colour matrices — it
is *not* demosaiced. Demosaic is a GPU pipeline stage (§5.2). is *not* demosaiced. Demosaic is a GPU pipeline stage (§5.2).
> **As built (FR-RAW-2, 0.15.0).** The trait is `dr_decode::Decoder`, and it takes bytes rather
> than a reader: `header_bytes` says how much of a file metadata needs, `metadata` and
> `orientation` read it, `locate_preview` says where the embedded preview sits so the caller's
> storage can fetch that range, and `preview` and `decode` take the bytes fetched. The split this
> section argues for survives; what moved is who reads the file, which is the storage layer
> (§3.1's `read_range`), not the decoder. `dr_decode::Rawler` is the one implementation, and only
> the places that start a job name `dr_decode::default()`; everything below them takes a
> `&dyn Decoder`.
### 3.3 Operation and descriptors ### 3.3 Operation and descriptors
The develop pipeline is a sequence of operations with a uniform interface. Polymorphism is by The develop pipeline is a sequence of operations with a uniform interface. Polymorphism is by
docs/dev/benchmarks.md
+7 -2
View File
@@ -58,8 +58,13 @@ Two of those rows carry a qualifier, and the qualifiers are the point.
**NFR-P1 — catalog open under 2 s.** The measured span is the four things the **NFR-P1 — catalog open under 2 s.** The measured span is the four things the
library view cannot paint without: `Catalog::open` (which connects, migrates and library view cannot paint without: `Catalog::open` (which connects, migrates and
**backfills**, and the backfill is three passes over the images table on every **backfills**, and the backfill is passes over the whole images table), `count`,
open), `count`, the first 400-row `window`, and the monthly `timeline`. Tagged the first 400-row `window`, and the monthly `timeline`. Since 0.17.0 the
backfill runs on the first open of a catalog in a process and is skipped by
later ones while nothing has changed ([catalog.md §2](catalog.md)), so
`catalog_open_ms` includes it and `catalog_open_warm_ms`, the second open in
the same process, does not — which is what a library reopened in the same
session costs. Tagged
`TRACES: NFR-P1` in [`tools/bench/src/catalog_open.rs`](../../tools/bench/src/catalog_open.rs), `TRACES: NFR-P1` in [`tools/bench/src/catalog_open.rs`](../../tools/bench/src/catalog_open.rs),
because a build that breaks it fails this gate. because a build that breaks it fails this gate.
docs/dev/catalog.md
+113 -13
View File
@@ -129,6 +129,37 @@ CREATE INDEX members_image ON collection_members(image_id);
partial index over the non-NULL subset is both smaller and what FR-CAT-9's reconnection-by-hash partial index over the non-NULL subset is both smaller and what FR-CAT-9's reconnection-by-hash
and FR-CAT-11's duplicate detection actually query. and FR-CAT-11's duplicate detection actually query.
**Made on first use, not by a migration.** A new `user_version` makes every older build refuse this
catalog's snapshot at sync (`sync::remote_is_mergeable` compares it and nothing else), and a tablet
a release behind would stop merging collections, keywords and people for a feature it does not
have. So what later releases added without needing old rows rewritten is created with `IF NOT
EXISTS` where it is first used, and an older build that meets it ignores it:
- `dedup_probes` (FR-CAT-11a, §3.5);
- the albums (FR-EXP-10, `core/dr-catalog/src/albums.rs`): `albums`, `album_exports` — one row per
file written into an album, keyed on the file name, since two crops of one photograph are two
files — and `album_folders`, this device's folder for each (§8.2);
- `keywords_term_version (keyword, version_id)`, made by `keywords::list`, which counts each word's
photographs on every selection change and without it read a `keywords` row per assignment to
learn its version;
- `faces_box (image_id, model_id, x, y, w, h)`, made by the merge's `match_faces`, which reads every
local face's box and model and without it opened each ~8 KB `faces` row to do so.
A failure to make one of the indexes — a read-only or busy catalog — is logged and the query runs
without it, as it did before.
**Opening does not repeat the backfill.** `schema::backfill` repairs what a write left owing — an
image a scan inserted without its default version, the RAW/JPEG pairing, a merged version's uuid, a
keyword assignment whose word has no term — and it used to run in every `Catalog::open`. Every
worker opens its own connection, so a develop landing paid it five times (~80 ms of CPU on the
reference catalog) to learn that nothing had changed. `core/dr-catalog/src/backfilled.rs` now
records, per path and per process, a stamp read before each backfill: the schema version, the
file's device and inode, and the newest image, version and keyword assignment by content as well as
id — because none of those tables is `AUTOINCREMENT`, a freed newest id is handed out again, and the
row that takes it is exactly one the backfill is owed. An open whose stamp matches skips it (~1 ms);
the first open in a process, a migration, a pull and a replaced file always run it. Kept in memory
rather than in the catalog, so nothing about it travels in the sync snapshot.
--- ---
## 3. Incremental scan ## 3. Incremental scan
@@ -249,6 +280,13 @@ what you actually have.
only when something needs it: import duplicate detection (FR-CAT-11), or reconnecting a moved source only when something needs it: import duplicate detection (FR-CAT-11), or reconnecting a moved source
(FR-CAT-9). Never during a routine scan. (FR-CAT-9). Never during a routine scan.
Consolidating the duplicates a library already holds (FR-CAT-11a, 0.16.0) does not wait for it. It
proves a group the same by `content_hash` where every copy has one, and otherwise by a digest of
each copy's first and last megabyte, read by range through the backend and kept in `dedup_probes`
keyed on the size and mtime it was taken at, so a second review reads nothing.
`dedup_probes` is created on first use rather than by a migration, because a schema bump would make
an older build refuse this catalog's snapshot at sync (`core/dr-catalog/src/duplicates.rs`).
--- ---
## 4. The library view ## 4. The library view
@@ -413,10 +451,33 @@ pub enum JobKind {
} }
``` ```
**Thumbnails are not queued (decided 2026-09-26, #73).** `Thumbnail` is kept only so its number
stays taken, and is listed in `JobKind::RETIRED`. Up to 0.16.0 every remote scan enqueued one per
photograph, and nothing claimed the kind — no `JobHandler` was ever registered for it, on desktop or
Android (the same `dr-ui`), and the catalog snapshot carries `jobs` but the merge never reads them
(§8.2). The reference catalog held 23,582 such rows, about 1 MB with its indexes. Thumbnails are
made another way, and by the right source of truth:
- the grid asks a worker for the cells it is drawing, which serves them from the thumbnail store or
range-fetches the embedded preview (§7.2);
- the thumbnail sweep's work list is *what the store does not hold* (`thumbnails_outstanding`).
The store is shared between devices (§7.3), so it is the only thing that knows another device already
made a thumbnail; a per-device queue row cannot. A queue row would be a second, staler record of the
same debt. Metadata is owed the same way — `metadata_state < 2` is the sweep's work list — so the local
walk no longer enqueues `ExtractMetadata` either.
The rows already queued are dropped by `jobs::drop_retired`, from `runner::recover` at every catalog
open, rather than by a migration: a schema bump would make a device still on an older build refuse the
synced snapshot, and "every open" rather than "once" because an older build sharing the catalog
queues them again on its next scan. With the rows gone it is one probe of the `(kind, subject_id)`
index. `every_queued_kind_has_a_consumer` (dr-catalog) holds the rule: it reads the shipping sources
and fails if any kind is enqueued that no handler or claim names.
### 6.2 Coalescing is the point ### 6.2 Coalescing is the point
`UNIQUE(kind, subject_id)` on `jobs` means enqueueing is idempotent: an image touched five times `UNIQUE(kind, subject_id)` on `jobs` means enqueueing is idempotent: an image touched five times
during a scan has one thumbnail job, not five. Enqueue is has one job of a kind, not five. Enqueue is
`INSERT … ON CONFLICT DO UPDATE SET priority = max(priority, excluded.priority)`, so a re-request at `INSERT … ON CONFLICT DO UPDATE SET priority = max(priority, excluded.priority)`, so a re-request at
higher priority promotes the existing row rather than duplicating it. higher priority promotes the existing row rather than duplicating it.
@@ -436,7 +497,8 @@ priority governs the whole app:
Visible-cell work is enqueued by the grid as it scrolls, at `Interactive`. The effect is that a Visible-cell work is enqueued by the grid as it scrolls, at `Interactive`. The effect is that a
freshly scanned library fills in *where the user is looking* first, and grinds through the rest freshly scanned library fills in *where the user is looking* first, and grinds through the rest
behind them. behind them. (As built, thumbnails and metadata get this ordering without the queue — the grid
requests its visible cells directly and the sweeps take what is left; see §6.1.)
### 6.4 Durability and failure ### 6.4 Durability and failure
@@ -655,21 +717,57 @@ mechanism.
### 8.2 What the file sync does and does not carry ### 8.2 What the file sync does and does not carry
Only **collections and their membership** merge. The rest of a catalog describes *local* state — Collections were the first thing merged, and the rules in §8.4 were written for them. What else
folder mtimes, cache file paths, job rows, `tier_actual` — and importing another device's version merges reuses those rules or keys on the same identities, and each has no other home:
of those would be actively wrong. The downloaded remote is read for its collections and discarded.
This is what keeps §6.12 substantially intact: nothing here makes the local database authoritative - **Collections and their membership** — by uuid and revision, membership as a set union.
for anything a rebuild could not recover. The catalog is still deletable. What syncs is one table - **Keywords** — the vocabulary by the same verdict, the assignments as a union.
pair that had no other home. - **People and identity judgements** — people by uuid and revision, and the confirmed and rejected
face assignments matched to local faces (`merge::match_faces`): by box first, and — since 0.18.0,
only on the photographs where a remote face is left over — by embedding, a pair being accepted
at cosine ≥ 0.7 when each is the other's best by a lead of ≥ 0.2 (#77; [faces.md §18.2](faces.md)). After every merge,
`dedup_people` folds people of one name whose confirmed faces agree, and a face held twice in
one photograph, through the ordinary `merged_into` redirect, which older builds already honour
(#78; [faces.md §19](faces.md)).
- **Albums** (FR-EXP-10, 0.17.0) — by uuid and revision with tombstones, and what went into each as
a set union keyed on the server's file id (a content hash on a folder library). An album's
server folder is a column of its row and travels with it; a folder on *this device* is in
`album_folders`, which the merge never reads and the upload snapshot drops (§8.3), because a
path or a SAF grant on one device means nothing on another.
- **Capture metadata** — the one exception inside `images`: a date, a camera, a lens and an ISO are
facts about the file's bytes, so a row this device has not yet read takes them from a peer that
has (`merge_metadata`), matched by `oc:fileid`.
The rest of a catalog describes *local* state — folder mtimes, cache file paths, job rows,
`tier_actual` — and importing another device's version of those would be actively wrong; the
downloaded remote is read for the tables above and discarded.
This is what keeps §6.12 substantially intact. The catalog is still deletable; what a rebuild from
sidecars cannot recover — collections and albums, which nothing in the filesystem records — is what
the sync exists to carry.
### 8.3 Two hazards the implementation must handle ### 8.3 Two hazards the implementation must handle
**A WAL database is not one file.** Committed transactions can sit in `catalog.sqlite-wal` with the **A WAL database is not one file.** Committed transactions can sit in `catalog.sqlite-wal` with the
main file lagging, so copying `catalog.sqlite` alone uploads a torn snapshot — internally consistent main file lagging, so copying `catalog.sqlite` alone uploads a torn snapshot — internally consistent
as of some older point, silently missing everything since. Upload therefore runs a `TRUNCATE` as of some older point, silently missing everything since. The upload therefore never copies the
checkpoint and then SQLite's backup API, which serialises against concurrent writers rather than live file. It builds the snapshot in an empty file. It attaches the catalog, creates each table
racing them. It never copies the live file. from the catalog's own schema, and fills it with `INSERT … SELECT`, all inside one transaction.
That transaction holds a single read snapshot of the catalog, so concurrent writers are serialised
rather than raced, as the backup API did before. The file is then checked with `quick_check` before
it goes anywhere.
**The face crops stay out of the upload.** A crop is a ~5 KB JPEG on each `faces` row. On a 19k-face
library they are 96 MB of a 158 MB catalog. The face shards carry them to other devices, once each.
The merge reads a remote face's box and model to match it to a local one — and, where the boxes cannot decide, its embedding — never its pixels. No
device adopts a downloaded catalog as its own: a fresh device starts empty and takes faces, crops
included, from the shards. So the snapshot's `crop` is NULL, and a merge never writes a local
crop. They were first stripped (2026-08) by copying the whole file with the backup API, setting
`crop` to NULL and `VACUUM`ing. That wrote the file about three times to upload 50 MB. Since #71
the snapshot is built without them. Its header is kept as it was (`user_version`, page size and
the WAL flag), so every earlier build merges it unchanged. NFR-R2 backups still use the backup API
and keep the crops, because a backup is a file the user may have to live on. `album_folders` is
dropped from the built file for the reason §8.2 gives.
**Integer primary keys are not identities.** Two devices each allocate `collections.id = 1` for **Integer primary keys are not identities.** Two devices each allocate `collections.id = 1` for
different collections, so a row-level merge keyed on the integer id would collide them. Collections different collections, so a row-level merge keyed on the integer id would collide them. Collections
@@ -685,6 +783,7 @@ integer ids stay local and are never compared across catalogs.
| Deletion | Tombstone (`deleted = 1`) carrying a revision | Without it, merging against a device that still holds the collection resurrects it. With a revision, deletion competes on equal footing with a rename | | Deletion | Tombstone (`deleted = 1`) carrying a revision | Without it, merging against a device that still holds the collection resurrects it. With a revision, deletion competes on equal footing with a rename |
| An image the remote has and we do not | Skip the membership row | It joins on a later merge, once a scan has catalogued the file. Not an error | | An image the remote has and we do not | Skip the membership row | It joins on a later merge, once a scan has catalogued the file. Not an error |
| A remote from a newer schema | Decline before attaching | Attempting it would fail mid-transaction rather than declining cleanly | | A remote from a newer schema | Decline before attaching | Attempting it would fail mid-transaction rather than declining cleanly |
| People with the same name | Folded after each merge when their faces agree ([faces.md §19](faces.md)) | Names typed separately on two devices otherwise stay two people for ever |
Merging is idempotent: running it twice reports no changes the second time. That property is tested, Merging is idempotent: running it twice reports no changes the second time. That property is tested,
because a merge that oscillates would upload on every sync forever. because a merge that oscillates would upload on every sync forever.
@@ -711,8 +810,9 @@ because if the SQLite path proves troublesome, this is the fallback with a known
membership needs to be *stable* — for manual ordering, or for a pinned set that must not shift membership needs to be *stable* — for manual ordering, or for a pinned set that must not shift
under the user — it needs materialising with an invalidation rule. Deferred until there is a under the user — it needs materialising with an invalidation rule. Deferred until there is a
concrete need. concrete need.
- **Multi-root capture-time collisions.** FR-CAT-11 detects duplicates on import; the same image - **Multi-root capture-time collisions.** FR-CAT-11 detects duplicates on import, and FR-CAT-11a
catalogued under two roots is a related but distinct case, not yet specified. consolidates the copies one root holds in several folders; the same image catalogued under two
roots is a related but distinct case, not yet specified.
- **Timeline granularity selection.** Which bucket size the UI picks for a given zoom is a UI - **Timeline granularity selection.** Which bucket size the UI picks for a given zoom is a UI
concern, but the catalog should probably suggest one from the query's date span rather than have concern, but the catalog should probably suggest one from the query's date span rather than have
the UI guess. the UI guess.
docs/dev/display-and-extension.md
+1 -1
View File
@@ -22,7 +22,7 @@ and the reason is that some of the work is done and untagged.
| FR-DSP-1 proxy rendering | **Done.** The develop view renders at viewport resolution, not source. | | FR-DSP-1 proxy rendering | **Done.** The develop view renders at viewport resolution, not source. |
| FR-DSP-2 tiled computation | **Absent, and §2 now says it should stay that way.** Measured: the fused pass is inside the budget everywhere. See [frame-budget.md](frame-budget.md). | | FR-DSP-2 tiled computation | **Absent, and §2 now says it should stay that way.** Measured: the fused pass is inside the budget everywhere. See [frame-budget.md](frame-budget.md). |
| FR-DSP-3 interactive latency | **Measured and asserted** for the fused path — `core/dr-gpu/tests/frame_budget.rs`. Missed by one operation, clarity, for the reason recorded as TD-4. | | FR-DSP-3 interactive latency | **Measured and asserted** for the fused path — `core/dr-gpu/tests/frame_budget.rs`. Missed by one operation, clarity, for the reason recorded as TD-4. |
| FR-DSP-4 progressive refinement | **Absent**, and §4's condition did not fire. Every render is full quality and can afford to be. | | FR-DSP-4 progressive refinement | **Built** (0.15.0), although §4's condition did not fire on the fused path: a half-resolution draft while a gesture moves, one sharp frame 120 ms after it stops, the histogram dimmed while it lags, and the draft faded out over 150 ms — `ui/dr-ui/src/refine.rs`. See [frame-budget.md](frame-budget.md). |
| FR-DSP-5 zoom and pan | **Done and tagged**, against tests that fail if the behaviour is removed — `core/dr-gpu/tests/zoom_resolution.rs`. `Framing::view` shrinks the sampled region while the render target keeps its size, so zooming *raises* the resolution the pipeline works at. That is FR-DSP-5's requirement, arrived at without tiles. | | FR-DSP-5 zoom and pan | **Done and tagged**, against tests that fail if the behaviour is removed — `core/dr-gpu/tests/zoom_resolution.rs`. `Framing::view` shrinks the sampled region while the render target keeps its size, so zooming *raises* the resolution the pipeline works at. That is FR-DSP-5's requirement, arrived at without tiles. |
| FR-DSP-6 colour management | **Done.** Output space is a parameter of composition. | | FR-DSP-6 colour management | **Done.** Output space is a parameter of composition. |
| FR-DSP-7 histogram and clipping | **Done**, GPU-side, no per-frame readback. | | FR-DSP-7 histogram and clipping | **Done**, GPU-side, no per-frame readback. |
docs/dev/distribution.md
+69 -48
View File
@@ -18,7 +18,7 @@ permission to a package rather than after.
| Platform | Channel | State | What it constrains | | Platform | Channel | State | What it constrains |
|---|---|---|---| |---|---|---|---|
| Linux | Arch source package — [`packaging/PKGBUILD`](../../packaging/PKGBUILD) | Built, in tree | Nothing. Full filesystem access, system Vulkan, system secret daemon | | Linux | Arch source package — [`packaging/PKGBUILD`](../../packaging/PKGBUILD) | Built, in tree | Nothing. Full filesystem access, system Vulkan, system secret daemon |
| Linux | Flatpak — [`packaging/flatpak/`](../../packaging/flatpak/) | Manifest in tree, **library selection does not work** (§4) | Portals only. No `--filesystem=`, no host mount table, no typed paths | | Linux | Flatpak — [`packaging/flatpak/`](../../packaging/flatpak/) | Manifest in tree; folders chosen through the FileChooser portal since 0.17.0, **never built or run here** (§4) | Portals only. No `--filesystem=`, no host mount table, no typed paths |
| Linux | AppImage | v1 channel, **recipe not yet written** (§5) | Oldest supported glibc, and no sandbox at all | | Linux | AppImage | v1 channel, **recipe not yet written** (§5) | Oldest supported glibc, and no sandbox at all |
| Android | F-Droid | v1 channel, not yet submitted | GPLv3-clean build, reproducible, no proprietary blobs | | Android | F-Droid | v1 channel, not yet submitted | GPLv3-clean build, reproducible, no proprietary blobs |
| Android | Play Store | **Not v1** (§6) | Would make ARCH §6.9 binding as policy rather than as engineering | | Android | Play Store | **Not v1** (§6) | Would make ARCH §6.9 binding as policy rather than as engineering |
@@ -45,10 +45,11 @@ somewhere:
`project_license` is GPL-3.0-or-later; those differ on purpose — see the `project_license` is GPL-3.0-or-later; those differ on purpose — see the
comment in the file. comment in the file.
- **Vulkan is a requirement, not a preference.** The develop pipeline is - **Vulkan is a requirement, not a preference.** The develop pipeline is
compute shaders through wgpu, and NFR-R8 — how far a CPU fallback goes — is compute shaders through wgpu, and NFR-R8 was decided on 2026-09-19 against a
still open, so today there is nothing behind it. A package that installs onto CPU pipeline: without a device the library, the grid and the judgements work
a machine with no working ICD produces an application that starts and cannot on embedded previews, and develop and export say they are unavailable. A
develop. package that installs onto a machine with no working ICD produces an
application that starts and cannot develop.
- **A Secret Service implementation, or an honest degraded mode.** FR-NC-2 is - **A Secret Service implementation, or an honest degraded mode.** FR-NC-2 is
explicit that the absence of a secrets daemon is a stated degraded mode and explicit that the absence of a secrets daemon is a stated degraded mode and
never a silent fall back to plaintext. Packages express this as an optional never a silent fall back to plaintext. Packages express this as an optional
@@ -59,6 +60,12 @@ somewhere:
the Flatpak manifest check the file size and refuse, because the alternative the Flatpak manifest check the file size and refuse, because the alternative
is a package whose face indexing fails inside the graph loader on a user's is a package whose face indexing fails inside the graph loader on a user's
machine rather than on the packager's. machine rather than on the packager's.
- **So are the manual's pictures.** Since 0.15.0 the Arch package, the Windows
installer and the APK carry `docs/manual/index.html` and its `media/`, where
`dr_ui::manual` looks for them (`/usr/share/darkroom/manual`, `manual\` beside
`darkroom.exe`, the APK's `assets/manual`), and each refuses a pointer where a
picture should be — shipped, a pointer is a manual of broken images that
nothing reports. The Flatpak manifest does not install it yet.
--- ---
@@ -71,7 +78,7 @@ The Arch package and an AppImage both hand the application the same
unrestricted process the developer runs it in, so neither can discover that a unrestricted process the developer runs it in, so neither can discover that a
design assumed unrestricted access. Flatpak takes that assumption away, and design assumed unrestricted access. Flatpak takes that assumption away, and
FR-PLAT-LIN-3 exists to make the discovery happen deliberately rather than in a FR-PLAT-LIN-3 exists to make the discovery happen deliberately rather than in a
bug report. §4 is what it discovered. bug report. §4 is what it discovered, and what has been done about it.
The same argument runs the other way on Android, where SAF has been the only The same argument runs the other way on Android, where SAF has been the only
option since before the first line was written (ARCH §6.9) and `SourceRef` option since before the first line was written (ARCH §6.9) and `SourceRef`
@@ -113,31 +120,55 @@ advance:
--- ---
## 4. What does not work: choosing a library ## 4. Choosing a library: built, not yet proved in the sandbox
**FR-PLAT-LIN-3 is not satisfied today, and the manifest does not pretend **FR-PLAT-LIN-3 was not satisfied up to 0.16.0, and is still not shown to
otherwise.** be.** What changed in 0.17.0 is the code; what has not changed is that no
Flatpak has been built here, so nothing below has been observed inside one.
A folder library is chosen by typing an absolute path. `dr-sync-folder`'s Up to 0.16.0 a folder library was chosen by typing an absolute path, and
provider declares `SignIn::EndpointOnly` with the placeholder nothing in the tree called the FileChooser portal. Inside a sandbox with no
`/home/you/Pictures`, and `normalise_endpoint` expands `~`, requires the path `--filesystem=`, `$HOME` still resolves to the real home *path* but that
to be absolute, and checks it with `std::fs`. Nothing in the tree calls the directory holds only the application's own `.var/app/…` tree, so a typed
FileChooser portal — there is no `ashpd`, no `rfd`, and no toolkit file dialog `~/Pictures` failed the `exists()` check and the launch screen said so — a
anywhere in `ui/`, `platform/` or `core/`. truthful message about a situation the user could not fix from inside the
application.
Inside a sandbox with no `--filesystem=`, `$HOME` still resolves to the real **Every folder the desktop asks for is now chosen in the platform's
home *path* but that directory holds only the application's own dialogue** (FR-EXP-6): the library folder on the launch screen (`Choose
`.var/app/…` tree. So a typed `~/Pictures` fails the `exists()` check and the folder…`), an import's source and second copy, a folder or file of
launch screen says `No folder at /home/you/Pictures.` — a truthful message Lightroom presets, and an album's folder on this device.
about a situation the user cannot fix from inside the application. `ui/dr-ui/src/folder_dialog.rs` asks through `rfd` with its `xdg-portal`
backend — `org.freedesktop.portal.FileChooser` over D-Bus, which Flatpak
always permits without a `--talk-name` — and the common item dialogue on
Windows. No path is typed anywhere on the desktop any more; Android keeps
its fields (`Pickers.local-paths`), because SAF returns document trees rather
than paths.
Import is blocked one step earlier. `dr_plat::volumes()` finds a camera card by What the portal hands back inside a sandbox is a path under
reading `/proc/self/mountinfo` and the `removable` flag under `/sys`. A `/run/user/$UID/doc/` that the document portal has exported, and the chosen
library goes through the same `normalise_endpoint` a typed one did, which
checks it with `std::fs`. That is the route this section used to ask for —
`rfd` drives `ashpd` underneath, the crate it named. Two things differ from
the plan, and both are stated rather than smoothed over:
- **It is not behind a platform seam.** The plan put the chooser beside
`LocalStorage::grant` in `dr-plat`, the one place a `Path` enters the
application. It is in `ui/`, and the path it returns reaches the folder
connector as a string, as a typed one did.
- **Nothing has confirmed the sandbox half.** Whether the exported path
still resolves after a restart — a library is remembered across launches,
so it has to — and whether the export is writable, so a sidecar can be
written beside a photograph, are the first two things a Flatpak build has
to check.
Import is still blocked one step earlier. `dr_plat::volumes()` finds a camera
card by reading `/proc/self/mountinfo` and the `removable` flag under `/sys`. A
sandboxed process is in its own mount namespace, so the table it reads sandboxed process is in its own mount namespace, so the table it reads
describes the sandbox; a card mounted at `/run/media/…` on the host is not in describes the sandbox; a card mounted at `/run/media/…` on the host is not in
it. `volumes()` correctly returns an empty list, which the interface presents it. `volumes()` correctly returns an empty list, which the interface presents
as "no card found" — right for the code, wrong for the user, who is looking at as "no card found" — but the import page now offers `Browse…` beside that
a card. message, and the dialogue it opens is the portal's, which can reach the card.
### The permission that would hide this, and why it is not in the manifest ### The permission that would hide this, and why it is not in the manifest
@@ -145,41 +176,31 @@ a card.
names as the alternative to portals. Granting it would mean the sandboxed build names as the alternative to portals. Granting it would mean the sandboxed build
never exercises the sandbox, which removes the entire reason for shipping one never exercises the sandbox, which removes the entire reason for shipping one
(§2). `--filesystem=xdg-pictures` is narrower and would be tempting, but it is (§2). `--filesystem=xdg-pictures` is narrower and would be tempting, but it is
still a static grant that lets a typed path resolve — it makes the same design still a static grant that lets a path resolve without the portal — it makes
work by not testing it, only in a smaller directory. the same design work by not testing it, only in a smaller directory.
So the manifest grants no filesystem access at all. The consequence is stated So the manifest grants no filesystem access at all, and a Flatpak built from
plainly: **a Flatpak built from this manifest can open photographs handed to it it reaches the user's photographs only through what the portal hands it.
and cannot yet be pointed at a library.**
### What closes it ### What closes it
Two changes, in this order: 1. **Build it and run it.** `flatpak-builder` is not installed on the machine
this is developed on, so the manifest has never produced a package.
1. **A portal file chooser behind a platform seam.** `ashpd`'s 2. **Removable volumes.** There is no portal for "list the mounted cards".
`OpenFileRequest` with `directory(true)` returns a URI the document portal Under a sandbox `imports_supported()` should report the same `false` it
has exported, which the sandbox can read and which stays valid across reports on Android, for the same reason — the volume list cannot be right
restarts. It resolves to a real path under `/run/user/$UID/doc/`, so however hard the user tries — and leave `Browse…` as the way to a card.
`normalise_endpoint` accepts it as it stands — `canonicalize()` on a fuse
path returns the path itself. The seam matters more than the crate: this
belongs beside `LocalStorage::grant` in `dr-plat`, which is already the one
place a `Path` enters the application, and must not become a second way for
`ui/` to learn about paths.
2. **Removable volumes through the same door.** There is no portal for "list
the mounted cards". The honest answer is that under a sandbox
`imports_supported()` should report the same `false` it reports on Android,
for the same reason it gives there — the operation cannot be performed
however hard the user tries — and the import flow should offer the folder
chooser instead of a volume list.
**Done when:** a Flatpak built from **Done when:** a Flatpak built from
[`packaging/flatpak/paris.tourolle.darkroom.yml`](../../packaging/flatpak/paris.tourolle.darkroom.yml), [`packaging/flatpak/paris.tourolle.darkroom.yml`](../../packaging/flatpak/paris.tourolle.darkroom.yml),
with its `finish-args` unchanged and no `flatpak override` applied, can select a with its `finish-args` unchanged and no `flatpak override` applied, can select a
library root, scan it, and write a sidecar back into it. library root, scan it, write a sidecar back into it, and open it again after a
restart.
### Running a Flatpak build before then ### Running a Flatpak build before then
For testing the rest of the application inside the sandbox, grant the access If the portal's path turns out not to hold across a restart, the rest of the
application can still be tested inside the sandbox by granting the access
per-installation rather than in the manifest, so the file that describes the per-installation rather than in the manifest, so the file that describes the
application keeps telling the truth: application keeps telling the truth:
docs/dev/faces.md
+34 -2
View File
@@ -1575,5 +1575,37 @@ It is a match, not an update in place, and that is why the per-face repairs exis
detection: where nothing about a face but one field needs doing, `record_updates` keeps the id and detection: where nothing about a face but one field needs doing, `record_updates` keeps the id and
there is nothing to judge. there is nothing to judge.
The merge's `match_faces` still matches by overlap alone across devices. It is the same question, Since #77 (0.18.0) the merge's `match_faces` answers it too, within a photograph's `file_id` and
and the same answer would serve it; it is not changed here. one embedder: box IoU ≥ 0.5, unique on both sides, first; then, only for photographs where a remote
face is left over and a local face is free, embedding cosine ≥ 0.7, mutual best, with a lead of
≥ 0.2 over the runner-up on both sides. A box match is never overruled by a low cosine (about 150
genuine cross-device pairs of tiny faces score below 0.45). On the reference desktop/tablet pair this
recovers 20 of 631 unmatched faces with no false matches; the rest are faces one device alone found.
The merge also keeps one person to one face per photograph: an incoming assignment is refused when
another local face already holds that person, unless it is a remote confirmation over a local
suggestion, which moves the suggestion. Refusals are counted in `faces_one_per_photograph`.
The threshold differs from `SAME_FACE_COSINE` (0.45) above on purpose: re-detection additionally
requires the boxes to overlap, while the merge's embedding route exists for boxes that don't.
## 19. Deduplicating people · 2026-09-26
`dr_catalog::dedup_people::run` runs after every successful sync merge (`sync::merge_remote`, on the
sync worker), in one transaction, and logs one `dedup:` line (#78).
**People.** Named people with the same name, trimmed and case-folded, merge into the one with the
most confirmed faces (ties go to the smaller uuid) when every shared embedder's confirmed-face
centroids agree at cosine ≥ 0.7 (distance < 0.3). Each side needs at least two confirmed faces to
compare; a namesake holding no faces merges outright; a face confirmed as one and rejected as the
other keeps them apart; unnamed and set-aside people are never touched. On the reference library the
same-person centroid median is 0.91, and different named people have a 99.9th percentile of 0.41.
**Faces.** Two faces in the same image and embedder with IoU ≥ 0.5 and cosine ≥ 0.7 are one: the
job keeps the stronger detector's face (`FaceDetector::outranks`), then the confirmed one, then the
lower id, and it takes both faces' assignment and rejections.
**Propagation.** The merge is `faces::merge_people`, whose `merged_into` redirect a 0.17.0 peer
already honours, so an older device never resurrects the duplicate. The job also follows redirects
left by earlier manual merges, moving this device's own assignments onto the person kept, and
breaks a mutual redirect at the smaller uuid, which every device computes alike. A merge now also
carries the merged-away person's rejections to the person kept.
docs/dev/frame-budget.md
+81
View File
@@ -264,6 +264,14 @@ base, computed at reduced resolution and correct at any moment the user stops.
"Render coarse while dragging, sharpen when it settles" would paper over the same "Render coarse while dragging, sharpen when it settles" would paper over the same
34 ms with a visible swap. Fix the stage. 34 ms with a visible swap. Fix the stage.
> **Since, 0.15.0.** The stage was fixed (§ The reduced base, below), and the
> draft was built anyway, in the form this section would accept: the develop
> view renders at half resolution while a gesture moves and once at full
> resolution 120 ms after it stops (`ui/dr-ui/src/refine.rs`), the histogram
> dims while it describes an older frame, and the last draft fades out over
> 150 ms rather than being swapped. It is not a mask over a slow stage; it
> spares a drag the full-resolution frames it does not need.
--- ---
## Which GPU, on a machine with more than one ## Which GPU, on a machine with more than one
@@ -431,3 +439,76 @@ difference between a quarter-scale and a half-scale base to 0.03 stops of peak
excursion and 2% of frame reach. But it is a change in reach rather than only excursion and 2% of frame reach. But it is a change in reach rather than only
in cost, it is what a tile scheduler would be handed, and it is worth knowing in cost, it is what a tile scheduler would be handed, and it is worth knowing
that the number moved rather than discovering it later as a seam. that the number moved rather than discovering it later as a seam.
---
## The fit view, again — 2026-09-25
**Status:** Measured in the commits named, not re-run for this file.
Two changes to the fused path made the `fit` rows above cheaper again,
each with its before and after in its commit message. Both were measured
on the laptop RTX 3050 with its clocks held at 420/810 MHz by the power
cap, on the synthetic 60 MP source of `examples/frame_budget.rs`, median of
five alternated runs; both leave the rgba8 output bit-identical.
**The source gather is read once per framing** (`1dc7b45`). At fit every
output pixel reads one texel on a stride through a source three or four
times its width, and that gather was most of the fused pass. The pass now
keeps a render-sized `rgba16float` cache of it, keyed on the framing, and
reads it back while only the adjustments move:
| scene | before | after |
|---|---:|---:|
| neutral, 2560 × 1600 fit | 10.62 ms | 3.88 ms |
| neutral, 3840 × 2160 fit | 21.05 ms | 7.11 ms |
| clarity, 3840 × 2160 fit | 42.20 ms | 27.88 ms |
| neutral, 2560 × 1600 1:1 (control) | 3.83 ms | 3.84 ms |
An interpolated read (straightening, lens warps, CA) is not cached, and the
cache is written on the second frame with a given key, so a crop or zoom
drag pays nothing for it.
**A detail pass that changes nothing is dropped** (`d430ec9`). Capture
sharpening at a scale too coarse to draw its radius emits an empty pass,
which cost a full read and write when another neighbourhood operation
followed it: sharpen with clarity at 2560 × 1600 fit went from 18.66 ms to
14.16 ms, and at 3840 × 2160 from 37.93 ms to 27.88 ms.
## Dehaze in two passes — 2026-09-26
**Status:** Measured in the commit named, not re-run for this file.
**Dehaze erodes each axis in one pass and recovers in the second**
(`bee5c58`, #74). It was five passes — a run and a span erosion along x,
the same along y, and the recovery — and cost 22.9 ms of a 2560 × 1600
frame on the laptop RTX 3050, 54.1 ms at 3840 × 2160, with the memory
clock held at 810 MHz by the power cap. At those clocks a detail pass costs
what it reads and writes rather than what it taps: a pass with an empty
body, one render-sized `rgba16float` read and write, measured 4.0 ms, and
each dehaze pass 4.4–4.6 ms, so the taps were about 2 ms of the 22 and the
four hand-offs between passes were the rest. Each axis now takes the
minimum over its whole window directly, and the recovery rides in the y
pass, which already holds the veil and the pixel's own colour: 36 texture
reads a pixel in place of 12, nearly all cache hits, and two passes in
place of five.
The same synthetic 60 MP source, only a detail parameter moving so the
fused pass is reused, 30 frames a scene after six of warm-up, five runs of
each binary alternated, median of the per-run p50:
| scene | before | after |
|---|---:|---:|
| dehaze, 2560 × 1600 fit | 22.88 ms | 9.06 ms |
| dehaze, 2560 × 1600 1:1 | 23.41 ms | 9.52 ms |
| dehaze, 3840 × 2160 fit | 54.09 ms | 28.12 ms |
| all five detail operations, 2560 × 1600 fit | 53.11 ms | 39.97 ms |
| all five detail operations, 2560 × 1600 1:1 | 67.48 ms | 56.42 ms |
| every operation with film, 2560 × 1600 fit | 57.59 ms | 44.19 ms |
| every operation with film, 2560 × 1600 1:1 | 71.83 ms | 57.93 ms |
The five detail operations are noise reduction, sharpening, clarity,
texture and dehaze; the scenes without dehaze moved within ±2%. The
picture is the same bits: a minimum is exact in any order, the window is
the one the split passes covered, and the rgba8 output hashed identically
before and after in all 64 scene, view and size combinations measured.
docs/dev/outstanding.md
+141 -61
View File
@@ -28,6 +28,16 @@ Android memory pressure and lost-root recovery (FR-PLAT-AND-5, FR-PLAT-AND-2), i
recorded where it appears rather than deleted, because a requirement that is *half* met is the recorded where it appears rather than deleted, because a requirement that is *half* met is the
one most likely to be reported as closed. one most likely to be reported as closed.
**Swept again on 2026-09-26, for 0.16.0.** Progressive refinement (FR-DSP-4) and the panorama
(§3.11) were built and are struck below; consolidating duplicate originals (FR-CAT-11a) now sits
beside re-import detection; the accessibility and localisation counts in §6 had not been read
against the tree since 2026-08-30 and are replaced; and §4a records what the develop and keyboard
work of 0.15.0 and 0.16.0 left open.
**And again for 0.17.0.** Folders are chosen through the platform's dialogue now, so FR-PLAT-LIN-3
in §5 is rewritten around what the Flatpak has still not shown; albums brought the first SAF code,
which FR-PLAT-AND-1's entry now describes, along with why its new tag overstates it.
--- ---
## 1. Plugins — post-v1 since 2026-09-19 ## 1. Plugins — post-v1 since 2026-09-19
@@ -80,8 +90,8 @@ somebody reads the matrix.
## 2. Culling — the stated differentiator, half built ## 2. Culling — the stated differentiator, half built
[D11](requirements.md) names culling "the core differentiator". FR-CULL-1, -2, -3, -4 and -8 through [D11](requirements.md) names culling "the core differentiator". FR-CULL-1 through -5 and -8 through
-12 are built. Three are not. -13 are built. Two are not.
**FR-CULL-3 — Raw-truth overlays. Built, all three bullets.** Focus peaking is **FR-CULL-3 — Raw-truth overlays. Built, all three bullets.** Focus peaking is
`core/dr-gpu/src/focus.rs` and `ui/dr-ui/src/peaking.rs`; the raw histogram and the raw clipping `core/dr-gpu/src/focus.rs` and `ui/dr-ui/src/peaking.rs`; the raw histogram and the raw clipping
@@ -115,8 +125,13 @@ labels — that comment was a forward reference and is now simply wrong, rather
And `dr_catalog::bursts::choose_representative` is written and tested but bound to no gesture, so And `dr_catalog::bursts::choose_representative` is written and tested but bound to no gesture, so
today the only override is expanding the burst. today the only override is expanding the burst.
`core/dr-catalog/src/dedup.rs` remains a different thing: re-import detection under FR-CAT-11, Neither is deduplication, which is about one file held twice rather than two frames that look
matching a file against one already catalogued, not two photographs against each other. alike. `core/dr-catalog/src/dedup.rs` is re-import detection under FR-CAT-11, matching a file on
a card against one already catalogued. Its other half, FR-CAT-11a, is built since 0.16.0:
`core/dr-catalog/src/duplicates.rs` groups the copies a library already holds (same root, camera,
capture instant and size), `ui/dr-ui/src/duplicates.rs` proves each group the same by digest and
compares their edits, and a group is folded onto one survivor with the others trashed in one
transaction, from the Duplicate originals review in the sidebar and in Settings.
**FR-CULL-6 — Compare and survey.** Absent. No side-by-side view, no synchronised zoom or pan. **FR-CULL-6 — Compare and survey.** Absent. No side-by-side view, no synchronised zoom or pan.
This is the one of the four with no adjacent machinery at all, and it is also the one that most This is the one of the four with no adjacent machinery at all, and it is also the one that most
@@ -166,10 +181,15 @@ Two measurements say it costs more than it saves on the interactive path. Neithe
about the export path or about a device under memory pressure, which is where the case for it about the export path or about a device under memory pressure, which is where the case for it
actually lives — and that is spike S6, which has not run. actually lives — and that is spike S6, which has not run.
**FR-DSP-4 — Progressive refinement.** Unbuilt. FR-DSP-1's proxy rendering and TD-4's **FR-DSP-4 — Progressive refinement. Built in 0.15.0.** While a gesture moves the canvas renders
quarter-resolution base are adjacent and are not it: both are fixed choices about what resolution to a half-resolution draft, and the sharp frame lands once, 120 ms after the last movement: the
compute at, where FR-DSP-4 asks for a first frame that is deliberately cheap and a second that decision is `ui/dr-ui/src/refine.rs`, a debounce whose every draft re-arms the settle timer, driven
replaces it. Nothing tracks a "this frame is provisional" state. through simulated timelines in its tests. The draft flag reaches the interface (`canvas-draft`,
`Levels.provisional`), so the histogram dims while it describes a frame older than the one on
screen, and the last draft is kept and faded out over 150 ms when the sharp frame arrives, so
refinement is not a swap. Draft frames themselves date from 2026-08-09; what this entry missed,
and 0.15.0 finished, was a settle that waited for the gesture to stop, a provisional state the
interface could see, and a refinement that was not a jarring swap.
**NFR-RES-2 — Images larger than GPU memory.** Half answered. NFR-R8's "decide explicitly" was **NFR-RES-2 — Images larger than GPU memory.** Half answered. NFR-R8's "decide explicitly" was
decided on 2026-09-19: there is no CPU render pipeline, the degraded mode is the viewer on decided on 2026-09-19: there is no CPU render pipeline, the degraded mode is the viewer on
@@ -181,28 +201,67 @@ settle both this and FR-DSP-2, and there is no evidence it has run.
--- ---
## 4a. Develop, masks and the keyboard — what the 0.15.0 and 0.16.0 work left open
Most of what these two releases built closed a clause outright, and is listed here only so that
nobody looks for it below: perspective correction (FR-DEV-20, Vertical and Horizontal in Compose),
the crop that says when it orphans a mask (FR-DEV-17), intersection as a third mask join
(FR-DEV-19a), the keyboard vocabulary for develop (FR-DEV-16, met and held by `gestures-check` in
both directions), colour labels set, shown and filtered (NFR-A11Y-3's first example), the RAW
decoder behind a trait (FR-RAW-2), and duplicate originals (FR-CAT-11a, §2 above). What they left:
- **FR-UI-5's pointer half.** Keys cover navigation, rating and common adjustments in every view,
and judging works in develop on the open photograph; but no develop slider takes the scroll
wheel, which the clause names. Its status note in the register says so rather than rounding up.
- **FR-RAW-2's second decoder.** The trait is built and a stub decoder is proved to reach the
scan, the preview ladder and export without a caller changing; LibRaw itself (D2) is not
built, and S7 is what would say when it is needed.
- **FR-DEV-19's M2 remainder** ([mask-editing.md §11](mask-editing.md)). The panel's Add, Subtract
and Intersect buttons join a painted part only, so an intersection with a gradient or a range is
painted where the mask should survive; per-part distance fields and folding two layers are
unbuilt. From M1, the brush has no ring drawn on the photograph and a layer's whole stroke
history is redrawn per dab.
- **FR-DEV-17's blind spot, by design.** A layer that adds a range or a region selection is never
reported, because the check has no label map to measure it with (`dr_pipeline::orphan`); a
false alarm on the common path would teach the notice to be dismissed unread.
The desktop's scrollbars (the develop column, the grid, the sidebar, Settings, the film list and
the help sheet) are drawn only where `dr_plat::is_touch_first()` is false; on Android the lists
still scroll by flick alone, which is deliberate rather than outstanding.
---
## 5. Android beyond running, and Flatpak ## 5. Android beyond running, and Flatpak
The Android app is not a stub — it builds an APK, runs the whole application, unpacks bundled face The Android app is not a stub — it builds an APK, runs the whole application, unpacks bundled face
models, and has been measured on a tablet ([faces.md §12.1](faces.md), models, has been measured on a tablet ([faces.md §12.1](faces.md)), and since 0.15.0 draws the
[technical-debt.md TD-1](technical-debt.md)). What is missing is the platform contract around it. develop view zero-copy as the desktop does ([technical-debt.md TD-1](technical-debt.md), paid off).
It carries the manual and opens it in a WebView. What is missing is the platform contract around it.
**FR-PLAT-AND-1 is untagged, and what it was tagged for was intent rather than code.** **FR-PLAT-AND-1 — SAF is built for export folders, not for the library.** The requirement demands
The requirement demands that library access be obtained *exclusively* through the Storage Access that library access be obtained *exclusively* through the Storage Access Framework. Until 0.17.0
Framework. There is no SAF code: no `ACTION_OPEN_DOCUMENT_TREE`, no `takePersistableUriPermission`, there was no SAF code at all, and the requirement's two tags rested on a `SourceRef::Document`
no `DocumentsContract`. Its two tags rested on a `SourceRef::Document` variant constructed only variant constructed only inside `#[cfg(test)]` and on `dr_plat::imports_supported`, which *returns
inside `#[cfg(test)]` — `LocalStorage::open` refuses it, and the test that proves so is named false on Android* and whose own documentation says it "stops being false when a SAF implementation
`a_reference_of_the_wrong_kind_is_refused_rather_than_guessed_at` — and on lands". Both were removed as intent rather than code — the "plumbing a future feature would use"
`dr_plat::imports_supported`, which *returns false on Android* and whose own documentation says it case [CONTRIBUTING.md](../../CONTRIBUTING.md) and [code-health.md CH-4](code-health.md) both warn
"stops being false when a SAF implementation lands". The second tag documented the absence of the about.
thing it was counted as evidence for. Both have been removed; this is the "plumbing a future feature
would use" case [CONTRIBUTING.md](../../CONTRIBUTING.md) and [code-health.md CH-4](code-health.md) both 0.17.0 brought the first real SAF code, for albums (FR-EXP-10): `FolderPicker.java` starts
warn about. Android reaches a library through a Nextcloud account or a folder, over paths, like the `ACTION_OPEN_DOCUMENT_TREE` from a translucent activity of its own (the main activity is
desktop. `NativeActivity`, whose results are not ours) and takes a persistable grant; `Saf.java` writes each
export through `DocumentsContract`; `ui/dr-ui/src/saf.rs` is the JNI bridge. They shipped tagged
`TRACES: FR-PLAT-AND-1`, which made the matrix count the requirement as covered, and that overstated
it: the mechanism is the one the requirement names, but its subject is the library, and Android
still reaches a library through a Nextcloud account or a folder, over paths, like the desktop. The
tags now say FR-EXP-10 alone (0.17.1), so FR-PLAT-AND-1 reads as uncovered again until the library
itself is reached through SAF.
That has a consequence for the rest of the cluster: **FR-PLAT-AND-2** — detecting the loss of a That has a consequence for the rest of the cluster: **FR-PLAT-AND-2** — detecting the loss of a
granted tree permission and marking images offline rather than deleting rows — cannot be built until granted tree permission and marking images offline rather than deleting rows — is still blocked for
there is a permission to lose. It is listed here as unbuilt, but it is blocked, not skipped. the library, because there is no library grant to lose. An album's folder has a grant now, and
nothing checks for its loss: an export into a folder whose grant has gone fails with whatever the
write raises, rather than the album saying beforehand that it needs a folder again.
**FR-PLAT-AND-4 — half built.** The runner is done (`core/dr-catalog/src/runner.rs`): the **FR-PLAT-AND-4 — half built.** The runner is done (`core/dr-catalog/src/runner.rs`): the
queue that `jobs.rs` always had is now claimed from, completed, failed and recovered after a queue that `jobs.rs` always had is now claimed from, completed, failed and recovered after a
@@ -211,11 +270,13 @@ the platform half — a foreground `Service`, `FOREGROUND_SERVICE` and `POST_NOT
manifest, and a stated Doze behaviour. The build step that blocked it is no longer a blocker: the manifest, and a stated Doze behaviour. The build step that blocked it is no longer a blocker: the
APK now compiles its own Java. APK now compiles its own Java.
Note also that **no handler is registered**, deliberately. The only enqueue site reachable in the Note also that **no handler is registered**, deliberately — and, since #73, nothing enqueues
shipping app produces remote thumbnail jobs already served by the async grid worker, and either. The remote scan's per-photograph `Thumbnail` jobs, and `walk::scan_root`'s `Thumbnail` and
`walk::scan_root` — which holds the other two enqueue sites — has no caller outside an example. `ExtractMetadata` jobs, duplicated debts the thumbnail store and `metadata_state` already record,
Wiring the sweep to claim from the queue is the honest next step and is an async rewrite of and were never claimed; they were removed and `Thumbnail` retired ([catalog.md §6.1](catalog.md)).
`library.rs`. Feeding FR-PLAT-AND-4's platform half is therefore a matter of choosing a kind whose work has no
better source of truth, registering its handler, and enqueuing it in the same change —
`every_queued_kind_has_a_consumer` refuses the enqueue without the handler.
**FR-PLAT-AND-5 — built.** A tiered eviction registry drives GPU caches, then proxies, then **FR-PLAT-AND-5 — built.** A tiered eviction registry drives GPU caches, then proxies, then
thumbnails, from `MainEvent::LowMemory` and `MainEvent::Stop`. thumbnails, from `MainEvent::LowMemory` and `MainEvent::Stop`.
@@ -226,14 +287,20 @@ Intent read over JNI, and an `ExportProvider` rooted at `getFilesDir()` rather t
been exercised on a device** — the tests read the manifest and the Java through `include_str!`, been exercised on a device** — the tests read the manifest and the Java through `include_str!`,
which catches a deleted filter but not a class loader that cannot find the class. which catches a deleted filter but not a class loader that cannot find the class.
**FR-PLAT-LIN-3 — packaged, not satisfied.** There is a Flatpak manifest now, granting no **FR-PLAT-LIN-3 — packaged and wired, not yet proved.** There is a Flatpak manifest, granting no
filesystem permission of any kind, plus AppStream metainfo and `docs/distribution.md`. The filesystem permission of any kind, plus AppStream metainfo and `docs/distribution.md`. Until 0.17.0
requirement is still not met, and cannot be met by packaging: a folder library is chosen by typing the requirement could not be met by packaging: a folder library was chosen by typing an absolute
an absolute path, nothing in the tree calls the FileChooser portal, and inside the sandbox `$HOME` path, nothing called the FileChooser portal, and inside the sandbox `$HOME` holds only
holds only `.var/app/...`. `dr_plat::volumes()` reads `/proc/self/mountinfo`, so a card mounted on `.var/app/...`. Since 0.17.0 every folder the desktop asks for — the library, an import's source
the host is invisible to a sandboxed process as well. The fix is an `ashpd` directory picker beside and second copy, Lightroom presets, an album's folder — is chosen in the platform's dialogue
`LocalStorage::grant`, not a change to the manifest. No Flatpak has been built here — (`ui/dr-ui/src/folder_dialog.rs`, `rfd` over the XDG portal), which is what a sandbox needs. Two
`flatpak-builder` is not installed — so the permission set is reasoned, not observed. things stop this entry being struck. No Flatpak has been built here — `flatpak-builder` is not
installed — so the portal path has never been seen to open a library, reopen it after a restart,
or take a sidecar inside the sandbox; [distribution.md §4](distribution.md) says what has to be
checked. And
`dr_plat::volumes()` still reads `/proc/self/mountinfo`, so a card mounted on the host is invisible
to a sandboxed process; the import page's `Browse…` reaches one through the portal instead. The
chooser landed in `ui/` rather than behind the `dr-plat` seam distribution.md had proposed.
**NFR-COMPAT-2 — distribution channels. Stated, which is all this requirement asks.** The paragraph **NFR-COMPAT-2 — distribution channels. Stated, which is all this requirement asks.** The paragraph
above cites [distribution.md](distribution.md) and it is the same document that answers this: §1 above cites [distribution.md](distribution.md) and it is the same document that answers this: §1
@@ -267,21 +334,29 @@ fixes users will need — is unaddressed, and there is no update mechanism of an
## 6. Accessibility and internationalisation — the hard half is done and the easy half is not ## 6. Accessibility and internationalisation — the hard half is done and the easy half is not
**NFR-A11Y-1 — Localisation.** `@tr(` appears **zero** times across 14,482 lines of Slint. That **NFR-A11Y-1 — Localisation.** One screen of twenty-eight Slint files is converted: the launch
number overstates the problem, because the part that is genuinely architectural was got right: screen wraps its strings in `@tr(`, and `ui/dr-ui/build.rs` records the mechanism — Slint's bundled
`LocalizedKey` keeps display strings out of `core/` entirely, every operation publishes a key rather translations, a `.po` per language under `ui/dr-ui/lang/`, extracted with `slint-tr-extractor` —
than a label, and `labels::resolve` is the single point where a key becomes text. What that single and why bundling rather than gettext (Android has no path a `.mo` could sit at). No `.po` exists
point does, however, is a hardcoded English `match` in Rust source — so changing a translation yet, so every build is the original English, and the rest of the interface (about 23,600 lines of
requires a recompile, which is the one thing the requirement explicitly forbids. There is no message markup) is unwrapped. The part that is genuinely architectural was got right: `LocalizedKey` keeps
catalogue in any format, no locale-resolution rule, and no decision recorded about RTL. display strings out of `core/` entirely, every operation publishes a key rather than a label, and
`labels.rs` is the single point where a key becomes text — but that point is a hardcoded English
`match`, and `build.rs` says the Rust-side mechanism is not built. Two things the requirement asks
remain against it even when the conversion is finished: a bundled translation is compiled in, so
changing one still needs a rebuild, which the clause forbids; and there is no locale-resolution rule
and no decision recorded about RTL.
The work left is therefore smaller than it looks and entirely mechanical: a catalogue format, a load **NFR-A11Y-2 — Accessibility. Tagged, and half tested.** `accessible-*` properties now appear on
path behind `resolve`, and `@tr(` around the Slint literals. The design it needs already exists. some 127 lines across eleven Slint files — the shared controls in `widgets.slint` and
`controls.slint`, every slider, the rail, the grid's cells (named by file), the rating stars, the
**NFR-A11Y-2 — Accessibility.** `accessible-*` appears five times in the whole interface, all five labels and the duplicates review — and `ui/dr-ui/tests/ui_controls_are_accessible.rs` fails when a
on one control — the parameter slider in `adjust.slint` — and nothing is set from the Rust side at shared control stops declaring its role or a slider is added without a name. The manual's recording
all. Everything else in eighteen Slint files is unnamed to AT-SPI and TalkBack. The requirement's own scripts find every control they press through the same names (`dr_ui::automation`), so a control a
caveat, that Slint's Android accessibility needs verifying, is spike S13, which has not run. screen reader cannot name is also one the manual cannot record. What is unchecked is stated in the
test: whether the words are good, and whether Android exposes any of it — spike S13, which has not
run. The requirement's other two clauses are debt with entries of their own: platform font scaling
([TD-7](technical-debt.md)) and WCAG AA contrast ([TD-6](technical-debt.md)).
**NFR-A11Y-3 — Colour-independent status.** Built, and now asserted. The clipping readout pairs a **NFR-A11Y-3 — Colour-independent status.** Built, and now asserted. The clipping readout pairs a
marker that appears or disappears with a figure in words; the rating strip is a solid star against marker that appears or disappears with a figure in words; the rating strip is a solid star against
@@ -438,7 +513,7 @@ requirement text that asks for it:
|---|---|---| |---|---|---|
| S6 | FR-DSP-2, NFR-RES-2 — tiling and images larger than GPU memory | Nothing; needs a device and a large image | | S6 | FR-DSP-2, NFR-RES-2 — tiling and images larger than GPU memory | Nothing; needs a device and a large image |
| S9 | R1's tolerance threshold, and therefore R1 | Nothing; the threshold is defined *by* running it | | S9 | R1's tolerance threshold, and therefore R1 | Nothing; the threshold is defined *by* running it |
| S10 | Whether SAF at 10k files meets NFR-P1/P3 | §5 — there is no SAF code to measure | | S10 | Whether SAF at 10k files meets NFR-P1/P3 | §5 — SAF reaches album folders only, never a library to enumerate |
| S11 | NFR-COMPAT-2, and whether Play makes SAF binding | Nothing | | S11 | NFR-COMPAT-2, and whether Play makes SAF binding | Nothing |
| S13 | NFR-A11Y-2 on Android | §6 — there is almost nothing to test with | | S13 | NFR-A11Y-2 on Android | §6 — there is almost nothing to test with |
@@ -453,18 +528,23 @@ whether something *should* be built — which is the opposite of the order §9 a
--- ---
## 11. Merging — specified 2026-09-19, nothing built ## 11. Merging — the panorama built, three clauses open
§3.11 was written on 2026-09-19 under D18, undeferring the panorama from §7 and leaving HDR merge §3.11 was written on 2026-09-19 under D18, undeferring the panorama from §7 and leaving HDR merge
and focus stacking there with their data model decided. Eleven `FR-MRG` clauses and two `NFR-MRG` and focus stacking there with their data model decided. Its clauses entered the register with no
figures entered the register at once with no code behind any of them, which is why the coverage code behind them, which is why the coverage figure fell from 83.0% to 77.2% on that day — and the
figure fell from 83.0% to 77.2% on the same day — a specification, not a regression. panorama was then built in the same week: alignment, projections, the chunked composite written as
a DNG beside its sources, the auto-crop and the model's border fill.
[panorama.md](panorama.md) §11 and §13 are where it stands.
[panorama.md](panorama.md) is the design, and its §10 is the order of work. Nothing starts before Three clauses carry no tag:
**S15**: whether rawler reads back a linear DNG the application writes, whether XFeat loads under
tract at a fixed shape, whether the working-space texture can be tapped where FR-MRG-2 needs it, - **FR-MRG-9 — the tablet.** Android runs the same code, but the stated ceiling on frame count and
and what a chunked blend of a 100 MP composite costs on the tablet. The first two are a day each source resolution, refused with a message rather than an out-of-memory kill, is not set.
and either can change the design, which is the reason they come first. - **NFR-MRG-1 — merge latency.** Measured on the desktop and inside its 60 s (panorama.md §11);
the tablet's figure is S15.4's open half, and nothing asserts either per commit.
- **NFR-MRG-2 — reproducible merges.** Nothing checks that the same sources and settings give a
byte-identical composite.
## 12. D12, which governed all of the above ## 12. D12, which governed all of the above
docs/dev/requirements.md
+57 -2
View File
@@ -179,6 +179,17 @@ card into the library. Both are needed.
original filename) and by content hash, offering skip or import-as-new. Camera filenames wrap at original filename) and by content hash, offering skip or import-as-new. Camera filenames wrap at
`IMG_9999`, so filename alone is insufficient. Existing catalog duplicates are detectable on demand. `IMG_9999`, so filename alone is insufficient. Existing catalog duplicates are detectable on demand.
**FR-CAT-11a — Consolidating catalog duplicates.** Files already in the library more than once
(same root, camera, capture time and size) shall be listed on demand as groups, from the library
and from Settings. Before anything moves, each group shall be proved the same file by a stored full
digest or by a digest of the first and last megabyte of every copy, and its copies' develop edits
compared; a group that differs in either is left out and the review says why. One copy survives —
not under a backup-looking folder, then camera-named, then oldest, overridable per group — and the
others' collections, keywords, highest rating, agreed flag and label, and faces are merged onto it,
with disagreements reported rather than decided silently. Each group is merged and its other copies
moved to the trash (FR-CAT-15) in one transaction: wholly consolidated or untouched. Nothing is
deleted.
**FR-CAT-12 — Versions (virtual copies).** An image may carry multiple named `Version`s, each with **FR-CAT-12 — Versions (virtual copies).** An image may carry multiple named `Version`s, each with
an independent edit graph, without duplicating source data. Versions are creatable, nameable, an independent edit graph, without duplicating source data. Versions are creatable, nameable,
deletable, and independently exportable; one is the default. deletable, and independently exportable; one is the default.
@@ -491,6 +502,13 @@ so history survives a restart. Named snapshots of an edit state.
**FR-DEV-6 — Presets.** Save, apply, and manage named presets covering a subset of the edit **FR-DEV-6 — Presets.** Save, apply, and manage named presets covering a subset of the edit
graph. Copy/paste settings between images. Batch-apply to a selection. graph. Copy/paste settings between images. Batch-apply to a selection.
A preset may name a film stock, which travels with the film node's own parameters. The
application ships a read-only collection of presets, updated with each release and never
written into the user's library; a user preset saved under a shipped name overrides it
until deleted or renamed. Shipped and imported presets change only the operations they
name, so a look applied to a corrected photograph keeps the correction; a copy or a saved
edit replaces everything in scope.
**FR-DEV-7 — Before/after.** Compare current edit state against the unedited original or against **FR-DEV-7 — Before/after.** Compare current edit state against the unedited original or against
a chosen history state. a chosen history state.
@@ -656,6 +674,12 @@ asynchronously, and the proxy result remains on screen until it is ready.
quality or resolution, refining to full quality when interaction settles. Refinement is visually quality or resolution, refining to full quality when interaction settles. Refinement is visually
smooth, not a jarring swap. smooth, not a jarring swap.
*Status (2026-09-26).* Met in 0.15.0. A gesture renders half-resolution drafts, and the sharp frame
lands once, 120 ms after the last movement rather than on a timer counted from the first
(`ui/dr-ui/src/refine.rs`). While a draft is up the histogram's display reading is dimmed, since it
describes the last settled frame; the last draft is kept over the sharp one and faded out in
150 ms, so the refinement is not a swap.
**FR-DSP-5 — Zoom and pan.** Fit, 1:1, and arbitrary zoom levels. At 1:1 and above, the pipeline **FR-DSP-5 — Zoom and pan.** Fit, 1:1, and arbitrary zoom levels. At 1:1 and above, the pipeline
operates on the visible crop at full source resolution. operates on the visible crop at full source resolution.
@@ -843,8 +867,16 @@ full-size TIFF alongside a 2048px sRGB JPEG.
**FR-EXP-6 — Naming and destination.** Output filenames are generated from a template supporting at **FR-EXP-6 — Naming and destination.** Output filenames are generated from a template supporting at
minimum: original filename, sequence number, capture date, export dimensions, and preset name. minimum: original filename, sequence number, capture date, export dimensions, and preset name.
Collision policy (overwrite / skip / auto-increment) is configurable. Destinations include a local Collision policy (overwrite / skip / auto-increment) is configurable. The destination is an album
path and a Nextcloud remote path (FR-NC-7). (FR-EXP-10), whose folder is on this device or on the library's server (FR-NC-7) — never inside the
library itself, where a scan would catalogue the exported files as photographs.
A folder is chosen by pointing, never by typing a path: the platform's own dialogue on the desktop
(the XDG desktop portal on Linux, which reaches the user's files from inside the Flatpak; the common
item dialogue on Windows), the Storage Access Framework's tree picker on Android, and an in-app
browser for a folder on the server. Every one of them can make a new folder as well as open one.
The same applies to the other folders the application asks for on the desktop — the library folder,
an import's source and second copy, and presets brought across from Lightroom.
**FR-EXP-7 — Batch export.** Export a selection with one or more presets, running in the background **FR-EXP-7 — Batch export.** Export a selection with one or more presets, running in the background
with progress and cancellation. Uses all available cores and the GPU. A failure on one image is with progress and cancellation. Uses all available cores and the GPU. A failure on one image is
@@ -857,6 +889,20 @@ strip GPS and personal metadata. Copyright and contact fields are settable per-p
regardless of what the display was showing — including the high-quality demosaic (FR-RAW-3), never regardless of what the display was showing — including the high-quality demosaic (FR-RAW-3), never
the fast preview method. the fast preview method.
**FR-EXP-10 — Albums.** An album is a named export destination listed in the library sidebar
beneath the collections. Its folder holds only the exported files; the catalog records, for each
file written there, the image it was rendered from, so that selecting the album shows the originals
behind its files and re-exporting after an edit is one selection away. The export sheet chooses an
album by name, and an export with no album chosen is refused and says so.
Albums sync between devices as collections do (FR-CAT-7): by uuid and revision, deletions as
tombstones, exports as a set union keyed on the server's file id. A folder on the server syncs with
the album; a folder on the device does not, because a path or a SAF grant on one device means
nothing on another — an album made elsewhere arrives with no folder here until one is chosen.
Deleting an album never deletes the files in its folder. The album tables are created on first use
rather than by a schema migration, so a device on an older build keeps merging the rest of the
catalog.
### 3.7 Nextcloud integration ### 3.7 Nextcloud integration
Mechanics below are verified against Nextcloud 34 documentation and server/desktop-client source. Mechanics below are verified against Nextcloud 34 documentation and server/desktop-client source.
@@ -1141,6 +1187,12 @@ protocol is unavailable, FR-DSP-8's stated fallback applies.
portals and credential storage uses the Secret Service portal, both verified to satisfy FR-NC-2 and portals and credential storage uses the Secret Service portal, both verified to satisfy FR-NC-2 and
FR-CAT-1 within the sandbox. FR-CAT-1 within the sandbox.
*Status (2026-09-26).* Not verified, because no Flatpak has been built. Since 0.17.0 every folder the
desktop asks for is chosen through the FileChooser portal (FR-EXP-6), so the filesystem half has the
code it needs; whether the path the portal returns opens, persists and takes a sidecar inside the
sandbox is unobserved ([distribution.md §4](distribution.md)). Credentials reach the session's secret
daemon through a talk hole rather than the Secret Service portal, for the reason the manifest gives.
#### Windows #### Windows
Specified in [windows.md](windows.md); a stated channel under NFR-COMPAT-2, not a v1 one. Specified in [windows.md](windows.md); a stated channel under NFR-COMPAT-2, not a v1 one.
@@ -2163,6 +2215,9 @@ public channel — and it is the position until those weights are replaced.
| Android | Signed release APK, sideloaded (`docker/android/assemble-apk.sh`, the release key of 2026-09-11) — **not** Play, **not** F-Droid | CI | | Android | Signed release APK, sideloaded (`docker/android/assemble-apk.sh`, the release key of 2026-09-11) — **not** Play, **not** F-Droid | CI |
| Windows | NSIS per-user installer cross-built from Linux (`docker/windows/`, FR-PLAT-WIN-3) | CI | | Windows | NSIS per-user installer cross-built from Linux (`docker/windows/`, FR-PLAT-WIN-3) | CI |
*Status (2026-09-26).* The Flatpak row is the decision, not yet the state: no workflow under
`.gitea/workflows/` builds it, and none has been built by hand (FR-PLAT-LIN-3).
Two consequences the channel decision has on the requirements above it. ARCH §6.9's SAF-only Two consequences the channel decision has on the requirements above it. ARCH §6.9's SAF-only
storage model was written because Play would reject anything else; a sideloaded build *could* storage model was written because Play would reject anything else; a sideloaded build *could*
request broader permissions, and FR-PLAT-AND-1 keeps SAF anyway, because the design is right on request broader permissions, and FR-PLAT-AND-1 keeps SAF anyway, because the design is right on
docs/dev/storage.md
+69 -2
View File
@@ -85,6 +85,9 @@ pub trait RemoteBackend: Send + Sync {
// transfer // transfer
async fn get(&self, id: &RemoteId, range: Option<Range<u64>>) async fn get(&self, id: &RemoteId, range: Option<Range<u64>>)
-> Result<Vec<u8>, RemoteError>; -> Result<Vec<u8>, RemoteError>;
async fn get_reporting(&self, id: &RemoteId, // defaulted
progress: &(dyn Fn(u64, Option<u64>) + Send + Sync))
-> Result<Vec<u8>, RemoteError>;
async fn put(&self, path: &RemotePath, body: Vec<u8>, precond: Option<Precondition>) async fn put(&self, path: &RemotePath, body: Vec<u8>, precond: Option<Precondition>)
-> Result<Validator, RemoteError>; -> Result<Validator, RemoteError>;
async fn put_many(&self, items: Vec<(RemotePath, Vec<u8>)>) // defaulted async fn put_many(&self, items: Vec<(RemotePath, Vec<u8>)>) // defaulted
@@ -111,6 +114,16 @@ Rules that are not obvious from the signatures:
- **`get` takes an optional range, and it is a hint.** A backend without cheap - **`get` takes an optional range, and it is a hint.** A backend without cheap
ranges may return the whole object; the caller slices. Correctness holds ranges may return the whole object; the caller slices. Correctness holds
either way and `Capabilities::range_reads` says whether it was cheap. either way and `Capabilities::range_reads` says whether it was cheap.
- **`get_reporting` is for the one transfer somebody is watching.** An original
opened in develop is tens of megabytes, and "downloading" alone for that long
reads as stuck. It reports the bytes received and the length the server
declared, if it declared one. The default is `get` whole and one report at the
end, which is right for a backend whose read is local; Nextcloud overrides it
to read the body chunk by chunk. The develop view reads the figures by path
from the in-flight registry in `ui/dr-ui/src/library/thumbnails_fetch.rs`,
because a step along the roll usually lands on a frame the prefetcher is
already fetching, and falls back on the catalog's file length when no length
was declared.
- **Chunked upload is not in the trait.** It is an implementation detail of - **Chunked upload is not in the trait.** It is an implementation detail of
`put`, chosen by body size. Exposing it would leak one server's protocol. `put`, chosen by body size. Exposing it would leak one server's protocol.
- **`move_to` must preserve identity where the backend has stable ids.** This - **`move_to` must preserve identity where the backend has stable ids.** This
@@ -118,7 +131,9 @@ Rules that are not obvious from the signatures:
allocates a new id, orphaning the thumbnail shard and turning a restore into a allocates a new id, orphaning the thumbnail shard and turning a restore into a
full re-download. full re-download.
- **`create_dir` makes parents and succeeds if the directory exists.** Callers - **`create_dir` makes parents and succeeds if the directory exists.** Callers
use it to guarantee a destination, not to claim they created one. use it to guarantee a destination, not to claim they created one. It is also
the server browser's `New folder` (§5.3), which then lists the parent again
rather than inserting the name it asked for: the server may have normalised it.
### 3.2 `Capabilities` — what is cheap ### 3.2 `Capabilities` — what is cheap
@@ -195,6 +210,7 @@ pub trait BackendProvider: Send + Sync {
fn endpoint_placeholder(&self) -> &'static str; fn endpoint_placeholder(&self) -> &'static str;
fn sign_in(&self) -> SignIn; fn sign_in(&self) -> SignIn;
fn normalise_endpoint(&self, input: &str) -> Result<String, String>; fn normalise_endpoint(&self, input: &str) -> Result<String, String>;
fn upgrade_endpoint(&self, stored: &str) -> Option<String>; // defaulted: None
fn account_for(&self, endpoint: &str) -> Result<Account, RemoteError>; // defaulted fn account_for(&self, endpoint: &str) -> Result<Account, RemoteError>; // defaulted
fn connect(&self, conn: &Connection) -> Result<Box<dyn RemoteBackend>, RemoteError>; fn connect(&self, conn: &Connection) -> Result<Box<dyn RemoteBackend>, RemoteError>;
} }
@@ -215,6 +231,18 @@ pub enum SignIn {
Nextcloud provider upgrades `http://` to `https://` here (`NFR-SEC-3`); the Nextcloud provider upgrades `http://` to `https://` here (`NFR-SEC-3`); the
folder provider canonicalises the path, so two spellings of one directory do folder provider canonicalises the path, so two spellings of one directory do
not become two accounts indexing the same photographs. not become two accounts indexing the same photographs.
- **`upgrade_endpoint` is for accounts already saved,** and is not a second
call to `normalise_endpoint`: the folder provider's needs the path to exist,
so running it on every launch would fail a library on an unplugged disk.
`AccountStore::upgrade_endpoints` runs it at launch; only Nextcloud answers,
rewriting an `http://` account an older build saved to `https://`
(#65). The keyring entry is filed under the endpoint, so the secret is
copied across before the record changes, and a move that would change
`namespace()` is refused rather than performed. Below both, the Nextcloud
client sets `https_only`, so no URL it sends — a typed one, the login flow's
poll endpoint, a redirect — can carry the app password in the clear, and
the login flow keeps the address the user typed rather than the server's
idea of its own URL.
- **`connect` is synchronous and cheap.** It validates configuration and builds - **`connect` is synchronous and cheap.** It validates configuration and builds
a client; it does not talk to the remote. Workers call it per task. a client; it does not talk to the remote. Workers call it per task.
- **`SignIn` is a shape, not a method.** It would be tidier to expose - **`SignIn` is a shape, not a method.** It would be tidier to expose
@@ -281,6 +309,7 @@ for a second backend:
| `bulk_upload` | yes, `POST /remote.php/dav/bulk` | | `bulk_upload` | yes, `POST /remote.php/dav/bulk` |
| `conditional_write` | yes, `If-Match` | | `conditional_write` | yes, `If-Match` |
| `server_previews` | `CommonFormatsOnly` — stock Nextcloud renders no RAW | | `server_previews` | `CommonFormatsOnly` — stock Nextcloud renders no RAW |
| `get_reporting` | overridden: the body is read chunk by chunk, against `Content-Length` |
| sign-in | `SignIn::Browser`, Login Flow v2, system browser, app password | | sign-in | `SignIn::Browser`, Login Flow v2, system browser, app password |
Also kept: the `oc:permissions` probe on a refused `PUT`, which is what Also kept: the `oc:permissions` probe on a refused `PUT`, which is what
@@ -302,7 +331,8 @@ which makes it the route that works on a machine with no secrets daemon at all.
| `bulk_upload` | no | | `bulk_upload` | no |
| `conditional_write` | yes, with a documented residual race | | `conditional_write` | yes, with a documented residual race |
| `server_previews` | `None` | | `server_previews` | `None` |
| sign-in | `SignIn::EndpointOnly` | | `get_reporting` | the default: a local read, reported once when it is done |
| sign-in | `SignIn::EndpointOnly`, the folder chosen in the platform's dialogue (§5.3) |
**Why `LocalEtags` and not `PropagatingEtags`.** A POSIX directory's mtime **Why `LocalEtags` and not `PropagatingEtags`.** A POSIX directory's mtime
changes when its own entry list changes and at no other time — not when a changes when its own entry list changes and at no other time — not when a
@@ -364,6 +394,43 @@ behind a network banner. An endpoint that is not a directory at all maps to
`RemoteError::Configuration`: nothing was unreachable and no credential was `RemoteError::Configuration`: nothing was unreachable and no credential was
wrong, so neither of the other two would send the user anywhere useful. wrong, so neither of the other two would send the user anywhere useful.
### 5.3 Choosing folders, and the folders exports go to
**Folders are pointed at, never typed** (FR-EXP-6). On the desktop,
`ui/dr-ui/src/folder_dialog.rs` asks the platform: the XDG desktop portal's
FileChooser on Linux, through `rfd`, and the common item dialogue on Windows.
That is how the folder connector's endpoint is chosen on the launch screen, and
the path it returns still goes through `normalise_endpoint` as a typed one did.
A folder on the server is chosen in the in-app browser, which lists with `list`
and makes a folder with `create_dir`; the launch screen and the album sheet
share it through `ui/dr-ui/src/remote_folders.rs`, which keeps both round trips
off the interface thread. Android has no filesystem dialogue, so its library
folder is still typed, and an album's folder there comes from the Storage
Access Framework's tree picker.
**An album is where exports go** (FR-EXP-10), and never inside the library: a
JPEG written into the tree a scan catalogues comes back as a photograph beside
the RAW it was made from. Its folder is one of two kinds
(`dr_catalog::albums::Place`):
- **On the server,** relative to the *account* root, not the library root. The
browser refuses a folder inside the library and says why; on a server whose
whole account is the library, every folder is inside it, and it says that
instead. Exports reach it through the outbox like any upload, and a queued
file's `.dest` record gains a third line saying its folder is relative to
the account — a third line rather than a leading slash, because a record
written before albums may carry a stray slash and has to keep the meaning it
was written with.
- **On this device,** a path, or on Android a SAF tree URI with a persisted
grant, written through `DocumentsContract` (`ui/dr-ui/src/saf.rs`). A
provider renames on a collision by itself, so the album records the name it
was given rather than the one asked for.
A server folder lives on the album row and syncs with it. A device folder lives
in `album_folders`, which the merge never reads and the upload snapshot drops,
because a path or a grant on one device means nothing on another: an album made
on the desktop arrives on the tablet with no folder until one is chosen there.
--- ---
## 6. Virtual filesystems ## 6. Virtual filesystems
docs/dev/traceability.md
+111 -109
View File
File diff suppressed because one or more lines are too long
docs/dev/windows.md
+6 -1
View File
@@ -130,8 +130,13 @@ in `core/` is touched, which is NFR-PORT-1 holding.
- **`volumes.rs`** — card detection reads `/proc/mounts` and `/sys/block` under - **`volumes.rs`** — card detection reads `/proc/mounts` and `/sys/block` under
`cfg(target_os = "linux")` and returns an empty list elsewhere. Windows gets no card detection `cfg(target_os = "linux")` and returns an empty list elsewhere. Windows gets no card detection
in this pass; the import flow's path picker still works. (A `GetDriveType`/`DRIVE_REMOVABLE` in this pass; the import page's `Browse…` still reaches a card. (A `GetDriveType`/`DRIVE_REMOVABLE`
implementation is a screen of code and a follow-up.) implementation is a screen of code and a follow-up.)
- **Folder dialogues** — since 0.17.0 every folder the desktop asks for (the library, an import's
source and second copy, Lightroom presets, an album's folder) is chosen in the platform's own
dialogue through `rfd` ([`folder_dialog.rs`](../../ui/dr-ui/src/folder_dialog.rs)), which on
Windows is the common item dialogue rather than the XDG portal. *Not verified*: nothing has
opened one under Wine or on Windows.
- **`display.rs`** — the X11 and Wayland colour-profile readers are `cfg(all(unix, not(android)))`; - **`display.rs`** — the X11 and Wayland colour-profile readers are `cfg(all(unix, not(android)))`;
the fallback is FR-DSP-8's stated one. Windows ICC profiles via `GetICMProfile` are a follow-up the fallback is FR-DSP-8's stated one. Windows ICC profiles via `GetICMProfile` are a follow-up
for the same reason. for the same reason.
docs/gestures.md
+143 -50
View File
@@ -5,7 +5,7 @@
Every entry here is extracted from the comment beside the code that implements it, so this file cannot describe a gesture the application does not have. Add one by writing a `GESTURE:` block next to the implementation; there is nowhere else to write it. Every entry here is extracted from the comment beside the code that implements it, so this file cannot describe a gesture the application does not have. Add one by writing a `GESTURE:` block next to the implementation; there is nowhere else to write it.
67 gestures, in 5 places. 76 gestures, in 7 places.
## Develop ## Develop
@@ -17,7 +17,18 @@ Every entry here is extracted from the comment beside the code that implements i
Sampling a neutral is the first move of the tonal pass — every colour judgement afterwards is measured against where the grey was put — and guessing at two sliders until a wall stops looking green is the wrong way round. One click is one sample and one step to undo; the sliders stay, because a sampled neutral is where the decision starts rather than where it ends. Sampling a neutral is the first move of the tonal pass — every colour judgement afterwards is measured against where the grey was put — and guessing at two sliders until a wall stops looking green is the wrong way round. One click is one sample and one step to undo; the sliders stay, because a sampled neutral is where the decision starts rather than where it ends.
<sub>`ui/dr-ui/ui/adjust.slint:158`</sub> <sub>`ui/dr-ui/ui/adjust.slint:159`</sub>
### Choose a film stock from the keyboard
- **Touch** — Tap the Film row, flick the list, tap a stock
- **Pointer** — Click the Film row, then a stock; the wheel or a drag scrolls the list
- **Keyboard** — With the list open, `↑` and `↓` move along it, `Enter` chooses, and `Escape` or `Back` closes it unchanged
- **See it** — [in the manual](manual/README.md#film)
The list is longer than it is tall, so a way to walk it that cannot be lost to the column's scrolling is part of it being reachable at all. Focus goes back to the photograph's keys when it closes.
<sub>`ui/dr-ui/ui/adjust.slint:1476`</sub>
### Magnify the photograph by any amount ### Magnify the photograph by any amount
@@ -28,7 +39,7 @@ Sampling a neutral is the first move of the tonal pass — every colour judgemen
Anchored on the fingers' midpoint, and on the pointer, so the gesture reads as magnifying the picture rather than sliding it about. Double-tap is the way to an exact 1:1; this is the way to everything in between. Past 1:1 the pixels are shown as they are, square and unsmoothed; below it, filtered. Anchored on the fingers' midpoint, and on the pointer, so the gesture reads as magnifying the picture rather than sliding it about. Double-tap is the way to an exact 1:1; this is the way to everything in between. Past 1:1 the pixels are shown as they are, square and unsmoothed; below it, filtered.
<sub>`ui/dr-ui/ui/app.slint:1885`</sub> <sub>`ui/dr-ui/ui/app.slint:1961`</sub>
### Move a magnified photograph about ### Move a magnified photograph about
@@ -39,7 +50,7 @@ Anchored on the fingers' midpoint, and on the pointer, so the gesture reads as m
Only once there is something outside the viewport to reach, which is why the cursor becomes a hand exactly then. The view is clamped to the frame: panning past the edge would show undefined area beside the photograph, and that reads as a rendering fault rather than as the end of the picture. Only once there is something outside the viewport to reach, which is why the cursor becomes a hand exactly then. The view is clamped to the frame: panning past the edge would show undefined area beside the photograph, and that reads as a rendering fault rather than as the end of the picture.
<sub>`ui/dr-ui/ui/app.slint:1981`</sub> <sub>`ui/dr-ui/ui/app.slint:2057`</sub>
### Paint a mask by hand ### Paint a mask by hand
@@ -49,7 +60,17 @@ Only once there is something outside the viewport to reach, which is why the cur
A model's mask stops inside a shoulder and leaks into the hair, and no single edge control fixes two errors that go opposite ways. The whole stroke is one step in the history, so taking a mark back costs one press however long it took to make. A model's mask stops inside a shoulder and leaks into the hair, and no single edge control fixes two errors that go opposite ways. The whole stroke is one step in the history, so taking a mark back costs one press however long it took to make.
<sub>`ui/dr-ui/ui/app.slint:2072`</sub> <sub>`ui/dr-ui/ui/app.slint:2148`</sub>
### Open this list
- **Touch** — Press "?" in the top bar, beside Settings, and Done to put it away
- **Pointer** — Press "?" in the top bar, beside Settings, and Done to put it away
- **Keyboard** — `F1`; the key that closes any sheet puts it away
Most of the keys are develop's, and a reference that could only be opened from the grid had to be looked up before opening the photograph they were wanted for.
<sub>`ui/dr-ui/ui/app.slint:2374`</sub>
### Take back the last change ### Take back the last change
@@ -60,7 +81,7 @@ A model's mask stops inside a shoulder and leaks into the hair, and no single ed
A whole drag is one step, so undo takes back a decision rather than a frame of a gesture. The list is there because arriving six steps back costs what arriving from one does. A whole drag is one step, so undo takes back a decision rather than a frame of a gesture. The list is there because arriving six steps back costs what arriving from one does.
<sub>`ui/dr-ui/ui/app.slint:2306`</sub> <sub>`ui/dr-ui/ui/app.slint:2404`</sub>
### Do it again after taking it back ### Do it again after taking it back
@@ -69,7 +90,7 @@ A whole drag is one step, so undo takes back a decision rather than a frame of a
- **Keyboard** — `Ctrl+Shift+Z`, or `Ctrl+Y` - **Keyboard** — `Ctrl+Shift+Z`, or `Ctrl+Y`
- **See it** — [in the manual](manual/README.md#history-snapshots-presets) - **See it** — [in the manual](manual/README.md#history-snapshots-presets)
<sub>`ui/dr-ui/ui/app.slint:2320`</sub> <sub>`ui/dr-ui/ui/app.slint:2418`</sub>
### Remove a repair ### Remove a repair
@@ -77,7 +98,7 @@ A whole drag is one step, so undo takes back a decision rather than a frame of a
- **Pointer** — Click it, then Delete Repair - **Pointer** — Click it, then Delete Repair
- **Keyboard** — `Delete` or `Backspace`, while repairing - **Keyboard** — `Delete` or `Backspace`, while repairing
<sub>`ui/dr-ui/ui/app.slint:2340`</sub> <sub>`ui/dr-ui/ui/app.slint:2438`</sub>
### Copy the settings from this photograph ### Copy the settings from this photograph
@@ -88,7 +109,7 @@ A whole drag is one step, so undo takes back a decision rather than a frame of a
The button is the copy that has to work: a tablet has no modifier key to hold and no menu bar to hang the action from. The shortcut is an accelerator for a control that is on screen either way. The button is the copy that has to work: a tablet has no modifier key to hold and no menu bar to hang the action from. The shortcut is an accelerator for a control that is on screen either way.
<sub>`ui/dr-ui/ui/app.slint:2359`</sub> <sub>`ui/dr-ui/ui/app.slint:2457`</sub>
### Paste the settings onto this photograph ### Paste the settings onto this photograph
@@ -99,7 +120,7 @@ The button is the copy that has to work: a tablet has no modifier key to hold an
The button names what would be pasted — "3 adjustments", and whether the crop is coming with it — which the shortcut cannot say. Both paste the same scope. The button names what would be pasted — "3 adjustments", and whether the crop is coming with it — which the shortcut cannot say. Both paste the same scope.
<sub>`ui/dr-ui/ui/app.slint:2372`</sub> <sub>`ui/dr-ui/ui/app.slint:2470`</sub>
### Choose which kinds of edit a copy carries ### Choose which kinds of edit a copy carries
@@ -110,7 +131,7 @@ The button names what would be pasted — "3 adjustments", and whether the crop
Lightroom's Copy Settings. Pasting a look across a shoot usually means leaving each frame's crop and rotation alone, and that is a choice to make at the moment of copying. Lightroom's Copy Settings. Pasting a look across a shoot usually means leaving each frame's crop and rotation alone, and that is a choice to make at the moment of copying.
<sub>`ui/dr-ui/ui/app.slint:2390`</sub> <sub>`ui/dr-ui/ui/app.slint:2488`</sub>
### Export this photograph as the last one was ### Export this photograph as the last one was
@@ -121,7 +142,7 @@ Lightroom's Copy Settings. Pasting a look across a shoot usually means leaving e
Every export runs on the defaults in Settings, so "as the last one was" is what the button already does. The chord is Lightroom's and darktable's, kept so hands that learned it there need not learn it again. Every export runs on the defaults in Settings, so "as the last one was" is what the button already does. The chord is Lightroom's and darktable's, kept so hands that learned it there need not learn it again.
<sub>`ui/dr-ui/ui/app.slint:2415`</sub> <sub>`ui/dr-ui/ui/app.slint:2513`</sub>
### Choose how to export, then export ### Choose how to export, then export
@@ -132,7 +153,7 @@ Every export runs on the defaults in Settings, so "as the last one was" is what
The export sheet is the export defaults alone with an Export button. What is chosen there is kept, so it is also what the next Ctrl+Shift+E uses. The export sheet is the export defaults alone with an Export button. What is chosen there is kept, so it is also what the next Ctrl+Shift+E uses.
<sub>`ui/dr-ui/ui/app.slint:2428`</sub> <sub>`ui/dr-ui/ui/app.slint:2526`</sub>
### Keep a crop that leaves a mask outside ### Keep a crop that leaves a mask outside
@@ -140,7 +161,7 @@ The export sheet is the export defaults alone with an Export button. What is cho
- **Pointer** — Press "Keep crop" on the notice, or "Undo crop" to take it back - **Pointer** — Press "Keep crop" on the notice, or "Undo crop" to take it back
- **Keyboard** — `Enter` keeps it; `Ctrl+Z` takes the crop back, like any other step - **Keyboard** — `Enter` keeps it; `Ctrl+Z` takes the crop back, like any other step
<sub>`ui/dr-ui/ui/app.slint:2493`</sub> <sub>`ui/dr-ui/ui/app.slint:2591`</sub>
### Go back to the grid ### Go back to the grid
@@ -150,7 +171,7 @@ The export sheet is the export defaults alone with an Export button. What is cho
Lightroom's key for the grid. Escape gets there too, but a step at a time — out of a mode, then out of a zoom — where this goes straight back. Lightroom's key for the grid. Escape gets there too, but a step at a time — out of a mode, then out of a zoom — where this goes straight back.
<sub>`ui/dr-ui/ui/app.slint:2510`</sub> <sub>`ui/dr-ui/ui/app.slint:2608`</sub>
### Nudge the control last moved ### Nudge the control last moved
@@ -160,7 +181,7 @@ Lightroom's key for the grid. Escape gets there too, but a step at a time — ou
Lightroom's keys for the selected slider. There is no focus ring on a slider here, so "selected" is the last one moved — the same control `R` puts back — which covers the framing sliders, perspective included, as well as the adjustments. Lightroom's keys for the selected slider. There is no focus ring on a slider here, so "selected" is the last one moved — the same control `R` puts back — which covers the framing sliders, perspective included, as well as the adjustments.
<sub>`ui/dr-ui/ui/app.slint:2539`</sub> <sub>`ui/dr-ui/ui/app.slint:2637`</sub>
### Change which group of adjustments is on screen ### Change which group of adjustments is on screen
@@ -171,7 +192,7 @@ Lightroom's keys for the selected slider. There is no focus ring on a slider her
The groups are whatever the operation set declares itself to be about, so there are as many as the pipeline has and no key can be assigned to one of them by name. Stepping is the binding that survives a node being added. The groups are whatever the operation set declares itself to be about, so there are as many as the pipeline has and no key can be assigned to one of them by name. Stepping is the binding that survives a node being added.
<sub>`ui/dr-ui/ui/app.slint:2567`</sub> <sub>`ui/dr-ui/ui/app.slint:2665`</sub>
### Look at the photograph at 1:1 ### Look at the photograph at 1:1
@@ -182,7 +203,7 @@ The groups are whatever the operation set declares itself to be about, so there
Noise reduction and capture sharpening are judgements about single pixels, and a fitted view averages several of the file's into each one on screen — so the frame looks softer than it is and the correction goes too far. The point and the magnification survive opening the next photograph, which is what makes checking the same eye across forty portraits forty keystrokes rather than forty pans. From 1:1 on the photograph is drawn as its own pixels, each a hard-edged square, rather than smoothed into a blur. Noise reduction and capture sharpening are judgements about single pixels, and a fitted view averages several of the file's into each one on screen — so the frame looks softer than it is and the correction goes too far. The point and the magnification survive opening the next photograph, which is what makes checking the same eye across forty portraits forty keystrokes rather than forty pans. From 1:1 on the photograph is drawn as its own pixels, each a hard-edged square, rather than smoothed into a blur.
<sub>`ui/dr-ui/ui/app.slint:2603`</sub> <sub>`ui/dr-ui/ui/app.slint:2701`</sub>
### Rate this photograph ### Rate this photograph
@@ -190,7 +211,7 @@ Noise reduction and capture sharpening are judgements about single pixels, and a
- **Pointer** — Click a star in the top bar - **Pointer** — Click a star in the top bar
- **Keyboard** — `0`–`5` - **Keyboard** — `0`–`5`
<sub>`ui/dr-ui/ui/app.slint:2660`</sub> <sub>`ui/dr-ui/ui/app.slint:2758`</sub>
### Pick or reject this photograph ### Pick or reject this photograph
@@ -200,7 +221,7 @@ Noise reduction and capture sharpening are judgements about single pixels, and a
The grid's keys, on the photograph that is open (FR-UI-5, 2026-09-19). Judging here does not move on to the next frame: that belongs to culling, and in develop the photograph in front of you is the one being worked on. The grid's keys, on the photograph that is open (FR-UI-5, 2026-09-19). Judging here does not move on to the next frame: that belongs to culling, and in develop the photograph in front of you is the one being worked on.
<sub>`ui/dr-ui/ui/app.slint:2666`</sub> <sub>`ui/dr-ui/ui/app.slint:2764`</sub>
### Give this photograph a colour label ### Give this photograph a colour label
@@ -211,7 +232,7 @@ The grid's keys, on the photograph that is open (FR-UI-5, 2026-09-19). Judging h
The grid's keys, on the photograph that is open, so labelling while stepping through a folder is one hand's work. The bar names the label in words beside its mark. The grid's keys, on the photograph that is open, so labelling while stepping through a folder is one hand's work. The bar names the label in words beside its mark.
<sub>`ui/dr-ui/ui/app.slint:2696`</sub> <sub>`ui/dr-ui/ui/app.slint:2794`</sub>
### Move to the next or previous photograph ### Move to the next or previous photograph
@@ -222,7 +243,7 @@ The grid's keys, on the photograph that is open, so labelling while stepping thr
The edit on screen is saved on the way out, so stepping through a folder is as much a departure as going back to the grid and loses nothing. A and D as well as the arrows, so the left hand steps along the roll while the right stays on the mouse. The edit on screen is saved on the way out, so stepping through a folder is as much a departure as going back to the grid and loses nothing. A and D as well as the arrows, so the left hand steps along the roll while the right stays on the mouse.
<sub>`ui/dr-ui/ui/app.slint:2721`</sub> <sub>`ui/dr-ui/ui/app.slint:2819`</sub>
### See the photograph before you edited it ### See the photograph before you edited it
@@ -233,7 +254,7 @@ The edit on screen is saved on the way out, so stepping through a folder is as m
Held rather than toggled, and no split screen: a split halves the working image on the tablet the column was sized for, and the comparison photographers describe making is a flick back and forth. It takes no history step, so checking whether a frame is overcooked costs nothing to undo afterwards. Held rather than toggled, and no split screen: a split halves the working image on the tablet the column was sized for, and the comparison photographers describe making is a flick back and forth. It takes no history step, so checking whether a frame is overcooked costs nothing to undo afterwards.
<sub>`ui/dr-ui/ui/app.slint:2851`</sub> <sub>`ui/dr-ui/ui/app.slint:2949`</sub>
### Put one control back to its default ### Put one control back to its default
@@ -317,7 +338,7 @@ The question a correction raises is whether it did what it was for — whether t
One key for "up one", innermost first: a question before the sheet under it, a sheet before the view, a view before the library. Nothing is left behind a dialogue that the key walked straight past. One key for "up one", innermost first: a question before the sheet under it, a sheet before the view, a view before the library. Nothing is left behind a dialogue that the key walked straight past.
<sub>`ui/dr-ui/ui/app.slint:975`</sub> <sub>`ui/dr-ui/ui/app.slint:1021`</sub>
### Do what a sheet offers ### Do what a sheet offers
@@ -325,7 +346,15 @@ One key for "up one", innermost first: a question before the sheet under it, a s
- **Pointer** — Press its button — Export, or Copy - **Pointer** — Press its button — Export, or Copy
- **Keyboard** — `Enter`, on the export and copy sheets - **Keyboard** — `Enter`, on the export and copy sheets
<sub>`ui/dr-ui/ui/app.slint:985`</sub> <sub>`ui/dr-ui/ui/app.slint:1031`</sub>
### Scroll by the scrollbar
- **Pointer** — Drag the bar along the right-hand edge of a list, or click the track above or below it to move by a page
A list cut off at its edge looks, to a mouse, like a list that ends there — the film stocks past the tenth read as deleted. The bar says there is more and where the view is in it, and it is the one way to scroll that needs neither a wheel nor a drag on the content, which may be a row that would take the click.
<sub>`ui/dr-ui/ui/widgets.slint:1131`</sub>
## Collections sidebar ## Collections sidebar
@@ -337,7 +366,7 @@ One key for "up one", innermost first: a question before the sheet under it, a s
The tree is inside a Flickable, which claims any drag beginning inside it — so with a finger a drag on a row is a scroll until something says otherwise. The hold is that something, and it is what every mobile list already uses to pick a row up. The row lifts the moment it fires, so the gesture says it has been understood before anything moves. The tree is inside a Flickable, which claims any drag beginning inside it — so with a finger a drag on a row is a scroll until something says otherwise. The hold is that something, and it is what every mobile list already uses to pick a row up. The row lifts the moment it fires, so the gesture says it has been understood before anything moves.
<sub>`ui/dr-ui/ui/collections.slint:351`</sub> <sub>`ui/dr-ui/ui/collections.slint:353`</sub>
### Act on a collection — rename, nest, un-nest, delete ### Act on a collection — rename, nest, un-nest, delete
@@ -347,7 +376,7 @@ The tree is inside a Flickable, which claims any drag beginning inside it — so
The hold arms a drag and opens this menu, and which one you get is decided by whether you moved — the same fork the grid uses. One menu for everything done to a row, because there is one hold per row: while the hold opened the offline question by itself, nothing else the tree can do had a touch route. The hold arms a drag and opens this menu, and which one you get is decided by whether you moved — the same fork the grid uses. One menu for everything done to a row, because there is one hold per row: while the hold opened the offline question by itself, nothing else the tree can do had a touch route.
<sub>`ui/dr-ui/ui/collections.slint:363`</sub> <sub>`ui/dr-ui/ui/collections.slint:365`</sub>
### Take a collection back out of the one it is nested in ### Take a collection back out of the one it is nested in
@@ -357,7 +386,7 @@ The hold arms a drag and opens this menu, and which one you get is decided by wh
Nesting is a drag of one row onto another, and its inverse had no gesture at all: "All photographs" refused every drop, which is right for a photograph — it is already in the library — and wrong for a collection, which has a top level to be returned to. Without it a collection dragged into another was in there permanently. Nesting is a drag of one row onto another, and its inverse had no gesture at all: "All photographs" refused every drop, which is right for a photograph — it is already in the library — and wrong for a collection, which has a top level to be returned to. Without it a collection dragged into another was in there permanently.
<sub>`ui/dr-ui/ui/collections.slint:374`</sub> <sub>`ui/dr-ui/ui/collections.slint:376`</sub>
### Rename a collection ### Rename a collection
@@ -368,7 +397,59 @@ Nesting is a drag of one row onto another, and its inverse had no gesture at all
Double-click is what a file manager and a Lightroom panel use for the same thing, so it needs no discovering — but nothing on screen says so, which is what the menu item is for. Double-click is what a file manager and a Lightroom panel use for the same thing, so it needs no discovering — but nothing on screen says so, which is what the menu item is for.
<sub>`ui/dr-ui/ui/collections.slint:388`</sub> <sub>`ui/dr-ui/ui/collections.slint:390`</sub>
### Review duplicate originals
- **Touch** — Tap "Duplicate originals" under the trash
- **Pointer** — Click "Duplicate originals" under the trash
- **See it** — [in the manual](manual/README.md#duplicate-originals)
Under the trash because the trash is where the spare copies go, and only while the catalog holds any: a row that is always there and usually empty is noise.
<sub>`ui/dr-ui/ui/collections.slint:898`</sub>
## Duplicate originals
### Check duplicate originals are the same file
- **Touch** — Tap "Check"
- **Pointer** — Click "Check"
- **See it** — [in the manual](manual/README.md#duplicate-originals)
Nothing is moved on the catalog's say-so. The check reads the first and last megabyte of each copy and its sidecar, keeps what it read, and drops any group whose bytes or edits differ.
<sub>`ui/dr-ui/ui/duplicates.slint:233`</sub>
### Move the spare copies to the trash
- **Touch** — Tap "Move N copies to trash"
- **Pointer** — Click "Move N copies to trash"
- **See it** — [in the manual](manual/README.md#duplicate-originals)
The one verb, named with its count, and off until a check has proved something. Each group is merged onto the copy that stays and the others go to the trash together or not at all; the trash view gives them back.
<sub>`ui/dr-ui/ui/duplicates.slint:248`</sub>
### Choose which copy of a duplicate stays
- **Touch** — Tap the copy's path
- **Pointer** — Click the copy's path
- **See it** — [in the manual](manual/README.md#duplicate-originals)
The rule picks the copy outside a backup folder that still has the camera's name. The path is the evidence, so the path is what is tapped to overrule it.
<sub>`ui/dr-ui/ui/duplicates.slint:313`</sub>
### Leave a duplicate group as it is
- **Touch** — Untick "Include" on the group
- **Pointer** — Untick "Include" on the group
- **See it** — [in the manual](manual/README.md#duplicate-originals)
The plan is every group the check proved; a group you want to keep in two places is taken out of it rather than argued with.
<sub>`ui/dr-ui/ui/duplicates.slint:323`</sub>
## People ## People
@@ -440,7 +521,7 @@ The right match confidence is a property of your library, not of the model. "Wha
Touch has no ctrl, so without a mode there is no way to select a second photograph — the first tap would open it. The hold is the fast way in and the button is the one that can be found. Touch has no ctrl, so without a mode there is no way to select a second photograph — the first tap would open it. The hold is the fast way in and the button is the one that can be found.
<sub>`ui/dr-ui/ui/library.slint:1698`</sub> <sub>`ui/dr-ui/ui/library.slint:1706`</sub>
### Add or remove one photograph ### Add or remove one photograph
@@ -450,7 +531,7 @@ Touch has no ctrl, so without a mode there is no way to select a second photogra
While selecting, a tap never opens. That is the whole point of the mode: one meaning per gesture at a time. Press Done to get tap-to-open back. While selecting, a tap never opens. That is the whole point of the mode: one meaning per gesture at a time. Press Done to get tap-to-open back.
<sub>`ui/dr-ui/ui/library.slint:1708`</sub> <sub>`ui/dr-ui/ui/library.slint:1716`</sub>
### Leave selecting ### Leave selecting
@@ -459,7 +540,7 @@ While selecting, a tap never opens. That is the whole point of the mode: one mea
- **Keyboard** — `Escape`, or `Back`; an open sheet closes first - **Keyboard** — `Escape`, or `Back`; an open sheet closes first
- **See it** — [in the manual](manual/README.md#selecting-several) - **See it** — [in the manual](manual/README.md#selecting-several)
<sub>`ui/dr-ui/ui/library.slint:1717`</sub> <sub>`ui/dr-ui/ui/library.slint:1725`</sub>
### Pick a photograph up to drag it ### Pick a photograph up to drag it
@@ -469,7 +550,7 @@ While selecting, a tap never opens. That is the whole point of the mode: one mea
A finger on a photograph might be starting a scroll, and for the first half-second the grid assumes it is. Holding says otherwise, and the ring is the grid saying it heard — from there the drag cannot be lost to a scroll. A mouse never waits: the cursor is precise enough that a sideways drag is unambiguous from the first pixel. A finger on a photograph might be starting a scroll, and for the first half-second the grid assumes it is. Holding says otherwise, and the ring is the grid saying it heard — from there the drag cannot be lost to a scroll. A mouse never waits: the cursor is precise enough that a sideways drag is unambiguous from the first pixel.
<sub>`ui/dr-ui/ui/library.slint:1748`</sub> <sub>`ui/dr-ui/ui/library.slint:1756`</sub>
### Select a range ### Select a range
@@ -480,7 +561,7 @@ A finger on a photograph might be starting a scroll, and for the first half-seco
This replaced a double tap, which had no visible state and could take forty photographs by accident. The run is resolved by the catalog rather than by what is on screen, so the grid can scroll between the two taps — the ranges that hurt on a tablet are longer than a screenful, which is exactly where a finger sweep runs out. This replaced a double tap, which had no visible state and could take forty photographs by accident. The run is resolved by the catalog rather than by what is on screen, so the grid can scroll between the two taps — the ranges that hurt on a tablet are longer than a screenful, which is exactly where a finger sweep runs out.
<sub>`ui/dr-ui/ui/library.slint:1814`</sub> <sub>`ui/dr-ui/ui/library.slint:1822`</sub>
### Take the blinks out of a burst ### Take the blinks out of a burst
@@ -490,7 +571,7 @@ This replaced a double tap, which had no visible state and could take forty phot
Face indexing reads each face's eyes. The chip drops frames where the chosen people are caught blinking, and leaves sunglasses and eyes it could not read alone. Face indexing reads each face's eyes. The chip drops frames where the chosen people are caught blinking, and leaves sunglasses and eyes it could not read alone.
<sub>`ui/dr-ui/ui/library.slint:2531`</sub> <sub>`ui/dr-ui/ui/library.slint:2540`</sub>
### Find photographs with two people in them ### Find photographs with two people in them
@@ -500,7 +581,7 @@ Face indexing reads each face's eyes. The chip drops frames where the chosen peo
"Any of them" is a union and "all of them" is an intersection. The tray is where both terms and the choice between them live, because a filter belongs on the filter bar. "Any of them" is a union and "all of them" is an intersection. The tray is where both terms and the choice between them live, because a filter belongs on the filter bar.
<sub>`ui/dr-ui/ui/library.slint:2561`</sub> <sub>`ui/dr-ui/ui/library.slint:2570`</sub>
### Show only photographs with one colour label ### Show only photographs with one colour label
@@ -510,7 +591,7 @@ Face indexing reads each face's eyes. The chip drops frames where the chosen peo
Each chip is the label's mark and its name, so the one you want is found by reading it; tap the lit chip again to show every label. Each chip is the label's mark and its name, so the one you want is found by reading it; tap the lit chip again to show every label.
<sub>`ui/dr-ui/ui/library.slint:2685`</sub> <sub>`ui/dr-ui/ui/library.slint:2694`</sub>
### Export the selection as the last export was ### Export the selection as the last export was
@@ -521,7 +602,7 @@ Each chip is the label's mark and its name, so the one you want is found by read
Lightroom's and darktable's chords. Every export runs on the saved defaults, so the plain chord opens them beside an Export button and the shifted one skips straight to exporting. Lightroom's and darktable's chords. Every export runs on the saved defaults, so the plain chord opens them beside an Export button and the shifted one skips straight to exporting.
<sub>`ui/dr-ui/ui/library.slint:3168`</sub> <sub>`ui/dr-ui/ui/library.slint:3164`</sub>
### Paste copied settings onto the selection ### Paste copied settings onto the selection
@@ -530,7 +611,7 @@ Lightroom's and darktable's chords. Every export runs on the saved defaults, so
- **Keyboard** — `Ctrl+V` - **Keyboard** — `Ctrl+V`
- **See it** — [in the manual](manual/README.md#copying-settings) - **See it** — [in the manual](manual/README.md#copying-settings)
<sub>`ui/dr-ui/ui/library.slint:3192`</sub> <sub>`ui/dr-ui/ui/library.slint:3188`</sub>
### Keyword the selection ### Keyword the selection
@@ -540,7 +621,7 @@ Lightroom's and darktable's chords. Every export runs on the saved defaults, so
Lightroom's keywording chord. The sheet opens with its field ready for typing, so the keys that judge in the grid are out of the way until it closes. Lightroom's keywording chord. The sheet opens with its field ready for typing, so the keys that judge in the grid are out of the way until it closes.
<sub>`ui/dr-ui/ui/library.slint:3221`</sub> <sub>`ui/dr-ui/ui/library.slint:3217`</sub>
### Show only photographs with some number of stars ### Show only photographs with some number of stars
@@ -551,7 +632,7 @@ Lightroom's keywording chord. The sheet opens with its field ready for typing, s
The chips say "this many or more". A range with a ceiling — the twos and threes still to be decided — is the keyboard's alone, and the bar says so in words while it holds. The chips say "this many or more". A range with a ceiling — the twos and threes still to be decided — is the keyboard's alone, and the bar says so in words while it holds.
<sub>`ui/dr-ui/ui/library.slint:3255`</sub> <sub>`ui/dr-ui/ui/library.slint:3251`</sub>
### Give photographs a colour label ### Give photographs a colour label
@@ -562,7 +643,7 @@ The chips say "this many or more". A range with a ceiling — the twos and three
Lightroom's keys, so hands that learned them there need not learn them again. Purple has no key there either, and is on the bar. Every mark carries its label's initial, so the label is read without telling the colours apart. Lightroom's keys, so hands that learned them there need not learn them again. Purple has no key there either, and is on the bar. Every mark carries its label's initial, so the label is read without telling the colours apart.
<sub>`ui/dr-ui/ui/library.slint:3305`</sub> <sub>`ui/dr-ui/ui/library.slint:3301`</sub>
### Pick or reject a photograph ### Pick or reject a photograph
@@ -572,7 +653,7 @@ Lightroom's keys, so hands that learned them there need not learn them again. Pu
The keys every culling tool uses, so muscle memory built elsewhere works here. The keys every culling tool uses, so muscle memory built elsewhere works here.
<sub>`ui/dr-ui/ui/library.slint:3329`</sub> <sub>`ui/dr-ui/ui/library.slint:3325`</sub>
### Move photographs to the trash ### Move photographs to the trash
@@ -582,7 +663,7 @@ The keys every culling tool uses, so muscle memory built elsewhere works here.
The bin acts on one photograph, so a stray click cannot trash a selection; the key acts on the selection because that is what every file manager's Delete does. Both are undone from the trash view. The bin acts on one photograph, so a stray click cannot trash a selection; the key acts on the selection because that is what every file manager's Delete does. Both are undone from the trash view.
<sub>`ui/dr-ui/ui/library.slint:3356`</sub> <sub>`ui/dr-ui/ui/library.slint:3352`</sub>
### Open this list ### Open this list
@@ -590,7 +671,7 @@ The bin acts on one photograph, so a stray click cannot trash a selection; the k
- **Pointer** — Press Help in the header, and Done to put it away - **Pointer** — Press Help in the header, and Done to put it away
- **Keyboard** — `F1`, and `Escape` to put it away - **Keyboard** — `F1`, and `Escape` to put it away
<sub>`ui/dr-ui/ui/library.slint:3381`</sub> <sub>`ui/dr-ui/ui/library.slint:3377`</sub>
### Rename the collection the grid is showing ### Rename the collection the grid is showing
@@ -598,7 +679,7 @@ The bin acts on one photograph, so a stray click cannot trash a selection; the k
- **Pointer** — Double-click it in the sidebar - **Pointer** — Double-click it in the sidebar
- **Keyboard** — `F2` - **Keyboard** — `F2`
<sub>`ui/dr-ui/ui/library.slint:3389`</sub> <sub>`ui/dr-ui/ui/library.slint:3385`</sub>
### Move through the grid ### Move through the grid
@@ -608,7 +689,7 @@ The bin acts on one photograph, so a stray click cannot trash a selection; the k
The cursor selects what it lands on, so walking and judging are one hand's work. The cursor selects what it lands on, so walking and judging are one hand's work.
<sub>`ui/dr-ui/ui/library.slint:3409`</sub> <sub>`ui/dr-ui/ui/library.slint:3405`</sub>
### Resize the thumbnails ### Resize the thumbnails
@@ -672,7 +753,7 @@ A folded burst draws its earliest frame, which is a fact about the clock and not
Distinct from Done, which leaves the mode entirely. Clearing keeps it, so the next selection can start straight away. Distinct from Done, which leaves the mode entirely. Clearing keeps it, so the next selection can start straight away.
<sub>`ui/dr-ui/ui/library.slint:4973`</sub> <sub>`ui/dr-ui/ui/library.slint:4987`</sub>
### Select everything the grid is showing ### Select everything the grid is showing
@@ -683,7 +764,7 @@ Distinct from Done, which leaves the mode entirely. Clearing keeps it, so the ne
A scoped grid of two hundred frames is two hundred taps otherwise, and "all of them, except those three" is a far more common shape than the taps it took to say it. A scoped grid of two hundred frames is two hundred taps otherwise, and "all of them, except those three" is a far more common shape than the taps it took to say it.
<sub>`ui/dr-ui/ui/library.slint:4992`</sub> <sub>`ui/dr-ui/ui/library.slint:5006`</sub>
### Take photographs out of a collection ### Take photographs out of a collection
@@ -693,4 +774,16 @@ A scoped grid of two hundred frames is two hundred taps otherwise, and "all of t
The badge on a cell says a photograph is filed in three collections and never which. This is the sheet that names them, and the only way out of one the grid is not currently scoped to. The badge on a cell says a photograph is filed in three collections and never which. This is the sheet that names them, and the only way out of one the grid is not currently scoped to.
<sub>`ui/dr-ui/ui/library.slint:5173`</sub> <sub>`ui/dr-ui/ui/library.slint:5187`</sub>
## Settings
### Review duplicate originals
- **Touch** — Tap "Review duplicate originals"
- **Pointer** — Click "Review duplicate originals"
- **See it** — [in the manual](manual/README.md#duplicate-originals)
Beside the other whole-library passes, because it is one; the sidebar offers the same page under the trash.
<sub>`ui/dr-ui/ui/settings.slint:533`</sub>
docs/manual/README.md
+121 -18
View File
@@ -15,15 +15,21 @@ The photographs are the author's. None show a person.
DarkRoom opens on a library: a folder on this machine, a folder a sync DarkRoom opens on a library: a folder on this machine, a folder a sync
client keeps, or a Nextcloud account. A folder needs no password and uploads client keeps, or a Nextcloud account. A folder needs no password and uploads
nothing. nothing. `Choose folder…` opens your desktop's own folder dialogue, which can
make a new folder too; the folder used last stays on the screen with
`Open folder` beside it, and `Choose another…` in place of `Choose folder…`.
On a Nextcloud account the library folder is chosen in a browser of the
server, whose `New folder` makes one there. (The browser is not pictured:
these recordings have no server behind them. Nor is the folder dialogue,
which is your desktop's rather than DarkRoom's.)
![The launch screen: a server field, a folder field, and which formats to scan for](media/launch.png) ![The launch screen: a Nextcloud server or an app password, or the folder used last with Open folder beside it](media/launch.png)
Once a folder is named, it is the library — you are not asked for it again, Once a folder is named, it is the library — you are not asked for it again,
and `Open library` opens it whole. `Subfolder…` narrows the scan to part of and `Open library` opens it whole. `Subfolder…` narrows the scan to part of
it. The formats ticked are what the scan looks for; RAW is on and JPEG off it. The formats ticked are what the scan looks for; RAW is on and JPEG off
by default, because a RAW editor's sensible default is the file the camera by default, because a RAW editor's sensible default is the file the camera
wrote first. wrote first. `Change library`, at the foot of the sidebar, comes back here.
![A folder chosen: the library, whether to scan a subfolder, and the formats](media/launch-folder.png) ![A folder chosen: the library, whether to scan a subfolder, and the formats](media/launch-folder.png)
@@ -39,7 +45,13 @@ file is (on this device, or only on the server).
### Rating and flagging ### Rating and flagging
Hover a cell and the stars appear; click one. The filter chips above the grid Hover a cell and the stars appear; click one. The filter chips above the grid
count what each rating holds, and clicking `3+` shows only those. count what each rating holds, and clicking `3+` shows only those. From the
keyboard, `0` to `5` set the stars on the photograph under the pointer or on
the selection; `P` picks, `X` rejects and `U` takes the flag off, and `Flag`
on the selection bar does the same with the pointer. In
develop the same keys judge the open photograph and stay on it, and the top
bar carries its stars and `Pick` and `Reject`; the roll marks each frame's
flag and stars.
![Rating two photographs, then filtering the grid to three stars and more](media/library-rating.gif) ![Rating two photographs, then filtering the grid to three stars and more](media/library-rating.gif)
@@ -56,7 +68,14 @@ and the same keys work there.
### Getting about ### Getting about
Drag the timeline to scrub through years; Ctrl and the wheel resize the Drag the timeline to scrub through years; Ctrl and the wheel resize the
thumbnails. thumbnails. On a desktop a scrollbar beside the grid says how far through the
library the view is — drag its thumb, or click the track to move a page — and
the sidebar, the develop column and Settings have one too whenever they run
past the window. On a tablet they scroll by flick alone.
`Help` in the header, or `F1`, opens the controls and shortcuts: every key and
gesture, screen by screen, with `See it` beside those this page shows, and
`Manual` to open this page. In develop it is the `?` beside `Settings`.
![Scrubbing the timeline](media/library-timeline.gif) ![Scrubbing the timeline](media/library-timeline.gif)
@@ -106,6 +125,39 @@ to have it show another, open it and click the ring on the frame you want.
With faces indexed, narrowing the grid to a person puts `Eyes open` beside With faces indexed, narrowing the grid to a person puts `Eyes open` beside
their name on the filter bar, which leaves out the frames where they blinked. their name on the filter bar, which leaves out the frames where they blinked.
### Duplicate originals
A library put together by hand often holds the same RAW more than once — a
dated folder, a `bck` folder beside it, a renamed copy another program
exported. When the catalog finds files with the same camera, capture time
and size in more than one place, `Duplicate originals` appears under the
trash in the sidebar with how many there are; Settings offers the same page
beside the other whole-library passes.
The page lists each group with its picture and its paths. One copy is
marked `Stays`: the one outside a folder named like a backup (`bck`,
`backup`, `copy`, `old`…), then the one still named the way the camera named
it (`_MG_4623`, `IMG_0001`, `DSC_0042`), then the one catalogued first. Tap
another path to keep that copy instead, and untick `Include` to leave a
group alone.
Nothing moves until `Check` has read the first and last megabyte of every
copy and each copy's sidecar. A group whose files differ, or whose copies
carry two different edits, is marked skipped and says why — two edits of
one frame are kept for virtual copies. What the check reads is kept, so a
second visit costs nothing. The line under each group says what the copy
that stays will gain: the highest rating, every keyword and collection, a
flag or label the copies agree on, faces and the names on them, and the
edit if only a copy had one. Where the copies disagree on a flag or a
label, the one that stays keeps its own and the line says so.
![Two frames with a copy each in a bck folder, checked and proved the same, the copy that stays marked on each](media/duplicates.png)
`Move N copies to trash` does it, one group at a time: each group is merged
and its spare copies moved together, or not at all. The copies go to the
trash, not away — select them in the trash view and `Restore` puts them
back where they were.
## Developing a photograph ## Developing a photograph
Click a thumbnail to open it. The column on the right is every adjustment; Click a thumbnail to open it. The column on the right is every adjustment;
@@ -135,8 +187,18 @@ the sensor recorded.
The roll along the foot of the canvas holds the photographs the grid was The roll along the foot of the canvas holds the photographs the grid was
showing; click one to open it. The right arrow, `D` or space opens the next, showing; click one to open it. The right arrow, `D` or space opens the next,
and the left arrow or `A` the one before. The edit on screen is saved on the and the left arrow or `A` the one before; held down, they go on past the
way, so stepping along a shoot loses nothing. stretch the roll has loaded, through everything the grid would show. The
edit on screen is saved on the way, so stepping along a shoot loses nothing.
On a Nextcloud library a photograph may not be on this device yet. Its
thumbnail from the grid stands in at once; if the original has to come down,
the thumbnail dims under *Not on this device yet*, with how far the download
has got — `Downloading — 12.4 of 38.0 MB` — and a bar, and the photograph
opens when it lands. Step on before then and the one you step to is the one
that opens: a download that arrives late is kept for later and never takes
the place of the photograph whose name is showing. (Not pictured: a folder
library, which these recordings use, never has a photograph to wait for.)
### White balance from the photograph ### White balance from the photograph
@@ -202,17 +264,31 @@ opacity are in the panel. Heal blends; clone copies.
The `Film` chooser at the head of Adjust applies a spectral simulation of a The `Film` chooser at the head of Adjust applies a spectral simulation of a
named stock; below it, the print exposure and push controls a film has and a named stock; below it, the print exposure and push controls a film has and a
sensor does not. sensor does not. The list opens over the column and scrolls on its own — by
wheel, drag or flick, or with `Up`, `Down` and `Enter` — down to the
black-and-white stocks at its end.
![Choosing Velvia, then holding Before](media/film.gif) ![Opening the film list, scrolling it, choosing Velvia, then holding Before](media/film.gif)
### History, snapshots, presets ### History, snapshots, presets
Every change is a step; `Undo` and the History panel walk them. `Snapshot` Every change is a step; `Undo` and the History panel walk them. `Snapshot`
keeps the current state under a name. `Presets…` saves the settings to keeps the current state under a name. `Presets…` saves the settings to
apply elsewhere, and imports `.xmp` from other applications. apply elsewhere, and imports Lightroom presets — `Folder…` for a folder of
them, `.xmp file…` for one.
![The presets sheet](media/presets.png) The sheet lists your own presets first, then the ones DarkRoom ships —
Essentials, Skies, and colour, cinema and black-and-white film, one measured stock
each. A shipped preset is a look: it changes what it names and leaves the
photograph's own corrections alone, as an imported Lightroom preset does.
Saving under a shipped preset's name makes your version the one that name
applies, marked *changed*; `Revert` brings the shipped one back, and renaming
yours makes it one of your own. A film preset carries its stock: choosing one
sets the `Film` chooser and leaves the rest of the edit where it was.
![The presets sheet: a name for the current edit, the shipped Essentials, importing, and what an apply carries](media/presets.png)
![Scrolling down the shipped presets to the black-and-white films, applying Ilford HP5 Plus, then holding Before](media/presets-film.gif)
### Copying settings ### Copying settings
@@ -243,19 +319,46 @@ as the first step in its history.
## Export ## Export
`Export` in the develop header, or `Export N` from a selection. Format, `Export` in the develop header, or `Export N` from a selection. Format,
size, colour space, sharpening, naming and where the file goes are in size, colour space, sharpening and naming are in Settings, and apply to every
Settings, and apply to every export until changed. An export with no folder export until changed. Where the file goes is an album.
set is refused, and the header says so.
![Export defaults in Settings](media/settings-export.png) An album is a folder exports go to, listed under **Albums** in the sidebar,
below the collections. Press `+` there, or `New album…` in the export sheet
(`Ctrl+E`), name it, and choose its folder: on this device, in the system's
folder dialogue (on the tablet, Android's folder picker), or on the server,
in a browser that can make a folder as well as open one. Not inside the
library — a JPEG exported there would come back from the next scan as a
photograph of its own, and the browser says so rather than letting you.
![A new album named, before its folder is chosen](media/album-sheet.png)
The folder holds only the exported files. The album remembers which
photograph each came from, so selecting it in the sidebar shows the
originals behind its JPEGs — edit one and export it again. Albums reach your
other devices as collections do; a folder on this device does not, so an
album made on the desktop asks the tablet for a folder of its own.
Right-click an album, double-click it or press the arrow on its row to
rename it, give it another folder or delete it; deleting an album leaves
its files where they are.
![Four New York frames exported to an album, then the album chosen in the sidebar](media/albums.gif)
![The album chosen: the four photographs behind its files](media/albums.png)
An export with no album chosen is refused, and the header says so. With one
chosen, the buttons name it: `Export to Exports` in develop, `Export 4 to
Exports` on the selection bar.
![Export defaults in Settings, ending with the album exports go to](media/settings-export.png)
## Settings ## Settings
![The settings page: background activity, indexing, storage, display](media/settings.png) ![The settings page: background activity, indexing, storage, display](media/settings.png)
Background activity with progress, thumbnail and face indexing, storage on Background activity with progress, thumbnail and face indexing, how many
this device, display and colour, export defaults, what is written to XMP duplicate originals the library holds and the way to their review, storage
sidecars, and a diagnostics bundle for a bug report. on this device, display and colour, export defaults, what is written to XMP
sidecars, the manual, and a diagnostics bundle for a bug report.
## People ## People
docs/manual/index.html
+105 -18
View File
@@ -110,6 +110,7 @@ th { color: var(--ink-dim); font-weight: 600; }
<li><a href="#selecting-several">Selecting several</a></li> <li><a href="#selecting-several">Selecting several</a></li>
<li><a href="#collections">Collections</a></li> <li><a href="#collections">Collections</a></li>
<li><a href="#bursts">Bursts</a></li> <li><a href="#bursts">Bursts</a></li>
<li><a href="#duplicate-originals">Duplicate originals</a></li>
</ul> </ul>
</li> </li>
<li><a href="#developing-a-photograph">Developing a photograph</a> <li><a href="#developing-a-photograph">Developing a photograph</a>
@@ -147,13 +148,19 @@ is only about what you see.</p>
<h2 id="opening-a-library">Opening a library</h2> <h2 id="opening-a-library">Opening a library</h2>
<p>DarkRoom opens on a library: a folder on this machine, a folder a sync <p>DarkRoom opens on a library: a folder on this machine, a folder a sync
client keeps, or a Nextcloud account. A folder needs no password and uploads client keeps, or a Nextcloud account. A folder needs no password and uploads
nothing.</p> nothing. <code>Choose folder…</code> opens your desktop's own folder dialogue, which can
<figure><img loading="lazy" src="media/launch.png" alt="The launch screen: a server field, a folder field, and which formats to scan for"><figcaption>The launch screen: a server field, a folder field, and which formats to scan for</figcaption></figure> make a new folder too; the folder used last stays on the screen with
<code>Open folder</code> beside it, and <code>Choose another…</code> in place of <code>Choose folder…</code>.
On a Nextcloud account the library folder is chosen in a browser of the
server, whose <code>New folder</code> makes one there. (The browser is not pictured:
these recordings have no server behind them. Nor is the folder dialogue,
which is your desktop's rather than DarkRoom's.)</p>
<figure><img loading="lazy" src="media/launch.png" alt="The launch screen: a Nextcloud server or an app password, or the folder used last with Open folder beside it"><figcaption>The launch screen: a Nextcloud server or an app password, or the folder used last with Open folder beside it</figcaption></figure>
<p>Once a folder is named, it is the library — you are not asked for it again, <p>Once a folder is named, it is the library — you are not asked for it again,
and <code>Open library</code> opens it whole. <code>Subfolder…</code> narrows the scan to part of and <code>Open library</code> opens it whole. <code>Subfolder…</code> narrows the scan to part of
it. The formats ticked are what the scan looks for; RAW is on and JPEG off it. The formats ticked are what the scan looks for; RAW is on and JPEG off
by default, because a RAW editor's sensible default is the file the camera by default, because a RAW editor's sensible default is the file the camera
wrote first.</p> wrote first. <code>Change library</code>, at the foot of the sidebar, comes back here.</p>
<figure><img loading="lazy" src="media/launch-folder.png" alt="A folder chosen: the library, whether to scan a subfolder, and the formats"><figcaption>A folder chosen: the library, whether to scan a subfolder, and the formats</figcaption></figure> <figure><img loading="lazy" src="media/launch-folder.png" alt="A folder chosen: the library, whether to scan a subfolder, and the formats"><figcaption>A folder chosen: the library, whether to scan a subfolder, and the formats</figcaption></figure>
<h2 id="the-library">The library</h2> <h2 id="the-library">The library</h2>
<figure><img loading="lazy" src="media/library.png" alt="The grid: collections on the left, the timeline beside it, the roll of thumbnails, and the filter bar above"><figcaption>The grid: collections on the left, the timeline beside it, the roll of thumbnails, and the filter bar above</figcaption></figure> <figure><img loading="lazy" src="media/library.png" alt="The grid: collections on the left, the timeline beside it, the roll of thumbnails, and the filter bar above"><figcaption>The grid: collections on the left, the timeline beside it, the roll of thumbnails, and the filter bar above</figcaption></figure>
@@ -163,7 +170,13 @@ The bar above the grid filters by rating, flag, colour label and where the
file is (on this device, or only on the server).</p> file is (on this device, or only on the server).</p>
<h3 id="rating-and-flagging">Rating and flagging</h3> <h3 id="rating-and-flagging">Rating and flagging</h3>
<p>Hover a cell and the stars appear; click one. The filter chips above the grid <p>Hover a cell and the stars appear; click one. The filter chips above the grid
count what each rating holds, and clicking <code>3+</code> shows only those.</p> count what each rating holds, and clicking <code>3+</code> shows only those. From the
keyboard, <code>0</code> to <code>5</code> set the stars on the photograph under the pointer or on
the selection; <code>P</code> picks, <code>X</code> rejects and <code>U</code> takes the flag off, and <code>Flag</code>
on the selection bar does the same with the pointer. In
develop the same keys judge the open photograph and stay on it, and the top
bar carries its stars and <code>Pick</code> and <code>Reject</code>; the roll marks each frame's
flag and stars.</p>
<figure><img loading="lazy" src="media/library-rating.gif" alt="Rating two photographs, then filtering the grid to three stars and more"><figcaption>Rating two photographs, then filtering the grid to three stars and more</figcaption></figure> <figure><img loading="lazy" src="media/library-rating.gif" alt="Rating two photographs, then filtering the grid to three stars and more"><figcaption>Rating two photographs, then filtering the grid to three stars and more</figcaption></figure>
<p>Colour labels work as Lightroom's do: <code>6</code> red, <code>7</code> yellow, <code>8</code> green, <code>9</code> <p>Colour labels work as Lightroom's do: <code>6</code> red, <code>7</code> yellow, <code>8</code> green, <code>9</code>
blue, on the photograph under the pointer or on the selection, and the same blue, on the photograph under the pointer or on the selection, and the same
@@ -175,7 +188,13 @@ and the same keys work there.</p>
<figure><img loading="lazy" src="media/library-labels.gif" alt="Labelling four frames with 6, 7, 8 and 9, taking one off with the same key, and filtering the grid to green"><figcaption>Labelling four frames with 6, 7, 8 and 9, taking one off with the same key, and filtering the grid to green</figcaption></figure> <figure><img loading="lazy" src="media/library-labels.gif" alt="Labelling four frames with 6, 7, 8 and 9, taking one off with the same key, and filtering the grid to green"><figcaption>Labelling four frames with 6, 7, 8 and 9, taking one off with the same key, and filtering the grid to green</figcaption></figure>
<h3 id="getting-about">Getting about</h3> <h3 id="getting-about">Getting about</h3>
<p>Drag the timeline to scrub through years; Ctrl and the wheel resize the <p>Drag the timeline to scrub through years; Ctrl and the wheel resize the
thumbnails.</p> thumbnails. On a desktop a scrollbar beside the grid says how far through the
library the view is — drag its thumb, or click the track to move a page — and
the sidebar, the develop column and Settings have one too whenever they run
past the window. On a tablet they scroll by flick alone.</p>
<p><code>Help</code> in the header, or <code>F1</code>, opens the controls and shortcuts: every key and
gesture, screen by screen, with <code>See it</code> beside those this page shows, and
<code>Manual</code> to open this page. In develop it is the <code>?</code> beside <code>Settings</code>.</p>
<figure><img loading="lazy" src="media/library-timeline.gif" alt="Scrubbing the timeline"><figcaption>Scrubbing the timeline</figcaption></figure> <figure><img loading="lazy" src="media/library-timeline.gif" alt="Scrubbing the timeline"><figcaption>Scrubbing the timeline</figcaption></figure>
<figure><img loading="lazy" src="media/library-thumbsize.gif" alt="Resizing the thumbnails with Ctrl and the wheel"><figcaption>Resizing the thumbnails with Ctrl and the wheel</figcaption></figure> <figure><img loading="lazy" src="media/library-thumbsize.gif" alt="Resizing the thumbnails with Ctrl and the wheel"><figcaption>Resizing the thumbnails with Ctrl and the wheel</figcaption></figure>
<h3 id="selecting-several">Selecting several</h3> <h3 id="selecting-several">Selecting several</h3>
@@ -207,6 +226,33 @@ the grid, and again to fold it back. A folded burst shows its earliest frame;
to have it show another, open it and click the ring on the frame you want.</p> to have it show another, open it and click the ring on the frame you want.</p>
<p>With faces indexed, narrowing the grid to a person puts <code>Eyes open</code> beside <p>With faces indexed, narrowing the grid to a person puts <code>Eyes open</code> beside
their name on the filter bar, which leaves out the frames where they blinked.</p> their name on the filter bar, which leaves out the frames where they blinked.</p>
<h3 id="duplicate-originals">Duplicate originals</h3>
<p>A library put together by hand often holds the same RAW more than once — a
dated folder, a <code>bck</code> folder beside it, a renamed copy another program
exported. When the catalog finds files with the same camera, capture time
and size in more than one place, <code>Duplicate originals</code> appears under the
trash in the sidebar with how many there are; Settings offers the same page
beside the other whole-library passes.</p>
<p>The page lists each group with its picture and its paths. One copy is
marked <code>Stays</code>: the one outside a folder named like a backup (<code>bck</code>,
<code>backup</code>, <code>copy</code>, <code>old</code>…), then the one still named the way the camera named
it (<code>_MG_4623</code>, <code>IMG_0001</code>, <code>DSC_0042</code>), then the one catalogued first. Tap
another path to keep that copy instead, and untick <code>Include</code> to leave a
group alone.</p>
<p>Nothing moves until <code>Check</code> has read the first and last megabyte of every
copy and each copy's sidecar. A group whose files differ, or whose copies
carry two different edits, is marked skipped and says why — two edits of
one frame are kept for virtual copies. What the check reads is kept, so a
second visit costs nothing. The line under each group says what the copy
that stays will gain: the highest rating, every keyword and collection, a
flag or label the copies agree on, faces and the names on them, and the
edit if only a copy had one. Where the copies disagree on a flag or a
label, the one that stays keeps its own and the line says so.</p>
<figure><img loading="lazy" src="media/duplicates.png" alt="Two frames with a copy each in a bck folder, checked and proved the same, the copy that stays marked on each"><figcaption>Two frames with a copy each in a bck folder, checked and proved the same, the copy that stays marked on each</figcaption></figure>
<p><code>Move N copies to trash</code> does it, one group at a time: each group is merged
and its spare copies moved together, or not at all. The copies go to the
trash, not away — select them in the trash view and <code>Restore</code> puts them
back where they were.</p>
<h2 id="developing-a-photograph">Developing a photograph</h2> <h2 id="developing-a-photograph">Developing a photograph</h2>
<p>Click a thumbnail to open it. The column on the right is every adjustment; <p>Click a thumbnail to open it. The column on the right is every adjustment;
the strip at its head narrows it to one group.</p> the strip at its head narrows it to one group.</p>
@@ -225,8 +271,17 @@ the sensor recorded.</p>
<h3 id="moving-between-photographs">Moving between photographs</h3> <h3 id="moving-between-photographs">Moving between photographs</h3>
<p>The roll along the foot of the canvas holds the photographs the grid was <p>The roll along the foot of the canvas holds the photographs the grid was
showing; click one to open it. The right arrow, <code>D</code> or space opens the next, showing; click one to open it. The right arrow, <code>D</code> or space opens the next,
and the left arrow or <code>A</code> the one before. The edit on screen is saved on the and the left arrow or <code>A</code> the one before; held down, they go on past the
way, so stepping along a shoot loses nothing.</p> stretch the roll has loaded, through everything the grid would show. The
edit on screen is saved on the way, so stepping along a shoot loses nothing.</p>
<p>On a Nextcloud library a photograph may not be on this device yet. Its
thumbnail from the grid stands in at once; if the original has to come down,
the thumbnail dims under <em>Not on this device yet</em>, with how far the download
has got — <code>Downloading — 12.4 of 38.0 MB</code> — and a bar, and the photograph
opens when it lands. Step on before then and the one you step to is the one
that opens: a download that arrives late is kept for later and never takes
the place of the photograph whose name is showing. (Not pictured: a folder
library, which these recordings use, never has a photograph to wait for.)</p>
<h3 id="white-balance-from-the-photograph">White balance from the photograph</h3> <h3 id="white-balance-from-the-photograph">White balance from the photograph</h3>
<p>Press <code>pick</code> in the White Balance group, then click something neutral — <p>Press <code>pick</code> in the White Balance group, then click something neutral —
a white wall, a grey card, the air conditioner here. The picker sets the a white wall, a grey card, the air conditioner here. The picker sets the
@@ -269,13 +324,25 @@ opacity are in the panel. Heal blends; clone copies.</p>
<h3 id="film">Film</h3> <h3 id="film">Film</h3>
<p>The <code>Film</code> chooser at the head of Adjust applies a spectral simulation of a <p>The <code>Film</code> chooser at the head of Adjust applies a spectral simulation of a
named stock; below it, the print exposure and push controls a film has and a named stock; below it, the print exposure and push controls a film has and a
sensor does not.</p> sensor does not. The list opens over the column and scrolls on its own — by
<figure><img loading="lazy" src="media/film.gif" alt="Choosing Velvia, then holding Before"><figcaption>Choosing Velvia, then holding Before</figcaption></figure> wheel, drag or flick, or with <code>Up</code>, <code>Down</code> and <code>Enter</code> — down to the
black-and-white stocks at its end.</p>
<figure><img loading="lazy" src="media/film.gif" alt="Opening the film list, scrolling it, choosing Velvia, then holding Before"><figcaption>Opening the film list, scrolling it, choosing Velvia, then holding Before</figcaption></figure>
<h3 id="history-snapshots-presets">History, snapshots, presets</h3> <h3 id="history-snapshots-presets">History, snapshots, presets</h3>
<p>Every change is a step; <code>Undo</code> and the History panel walk them. <code>Snapshot</code> <p>Every change is a step; <code>Undo</code> and the History panel walk them. <code>Snapshot</code>
keeps the current state under a name. <code>Presets…</code> saves the settings to keeps the current state under a name. <code>Presets…</code> saves the settings to
apply elsewhere, and imports <code>.xmp</code> from other applications.</p> apply elsewhere, and imports Lightroom presets — <code>Folder…</code> for a folder of
<figure><img loading="lazy" src="media/presets.png" alt="The presets sheet"><figcaption>The presets sheet</figcaption></figure> them, <code>.xmp file…</code> for one.</p>
<p>The sheet lists your own presets first, then the ones DarkRoom ships —
Essentials, Skies, and colour, cinema and black-and-white film, one measured stock
each. A shipped preset is a look: it changes what it names and leaves the
photograph's own corrections alone, as an imported Lightroom preset does.
Saving under a shipped preset's name makes your version the one that name
applies, marked <em>changed</em>; <code>Revert</code> brings the shipped one back, and renaming
yours makes it one of your own. A film preset carries its stock: choosing one
sets the <code>Film</code> chooser and leaves the rest of the edit where it was.</p>
<figure><img loading="lazy" src="media/presets.png" alt="The presets sheet: a name for the current edit, the shipped Essentials, importing, and what an apply carries"><figcaption>The presets sheet: a name for the current edit, the shipped Essentials, importing, and what an apply carries</figcaption></figure>
<figure><img loading="lazy" src="media/presets-film.gif" alt="Scrolling down the shipped presets to the black-and-white films, applying Ilford HP5 Plus, then holding Before"><figcaption>Scrolling down the shipped presets to the black-and-white films, applying Ilford HP5 Plus, then holding Before</figcaption></figure>
<h3 id="copying-settings">Copying settings</h3> <h3 id="copying-settings">Copying settings</h3>
<p><code>Copy</code> in the top bar, or Ctrl+C, takes this photograph's settings; <code>Paste</code>, <p><code>Copy</code> in the top bar, or Ctrl+C, takes this photograph's settings; <code>Paste</code>,
or Ctrl+V, puts them on another, and says what it would paste — how many or Ctrl+V, puts them on another, and says what it would paste — how many
@@ -297,15 +364,35 @@ as the first step in its history.</p>
<figure><img loading="lazy" src="media/panorama-filled.png" alt="The same, with the ragged border filled by the model rather than cropped away"><figcaption>The same, with the ragged border filled by the model rather than cropped away</figcaption></figure> <figure><img loading="lazy" src="media/panorama-filled.png" alt="The same, with the ragged border filled by the model rather than cropped away"><figcaption>The same, with the ragged border filled by the model rather than cropped away</figcaption></figure>
<h2 id="export">Export</h2> <h2 id="export">Export</h2>
<p><code>Export</code> in the develop header, or <code>Export N</code> from a selection. Format, <p><code>Export</code> in the develop header, or <code>Export N</code> from a selection. Format,
size, colour space, sharpening, naming and where the file goes are in size, colour space, sharpening and naming are in Settings, and apply to every
Settings, and apply to every export until changed. An export with no folder export until changed. Where the file goes is an album.</p>
set is refused, and the header says so.</p> <p>An album is a folder exports go to, listed under <strong>Albums</strong> in the sidebar,
<figure><img loading="lazy" src="media/settings-export.png" alt="Export defaults in Settings"><figcaption>Export defaults in Settings</figcaption></figure> below the collections. Press <code>+</code> there, or <code>New album…</code> in the export sheet
(<code>Ctrl+E</code>), name it, and choose its folder: on this device, in the system's
folder dialogue (on the tablet, Android's folder picker), or on the server,
in a browser that can make a folder as well as open one. Not inside the
library — a JPEG exported there would come back from the next scan as a
photograph of its own, and the browser says so rather than letting you.</p>
<figure><img loading="lazy" src="media/album-sheet.png" alt="A new album named, before its folder is chosen"><figcaption>A new album named, before its folder is chosen</figcaption></figure>
<p>The folder holds only the exported files. The album remembers which
photograph each came from, so selecting it in the sidebar shows the
originals behind its JPEGs — edit one and export it again. Albums reach your
other devices as collections do; a folder on this device does not, so an
album made on the desktop asks the tablet for a folder of its own.
Right-click an album, double-click it or press the arrow on its row to
rename it, give it another folder or delete it; deleting an album leaves
its files where they are.</p>
<figure><img loading="lazy" src="media/albums.gif" alt="Four New York frames exported to an album, then the album chosen in the sidebar"><figcaption>Four New York frames exported to an album, then the album chosen in the sidebar</figcaption></figure>
<figure><img loading="lazy" src="media/albums.png" alt="The album chosen: the four photographs behind its files"><figcaption>The album chosen: the four photographs behind its files</figcaption></figure>
<p>An export with no album chosen is refused, and the header says so. With one
chosen, the buttons name it: <code>Export to Exports</code> in develop, <code>Export 4 to Exports</code> on the selection bar.</p>
<figure><img loading="lazy" src="media/settings-export.png" alt="Export defaults in Settings, ending with the album exports go to"><figcaption>Export defaults in Settings, ending with the album exports go to</figcaption></figure>
<h2 id="settings">Settings</h2> <h2 id="settings">Settings</h2>
<figure><img loading="lazy" src="media/settings.png" alt="The settings page: background activity, indexing, storage, display"><figcaption>The settings page: background activity, indexing, storage, display</figcaption></figure> <figure><img loading="lazy" src="media/settings.png" alt="The settings page: background activity, indexing, storage, display"><figcaption>The settings page: background activity, indexing, storage, display</figcaption></figure>
<p>Background activity with progress, thumbnail and face indexing, storage on <p>Background activity with progress, thumbnail and face indexing, how many
this device, display and colour, export defaults, what is written to XMP duplicate originals the library holds and the way to their review, storage
sidecars, and a diagnostics bundle for a bug report.</p> on this device, display and colour, export defaults, what is written to XMP
sidecars, the manual, and a diagnostics bundle for a bug report.</p>
<h2 id="people">People</h2> <h2 id="people">People</h2>
<p>Face detection and identity run over the library and group faces by person; <p>Face detection and identity run over the library and group faces by person;
the <code>Identity</code> page is where suggestions are confirmed, rejected and split, the <code>Identity</code> page is where suggestions are confirmed, rejected and split,
docs/manual/media/album-sheet.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/albums.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/albums.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/compose-perspective.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/compose.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/crop-orphan.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/develop-groups.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/develop-light.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/develop-wb.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/develop-zoom.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/develop.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/duplicates.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/film.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/launch.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-collection-menu.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-collections.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-keywords.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-labels.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-nesting.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-nesting.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-rating.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-selection.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-thumbsize.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library-timeline.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/library.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/local-categories.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/local-intersect.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/local-paint.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/local-segment.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/panorama.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/presets-film.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/presets.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/repair.gif
LFS
BIN
View File
Binary file not shown.
docs/manual/media/settings-export.png
LFS
BIN
View File
Binary file not shown.
docs/manual/media/settings.png
LFS
BIN
View File
Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More