Seven names are two or three live people on both devices: Ian (756
confirmed faces, and a second Ian with none), Jessie three times,
Claudine, Mathias, Noemi, Pascal and PJ. Each was typed on its own
device and carried across by sync, which keys people on their uuid and
so keeps both. Each half of a person shows half their photographs.
dedup_people::run, in one transaction:
- Same-name people (trimmed, case-folded as the Identity screen folds
them) merge into the one with the most confirmed faces, ties to the
smaller uuid, through faces::merge_people_within, so confirmations,
rejections and the survivor's name are kept. A person holding no
faces at all merges: there is nothing to compare or to carry. Anyone
else needs >= 2 confirmed faces per shared embedder on both sides and
centroids at cosine >= 0.7 in each. A face confirmed as one and
rejected as the other keeps them apart. Unnamed and set-aside people
are never merged by name.
- Faces held twice (one image, one embedder, IoU >= 0.5, cosine >= 0.7)
keep the stronger detector's row (FaceDetector::outranks), then the
confirmed one, then the older. The survivor takes the confirmed
assignment and both rows' rejections. A pair confirmed as two
different people is left and counted.
- Judgements still on a merged-away person move to the person at the
end of its redirects, and a redirect cycle (two devices merging one
pair in opposite directions) is broken at the smaller uuid.
Measured on copies of the desktop catalog and the tablet's server
snapshot, w600k_mbf, confirmed faces only:
- Centroids of differently named people: 2,699 pairs, median 0.02,
99.9th percentile 0.41. One pair reaches 0.70 (0.700 desktop, 0.705
tablet), "Michelle Casanonve" and "Michelle Casanova", one person
typed two ways. Next is 0.62/0.64, "Boris Jost" and "Boris". The
highest pair that is plainly two people is 0.43/0.44.
- One person split in random halves: minimum 0.69, median 0.91 over 72
people. Four faces against twenty-two reach 0.7 in 97% of draws.
One face against twenty of somebody else's reached 0.74 in 3,000
draws, and two faces reached 0.61, hence the two-face minimum.
- Pascal (22 and 4 confirmed) is at 0.57 and PJ (14 and 7) at 0.50,
under 0.7 on both devices, so both pairs stay apart and are logged.
The desktop's second Ian holds 4 suggestions and no confirmations,
at 0.38 against Ian's centroid, and stays apart. On the tablet it
holds nothing and merges.
Why a merge made here survives a peer on 0.17.0: the merged-away
person stays as a merged_into redirect with a bumped revision, which
the catalog merge has always taken on revision. The peer hides the
duplicate and never sends it back as a live person. Its own
confirmations of that person stay on the redirect, because a merge
never overwrites a local confirmation. The manual merge has always
left them there too. They follow the redirect when the peer runs this
job. A test syncs two catalog files through the previous merge code
and back, and the people converge and stay converged.
Once a catalog is clean the job reads 80 redirects, the named people,
and the face boxes from the covering faces_box index. That is ~10 ms
on the reference library. There is no schema change. The index is
created IF NOT EXISTS, as the merge already does.
merge_people moved a person's faces onto the target and left the
"not this person" rejections on the redirect. A rejection there binds
nothing: once Annie is Anna, the grouping pass is free to suggest the
face the user pushed away from Annie as Anna, which is the behaviour
rejections exist to prevent. The Identity screen's merge has done this
since it was written, and the deduplication job for #78 merges through
the same function, so it would have done it for every same-name pair.
The source's rejections now move to the target (INSERT OR IGNORE, so
one the target already holds is not doubled). Where the two halves
disagree about one face, confirmed as one and rejected as the other,
the confirmation stands, as `confirm` already rules for one face; and a
moved rejection withdraws a suggestion of the same face, as `reject`
already does. Confirmations and names are unchanged.
The body is split into merge_people_within, taking the caller's
transaction, so a job that merges several pairs commits once
(unchecked_transaction cannot nest). merge_people keeps its signature
and its one transaction.
The grouping pass never puts two faces of one photograph in the same
group (the cannot-link in dr_face::cluster). The merge did not check
this. When the two devices disagree about which face in a frame is a
person, merge_people_within applied the remote's confirmation, or the
anchor of a set-aside group, to face X. This device already held the
same person on face Y of the same photograph, so the person ended up on
both faces.
The reference library has 80 such person/photograph pairs on the
desktop and 89 on the tablet: 79/88 unnamed set-aside groups and one
named person confirmed on two faces. There are no duplicate faces (no
pair of faces in one image and embedder with IoU >= 0.5).
An incoming assignment is now refused when another local face of the
same photograph already holds that person. The one exception is an
incoming confirmation against a local suggestion: the suggestion is
withdrawn and the confirmation is applied. Two confirmations stay as
this device has them, the same rule as a local confirmation outranking
a remote one. A face that already holds the person is not a rival to
itself, so a steady-state pass is unaffected. On the reference pair
this refuses 0 assignments and writes the same 8,954 as before; it only
changes what a future disagreement does. The refusals are counted in
MergeReport::faces_one_per_photograph.
Existing pairs are left alone. They are two different faces (cosine
0.31 for the named one), not one face twice, so there is nothing to
fuse, and which face is the wrong one is not the merge's to guess.
On the reference library, 631 of the faces in the tablet's snapshot
match no desktop face by box (IoU >= 0.5), so a name on them stays on
one device. Twenty of those are the same face with the box drawn
somewhere else. Whole photographs sit at IoU 0-0.48 with cosines of
0.72-0.96 between the two devices' vectors, and ten of them already
carry the same person on both sides. The merge never read the
1 KB embedding every face row carries.
match_faces now runs two passes. The box pass is unchanged except that
a pair must now be unique on both sides: a remote face with two
overlapping local faces, or a local face overlapped by two remote ones,
is no longer settled by whichever overlap is larger. Only for the
photographs where a remote face is left over, and a local face is still
free, does it read vectors: one json_each statement per side, keyed by
row id. That was 357 photographs on the reference library, not all
19 MB of vectors. A left-over face pairs with the local face it
resembles most when:
- the cosine is >= 0.7,
- the two are each other's best,
- each leads its runner-up by >= 0.2, and
- a box has not already claimed the local face.
Anything less decisive stays unmatched, so a new face stays new.
Why the threshold is safe, measured on both catalogs (w600k_mbf):
- Of 169,548 pairs of different faces in one photograph, 4 reach 0.7
(lookalikes in one frame) and the maximum is 0.82.
- At 0.6 the rule would claim two pairs that carry different people on
the two devices. At 0.7 it claims 20, none contradicted and 10
corroborated, each leading its runner-up by more than 0.5.
- It only compares faces the boxes left unmatched on both sides: 737
such pairs, so about 0.02 false pairs expected.
- A low cosine never overrules a box. About 150 box-matched pairs fall
below 0.45, because two detectors cut the same tiny face differently.
73 of them carry the same person on both devices.
- Faces are compared only within one file_id and one embedder, because
the same person in another photograph reaches cosine 1.0.
- `Embedding::cosine` refuses a comparison across models.
Before -> after on the reference pair: matched by box 18,348 -> 18,348,
by embedding 0 -> 20, unmatched 631 -> 611, ambiguous 0 -> 0. The
report counts the embedding matches. No schema change.
The bench merged the catalog with a copy of itself. Every face in that
merge matches its own box, so the pass never reaches the faces the two
devices disagree about. On the reference library that is 631 of the
tablet's 19,052 faces, and it is the work #77 adds to.
`--remote PEER.sqlite` now also times `merge_remote_catalog` against a
copy of the peer file, and prints the first pass's report so two builds
can be checked for agreement. The first run writes what the peer
brought. The runs after it are the steady state, so compare builds from
two fresh copies of one catalog.
Blue sky, Deep blue sky, Polariser, and Blue sky with golden land, in a
Skies section after Essentials. Each darkens the colour mixer's azure and
blue bands and adds chroma to them — what a polarising filter does to a
clear sky — and brings the highlights down with it, so a white cloud does
not read as a cut-out against the deeper blue. The stronger ones nudge
azure towards blue and add dehaze.
They are looks and work only on the hues a sky occupies, so an overcast
frame is left nearly alone: there is no blue for them to deepen, and
tinting grey cloud blue would be worse than doing nothing. Tuned by eye on
the demo library's alpine and Manhattan frames, with an overcast Étretat
frame as the control.
The backfill stamp read max(id) of images and versions and max(rowid)
of keywords. None of those tables is AUTOINCREMENT, so SQLite hands a
freed newest id out again: empty the trash of the newest photograph and
scan a new one, or let a local folder's walk delete a renamed file's row
and insert the new name in the same pass, and the new image takes the
old id. max(id) does not move, nor does count(*), and when a newer
version elsewhere keeps max(versions.id) still too, the stamp matched
and the open skipped the backfill.
That row is exactly one that needs it. Neither scan path creates the
default version: scan::persist and walk insert the image and leave the
version, the RAW/JPEG pairing and the keyword terms to the next open.
Skipped, the image went without them until the app restarted, so a
rating or a pulled sidecar judgement had no version to land on and a
JPEG beside its RAW showed twice.
The stamp now carries the newest row's content: the newest image's id,
path, added time and whether it has a version; the newest version's id
and image; the newest assignment's rowid, version and word. Whether the
newest image has a version is the part that cannot be fooled - after a
backfill every image has one, and a row that has just taken a freed id
has none - so the two stamps differ even when the same file comes back
at the same id in the same second. Still one statement: three reverse
rowid scans that stop at the first row, and one probe of versions_image.
An open that skips still costs ~1 ms on the reference catalog copy.
This closes the hole in the stamp itself rather than by a forget() at
each delete site, so a delete path added later, or one in another
process, cannot reopen it. Two tests delete the newest image and insert
another at the freed id on a separate connection, with a newer version
elsewhere holding max(versions.id); both fail against the old stamp.
Catalog::open ran schema::backfill every time, and every worker thread
opens its own connection. A develop landing made five opens, and each
paid the RAW/JPEG pairing, the default-version anti-join over every
image, the uuid pass over every default version and the keyword check:
17 ms of CPU an open on a copy of the reference catalog, ~80 ms a
landing, to confirm that nothing had changed since the open before.
Everything the backfill repairs is a row some write added: an image a
scan inserted, a version or keyword assignment a merge brought in. So
the open now reads a stamp - user_version, max(id) of images and
versions, max(rowid) of keywords, and the file's device and inode - and
skips the backfill when the stamp matches the one recorded at this
path's last backfill in this process. The maxima are each the last page
of a b-tree; an open that skips costs ~1 ms.
The backfill still runs:
- on the first open in a process (nothing recorded yet);
- on any open that migrated the schema, unconditionally;
- after a pull: merge_remote forgets the path, so the next open
backfills even when every incoming row collided and nothing moved;
- when the file is replaced under its name: the inode is in the stamp,
and recovery::set_aside, the first step of a restore and a rebuild,
forgets the path;
- when another process or thread adds rows, because the stamp is read
from the file, not from anything this process did.
The stamp is taken before the backfill, not after. Read after, it would
describe the backfill's own inserts, and could record an image another
connection inserted in between as covered when it was not. Read before,
the worst case is one redundant pass after a backfill that did real work.
Kept in memory rather than in the catalog: a stamp row would need a
table an older build does not have and would travel in the sync
snapshot, where a flag from another device's catalog says nothing about
this one. No schema version bump, so the tablet on 0.16.0 still reads
the snapshot. Tests cover the skip, a scan's new image, a migration, a
pull and a replaced file.
Landing on a photograph in develop opens the catalog once to fetch the
original and once more per prefetched neighbour to ask whether the cache
already holds it: five opens, each running the whole backfill. The bench
timed one open but not the landing, so the cost of the shape was not
visible and a fix to it could not be measured.
Two figures now, both against an empty cache so the question is asked the
same way whatever the answer: the five-open shape the app had, and the
two-open shape where the prefetch worker keeps one connection for its
batch. On a copy of the reference catalog (23,582 images) under load, the
five-open landing costs ~80 ms of CPU.
Dehaze cost 22.9 ms of a 2560x1600 frame on the reference laptop RTX 3050,
and 54.1 ms at 3840x2160, with the memory clock held at 810 MHz by the power
cap (graphics 1762 MHz). It ran five passes: a run and a span erosion along
x, the same along y, and the recovery. At those clocks a detail pass costs
what it reads and writes, not what it taps: a pass with an empty body -
one render-sized rgba16float read and write - measured 4.0 ms, and each
dehaze pass 4.4-4.6 ms, so the taps were about 2 ms of the 22 and the four
hand-offs between passes were the rest.
Each axis is now one pass that takes the minimum over the whole window
directly, and the recovery rides in the y pass, which already holds the
veil and the pixel's own colour. That is 36 texture reads per pixel at
2560x1600 in place of 12, nearly all of them cache hits, and two passes in
place of five.
The picture is the same bits. A minimum is exact in any order, and the
window is the one Split always covered, the surplus pixel on the far side
included (Split::first and Split::width). The veil crossing the removed
hand-offs was already exactly representable in rgba16float - a minimum of
channels read from rgba16float, floored at zero - so storing it between
passes never rounded anything that the fused form now keeps unrounded.
Measured with a scratch probe that renders the synthetic 60 MP frame from
examples/frame_budget.rs, only a detail parameter moving so the fused pass
is reused, 30 frames per scene after six of warm-up, five runs of each
binary alternated, median of the per-run p50:
scene before after
dehaze 2560 fit 22.88 ms 9.06 ms
dehaze 2560 1:1 23.41 ms 9.52 ms
dehaze 3840 fit 54.09 ms 28.12 ms
all detail 2560 fit 53.11 ms 39.97 ms (NR, sharpen, clarity,
all detail 2560 1:1 67.48 ms 56.42 ms texture, dehaze)
every op 2560 fit 57.59 ms 44.19 ms (with film)
every op 2560 1:1 71.83 ms 57.93 ms
controls without dehaze (NR, sharpen, clarity, texture): within +-2%
The rgba8 output hashed identically before and after for every scene -
dehaze alone, all five detail operations, every operation with film, and
each other detail operation alone - at fit and 1:1, at 2560x1600,
3840x2160, 1917x1203 and 333x211: 64 of 64.
rustfmt over the files the albums work touched, and the album merge's
incoming row as a named struct rather than an eight-field tuple, which
clippy's type_complexity refused.
Export took a path typed into the settings page, or a folder inside the
library on the server. The first is how exports end up somewhere nobody
looks; the second put JPEGs into the tree a scan catalogues, where they
came back as photographs beside the RAWs they were made from.
The destination is now an album (FR-EXP-10), chosen by name in the
export sheet. Albums are listed under the collections in the sidebar;
"+" there, or "New album…" in the sheet, opens a sheet for its name and
its folder — on this device through the platform's dialogue, or on the
server through the browser with "New folder". A server folder inside
the library is refused, and the sheet says why. Selecting an album
narrows the grid to the photographs behind its files: library::Scope
is Collection or Album, and scope_clause is the one place the two are
spelled, which also retires the two copies of the collection predicate
total_images_scoped and read_cells_scoped had inlined.
A batch resolves the album when it starts, and refuses in words when
none is chosen, it has gone, or its folder is local to another device.
Each item reports the image it came from, and the files written are
recorded against the album in one transaction when the batch ends.
A server album lives outside the library, so its queued uploads are
relative to the account root. That is a third line in the outbox's
.dest record rather than a leading slash, because a record written
before albums may carry a stray slash and must keep the meaning it was
written with.
An export folder set before albums becomes an album called "Exports"
on first open, so upgrading does not lose where exports were going.
The old destination fields stay in ExportSettings so older settings
files still read.
An album is a named export destination. Its folder holds only the
exported files; the catalog records, per file, the image it was
rendered from, so an album can show the originals behind its JPEGs
(FR-EXP-10).
The tables are created on first use (CREATE TABLE IF NOT EXISTS), the
way dedup_probes is, rather than by a schema migration: a new
user_version makes every older build refuse this catalog's snapshot at
sync, and the 0.16.0 tablet would stop merging collections, keywords
and people for a feature it does not have.
Albums merge as collections do: by uuid and revision, tombstones on
delete, exports as a set union keyed on the server's file id (content
hash for a folder library). A folder on the server lives on the album
row and syncs; a folder on this device lives in album_folders, which
the merge never reads and the upload snapshot drops, because a path or
a SAF grant on one device means nothing on another.
Exports are keyed on the file name, not the image: two crops of one
photograph are two files and two rows, and an overwrite re-points the
name at whatever wrote it last.
Each sync pass spent 0.8-2.0 s of CPU and 1.0-5.4 s wall on the upload
snapshot of the reference catalog (24k images, 18,871 faces), ahead of the
rest of the pass. The upload itself had been crop-less since the crops
moved to the face shards. The cost was in how it got that way. The backup
API copied all 158 MB of the catalog, 96 MB of it the ~5 KB JPEG crop on
every faces row. Then `UPDATE faces SET crop = NULL` rewrote 18.9k rows
and freed their overflow chains, and VACUUM rebuilt the file again. That
wrote the catalog about three times over to upload 50 MB.
The snapshot is now built rather than copied. An empty file attaches the
catalog, creates each table from the catalog's own sqlite_master and fills
it with INSERT ... SELECT, with faces.crop selected as NULL. Indexes,
triggers and views follow, and user_version, application_id, page size and
the WAL header flag are carried over. It all runs in one transaction on the
snapshot's connection, so the catalog is read as of one moment and
concurrent writers are serialised, not raced, as the backup API did. The
build journal is in memory with synchronous off, because the file is
scratch that is rebuilt every pass and quick_check'd before upload. Foreign
keys are off on that connection. The bundled SQLite enables them, and then
a multi-row INSERT into images scans images for children of each new row
(shadowed_by is a self-reference with no index), which cost 1.2 s alone.
Measured on a .backup copy of the reference catalog with catalog_bench,
old and new binaries back to back on a loaded machine:
before best 1.0-5.4 s wall, 0.84-1.98 s cpu, 49.8 MB
after best 0.40-2.1 s wall, 0.39-0.96 s cpu, 50.4 MB
With the machine quiet the new build takes 0.31-0.43 s.
What a receiving device gets is unchanged. It is the same schema, the same
rows and a NULL crop, which is what 0.16.0 already uploads and merges. The
merge reads only a remote face's box and model (merge::match_faces) and
never writes a local crop. No device adopts a downloaded catalog as its
own, and a fresh one takes faces and crops from the shards. There is no
schema bump, so older builds still merge it. NFR-R2 backups keep using the
backup API and keep their crops.
Tests: the snapshot matches the catalog in schema, row counts, pragmas and
WAL header. A leftover file is replaced. Merging a crop-less snapshot
carries a confirmed name across by box and leaves the local crop
untouched, and does so idempotently.
A Lightroom preset changes the settings it was saved with and leaves every
other one where the photograph had it. Imported as a whole edit, a preset
holding only a grade reset the exposure, white balance and noise reduction
it was put on top of — the opposite of what the photographer had in
Lightroom.
Imported presets now reach only the operations they name
(`Reach::Named`), the rule the shipped presets already follow.
The six starter presets were copied into the photographer's own library
on a first run and were theirs from then on. That cannot grow into a
real collection: a copy is frozen at the release that wrote it, so an
improved preset reaches nobody who had the old one, and re-seeding would
overwrite a preset someone had tuned.
`dr_pipeline::bundled` now holds the shipped presets as `.drpl` files
compiled into the binary, in sections — Essentials (the former six) and
three sections of film presets, one per measured stock in dr-film,
printed on the paper its profile names — and never writes them to the
user's file. Every shipped preset is a look (`Reach::Named`), so applying
one keeps the corrections a photograph already has.
A name links a photographer's copy to a shipped preset. Saving over a
shipped name makes their version the one that name applies; it is listed
in the shipped section, marked as changed, and deleting it reverts to the
shipped one. Renaming it makes it one of their own and the shipped preset
reappears. Keyed on the name because that is what the photographer sees
and chooses by.
Copies an older first run seeded are forgotten on load where they are
still exactly as seeded — otherwise all six would list as changed and
stay frozen at their old values. A tuned one is kept and now overrides.
The sheet lists "Yours" first, then each shipped section, with headings.
Shipped rows apply and nothing else; a changed row offers Revert where
the photographer's own offer Delete. A dr-ui test checks every shipped
film names a stock this build can bake, on that stock's own paper,
because dr-pipeline does not link the profile database.
The film presets name stocks by id; the measurements behind them are
spektrafilm's (CC BY-SA 4.0), attributed in each file as in dr-film.
A preset could not choose a film stock. The stock is a choice of material
rather than a parameter, so `Preset` — a map of `op.param = value` — had
nowhere to hold it, and "Portra 400, printed" could not be saved, copied
or shipped as a look. Worse, the film node's own sliders *were*
parameters: a paste moved one stock's exposure and push onto whatever
stock the target was on, and left the target's tables baked from the
values it had just replaced.
A preset now carries a `FilmRef` beside its parameters. It travels under
whichever scope carries the film node, so the stock and its sliders are
never split, and by the replacement rule every other parameter follows:
applied at that scope, a preset without a film develops the target
without one. `Preset::apply` returns the `FilmRebake` it owes, as
`EditGraph::set_state` already did, because this crate cannot bake a
stock; the develop session pays it before recording the step, and the
batch paste writes the stock into each sidecar through `film_for`. The
library file spells it `film =` / `film_print =`, as a sidecar does, and
an older build keeps those lines as ones it does not understand.
`EditState` keeps the film in its own field only: the parameters it
captures leave it out, so one edit has one place to say which stock it
is on.
Second, a preset now has a reach. Replacement is right for a copy of a
whole edit — "make these match" — and wrong for a look: a stock-only
"Portra 400" applied that way would put the photograph's exposure, white
balance and noise reduction back to default. `Reach::Named` replaces only
the operations a preset names (whole operations, so a look that sets the
blacks resets the whites beside them) and the film only if it names one.
Saved edits and the clipboard keep `Reach::Whole`; the line `reach =
named` is written only for the other, so existing libraries write the
same bytes.
After 9cff677 the loop over the other device's confirmed and ignored faces
(13,000 on the reference library) still asked three cached statements per
face -- the person by uuid, the face's current assignment, and whether this
pair was rejected here. It was 51 ms of a steady-state merge.
The person is now resolved in the statement that reads the incoming rows,
by the local `people.uuid` key:
SCAN fp
SEARCH p USING INTEGER PRIMARY KEY (rowid=?)
SEARCH lp USING COVERING INDEX sqlite_autoindex_people_1 (uuid=?)
and the local `face_person` (16,800 rows) and `face_person_rejected` are
each read once into memory and looked up there. A write goes to the table
and to the map, so a second remote face matched to the same local face sees
what the first left, as it did when each face re-read the table. The
incoming rows are ordered by face id -- the order the table was already
walked in -- since which of two such faces is applied last decides the
answer. An inner join to `people` drops the rows the old loop skipped for
want of a local person, and the counts in the report are unchanged.
After: the loop 10-12 ms. The merge as a whole, with the two changes before
this, went from 228-231 ms to 135 ms best of 5, and every catalog table
checksums the same after the bench as after the old build's run.
`merge::match_faces` reads every local face's box and model to pair the
other device's faces with ours. It took 54 ms of a steady-state merge on the
reference library (19,000 faces).
A `faces` row is eight kilobytes -- the embedding, the crop, the dense
landmarks -- and `model_id` sits past the embedding, so reading it opened
each row's overflow pages:
SCAN f
SEARCH r USING INTEGER PRIMARY KEY (rowid=?)
`faces_box (image_id, model_id, x, y, w, h)` holds every column the scan
asks for:
SCAN f USING COVERING INDEX faces_box
SEARCH r USING INTEGER PRIMARY KEY (rowid=?)
The local scan went from 38 ms to 8 ms (sqlite3 on a copy, aggregated so
output formatting is not timed), and `match_faces` from 54 ms to 30-37 ms;
what remains is the other device's half. That is read from its snapshot,
which has whatever indexes its build made -- this one will carry
`faces_box` in its uploads -- and whose rows have had their crops stripped.
The bench merges a full copy with crops, so it overstates that half.
Created on first use in `match_faces`, with CREATE INDEX IF NOT EXISTS,
rather than by a migration, for the reason `keywords::ensure_term_index`
gives: a schema version bump makes older builds refuse the snapshot, and an
extra index is invisible to them. The first merge after the upgrade builds
it (about a second, once). Its prefix duplicates `faces_image_model`, which
is left alone; the planner takes either for an (image_id, model_id) probe.
Tables checksum the same after the bench run as after the old build's.
`merge_remote_catalog` on the reference library (catalog_bench, a copy
merged with itself: the steady state of a sync pass) cost 228-231 ms best
of 5. Timing its phases put 91 ms in the keyword half, not in the faces the
issue named.
Both assignment unions refuse a word this device holds only as a tombstone,
with a correlated `NOT EXISTS (... deleted = 1) OR EXISTS (... deleted = 0)`
per incoming assignment. The `deleted = 1` half has no index to use --
`keyword_terms_name` is partial on `deleted = 0` -- so it scanned the whole
vocabulary for each of the 10,800 rows:
SCAN rk
CORRELATED SCALAR SUBQUERY 1
SCAN t
CORRELATED SCALAR SUBQUERY 2
SEARCH t USING COVERING INDEX keyword_terms_name (name=?)
The refused words are one set for the whole statement, so it is asked once:
`rk.keyword NOT IN (tombstoned names EXCEPT live names)`, which is the same
condition -- refused exactly when deleted under some identity and live under
none -- and which SQLite builds as a list before the walk:
SCAN rk
LIST SUBQUERY 2
MERGE (EXCEPT) ...
The file-id union alone went from 72 ms to 11 ms (sqlite3 on a copy), and
the keyword phase of the merge from 91 ms to 28-35 ms. Every table of the
catalog checksums the same after the bench as after the old build's run,
and the merge tests for tombstones and renames pass unchanged.
The grid's total is read on every scroll reload (`load_window` compares it
to notice a delete). On the reference library it cost 1.3-1.5 ms best-of-50
by catalog_bench, 2-3.6 ms on a busy machine, and the issue measured 4 ms.
`uncollapsed` asked every visible image whether a collapsed burst stands in
for it -- two primary-key probes per image, 19,000 times, on a library with
no bursts at all:
SCAN i USING INDEX images_grid_order
CORRELATED SCALAR SUBQUERY
SEARCH bm USING INTEGER PRIMARY KEY (rowid=?)
CORRELATED SCALAR SUBQUERY
SEARCH be USING INTEGER PRIMARY KEY (rowid=?)
`total_images_filtered` now counts what the filter keeps and subtracts the
frames `bursts::collapsed_away_frames` lists, under the same filter:
SCALAR SUBQUERY: SCAN i USING INDEX images_grid_order
SCALAR SUBQUERY: SCAN bm; SEARCH be ...; SEARCH i USING INTEGER PRIMARY KEY
The second half walks only `burst_members`. Each image is in it at most
once (it is the key), and the filter is applied to both halves, so the
subtraction removes exactly the rows the predicate used to drop. The new
fragment sits beside `not_collapsed_away` in bursts.rs, and a test holds
the two to the same rows with bursts open and closed.
After: 0.3 ms, the same count (19,152). The cells query keeps the predicate:
it is a window with a LIMIT and needs the rows, not their number. The
rated grid count (3.5-4 ms with a one-star filter) is unchanged: its cost
is the rating subquery per image, and changing how `RatingFilter` spells
it changes every grid and timeline query, which is left for its own change.
`keywords::list` is the vocabulary with a per-word photograph count, and
`keywords::for_images` calls it on every selection change to redraw the
keyword panel. On the reference library (58 words, 10,800 assignments) it
cost 3.0-3.5 ms best-of-50 by catalog_bench, `for_images` 3.1-3.6 ms (6 and
5 ms on a busy machine).
Per word, the count walks `keywords_term (keyword)` and, for each
assignment, reads the `keywords` row to learn its version before probing
`versions` for the image:
SEARCH k USING INDEX keywords_term (keyword=?)
SEARCH v USING INTEGER PRIMARY KEY (rowid=?)
With `keywords_term_version (keyword, version_id)` the first step is
index-only:
SEARCH k USING COVERING INDEX keywords_term_version (keyword=?)
SEARCH v USING INTEGER PRIMARY KEY (rowid=?)
After: `list` 1.3 ms, `for_images` 1.5 ms, with the same answers (digests of
both outputs compared on the reference library).
The index is created on first use by `list`, with CREATE INDEX IF NOT EXISTS,
not by a migration: a new schema version makes every older build refuse this
catalog's snapshot at sync (`sync::remote_is_mergeable` compares
`user_version` and nothing else), and a build that meets an extra index
ignores it. Once the index exists the statement is a schema lookup, 8 us. A
failure to create it -- a read-only or busy catalog -- is logged and the
list is read without it, as before.
`library::local_original_count` feeds the "On this device" chip and runs
beside the rating counts on every star keystroke. On the reference library
it cost 1.3-1.4 ms best-of-50 (3 ms on a busy machine) to find 254
originals among 19,000 visible images.
It was a correlated EXISTS per visible image:
SCAN i USING INDEX images_grid_order
SEARCH ic EXISTS USING INTEGER PRIMARY KEY (rowid=?)
`image_cache` holds a row only for what has been fetched, so the question
is driven from it: `i.id IN (SELECT image_id FROM image_cache WHERE
tier_actual >= Original)`, which SQLite plans as the list first and a probe
of `images` by id for each entry:
SEARCH i USING INTEGER PRIMARY KEY (rowid=?)
LIST SUBQUERY 1
SCAN image_cache
`image_id` is the cache's primary key, so each image is in the list at most
once and the count is the one the EXISTS gave (254). After: 0.05 ms. On a
library whose every original is cached this is as much work as before,
which is the proportion the rule asks for.
The count stays on the keystroke path: dropping it there would leave the
chip stale after a background download until something else refreshed it,
and at this cost there is nothing left to save. catalog_bench spells the
query as dr-ui does, so its copy changes with it.
`label_histogram` runs on every label keystroke and after every batch of
judgements is saved. On the reference library it cost 7.2-8.4 ms best-of-50
by catalog_bench (13 ms on a busy machine), to report that none of 23,500
images carried a label.
The join was the rating histogram's, with one thing worse: the index does
not carry `label`, so each probe went on to read the version's row.
SCAN i USING COVERING INDEX images_folder
SEARCH v USING INDEX versions_judgement (image_id=?) LEFT-JOIN
USE TEMP B-TREE FOR GROUP BY
It now takes the rating histogram's shape: only labelled default versions
are grouped, and the unlabelled slot is what is left of `judged_rows`.
SCAN versions USING INDEX versions_judgement
USE TEMP B-TREE FOR GROUP BY (the labelled rows only)
That pass still reads each default version's row for `label`, but in the
index's order, which follows the table's; a partial index on the labelled
rows would make it index-only, and was not worth a new index for the
remaining 1 ms. After: 1.7-2.0 ms, the same answer on the reference library,
and a test that compares it with the old join over the awkward states the
rating test uses (a second default's label counted, unknown codes and zero
folded into unlabelled).
`rating_histogram` runs on every star keystroke. On the reference library
(24k images, 1,200 of them rated) it cost 6.3 ms best-of-50 by
catalog_bench, and up to 10-14 ms when the machine is busy.
It was `images LEFT JOIN versions ON ... AND is_default = 1 GROUP BY
rating`. The plan:
SCAN i USING COVERING INDEX images_folder
SEARCH v USING COVERING INDEX versions_judgement (image_id=?) LEFT-JOIN
USE TEMP B-TREE FOR GROUP BY
A probe of the index per image, then a sort of all 23,500 rows, to put
22,000 of them in slot zero.
Now the rated rows are grouped on their own (`rating != 0`: one pass over
`versions_judgement`, a sort of 1,200 rows), and slot zero is what is left
of the join's row count. That count is three index-only aggregates -- the
library size, the default versions, and the images holding one -- so an
image with no version is still unrated, and an image with two default
versions still counts twice, exactly as the join counted it:
SCAN versions USING COVERING INDEX versions_judgement (x3)
SCAN images USING COVERING INDEX images_folder
`count(DISTINCT image_id)` has its own statement because alone it reads the
distinct values off the index order; beside other aggregates SQLite builds a
temporary b-tree for it.
After: 1.2 ms. The histogram is the same on the reference library
([22364, 663, 19, 47, 115, 374]), and a new test compares it with the old
join on a catalog holding every state the schema allows: no version, only a
virtual copy, two defaults, ratings below zero and above five.
Issue #75 lists catalog reads paid on interactive paths rather than once:
the rating and label chip counts on every judgement keystroke, the "On this
device" count beside them, the keyword panel's vocabulary on every
selection change, and the grid's total on every scroll reload. catalog_bench
now times each of them against a real catalog and prints their answers, so
a change to any of them can be checked for giving the same numbers.
Two of them live in dr-ui's private `library` module; their SQL is spelled
in the bench as it is spelled there, which the module comment says.
Reference library (24k images), best of 50, CPU, on a loaded machine:
rating_histogram 9.0 ms, local_original_count 2.0, label_histogram 10.0,
keywords::list 4.0, keywords::for_images 5.0, grid count 1.9, grid count
with a one-star filter 4.0.
The queue coalesces, so a producer with no consumer never fails: it
leaves one row per subject for ever. That is how 23,582 Thumbnail jobs
accumulated unnoticed (#73), and nothing at runtime would have said so.
every_queued_kind_has_a_consumer reads the shipping sources of every
crate under core/, ui/, apps/ and platform/ (cfg(test) items dropped)
and pairs the JobKind named at each enqueue( call with the kinds named
in a fn kinds( body or a claim_next_matching( call. An enqueue that does
not spell its kind is refused, since the pairing could not be checked.
It guards against passing over nothing: the queue's own files and the
scan must have been read. A second test runs the reader over fixed
snippets so a parsing bug shows up as a failure. Run against master's
scan.rs and walk.rs it names all three orphan enqueues.
Refs #73
Stopping the enqueue leaves the rows already queued: 23,582 on the
reference catalog, about 1 MB of table and indexes that every query over
jobs pays for.
A migration would be the usual tool and is the wrong one here. A schema
bump makes an older build refuse the synced catalog snapshot, and the
tablet is on 0.16.0. So the rows are dropped at runtime instead, by
jobs::drop_retired over a new JobKind::RETIRED list, from runner::recover
- which already runs exactly once per catalog open, before any worker.
It runs every open rather than once because an older build sharing the
catalog queues them again on its next scan. kind leads the
UNIQUE(kind, subject_id) index, so with nothing left it is one index
probe. Measured on a copy of the reference catalog: 23,582 rows dropped
in 40 ms on the first open, 0.07 ms after.
Thumbnail stays in the enum so its number is never reused for a kind
that would then inherit old rows. The runner tests that call recover
move to a live kind; the jobs.rs tests of queue mechanics never call
it and are unchanged.
Refs #73
walk::scan_root enqueued an ExtractMetadata and a Thumbnail job for every
image it inserted or found changed. No handler claims either kind. The
walk is only reachable from the scan_local example today, so no real
catalog holds these rows, but it is the same leftover the remote scan
carried (#73) and it is what a local library would inherit.
Both debts are already recorded where their consumers look: an inserted
or changed image is written at metadata_state 1, which is the metadata
sweep's work list, and the thumbnail store answers for itself.
The tests that used job rows as the measure of "this image owes work"
now read metadata_state, which is the record the sweep actually uses;
the no-requeue test marks the first image read before the second scan,
so it still proves an unchanged neighbour is not put back in debt.
Refs #73
The develop view reported a remote original on its way through the
error message, so it read "Could not load image" over "Downloading…".
It did so on every step along the roll, including a cached frame that
was ready within a tick, so each step flashed the error.
Waiting is now its own state. On the step, the grid's thumbnail of the
photograph stands in at once. Only when a transfer is really on the
wire does it dim under "Not on this device yet", with a line like
"Downloading — 12.4 of 38.0 MB" and a progress bar.
The bytes come from a new RemoteBackend::get_reporting. The Nextcloud
backend overrides it to read the body chunk by chunk; the default
reports once at the end. Progress is kept in the in-flight registry by
path, because a step usually lands on a frame the prefetcher is already
fetching. The catalog's file length stands in when the server sends no
Content-Length.
The library holds the same RAW in several folders: a dated folder, a
bck/ beside it, a renamed Darktable export tree. dr_catalog::duplicates
is the catalog half of consolidating them (#67).
candidates() is one grouped query over root, camera, capture instant and
size, joined back for the rows; count() is the same grouping under
COUNT. On a copy of the reference catalog (23,582 images) both take
10-35 ms and find 1,836 groups holding 3,379 spare copies.
survivor() prefers a copy outside a backup-looking folder, then one
still named the way the camera named it, then the oldest, then the
lowest id.
consolidate() re-checks the plan against the catalog, merges the copies'
judgements onto the survivor (highest rating, keywords unioned,
collections unioned with the survivor keeping its place, a flag or label
the copies agree on, faces via faces::carry_onto_copy) and records the
copies as trashed, all in one transaction, so a failure part way leaves
the group untouched. preview() runs the same code and rolls it back.
Sameness probes are kept in dedup_probes, created on first use rather
than by a migration: a schema bump would make older builds refuse this
catalog's snapshot at sync. trash::record_trashed_within lets the trash
write share the merge's transaction.
Capture sharpening at a scale too coarse to draw its radius emits one pass
with an empty body (`nothing_to_sharpen`), so that a chain still ends in
something that performs the output transform. At fit on any modern sensor
that is most of the time. When another neighbourhood operation follows it
- dehaze, clarity, texture - the pass is not last and does nothing: it
reads the rgba16float intermediate and writes the same texels to the other
one. It still cost a full render-sized read and write every frame:
scene before after
sharpen+clarity 2560x1600 fit 18.66 ms 14.16 ms
sharpen+clarity 3840x2160 fit 37.93 ms 27.88 ms
every operation 2560x1600 fit 71.80 ms 67.24 ms
clarity alone 2560x1600 fit 14.23 ms 14.20 ms (control)
sharpen+clarity 2560x1600 1:1 23.01 ms 22.97 ms (control: resolves)
(Laptop RTX 3050 held at 420/810 MHz by its power cap, synthetic 60 MP
source, median of five alternated runs of forty frames each.)
`compose_detail_with` now drops such a pass where dropping it is exact:
not the last pass, whose output transform would otherwise move onto the
previous pass's f32 result and round differently; and not a pass right
after a reduced one, because a full-resolution pass is what closes the
reduced chain for the operation after it. `DetailPass::is_identity` says
what "changes nothing" means: full size, nothing bound at binding 3, and a
body with no code in it.
The rgba8 output is bit-identical: every scene above hashed the same
before and after, and the sharpen+clarity frame hashes the same as clarity
on its own, which is the claim in one line. A new dr-pipeline test pins
the three cases - dropped ahead of another operation, kept when last, kept
when alone.
At fit, every output pixel of the fused pass loads one texel from a source
three or four times its width, on a stride. The memory system fetches the
texels it skips along with the one it wanted, so on a 60 MP rgba16float
source that gather was most of what the fused pass cost: 10.6 ms of a
2560x1600 frame against 3.8 ms for the same shader reading a contiguous
window (the 1:1 view). At 3840x2160 it was 21.1 ms. Those are the laptop
RTX 3050 with its clocks held at 420/810 MHz by the power cap; unthrottled
the same frames were about 2.0 and 3.2 ms, and the gather is the same
share of them.
Which texel an output pixel reads depends only on the framing prologue,
the framing and warp uniforms, the source and the render size. None of
those move during a slider drag, so the gather is the same work every
frame. The fused shader now takes a render-sized rgba16float cache of it
(bindings 6 and 7, declared in every generated shader like the masks) and
a pair of uniform flags: write what was gathered, or read it back at the
pixel's own coordinate. AdjustPass keeps the cache and decides per
dispatch. The composer supplies `ComposedShader::sample_key`, a hash of
the prologue and those uniforms, and AdjustPass adds the image and the
size; an image gets a process-unique id for this rather than being held
alive by the key.
The picture is bit-for-bit the same. The source is rgba16float and so is
the cache, so the stored texel is the texel, and only the path that reads
a texel whole takes part: an interpolated sample (straightening, lens
warps, CA) is a blend that f16 could not hold exactly, so the composer
gives it no key and it reads directly as before.
The cache is written on the second frame with a given key, not the first:
a crop or zoom drag changes the key every frame, and writing then would
add a render-sized write to exactly the gestures that can afford it least.
It is kept only up to 3840x2400, so an export never parks a full-frame
copy on the device, and `release_caches` drops it.
Measured with a scratch probe rendering the synthetic 60 MP frame from
examples/frame_budget.rs, forty frames per run after six warm-up, five
runs of each binary alternated, median of the per-run p50 (GPU idle apart
from the power cap):
scene before after
neutral 2560x1600 fit 10.62 ms 3.88 ms
exposure 2560x1600 fit 10.83 ms 3.87 ms
nr chroma 2560x1600 fit 19.84 ms 12.69 ms
neutral 3840x2160 fit 21.05 ms 7.11 ms
exposure 3840x2160 fit 21.08 ms 6.94 ms
clarity 3840x2160 fit 42.20 ms 27.88 ms
neutral 2560x1600 1:1 3.83 ms 3.84 ms (control: nothing to gain)
The rgba8 output of every scene hashed identically before and after, in
isolated runs and across all 38 scene/size/view combinations of the
probe. New tests walk a pass through direct, write and read frames, a
slider move, a neighbourhood operation and a framing change, and compare
every frame with a fresh pass that can only have read directly.
`persist` runs after every scan, for every photograph the scan listed. On
a settled library that is the folders whose ETag changed -- a sidecar
written there by a rating is enough -- so one relisted folder of 1,600
images is an ordinary pass, and a first scan is all 24,000.
Per photograph it prepared four statements from their SQL (a folder
lookup, the image upsert, the id read-back, the remote upsert) and then,
after the commit, found the image again by path and enqueued its
thumbnail job as an autocommitting statement of its own -- a commit per
photograph, for rows that were almost all already queued.
Now the statements are prepared once per pass, a folder's id is looked up
once per folder rather than once per photograph in it, and the job is
enqueued inside the transaction with the id already in hand. That also
makes the job atomic with the row it points at, which is what the old
ordering after the commit was trying to guarantee. `jobs::enqueue` uses a
cached statement for the same reason.
persist_bench on a copy of the reference catalog, CPU, best of runs:
largest folder (1,589 images) 102-118 ms -> 10-13 ms
whole library (23,582 images) 1.55-2.19 s -> 188-192 ms
The fingerprint of images, remote, jobs and folders after the run is the
same for both builds.
`adopt_orphan_terms` runs in the backfill on every catalog open. Its
check -- is there a vocabulary row for this word, tombstones included --
cannot use `keyword_terms_name`, which is partial on `deleted = 0`, so the
correlated subquery scanned the vocabulary once for each of the 10,800
assignment rows before `DISTINCT` threw the repeats away: 3.5 ms per open
on the reference library.
The distinct words are taken first and the check runs once per word -- a
few dozen scans of a few dozen rows. Same rows out, since `DISTINCT` over
the assignments is exactly the set of words.
catalog_bench, best of 20: 3.45 ms -> 0.30 ms.
`Catalog::open` runs the backfill every time, and every worker thread
opens its own catalog: the develop view does it to fetch each original and
again for each neighbour it prefetches, and the sync, sweep, burst and
thumbnail workers each do it too. On the reference library (24k images)
an open cost 26 ms of CPU, and most of it was `pair_raw_and_jpeg` reading
all 17,000 RAWs into a map of lowercased stems to find partners for the
1,900 JPEGs that have none -- the same 1,900 on every open.
It now starts from the small side. The unpaired JPEGs are read first, and
it stops there if there are none; otherwise it reads the RAWs in the
folders those JPEGs sit in (plus the unfiled ones when an unfiled JPEG is
waiting), which is 142 on the reference library. A pair is same-folder by
definition, so no pairing is lost; the RAWs are read in id order, so where
two share a stem the later one still wins as it did in the table scan; and
a pass with nothing to pair no longer opens and commits an empty write
transaction.
catalog_bench, best of 20, CPU: `Catalog::open` 26 ms -> 12 ms together
with the next commit (the backfill 24 ms -> 11 ms; this step is ~10 ms of
that). A test covers pairs found among other folders and unfiled images.
Every sync pass exports this device's faces to the shard store and imports
what peers sent, and both walked the whole library asking the store's index
about one image at a time: the export 19,000 `indexed_at` lookups (one per
face marker), the import 23,000 `held_model` lookups (one per image with a
server id), each a statement prepared and run against the index. With
nothing new either way -- the usual pass -- that was all they did.
Measured with catalog_bench against copies of the reference catalog and
face store, best of 5, CPU:
export_to_shards (steady) 119 ms -> 27 ms
import_from_shards (steady) 250 ms -> 87 ms
Each now reads the index in one statement into a map. The import's query
is `held_model`'s, ordered the same way, keeping the first row per file,
and nothing in the loop changes which pipeline a file is held under
(`set_indexed_at` touches only a file already decided; candidates are
distinct files). The export's `put_image_at` does rewrite entries -- but
only its own file's, its generation and the siblings it supersedes -- so a
file already written in this pass is asked of the store again, and every
other answer is the one the lookup would have given. An index that cannot
be read gives an empty map, which is what each failed lookup returned.
The store index and the catalog are identical after the old and new
builds' runs.
A sync pass that brought nothing new cost 450-540 ms of CPU in
`merge_remote_catalog` on the reference library (24k images, 19k faces),
measured by catalog_bench merging a copy of the catalog with itself.
Most of it was the loop over the other device's confirmed faces and the
faces under its ignored groups -- 13,000 rows. For each one it prepared
three statements from scratch (`query_row`/`execute` with a SQL string
compile the statement every call) and then rewrote the `face_person` row
with the values it already held, dirtying a page per face on every pass.
The rejection loop prepared three more per row.
The statements are now `prepare_cached`, the local assignment is read once
per face (whether it is confirmed, and what it holds, come from the same
row), and the upsert is skipped when the row already says exactly that.
`faces_assigned` is still counted for those rows, so the report is the one
the old code gave, and nothing else reads the difference: the row is
byte-for-byte what the upsert would have written.
After: 279 ms (best of 5, CPU), with every catalog table identical after
the run to the old build's.
Two benches for reading side by side before and after a change, against a
copy of a real catalog, in the manner of identity_bench:
- `dr-catalog --example catalog_bench CATALOG [FACES_DIR]` times
`Catalog::open` and the backfill inside it step by step, the upload
snapshot, a merge of the catalog with a copy of itself, and the face
shard export and import in the steady state where nothing is new.
- `persist_bench`, an ignored test in dr-ui's scan module because
`persist` and `apply_judgement` are private to it, replays the
catalog's own rows through `persist` (the largest folder, and the whole
library) and looks up every `.drsc` sidecar the catalog has read. It
works on a scratch copy and prints a fingerprint of what `persist` left,
so two builds can be shown to agree.
Both print best, median and CPU time; the CPU figure is the one to compare
while other builds share the machine.
With Skia drawing pre-rotated on wgpu's Vulkan swapchain, Android no
longer needs to draw with Skia over OpenGL, which was the only reason
the develop view read its frame back through memory (TD-1).
So `unstable-wgpu-29` moves back to the common slint dependency. The
android-activity backend then builds `SkiaRenderer::default_wgpu_29`,
and `shared_gpu` loses its Android arm. The one wgpu device is handed to
Slint through `BackendSelector::require_wgpu_29` on both platforms.
`slint::android::init_with_event_listener` runs before `dr_ui::run`, so
the selector reaches the Android adapter before its window exists.
`renderer-femtovg-wgpu` stays desktop-only, since Android has no FemtoVG.
The two `#[cfg(target_os = "android")]` readbacks in `develop::render`
(the frame through `export_pixels` and the focus overlay through
`read_overlay`) are gone. `read_overlay` stays for the tests that check
what the overlay marks.
Built for arm64 and release-signed. Not yet run on the tablet.
There was no perspective transform anywhere in the pipeline: framing
offered a ±45° straighten, quarter turns and flips, and a building shot
looking up kept its leaning walls.
Framing gains a vertical and a horizontal keystone (-100..100). They are
parameters of framing rather than a new stage, so they carry its Compose
attribute, persist in the sidecar under framing, and are withheld from a
default paste exactly as the crop is. In the prologue the keystone runs
after the crop and the straightening and before the stored orientation
and the lens warp, so "vertical" is the photograph's displayed height and
the lens still sees its whole frame.
The map takes the output frame onto a trapezoid inside the source, built
as a homography from four corners and uploaded as three columns in the
framing uniform block (which grows from two vec4s to five). A keystone on
its own therefore never exposes an empty corner and leaves any crop valid.
Combined with a straightening angle the empty area is a pulled-back
quadrilateral the closed-form inscribed rectangle cannot describe, so
max_inscribed_crop searches for the largest centred rectangle whose
corners all have a source pixel behind them. source_at and output_at
apply the same map, so masks, gradients and spot handles follow it.
Colour labels could be read from a Lightroom sidecar and queried by the
selector, but nothing drew one or set one, so the only labels a library
held were ones another program had written.
Every mark carries its label's initial on its colour — R, Y, G, B, P —
so a label is read without telling red from green, which is what
NFR-A11Y-3 asks of colour labels by name. A grid cell shows the mark
before its filename. In the grid, 6, 7, 8 and 9 set red, yellow, green and
blue as Lightroom's keys do, on the photograph under the pointer or on
the selection by the rule the star keys follow; the same key again takes
the label off, and over a mixed selection it sets it on all. The
selection bar gains Label, which opens the six choices — each a mark and
a name — and purple, which has no key, is there. In develop the top bar
says "Label: Green" beside the mark, opens the same choices, and 6-9
label the open photograph.
Each gesture is one catalog transaction, then the grid, the counts and
both sidecars are written as a rating's are. The filter bar gains a chip
per label, its mark and its name with a count, one at a time; the filter
is one SQL term, travels in the place record, and "All" clears it.
A rating and a flag are written to DarkRoom's sidecar as well as the
catalog, because the catalog is a disposable index and the sidecar is
how a judgement reaches the photographer's other devices. A label had no
place there, so once labels could be set, one would have lived only in
the catalog of the device it was set on and gone with it.
The sidecar version now carries `label` (0 none, 1-5 as the catalog
codes it), written only when set. It merges under the rating's rule, so a
device that never labelled a frame cannot clear another device's label,
and a code this build does not know reads as none rather than as some
other colour. A judgement write carries the catalog's label with the
stars, and the scan takes a sidecar's label into the catalog when it has
one. An older build keeps the line as an unknown key and writes it back.
Colour labels reached `versions.label` only from an XMP sidecar: nothing
in the catalog could set one, clear one, or read it back alongside the
stars, so there was nothing for an interface to call.
`set_label` and `set_label_many` write it the way ratings are written,
the bulk form in one transaction so a key over a selection is one commit.
`toggled_label` holds Lightroom's rule for a label key: it clears only
when every image already carries that label, and otherwise sets it on all
of them, so a half-red selection comes out red rather than inverted.
`Judgement` carries the label, so the grid's one window query brings it
with the stars, and `label_histogram` counts each label in one grouped
statement for the filter chips. A label does not make a frame "judged":
it is a pile of the photographer's own, not a cull decision.
The doc comment for `default_version_id` had been stranded above
`label_code` when that was inserted; it is back on its function.
Mask geometry is stored in source coordinates, so re-cropping tighter
never destroys a layer. It makes it invisible: the layer stays in the
panel and the sidecar, its adjustment lands on pixels nobody will see,
and nothing says so. The spec had no clause for this; FR-DEV-17 now
states it, under the ID issue #10 reserved.
dr_pipeline::orphan samples each layer's mask on a 64x64 lattice over
the source, with the gradient, radial, brush and model-raster geometry
the mask shader uses, folds the parts by their joins and inversions, and
maps the samples through the framing to see how much of the coverage
the crop keeps. `hidden_by_crop` reports the layers whose share fell
below a tenth, and only those the change newly hid, so an already
stranded layer is not announced again on every later adjustment.
Ranges follow the picture and region selections need a label map this
crate does not hold, so a layer that adds either is never reported: a
false alarm on the common path would teach the notice to be dismissed
unread.
With the trait in place the claim still meant nothing while every caller
named dr_decode's free functions: a second decoder would have had to be
threaded through the scan, the thumbnail ladder, import, the viewer,
export, merge and repairs at the moment it arrived.
Each of those now takes a &dyn Decoder and reads headers, previews,
orientation and sensor data through it, including the header budget a
remote fetch asks for (header_bytes) and where it finds the embedded
preview (locate_preview). Only the places that start a job name
dr_decode::default(): the thumbnail, sweep and thumbnail-sweep threads,
the viewer's open handlers, and the request structs a job is handed
(BatchRequest, MergeRequest, the import Request, the repairs Toolkit),
so a caller can be given another decoder by changing what it is handed.
The default is rawler through the same free functions as before, so
nothing a user sees changes. The trait gains Debug as a supertrait so
request structs that derive Debug can carry one.
FR-RAW-2 says a second decoder may be added for broader camera coverage
without changing callers, and D2 names LibRaw as that second decoder.
Nothing tested the claim: dr_decode was one decoder reached through free
functions, so adding another would have meant editing every caller at
the moment there was most pressure not to.
Decoder is an object-safe trait over bytes: header_bytes, metadata,
orientation, locate_preview, preview and decode. Rawler implements it by
delegating to the existing free functions, so behaviour is unchanged,
and dr_decode::default() hands it out as a &'static dyn Decoder, which
is what the places that start work will name. JPEG recognition, decoding
and completeness checks stay free functions: they are not a RAW
decoder's to vary.
Nothing in the trait takes a path or a SourceRef; the decoder states how
much of a file it needs and where its preview sits, and the caller's
storage fetches that.
A layer's parts could be added to the mask or taken out of it, and nothing
else. The selections that need composing most are the ones that are
neither: the sky that is also bright, the subject that is also skin. With
union and subtract alone, "this and that" had to be spelled as "this minus
everything that is not that", which needs a second part that selects the
complement and rarely exists.
Join gains Intersect, stored as "intersect" in the part block of a sidecar.
It is the product of the two coverages, dst * src, which is one more
fixed-function blend state beside union's max and subtract's
dst * (1 - src) (mask-editing.md 5.2): the same scratch texture, the same
three vertices, no shader arithmetic. The product equals the minimum
wherever either side is fully in or out, and is the softer reading where
two soft edges overlap. Join::apply spells the three operations on the CPU
so the GPU tests can be held to one definition.
A layer that intersects with a part covering nothing now reports that it
covers nothing, so it is not rasterised as an empty slice. Old sidecars
never contain the word, so they read as before; a build from before this
reads "intersect" as a union, the existing unknown-join fallback, which
keeps the part visible rather than dropping it. Join::ALL keeps union and
subtract at indices 0 and 1 so a stored panel index still means the same
join.
Before the previous commit, browser sign-in could store an account as
http://, and the client now refuses to send to one. Left alone, such a
library would fail to open with a configuration error, so its stored
endpoint is rewritten before anything reads it.
The endpoint is half of two keys, and both are handled:
- The keyring entry is filed under it. Rewriting only the record would
strand the app password under the old key and sign the user out, so
AccountStore::move_endpoint copies the secret across first, rewrites
the record in place (the last record is the one resumed), and deletes
the old entry only once nothing refers to it.
- namespace() is built from it and names the catalog directory. For an
http to https rewrite it does not change, because the namespace strips
either scheme. A move that would change it is refused, not performed,
so no later rewrite can abandon a catalog either.
The rewrite is a new BackendProvider::upgrade_endpoint hook, which does
nothing by default, and not a second call to normalise_endpoint. The
folder connector's normalise_endpoint canonicalises the path and needs
it to exist, so running it on every launch would fail a library on an
unplugged disk, or rename one whose path now resolves differently. Only
Nextcloud implements the hook.
If the move fails (for example, a locked keyring), it is logged, the
account is left as it was, and the move is tried again on the next
launch.
Closes#65.
dtourolle