31 Commits

Include renames

Author	SHA1	Message	Date
dtourolle	1d7115437b	Date a photograph from its name when its header has none ... WhatsApp strips every EXIF tag and names the file "WhatsApp Image 2023-06-15 at 07.00.42.jpeg"; Windows Phone, Android cameras and darktable's import put the date in the name too. Those images sorted after everything else and were absent from the timeline. name_dates reads a date (and a time, when one follows) from the file name, then from the innermost folder that states one. A sequence number after a date is not read as a time, and a bare year folder is not a date. EXIF always wins: only examined rows still undated are filled. The sweep and the metadata repair fill as they mark an image examined, and the open backfill fills catalogs examined by earlier builds. On the reference library that takes 274 undated images to 10; the no-op case is a seek on images_captured, 0.6 ms an open.	2026-09-30 21:45:54 -04:00
dtourolle	ffdd640170	Backfill the catalog once per state, not on every open ... Catalog::open ran schema::backfill every time, and every worker thread opens its own connection. A develop landing made five opens, and each paid the RAW/JPEG pairing, the default-version anti-join over every image, the uuid pass over every default version and the keyword check: 17 ms of CPU an open on a copy of the reference catalog, ~80 ms a landing, to confirm that nothing had changed since the open before. Everything the backfill repairs is a row some write added: an image a scan inserted, a version or keyword assignment a merge brought in. So the open now reads a stamp - user_version, max(id) of images and versions, max(rowid) of keywords, and the file's device and inode - and skips the backfill when the stamp matches the one recorded at this path's last backfill in this process. The maxima are each the last page of a b-tree; an open that skips costs ~1 ms. The backfill still runs: - on the first open in a process (nothing recorded yet); - on any open that migrated the schema, unconditionally; - after a pull: merge_remote forgets the path, so the next open backfills even when every incoming row collided and nothing moved; - when the file is replaced under its name: the inode is in the stamp, and recovery::set_aside, the first step of a restore and a rebuild, forgets the path; - when another process or thread adds rows, because the stamp is read from the file, not from anything this process did. The stamp is taken before the backfill, not after. Read after, it would describe the backfill's own inserts, and could record an image another connection inserted in between as covered when it was not. Read before, the worst case is one redundant pass after a backfill that did real work. Kept in memory rather than in the catalog: a stamp row would need a table an older build does not have and would travel in the sync snapshot, where a flag from another device's catalog says nothing about this one. No schema version bump, so the tablet on 0.16.0 still reads the snapshot. Tests cover the skip, a scan's new image, a migration, a pull and a replaced file.	2026-09-26 14:26:10 -04:00
dtourolle	1f26e1e627	Pair RAW and JPEG from the unpaired JPEGs, not from every RAW ... `Catalog::open` runs the backfill every time, and every worker thread opens its own catalog: the develop view does it to fetch each original and again for each neighbour it prefetches, and the sync, sweep, burst and thumbnail workers each do it too. On the reference library (24k images) an open cost 26 ms of CPU, and most of it was `pair_raw_and_jpeg` reading all 17,000 RAWs into a map of lowercased stems to find partners for the 1,900 JPEGs that have none -- the same 1,900 on every open. It now starts from the small side. The unpaired JPEGs are read first, and it stops there if there are none; otherwise it reads the RAWs in the folders those JPEGs sit in (plus the unfiled ones when an unfiled JPEG is waiting), which is 142 on the reference library. A pair is same-folder by definition, so no pairing is lost; the RAWs are read in id order, so where two share a stem the later one still wins as it did in the table scan; and a pass with nothing to pair no longer opens and commits an empty write transaction. catalog_bench, best of 20, CPU: `Catalog::open` 26 ms -> 12 ms together with the next commit (the backfill 24 ms -> 11 ms; this step is ~10 ms of that). A test covers pairs found among other folders and unfiled images.	2026-09-25 22:06:58 -04:00
dtourolle	84fade99ec	Put the developer docs under docs/dev and index the folder for users first ... docs/ had 26 developer documents flat beside the manual, and the two audiences are very differently sized: most readers want the manual and the gesture reference, a few want the register, the designs and the measurements. The manual and gestures.md stay at the top; everything for someone changing the code moves to docs/dev/, and the two documents that name their own successors — the v0.1 milestone and the UI-refinement plan — go to docs/dev/archive/ rather than being deleted, since both are still cited. docs/README.md is the index, users first. Every reference follows: code comments, Cargo manifests, the workflows, the pre-commit hook, the bench and traceability tools (which locate the repo root by docs/dev/requirements.md now), packaging, the Docker READMEs, CLAUDE.md, CONTRIBUTING.md and the README. The matrix links one level deeper and is regenerated. Links out of the moved documents into the tree gain a level; a link checker over every Markdown file finds none broken.	2026-09-20 21:16:03 +02:00
dtourolle	a437363bd6	Schema V20: put the mis-spelled run markers right ... The markers the previous commit stops writing are already in the catalogs — 2 on the desktop, 429 on the tablet — and in the shards both have exchanged. Renaming them to the faces' own id with a fresh time is what makes the export send each image again, under an entry newer than the empty one `held_model` would otherwise pick. Where the old write had inserted its marker beside the right one, the wrong one goes and the right one is refreshed for the same reason: its entry in the shards is older than the empty one. Images V14 left with faces and no marker are not touched. That state is the quality pass's cue, and the fixed write marks them correctly when it reaches them. Checked against copies of both real catalogs: the desktop renames 2, the tablet deletes 429, both in under 200 ms.	2026-09-20 13:00:59 +02:00
dtourolle	388bda6af3	Count the outstanding repairs from the faces, on partial indexes ... "How many images still owe a quality reading" was a correlated EXISTS per image over `faces`, and the face row is 8 KB of embedding and crop before the column it looks at, so each count opened every row. Six such counts run on every open of the Identity screen and at the end of every sweep: 160 ms on the reference library. V19 adds three partial indexes holding only the faces still owing each pass, keyed on the image and carrying the model id the predicate reads, and replaces `faces_image` with `(image_id, model_id)` so "does this image hold this embedder's faces" is answered from the index too. The planner takes a partial index when the count is driven from `faces` and ignores it inside the EXISTS, so `Needs::Face` carries the per-face fragment and `repairs::count` spells the query from the faces' side; the list and the per-image check keep the EXISTS. A test holds the two spellings to the same answer for every repair.	2026-09-20 10:56:37 +02:00
dtourolle	5c00942b84	One completeness job over a registry of repairs, and a re-index button ... A library's records are never all complete at once. A face found before its quality was kept has no quality; one found before the eye models existed has no reading; one adopted from a peer's shard has no crop; an image the fast detector examined on a 1024 px proxy has boxes the current detector would not have drawn; an image the scan stat'ed has no capture date. On the reference library that is 17,762 faces under the bare w600k_mbf id with no quality, no reading and no dense landmarks, 4,144 of them without a crop, beside 12,217 images the fast detector examined and found nothing in. Every one of those gaps was its own pass — V14's measuring pass, §17.5's eye pass, the sweep's proxy repair, the sweep's detector upgrade — with its own work list, its own count and its own idea of done, and adding a per-face field meant adding a pass. There was no pass at all for the case the library is actually in: boxes and landmarks drawn by a weaker detector on a proxy, which every later per-face pass would have read from. dr_ui::repairs replaces them with one job over a registry. A Repair names one thing a record can lack — the predicate that says which images still owe it, the input its handler needs (a header, the original, or a native render), the handler, and what to record for an image that can never be done. The job unions the predicates into one work list, fetches each image once at the most any claimant asks for, renders it at most once, and runs every handler whose predicate that image still matches, checked again before each because a detection writes every field a per-face handler would fill. The registry today: face-proxy, face-quality, face-eyes, face-crop, face-detection, face-upgrade, metadata — the last there to say that this is not a face job. Adding a field is one entry. A repair's predicate is the only definition of its work: the count the settings page shows, the list the job fetches and the check before its handler run are one predicate, so the job converges. That is why the registry is cut to what the device can do rather than listing what it skips — an entry is a count and a set of originals to fetch — and why an eye reading that cannot be cut is not a criterion. The catalog side is generic to match: record_updates writes whichever fields a FaceUpdate carries and re-marks the image so the shards export it; faces_needing and count_needing answer a predicate the caller supplies, replacing the measuring pass's three special cases. Two buttons on the settings page run the job and differ in one predicate. "Index faces" converges on coverage: has anything examined this image. "Re-index every face" converges on provenance: face-detection claims every image with no marker under the chosen detector, in either of its forms (FaceDetector::model_ids, so a desktop in f32 and a tablet on the Hexagon do not re-index each other's work), and a marker saying a weaker one looked is not that. An original over the fetch budget is left exactly as it was under the re-index, where the sweep marks it examined: a re-detection with nothing found would delete the faces, and "cannot fetch" is not "no faces".	2026-09-19 18:52:13 +02:00
dtourolle	facb44cb55	Keep the dense landmarks behind each eye reading, packed ... The 106 points the eye boxes were cut from, stored beside the reading as 16-bit fixed point over the frame: 424 bytes a face, a seventh of a pixel on a 6000-pixel frame, where f16 at the same size would have been six. Derived data like the embedding, kept for the same reason — it cost a fetch and a model run, and the next per-face pass should run from the catalog. Shards carry it; a peer's shard from before it is still read.	2026-09-19 14:24:15 +02:00
dtourolle	85cc2b1dcc	Trace the eye reading to FR-CULL-8a and the chip to FR-CULL-13 ... The register grew both clauses the same day this was built: FR-CULL-8a is the per-face state the reading is, and FR-CULL-13 is the rule that a signal is shown and filtered and never writes a judgement. The tags, faces.md §17 and catalog.md now say which is which; FR-CULL-8a records what of it is built, and that its third model is under the InsightFace grant by the same decision as the pair.	2026-09-19 14:06:59 +02:00
dtourolle	d706c12d77	Cover the eyes-open subquery with an index ... The people filter was served from faces_image without touching a row; reading the eye columns in the same subquery touched every one, and ALTER TABLE had put those seven floats after the embedding and the crop blob. One count took 24 seconds on the reference library, thirteen of them system time. faces_eyes covers the subquery again: five milliseconds.	2026-09-19 14:05:52 +02:00
dtourolle	54b543fb77	Store seven eye numbers per face rather than three ... Per eye P(open), the pixels across its box and the sharpness of the patch; and P(sunglasses). The verdict — open, closed, sunglasses, unclear — stays a rule in dr_face::eyes so the floors can move without re-measuring twenty thousand faces. Shards carry the same seven, and a peer's shard from before any of them is still read.	2026-09-19 14:04:08 +02:00
dtourolle	b908d861e0	Keep each face's eye reading in the catalog and in its shard ... Three nullable columns beside quality — P(open) for each eye and P(sunglasses) — because the verdict is a rule with thresholds in it and a rule belongs in code, not in rows that would have to be re-measured. NULL is "never read": a face from before the models, or from a device without them, and every reader treats it as unknown rather than as closed. The measuring pass V14 built for the embedding's length is what fills them, so the sweep's work list now also names faces with no eye reading — but only on a device that has the models, or it would fetch every original to do nothing to it. A peer's shard without the reading is still adopted, unlike one without the quality: the pass finds this work by the NULL rather than by the run marker, so adoption costs it nothing.	2026-09-19 14:04:06 +02:00
dtourolle	9b627e7713	Let a catalog writer wait for its turn instead of losing its work ... SQLite's busy timeout defaults to zero, and nothing ever set one: the loser of a write race got SQLITE_BUSY at the moment it asked. WAL does not cover this — it makes one writer and many readers free, and this application constantly has two writers, the face sweep committing a batch while the derived sync imports shards or reclustering reads. The cost was not a retry but lost work. A sweep that had already paid for the detection and the embedding — seconds per image, the expensive part — discarded the result on "storing faces for 214: database is locked" and moved on to the next image. Both the desktop and the tablet logged runs of those on consecutive images, which is a face sweep quietly failing to store the faces it had just computed. Ten seconds, on every connection, set in configure() so that nothing can open the catalog without it — the figure the job runner's own tests have used for this reason since they were written. It is far longer than any transaction here, so it bounds pathology rather than making anyone wait.	2026-09-14 20:05:53 +02:00
dtourolle	896188a489	Read the sidecars other editors write, and write them back on request ... FR-CAT-13 asked for standard XMP and `core/dr-xmp` answered the file: it has read and written `dc:subject`, `xmp:Rating`, `xmp:Label` and the IPTC core since 5fa4c07, under an ownership rule that leaves everything else in the document untouched. What nothing did was call it. No scan found an `.xmp` beside a raw, no catalog row was filled from one, no judgement wrote one back, and the "external modification detected, reload offered" clause had no mechanism. A library imported from Lightroom came in and could not go back out. The scan collects `.xmp` beside `.drsc` from the listings it was already paying for, and the pull reads each one whose ETag has moved. Both namings resolve: darktable's `IMG_0001.CR3.xmp` names its file exactly, Lightroom's `IMG_0001.xmp` names the stem, and under the stem the JPEG beside a RAW is the same photograph and takes the same document, as DarkRoom's own sidecar already does. Each is reconciled with the catalog winning — keywords union, a rating or label taken only where the catalog has none — because a standard XMP carries nothing that could say whether its value is newer. A genuine disagreement is not resolved; it is written to a table, and the settings page offers the sidecars' values against it. That button is the reload the requirement asks to be offered, and the ETag that moved is the detection it asks for: an `.xmp` edited elsewhere is exactly a file the pull's ordinary incrementality re-reads. Writing goes the other way behind a setting that starts off, since NFR-R4 makes writes beside somebody's originals theirs to switch on. With it on, a judgement or a keyword rewrites the sidecar of whichever spelling exists, or creates Lightroom's. The record is read from the catalog whole at that moment rather than carried from the gesture, so a rating and a keyword a second apart are two writes of one file that agree. And the file's own title, caption, copyright and hierarchy come through the rewrite: the catalog has no columns for them, `rewrite` replaces the owned set wholesale, and a record that said nothing about them would have deleted them from a Lightroom sidecar on every star. The rating's two axes cross the format's one field both ways: a rejection is Adobe's `-1` and stars are stars, and stars arriving on a rejected frame lift the rejection, since the file said it was worth a number. An unrated file says nothing and clears nothing, on the rule the `.drsc` merge keeps. `versions.label` finally has a reader and a writer, with the code table moved out of the query so the two cannot drift.	2026-09-12 01:08:11 +02:00
dtourolle	16f3fb41a3	Measure the faces already found rather than finding them again ... Every face stored before its quality was kept holds a unit vector, and V14 forgot the run marker of each image holding one so that the next sweep would look again. Looking again meant detecting again: a whole re-detection per image, with every suggestion on it thrown away and the confirmations carried across by box overlap, to recover one number. The sweep now has a measuring pass between the proxy repair and the un-indexed images. It lists every image holding an unmeasured face, fetches the original once, warps each stored face from the landmarks it already has, embeds it, and writes the raw vector and its length over the old row. Ids, boxes and identities are untouched; the marker is re-written fresh so the sync exports the measured vectors. A face whose landmarks no longer make a warp is dropped, as detection would have refused to store it. `faces_unindexed` leaves those images to the measuring pass, so the V14 deletion no longer costs a second detection.	2026-09-11 21:50:12 +02:00
dtourolle	8b3abdb787	Keep each face's quality, and never compare against a poor one ... The embedder's raw output has a length, and the length is a reading of how recognisable the crop was: a blur, an occlusion or a hard profile comes out short. Normalising threw it away. A short vector sits near the middle of the sphere and matches a little of everyone, which is how one bad crop bridges two people in a grouping pass. So the length is kept — the store now holds the raw vector, re-normalised on load, with the length beside it as `faces.quality` — and a face under MIN_GALLERY_QUALITY (14) is a probe: measured against the gallery and placed where it fits, but never what another face is measured against. Two probes are never paired, and a probe is nobody's evidence for a confidence. The People screen shows the number as "Quality 17.3", dimmed below the floor. Faces indexed before this stored unit vectors and have no reading; they are admitted to the gallery, and schema V14 forgets the run marker of every image holding one so the next indexing pass measures them. A peer's unmeasured shard faces are not adopted, or a sync would write that marker back.	2026-09-11 21:50:12 +02:00
dtourolleandClaude Opus 5	1ad35e2b87	Read the library's sidecars, so a cull done elsewhere arrives ... Judgements only ever travelled outward. A rating went to the catalog and to the photograph's sidecar, the sidecar reached the server, and there it stopped: the scan indexes files, `derived_sync` exchanges thumbnails, face shards and collections, `dr_catalog::merge` reconciles everything in a catalog except `versions.rating` and `versions.flag`, and the one sidecar reader that existed ran when a single photograph was opened in develop and handed its answer to the develop graph. `JobKind::ReadSidecar` was declared for exactly this when the job queue was written and was never enqueued or handled anywhere. The grid draws `versions.rating`. So a day of culling on the tablet could not reach the laptop by any path the application had, and the laptop's catalog says so plainly: 23,568 images, one of them judged. `pull_sidecars` closes it, off the back of work the scan already does. `dr_sync::scan` reports the `.drsc` files it meets in listings it was making anyway — no extra request, and a directory whose ETag is unchanged is still pruned before it is listed at all. A new `sidecars` table records the ETag of each one this device has taken in, so the fetch is one GET per sidecar that genuinely changed rather than one per photograph. A library nobody has edited costs nothing. The judgement is taken rather than maximised. The sidecar is the authoritative store and the fuse has already settled any contest between devices on `revision`, so lowering a rating from four to one on the tablet lowers it here — taking the larger would have refused every demotion the photographer ever made, which is most of what a second pass over a shoot is. A zero is the exception: it means *never judged*, not "judged zero", so a sidecar carrying none cannot erase a star this device holds. That is `merge_judgement`'s asymmetry and it carries the same known cost — clearing a rating does not propagate. A sidecar names a stem, so both halves of a RAW-and-JPEG pair are judged: they are one photograph (FR-CAT-11) sharing one document, and judging only one of them would leave the grid disagreeing with itself over which it drew. The `LIKE` that finds them is a filter, not the decision — `sidecar_path` is applied to every candidate, because a folder is entitled to contain a `%` and a rating landing on the wrong frame would be silent and permanent. Failing to read one is not a failure to scan: the ETag goes unrecorded, the ratings already here stay where they are, and the next scan tries again. The count is reported to the status line as well as the log, because a grid that silently gains three hundred stars is indistinguishable from one that has gone wrong — and because while this number was structurally zero there was nothing to tell the photographer their cull had not arrived. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-05 16:14:03 +02:00
dtourolleandClaude Opus 5	ca2a135e28	Let two devices name the same photograph's version the same way ... A version's uuid is the identity a cross-device merge keys on, and it was minted at random, per catalog, per image. Two devices indexing one Nextcloud library therefore held two different uuids for the same photograph — so the sidecar they shared collected a `default = 1` block each, `Version::merge` was never handed a matching pair to reconcile, and an afternoon's culling on the tablet did not exist as far as the laptop was concerned. `crate::merge` has said so in a comment since it was written: version uuids do not reconcile across devices, a uuid-keyed join unions nothing, so keywords are landed on the local default version instead. It named the problem and worked around it. `rating`'s own comment asserted the opposite — that generating the uuid here was what made it a cross-device identity — and `library::amend` repeated the claim. Uniqueness was never the difficulty; agreement was. `derived_version_uuid` computes it from `oc:fileid` instead. The server assigns that integer, every client pointed at the library sees the same one, and it survives a server-side rename and move — the three properties that already made `ASSIGN_BY_FILE_ID` prefer it to a content hash. The layout is a UUIDv8 (RFC 9562, an application-defined form) carrying all sixty-four bits verbatim across the variable fields with a fixed tag in the node field, so the mapping is injective by construction rather than by a hash's good behaviour, and a uuid in a sidecar can be read back to the file it belongs to by eye. A library with no server behind it has no shared identity to derive and keeps a generated one. The split is still reachable there if the folder is synced by something else; `Sidecar::fuse_default_versions` repairs that case rather than preventing it. Deriving it for new rows alone would have fixed nothing — every image in an existing library already has a version, so every one of them would have carried on writing to its own rival identity. `align_default_version_uuids` moves them, and runs from `schema::backfill` on every catalog open. It selects on the tag in SQL, so a catalog already realigned matches no rows and writes nothing, and it declines rather than fails where a virtual copy already holds the target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-05 16:13:58 +02:00
dtourolleandClaude Opus 5	e7b526c550	Specify face indexing at native resolution, and say what the proxy cost ... FR-CULL-8 said detection runs against the thumbnail or proxy tier and never a full decode, and faces.md §5 said the aligned crop is sampled from that same proxy. Both are wrong in the same place: they treat detection and cropping as one resolution problem when they are two, with opposite answers. Detection does not care. §4.1 fixes the graph's input at 640x640 and letterboxes whatever arrives, so a face filling 2% of the frame reaches the model at 12px whether the buffer handed over is 1024px or 6000px. Every pixel above the detector's own input is discarded before inference. The crop cares about nothing else. §5's warp produces the fixed 112x112 ArcFace sees, so source resolution converts directly into whether those 112 pixels were photographed or interpolated. Reading crop_px across the 18,671 faces the proxy-tier implementation stored: 47.3% were upsampled to reach the embedder, 314 of them by more than 2x, the smallest from 34 source pixels. An upsampled crop does not fail loudly -- it yields a confident embedding of detail that was never there, and the damage appears three stages later as clusters that will not separate. So FR-CULL-8 now specifies four stages with the resolutions named separately: render native through FR-EXP-9's pipeline, downscale for the detector, map boxes and landmarks back to native, crop and align from the native render. The affordability the old rule bought is met instead by when the pass runs -- background, preempted, resumable -- and the requirement says plainly what it now costs on a remote library: the original rather than FR-NC-3's byte range, 412 GB across the reference library's 19,107 images, so a whole-library pass is a transfer under FR-NC-6 rather than something that may start on its own. MIN_CROP_EDGE replaces the MIN_DETECT_EDGE this branch briefly had. Same number, guarding the quantity that turned out to matter. faces.md §7b records both measurements, and marks the second as unexplained rather than dressing it as a finding. Grouped by the buffer detection ran against, faces per image was 0.078 at 1024 or below and 1.82 at 2048 or better, controlled for file type and size. That gap is real and reproducible and I cannot account for it, because the letterbox above says detector input should not matter. M4 is where it gets settled. The crop measurement does not depend on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-30 19:41:20 +02:00
dtourolleandClaude Opus 5	0a6509de93	Forget the face runs made on proxies too small to see a face ... The floor added in the previous commit stops this happening again; it does nothing about the 1,824 images in the reference library that already carry a face_index row written against a proxy of 1024 or less. Those rows are why the damage is permanent rather than merely past. The work list is "images with no row for this model", so an image examined against a 1024px proxy -- 0.078 faces per image, nine in ten finding nothing -- is indistinguishable from one examined properly, and no later pass will ever offer it to the detector again. V12 deletes exactly those markers, and nothing else. The faces those runs did find stay in place and keep drawing the People screen until a better pass replaces them, and record_detections re-attaches the user's confirmed names across that replacement by box overlap, so a library somebody has spent an evening naming does not lose that evening. The cost is a re-fetch of the affected images. Deleting the marker rather than teaching the work-list query to select on source_edge, which was the other option and is worse. A standing `source_edge < floor` predicate never lets go: an image whose largest embedded preview is genuinely smaller than the floor would be re-fetched on every sweep for ever, because the next pass cannot do any better than the last one did. A one-off deletion gives each affected image exactly one more attempt through the good path and then lets the ordinary "has a row" rule settle it. The threshold is written out in the SQL instead of referring to dr_face::MIN_DETECT_EDGE. A migration has to keep meaning what it meant when it ran; binding it to a constant someone may raise later would quietly change what an old catalog gets migrated to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-30 19:40:26 +02:00
dtourolleandClaude Opus 5	5226f7223b	Group the frames of one moment, by when they were taken and what they look like ... A burst is the commonest thing in a cull and the least interesting: twelve frames of the same gull at 10 fps occupy twelve cells, are scrolled past twelve times, and end with the photographer keeping one. FR-CULL-5 asks for them to collapse to one representative and be judged as a unit. Two signals, because neither alone survives a real library. Time alone groups a whole wedding ceremony -- a photographer working steadily never leaves the gap that would end the run. Similarity alone groups a studio setup shot across two days, which is a project rather than a moment. Together they are specific: adjacent in time *and* looks like the frame before it. Two seconds is the time bound, and the reason is worth recording because the figure looks absurd next to a 10 fps camera. `images.captured_at` is whole seconds -- EXIF's DateTimeOriginal has no sub-second field and SubSecTimeOriginal is optional and widely omitted -- so a burst arrives in the catalog as ten frames sharing one timestamp. Any threshold finer than a second is a threshold on information that is not there. Where the pace really is faster, the similarity bound is what separates the frames. Similarity is a 64-bit difference hash over a 9x8 box-averaged reduction, compared between *adjacent* frames only. Chained rather than anchored on the first frame, because by frame twenty a camera following a bird has nothing in common with frame one while no two neighbours differ by much; the time bound is what stops the chain running away. There is no all-pairs step and there must never be one -- that is what turns a grouping pass into something nobody can afford to run over 50k images. Nothing here ranks a frame. FR-CULL-5 names the failure it is avoiding, which is rejecting the only frame of an important moment because somebody blinked, so there is no sharpness score and no best-of-burst. The representative is the earliest frame -- a fact about the clock, not a judgement about the photograph -- and the user's own choice lives in its own table so that rebuilding the grouping cannot erase it. Same argument `people.ignored` makes one subsystem over: nothing short of remembering a decision survives re-clustering. A newly found burst is recorded *open*. Collapsing on discovery would be tidier, and would also mean a background pass taking photographs off the screen part way through a cull. The pass marks; the user folds. It is a pass rather than a job kind for the reason catalog.md 10.2 gives for face clustering: a burst is a property of a run of frames and has no natural subject_id, so a per-image job would rebuild the world once per photograph. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-29 23:18:46 +02:00
dtourolleandClaude Opus 5	4ed10f7b23	Let a person cross from one device to another ... The face shards carry boxes, landmarks and embeddings. What they deliberately do not carry is who anybody **is** — the person rows, their names, and the assignments joining the two. Those travel in the catalog snapshot, which is a whole-file copy and does contain them. But the snapshot is *merged*, not adopted, and this merge only ever looked at collections and keywords. `face_shard`'s own module note says people travel in the snapshot; nothing implemented it. So a second device received every face and no people at all, and drew an empty People screen over a full catalog. Exactly what a tablet showed after syncing thousands of faces from a laptop. What travels is what the user decided, following the rule the rest of this module already follows — judgements travel, inference is rebuilt: - **People**, by uuid on `revision`, exactly as a collection is: the name, and whether the group was set aside. - **Confirmations**, and **rejections** — "this is not her" is a fact too, and is why re-clustering does not put it back. - **The suggestions inside an ignored group**, which are otherwise ordinary inference but are what anchors the ignore. Without them a group set aside on one device reappears on the other, the same fault that made "Not interested" not stick locally. Ordinary suggestions are not carried. Both devices hold the same embeddings and clustering is deterministic, so each recomputes them and arrives at the same answer; shipping them would double the merge for no new information. **A face has no cross-device identity**, and unlike a collection there is no uuid to give it one. Both devices do agree on `oc:fileid` and roughly on the box, so a remote face is matched to the local face on the same photograph whose box overlaps it most, above 0.5 IoU. That is not a new rule — it is the one `record_detections` already uses to carry a confirmation across a re-index, and it is loose on purpose: the question is "the same face in the frame", not "the same rectangle". A local confirmation is never overwritten. Two devices confirming one face as different people is a real disagreement and an assignment carries no revision to settle it with; taking the remote's answer would let a sync undo what the user just did on the device in their hands. The remote's schema is probed rather than assumed: `remote_is_mergeable` admits any catalog at or below this version, so one written before faces existed, or before V10 added `ignored`, is ordinary. An absent table skips this half instead of aborting a merge that would otherwise have succeeded. Nine tests, including that the name lands on the overlapping face and not its neighbour in the same frame, that a set-aside group stays set aside, that an ordinary suggestion does not travel, and that merging twice changes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 09:27:17 +02:00
dtourolleandClaude Opus 5	79c0520506	Keep the face, not just a way to find it again ... A face was drawn by decoding the 1024px proxy it was found on and cutting the box out again, every time the People screen opened. That made the screen a derivative of the thumbnail cache: evict a proxy — which the cache may do at any moment — and the cell goes blank, with no way back short of re-fetching the original over the network and re-detecting it. It also cost a full JPEG decode per image, per visit, to show a 96px cell. So the crop is cut once, when the pixels are already in hand at detection time, and kept. A 160px JPEG is a few KB against the ~250 KB proxy it replaces reading. Where it lives is the interesting part. The catalog snapshot is uploaded *whole* on every sync and downloaded by every device, so a crop column there would put tens of MB on every round trip — the exact cost `face_shard`'s 25 MB cap exists to bound, and the reason bulk per-face data lives in shards already. Crops therefore travel in the face shards, beside the embeddings, and `snapshot_for_upload` strips them from the copy it writes. Nothing reads a crop out of a merged remote catalog — the merge touches collections and keywords only — so a receiving device loses nothing. A shard carrying crops holds around 3,500 faces rather than 22,000, which is the price of a second device showing People immediately instead of re-fetching every proxy. The column is nullable and the reader falls back to the proxy, so a face indexed before this still works and the next indexing pass fills it in. V10 also adds `people.ignored`, for a person the user has looked at and does not want to identify. Most clusters in a real library are strangers — passers-by, other people's guests, a face on a poster — and there is no way to tell "not yet looked at" from "looked at, don't care" without recording the second. It is a column rather than a deletion because a deleted cluster comes straight back on the next Regroup: the faces are still there and still similar, and nothing short of remembering the judgement survives re-clustering. Same argument `face_person_rejected` makes one level down. And `prune_empty_unnamed`, for what clustering leaves behind. Regroup creates a person per unanchored group and never removed the previous run's now-empty ones, so pressing it twice added a rail entry per group it no longer believed in. Named people are never touched however empty — a name is user data — nor is a merge tombstone, which must outlive its faces to keep redirecting. 298 tests pass, including that the snapshot carries no crops while the live catalog keeps them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-27 21:17:44 +02:00
dtourolleandClaude Opus 5	26a1eb7e28	Record that face detection has run, not just what it found ... An image with no faces in it was indistinguishable from one that had never been looked at, so every landscape, still life and document scan in the library was re-detected on every pass, for ever. In a real library that is most of it: on the 23,527-image test library, 64 of the first 110 images indexed contain no face at all. Schema v9 adds face_index, a run marker per (image, model) carrying the face count and the proxy edge it read. Keyed on the model, so a model change puts every image back in the queue by itself. That makes a coverage figure possible, which is the thing a user actually wants to see. The audit also splits the outstanding set by whether a proxy exists, because 23,417 awaiting a proxy and 110 ready to index are different problems, and telling the user to run indexing again would not fix the first. The Identity screen gains Index faces, Stop, and the coverage line. examples/face_index.rs is the same check and sweep without a window, which is the right shape for an overnight pass. Measured on the real library in release: 3.5 images/second, 110 images and 125 faces in 30 seconds, and a second run correctly finds nothing left to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 22:13:41 +02:00
dtourolleandClaude Opus 5	aac3136407	Store faces and the people they belong to ... Schema v8: people, faces, face_person, face_person_rejected, and the per-library calibration. Follows catalog.md 10.1 with two additions the spec work turned up. crop_px, because at the 1024px proxy tier a group shot reaches the embedder at ~50 source pixels upsampled to 112 and a portrait at 340. FR-CULL-9 names face size as an axis along which an uncalibrated similarity misbehaves, so it is a stored feature rather than a UI hint. face_person_rejected, because rejection is not the absence of an assignment. Without it the next clustering pass re-suggests exactly the face the user just pushed away, and the tool feels broken. record_detections replaces rather than appends, since DetectFaces is coalesced per image -- and carries confirmations across the replacement by box overlap, so re-indexing with a better model cannot discard the user's own labelling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 20:02:52 +02:00
dtourolleandClaude Opus 5	6d6ef8d34b	Page the grid along an index instead of sorting the library each time ... Scrolling jittered, and this was the largest single reason. Every window the grid loads is `ORDER BY ... LIMIT n OFFSET k`, and neither half of that was being answered the cheap way. **The sort.** `GRID_ORDER` leads with `captured_at IS NULL`, so undated frames fall to the end. No ordinary index answers that — the leading term is an expression, not a column — so SQLite sorted the whole library into a temp b-tree on every window read, then threw away the first `k` rows of it. Schema V7 indexes the expression exactly as the query writes it, partial on the same `shadowed_by IS NULL AND trashed_at IS NULL` the grid filters by, so the read becomes a walk along the index. **The join.** `LEFT JOIN remote` was paged *after* it was joined, so reading 280 cells at offset 20,000 first seeked into `remote` for all 24,000 rows and then discarded 23,720 of them. The file ids are now fetched for the 280 rows that survived — the shape the badge and rating reads already use, one query for the window rather than one per cell. Measured together on 24,000 images at offset 20,000: **15.2 ms → 0.36 ms**, inside a scroll handler that has 16.7 ms to draw a frame. The test asserts on the query plan rather than on a duration, because there is no other symptom. A `GRID_ORDER` edited out of step with the index, or a column added back that drags `remote` in again, both still return exactly the right cells — just after sorting the library — and the jitter would come back with nothing to point at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-25 20:01:04 +02:00
dtourolleandClaude Opus 5	2147eaa6a5	Put a keyword on a photograph, not only search for one ... The catalog has been able to *find* by keyword since v1 — query.rs joins the keywords table, matches it exactly, and substring-matches it for free text — and nothing anywhere could ever put a word there. A user could filter to a keyword they had no way to apply. This is the missing half: create, rename, delete, list, assign, unassign, and the two reads a panel needs. Bulk-only for assignment, because keywording a selection is the common case rather than the exception — the photographer picks out the frames with the puffin in them and applies "puffin" once, in one transaction. Schema v6 adds `keyword_terms`, and deliberately does *not* touch the v1 join. The assignment keeps the word as text because the catalog is a rebuildable index and the durable copies of that fact — the sidecar, XMP dc:subject — both carry a string; a foreign key would mean a catalog rebuilt from sidecars had to invent identity rows before it could record anything, and would break the query path that already works. So the text is the fact, and the new table is only the identity a rename and a deletion can be keyed on. `keyword_terms.name` carries no unique index, which looks like an oversight and is not: two devices that each type "Iceland" are both right until they meet, and a constraint would abort the merge at that moment. Uniqueness is converged upon instead — create resolves an existing name, fuse_duplicates collapses a cross-device pair onto the smaller uuid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 15:50:49 +02:00
dtourolleandClaude Opus 5	03326242a1	Make the CI checks say what they mean, and format the workspace ... The Android job's "Verify minimum API level" step has never verified the minimum API level. It took the first `*.so` anywhere under the target directory, which is a host proc-macro from debug/deps — an x86-64 object built by the runner's gcc, whose .comment section cannot mention Android and so can never contradict the expected value. It now reads the artifact under the target triple, compares against MIN_API parsed from the Dockerfile rather than a second copy of the number, and fails on a mismatch. Both sides are checked non-empty first: two failed parses would otherwise compare equal and pass, which is the same silent success in a new costume. The Android image installs one SDK package per layer and keeps the output. sdkmanager is a JVM program that aborts when it cannot get memory, and the single `> /dev/null` step reported that as a bare "exit code 134" while a retry re-downloaded everything that had already succeeded. tools/ci-local.sh runs all four jobs — desktop, android, layering, traceability — against the host toolchain, which is pinned to the same 1.92.0 CI installs. Its matrix check compares regeneration against the working tree rather than against HEAD: CI starts from a clean checkout, so git's answer is the right one there and reports every local run stale here. The rest is rustfmt across the workspace, and the clippy findings that surfaced once it did: manual_contains in dr-thumbs and collections_ui, a map iterated as pairs for its keys, an index loop over a slice, and two runtime assertions on a constant now made at compile time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 12:02:51 +02:00
dtourolleandClaude Opus 5	fa12afed18	Keep originals on this device, by pin and by use ... Fills in `image_cache`, which the previous commit's "On this device" filter read but nothing wrote. Also carries in-flight work that shared these files: the Android TLS root store, the settings page, and a regenerated traceability report. # Two populations, deliberately separate An original is kept here for one of two reasons, and conflating them produces the exact failure the feature exists to prevent. **Pinned** originals were asked for. Pinning a collection before a trip is a promise, so pinned rows are never evicted and never counted against the budget — a cap that could silently delete a pinned trip would make pinning worthless, because it could not be relied on without checking. **Passively cached** originals are a side effect of working: develop already downloads the whole file, so keeping it costs no bandwidth and saves the entire transfer next time. This population is what the budget bounds, evicted least-recently-used, because it otherwise grows until a day of culling fills a disk. Sharing one budget would let a large pin starve the passive cache, or let browsing evict a pin. They are separate. # What was built `dr_catalog::cache` owns the bookkeeping — held tier, size, last use, pinned — and writes the bytes; deciding to download stays with the caller, which is what keeps a crate with no network out of the network's business. Files are written to a temporary and renamed, so a dropped connection cannot leave a truncated file recorded as a complete original. They are named by image id, not filename: `Photos/IMG_0001.CR2` and `Trips/IMG_0001.CR2` are different photographs, and a flat cache keyed on the name would serve one for the other. `spawn_full_fetch` became read-through. A hit is a disk read; a miss stores what it downloads and enforces the budget. A cache that cannot be opened is a miss, not a failure to open the photograph. Pinning writes intent — `tier_desired` — without downloading, so the button responds immediately, and `spawn_pin_fetch` fills it in sequentially afterwards. Sequential because these are tens of megabytes each: the lanes that make the thumbnail sweep fast buy little against one connection's bandwidth and cost a great deal of memory. A pin interrupted by a lost connection resumes from where it stopped. Schema v5 adds `pinned` and `path`. `pinned` is a column rather than something inferred from `pinned_by_rule`, which is ON DELETE SET NULL and so cannot answer for an image whose rule was deleted. A v4 catalog migrates in place; existing rows default to unpinned, the safe direction. The budget and "keep opened originals" come from the settings page rather than a constant, and are applied at startup rather than only on change — a cache capped at 2 GB last session would otherwise spend this one filling to the default. Turning off keeping leaves what is already cached readable: those bytes are paid for, and refusing them would re-download images sitting right there, including pinned ones. Also removes a doubled `#[test]` introduced in the previous commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-11 21:12:01 +02:00
dtourolle	d5b1f6bff5	Add collections, ratings, and soft delete to the catalog ... Three features over a shared schema migration. Collections: a tree of manual collections plus smart collections whose membership *is* their stored selector. Dropping images onto a smart collection is refused rather than silently discarded, so the UI can say why the drop did nothing — member rows there would be a second source of truth that nothing reads. Ratings: the star and pick/reject axes, kept independent. Trash: soft delete to a folder, then permanent delete. Catalog::open now backfills after migrating. A migration adds a column but cannot know what the value should be for rows that already existed; backfilling on open is what stops those rows being silently partial. Timeline queries exclude shadowed JPEGs, which would otherwise double every paired shot in the histogram, and gain a range-bounded variant so zooming in returns finer buckets rather than the same coarse ones with the ends cropped. Assisted-by: LLM	2026-08-09 20:42:13 +02:00
dtourolle	c8bb08e661	Add folder scan with format selection; validate A3 on a real library ... Library setup as the user described it: pick a folder, choose which RAW types to look for, scan recursively. dr-types::FormatFilter the tick-box selection, seeing through VFS placeholder suffixes so a dehydrated CR2 still matches as a CR2 dr-sync::scan recursive walk, Depth:1 per directory, pruning unchanged subtrees where the backend propagates directory ETags Verified against nextcloud.tourolle.paris (34.0.2) on a real library: browse root 32 entries, 98ms scan PhotosRaw 17,185 RAW files in 334 directories, 34.1s (7,836 CR2 + 9,349 DNG) range read 262KB of a 21.5MB DNG in 119ms — 1.22% of the file, and enough to read "Canon EOS 6D | ISO 100" That last line is assumption A3 validated on real data. Cataloguing this library by whole-file fetch would move roughly 370GB; the range path moves a few MB. Pruning is capability-gated rather than assumed: with per-entry ETags a probe costs a request and proves nothing about children, so it is skipped entirely. A test asserts zero probes in that case. Still unresolved: /core/preview returns 400 for every parameter combination tried, including on a JPEG the server reports as having a preview. Not a request-shape bug — it fails identically bare. Recorded rather than worked around; ARCH §6.7 already treats server previews as opportunistic, so nothing depends on it.	2026-08-09 12:22:31 +02:00