diff --git a/docs/dev/catalog.md b/docs/dev/catalog.md index ae442d4..61a85ec 100644 --- a/docs/dev/catalog.md +++ b/docs/dev/catalog.md @@ -129,6 +129,17 @@ CREATE INDEX members_image ON collection_members(image_id); partial index over the non-NULL subset is both smaller and what FR-CAT-9's reconnection-by-hash and FR-CAT-11's duplicate detection actually query. +**Made on first use, not by a migration.** A new `user_version` makes every older build refuse this +catalog's snapshot at sync (`sync::remote_is_mergeable` compares it and nothing else), and a tablet +a release behind would stop merging collections, keywords and people for a feature it does not +have. So what later releases added without needing old rows rewritten is created with `IF NOT +EXISTS` where it is first used, and an older build that meets it ignores it: + +- `dedup_probes` (FR-CAT-11a, §3.5); +- the albums (FR-EXP-10, `core/dr-catalog/src/albums.rs`): `albums`, `album_exports` — one row per + file written into an album, keyed on the file name, since two crops of one photograph are two + files — and `album_folders`, this device's folder for each (§8.2). + --- ## 3. Incremental scan @@ -686,13 +697,29 @@ mechanism. ### 8.2 What the file sync does and does not carry -Only **collections and their membership** merge. The rest of a catalog describes *local* state — -folder mtimes, cache file paths, job rows, `tier_actual` — and importing another device's version -of those would be actively wrong. The downloaded remote is read for its collections and discarded. +Collections were the first thing merged, and the rules in §8.4 were written for them. What else +merges reuses those rules or keys on the same identities, and each has no other home: -This is what keeps §6.12 substantially intact: nothing here makes the local database authoritative -for anything a rebuild could not recover. The catalog is still deletable. What syncs is one table -pair that had no other home. +- **Collections and their membership** — by uuid and revision, membership as a set union. +- **Keywords** — the vocabulary by the same verdict, the assignments as a union. +- **People and identity judgements** — people by uuid and revision, and the confirmed and rejected + face assignments matched to local faces by box (`merge::match_faces`). +- **Albums** (FR-EXP-10, 0.17.0) — by uuid and revision with tombstones, and what went into each as + a set union keyed on the server's file id (a content hash on a folder library). An album's + server folder is a column of its row and travels with it; a folder on *this device* is in + `album_folders`, which the merge never reads and the upload snapshot drops (§8.3), because a + path or a SAF grant on one device means nothing on another. +- **Capture metadata** — the one exception inside `images`: a date, a camera, a lens and an ISO are + facts about the file's bytes, so a row this device has not yet read takes them from a peer that + has (`merge_metadata`), matched by `oc:fileid`. + +The rest of a catalog describes *local* state — folder mtimes, cache file paths, job rows, +`tier_actual` — and importing another device's version of those would be actively wrong; the +downloaded remote is read for the tables above and discarded. + +This is what keeps §6.12 substantially intact. The catalog is still deletable; what a rebuild from +sidecars cannot recover — collections and albums, which nothing in the filesystem records — is what +the sync exists to carry. ### 8.3 Two hazards the implementation must handle @@ -714,7 +741,8 @@ crop. They were first stripped (2026-08) by copying the whole file with the back `crop` to NULL and `VACUUM`ing. That wrote the file about three times to upload 50 MB. Since #71 the snapshot is built without them. Its header is kept as it was (`user_version`, page size and the WAL flag), so every earlier build merges it unchanged. NFR-R2 backups still use the backup API -and keep the crops, because a backup is a file the user may have to live on. +and keep the crops, because a backup is a file the user may have to live on. `album_folders` is +dropped from the built file for the reason §8.2 gives. **Integer primary keys are not identities.** Two devices each allocate `collections.id = 1` for different collections, so a row-level merge keyed on the integer id would collide them. Collections diff --git a/docs/dev/outstanding.md b/docs/dev/outstanding.md index 9a1e408..ff531dc 100644 --- a/docs/dev/outstanding.md +++ b/docs/dev/outstanding.md @@ -34,6 +34,10 @@ beside re-import detection; the accessibility and localisation counts in §6 had against the tree since 2026-08-30 and are replaced; and §4a records what the develop and keyboard work of 0.15.0 and 0.16.0 left open. +**And again for 0.17.0.** Folders are chosen through the platform's dialogue now, so FR-PLAT-LIN-3 +in §5 is rewritten around what the Flatpak has still not shown; albums brought the first SAF code, +which FR-PLAT-AND-1's entry now describes, along with why its new tag overstates it. + --- ## 1. Plugins — post-v1 since 2026-09-19 @@ -234,22 +238,30 @@ models, has been measured on a tablet ([faces.md §12.1](faces.md)), and since 0 develop view zero-copy as the desktop does ([technical-debt.md TD-1](technical-debt.md), paid off). It carries the manual and opens it in a WebView. What is missing is the platform contract around it. -**FR-PLAT-AND-1 is untagged, and what it was tagged for was intent rather than code.** -The requirement demands that library access be obtained *exclusively* through the Storage Access -Framework. There is no SAF code: no `ACTION_OPEN_DOCUMENT_TREE`, no `takePersistableUriPermission`, -no `DocumentsContract`. Its two tags rested on a `SourceRef::Document` variant constructed only -inside `#[cfg(test)]` — `LocalStorage::open` refuses it, and the test that proves so is named -`a_reference_of_the_wrong_kind_is_refused_rather_than_guessed_at` — and on -`dr_plat::imports_supported`, which *returns false on Android* and whose own documentation says it -"stops being false when a SAF implementation lands". The second tag documented the absence of the -thing it was counted as evidence for. Both have been removed; this is the "plumbing a future feature -would use" case [CONTRIBUTING.md](../../CONTRIBUTING.md) and [code-health.md CH-4](code-health.md) both -warn about. Android reaches a library through a Nextcloud account or a folder, over paths, like the -desktop. +**FR-PLAT-AND-1 — SAF is built for export folders, not for the library.** The requirement demands +that library access be obtained *exclusively* through the Storage Access Framework. Until 0.17.0 +there was no SAF code at all, and the requirement's two tags rested on a `SourceRef::Document` +variant constructed only inside `#[cfg(test)]` and on `dr_plat::imports_supported`, which *returns +false on Android* and whose own documentation says it "stops being false when a SAF implementation +lands". Both were removed as intent rather than code — the "plumbing a future feature would use" +case [CONTRIBUTING.md](../../CONTRIBUTING.md) and [code-health.md CH-4](code-health.md) both warn +about. + +0.17.0 brought the first real SAF code, for albums (FR-EXP-10): `FolderPicker.java` starts +`ACTION_OPEN_DOCUMENT_TREE` from a translucent activity of its own (the main activity is +`NativeActivity`, whose results are not ours) and takes a persistable grant; `Saf.java` writes each +export through `DocumentsContract`; `ui/dr-ui/src/saf.rs` is the JNI bridge. `saf.rs` and the export +path now carry `TRACES: FR-PLAT-AND-1`, and the matrix counts the requirement as covered. **That +overstates it.** The mechanism is the one the requirement names, but its subject is the library, and +Android still reaches a library through a Nextcloud account or a folder, over paths, like the +desktop. Either the tags narrow to FR-EXP-10 or the requirement is met for the library too; until +one of those, read the coverage figure with this one subtracted. That has a consequence for the rest of the cluster: **FR-PLAT-AND-2** — detecting the loss of a -granted tree permission and marking images offline rather than deleting rows — cannot be built until -there is a permission to lose. It is listed here as unbuilt, but it is blocked, not skipped. +granted tree permission and marking images offline rather than deleting rows — is still blocked for +the library, because there is no library grant to lose. An album's folder has a grant now, and +nothing checks for its loss: an export into a folder whose grant has gone fails with whatever the +write raises, rather than the album saying beforehand that it needs a folder again. **FR-PLAT-AND-4 — half built.** The runner is done (`core/dr-catalog/src/runner.rs`): the queue that `jobs.rs` always had is now claimed from, completed, failed and recovered after a @@ -501,7 +513,7 @@ requirement text that asks for it: |---|---|---| | S6 | FR-DSP-2, NFR-RES-2 — tiling and images larger than GPU memory | Nothing; needs a device and a large image | | S9 | R1's tolerance threshold, and therefore R1 | Nothing; the threshold is defined *by* running it | -| S10 | Whether SAF at 10k files meets NFR-P1/P3 | §5 — there is no SAF code to measure | +| S10 | Whether SAF at 10k files meets NFR-P1/P3 | §5 — SAF reaches album folders only, never a library to enumerate | | S11 | NFR-COMPAT-2, and whether Play makes SAF binding | Nothing | | S13 | NFR-A11Y-2 on Android | §6 — there is almost nothing to test with | diff --git a/docs/dev/storage.md b/docs/dev/storage.md index dcc7d25..d8c0bf3 100644 --- a/docs/dev/storage.md +++ b/docs/dev/storage.md @@ -85,6 +85,9 @@ pub trait RemoteBackend: Send + Sync { // transfer async fn get(&self, id: &RemoteId, range: Option>) -> Result, RemoteError>; + async fn get_reporting(&self, id: &RemoteId, // defaulted + progress: &(dyn Fn(u64, Option) + Send + Sync)) + -> Result, RemoteError>; async fn put(&self, path: &RemotePath, body: Vec, precond: Option) -> Result; async fn put_many(&self, items: Vec<(RemotePath, Vec)>) // defaulted @@ -111,6 +114,16 @@ Rules that are not obvious from the signatures: - **`get` takes an optional range, and it is a hint.** A backend without cheap ranges may return the whole object; the caller slices. Correctness holds either way and `Capabilities::range_reads` says whether it was cheap. +- **`get_reporting` is for the one transfer somebody is watching.** An original + opened in develop is tens of megabytes, and "downloading" alone for that long + reads as stuck. It reports the bytes received and the length the server + declared, if it declared one. The default is `get` whole and one report at the + end, which is right for a backend whose read is local; Nextcloud overrides it + to read the body chunk by chunk. The develop view reads the figures by path + from the in-flight registry in `ui/dr-ui/src/library/thumbnails_fetch.rs`, + because a step along the roll usually lands on a frame the prefetcher is + already fetching, and falls back on the catalog's file length when no length + was declared. - **Chunked upload is not in the trait.** It is an implementation detail of `put`, chosen by body size. Exposing it would leak one server's protocol. - **`move_to` must preserve identity where the backend has stable ids.** This @@ -118,7 +131,9 @@ Rules that are not obvious from the signatures: allocates a new id, orphaning the thumbnail shard and turning a restore into a full re-download. - **`create_dir` makes parents and succeeds if the directory exists.** Callers - use it to guarantee a destination, not to claim they created one. + use it to guarantee a destination, not to claim they created one. It is also + the server browser's `New folder` (§5.3), which then lists the parent again + rather than inserting the name it asked for: the server may have normalised it. ### 3.2 `Capabilities` — what is cheap @@ -294,6 +309,7 @@ for a second backend: | `bulk_upload` | yes, `POST /remote.php/dav/bulk` | | `conditional_write` | yes, `If-Match` | | `server_previews` | `CommonFormatsOnly` — stock Nextcloud renders no RAW | +| `get_reporting` | overridden: the body is read chunk by chunk, against `Content-Length` | | sign-in | `SignIn::Browser`, Login Flow v2, system browser, app password | Also kept: the `oc:permissions` probe on a refused `PUT`, which is what @@ -315,7 +331,8 @@ which makes it the route that works on a machine with no secrets daemon at all. | `bulk_upload` | no | | `conditional_write` | yes, with a documented residual race | | `server_previews` | `None` | -| sign-in | `SignIn::EndpointOnly` | +| `get_reporting` | the default: a local read, reported once when it is done | +| sign-in | `SignIn::EndpointOnly`, the folder chosen in the platform's dialogue (§5.3) | **Why `LocalEtags` and not `PropagatingEtags`.** A POSIX directory's mtime changes when its own entry list changes and at no other time — not when a @@ -377,6 +394,43 @@ behind a network banner. An endpoint that is not a directory at all maps to `RemoteError::Configuration`: nothing was unreachable and no credential was wrong, so neither of the other two would send the user anywhere useful. +### 5.3 Choosing folders, and the folders exports go to + +**Folders are pointed at, never typed** (FR-EXP-6). On the desktop, +`ui/dr-ui/src/folder_dialog.rs` asks the platform: the XDG desktop portal's +FileChooser on Linux, through `rfd`, and the common item dialogue on Windows. +That is how the folder connector's endpoint is chosen on the launch screen, and +the path it returns still goes through `normalise_endpoint` as a typed one did. +A folder on the server is chosen in the in-app browser, which lists with `list` +and makes a folder with `create_dir`; the launch screen and the album sheet +share it through `ui/dr-ui/src/remote_folders.rs`, which keeps both round trips +off the interface thread. Android has no filesystem dialogue, so its library +folder is still typed, and an album's folder there comes from the Storage +Access Framework's tree picker. + +**An album is where exports go** (FR-EXP-10), and never inside the library: a +JPEG written into the tree a scan catalogues comes back as a photograph beside +the RAW it was made from. Its folder is one of two kinds +(`dr_catalog::albums::Place`): + +- **On the server,** relative to the *account* root, not the library root. The + browser refuses a folder inside the library and says why; on a server whose + whole account is the library, every folder is inside it, and it says that + instead. Exports reach it through the outbox like any upload, and a queued + file's `.dest` record gains a third line saying its folder is relative to + the account — a third line rather than a leading slash, because a record + written before albums may carry a stray slash and has to keep the meaning it + was written with. +- **On this device,** a path, or on Android a SAF tree URI with a persisted + grant, written through `DocumentsContract` (`ui/dr-ui/src/saf.rs`). A + provider renames on a collision by itself, so the album records the name it + was given rather than the one asked for. + +A server folder lives on the album row and syncs with it. A device folder lives +in `album_folders`, which the merge never reads and the upload snapshot drops, +because a path or a grant on one device means nothing on another: an album made +on the desktop arrives on the tablet with no folder until one is chosen there. + --- ## 6. Virtual filesystems