Let the manual's scripts find a control by its name

Every scene in tools/manual aimed at window pixels written in by hand, so
a panel that gained a row moved every slider under it and the recording
went on dragging where the slider used to be. The develop column has
already moved that way (Compose now sits above Adjust), and nothing said.

A build with the `automation` feature listens on the Unix socket named
by DR_AUTOMATION and answers where an element is: by its accessible
label, the name a screen reader reads, or by its markup id for the few
things that are not controls (the canvas, the crop rectangle). It uses
Slint's element queries, which need the compiler's debug tables, so the
feature also turns those on in build.rs. It only answers questions; the
input is still xdotool's real pointer. No default build has the feature,
and one that has it listens only when the variable is set.

drive.py gains click-on, drag-on, hold-on, wait-for, wait-gone, labels
and ids. The grid's cells are now named by their file, each rating star
by its value, the sidebar's + as "New collection", and the Adjust
heading's reset as "Reset all adjustments" - controls a screen reader
could not reach before either.

ui/dr-ui/Cargo.toml

@@ -104,6 +104,10 @@ anyhow.workspace = true
 thiserror.workspace = true
 log.workspace = true
 pollster.workspace = true
+# Slint's element queries, for the `automation` feature only. Pinned to the
+# exact Slint release: it reaches into i-slint-core, which is versioned in
+# lockstep and has no stable API of its own.
+i-slint-backend-testing = { version = "=1.17.1", default-features = false, optional = true }
 # Runtime YAML only for `live-style`; release builds read the tokens the
 # Slint compiler folded in at build time and never touch style.yaml.
 serde_norway = { workspace = true, optional = true }
@@ -143,6 +147,12 @@ serde_norway.workspace = true
 scene-model = ["dr-segment/embedded-scene-model"]
 
 default = []
+# Answer "where is the control labelled X?" over a Unix socket named by
+# `DR_AUTOMATION`, for the scripts that record the manual (src/automation.rs,
+# tools/manual). Off by default and never in a shipped build: it adds Slint's
+# element-query crate, and a listener nobody asked for has no place in a
+# release. `tools/manual/record.sh` builds with it.
+automation = ["dep:i-slint-backend-testing"]
 # Debug convenience: re-read style.yaml at startup so a palette can be tuned
 # without rebuilding. Costs the constant-folding of every token, so it stays
 # off by default and has no business in a release build.

ui/dr-ui/build.rs

@@ -134,6 +134,13 @@ fn main() {
         config = config.with_bundled_translations(lang);
     }
 
+    // The recording hook's element queries walk a tree the compiler only
+    // describes when asked to (src/automation.rs). Only that build asks: the
+    // extra tables are dead weight to everyone else.
+    if std::env::var_os("CARGO_FEATURE_AUTOMATION").is_some() {
+        config = config.with_debug_info(true);
+    }
+
     slint_build::compile_with_config(entry(&out_dir, live), config).expect("compiling app.slint");
 }
 

ui/dr-ui/src/automation.rs

@@ -0,0 +1,184 @@
+//! Where a control is, asked by its name — for the scripts that record the
+//! manual, and for nobody else.
+//!
+//! `tools/manual/drive.py` moves a real pointer over the window with xdotool,
+//! and until this existed it aimed at pixel coordinates written into
+//! `scenes.py` by hand. A panel that grew a row moved every slider below it,
+//! and the recording went on clicking where the slider used to be: no error,
+//! just a picture of the wrong thing. This answers "where is *Exposure*?" from
+//! the element tree itself, by the same accessible label a screen reader reads
+//! (NFR-A11Y-2), so a script that names controls survives a layout change the
+//! way a screen-reader user does.
+//!
+//! **Absent from every normal build.** The module exists only under the
+//! `automation` cargo feature, which no default build enables and
+//! `tools/manual/record.sh` does; and even in a build that has it, nothing
+//! listens unless `DR_AUTOMATION` names a socket path. It only reads the tree
+//! and answers: the input itself still arrives as ordinary pointer and key
+//! events from outside, so a scene exercises exactly what a person's hand
+//! would, gesture recognisers and all.
+//!
+//! ## The protocol
+//!
+//! One request per line on a Unix socket, one JSON line back:
+//!
+//! - `ping` — `"ok"`.
+//! - `locate LABEL` — every element whose accessible label is exactly
+//!   `LABEL`: `[{"label","role","x","y","w","h","opacity","enabled",
+//!   "checked","value"}, …]`, in physical window pixels, tree order.
+//! - `labels [SUBSTRING]` — every labelled element, or those whose label
+//!   contains `SUBSTRING`; the same objects. For finding what a scene can
+//!   name.
+//! - `locate-id ID` — the same, for elements by their id in the markup
+//!   (`canvas-image`, or qualified, `CropOverlay::move-area`) or by
+//!   component type (`Timeline`). For the few things a scene has to aim at
+//!   that are not controls — the photograph, the crop rectangle — and so
+//!   have no accessible name to give.
+//! - `ids [SUBSTRING]` — every element with an id, for finding those.
+//! - `window` — `{"w","h","scale"}`.
+//! - `canvas` — `{"w","h"}` of the develop canvas's current image, so a
+//!   script can place a point on the photograph within `canvas-image`.
+//!
+//! Only elements Slint considers visible are answered — not hidden, not
+//! clipped away by a scrolled list — so "on screen" is the default.
+//!
+//! Waiting is the client's job (poll `locate`), which keeps this stateless.
+
+use std::io::{BufRead, BufReader, Write};
+use std::os::unix::net::{UnixListener, UnixStream};
+use std::sync::mpsc;
+use std::time::Duration;
+
+use i_slint_backend_testing::{ElementHandle, ElementRoot as _};
+use serde_json::{json, Value};
+use slint::ComponentHandle as _;
+
+use crate::AppWindow;
+
+/// Listen on `DR_AUTOMATION` if it is set. Called once, before the event loop.
+pub(crate) fn attach(window: &AppWindow) {
+    let Some(path) = std::env::var_os("DR_AUTOMATION") else {
+        return;
+    };
+    let _ = std::fs::remove_file(&path);
+    let listener = match UnixListener::bind(&path) {
+        Ok(l) => l,
+        Err(e) => {
+            log::warn!("automation: cannot listen on {path:?}: {e}");
+            return;
+        }
+    };
+    log::info!("automation: listening on {path:?}");
+    let weak = window.as_weak();
+    std::thread::Builder::new()
+        .name("automation".into())
+        .spawn(move || {
+            for stream in listener.incoming().flatten() {
+                let weak = weak.clone();
+                std::thread::spawn(move || serve(stream, weak));
+            }
+        })
+        .ok();
+}
+
+fn serve(stream: UnixStream, weak: slint::Weak<AppWindow>) {
+    let Ok(mut out) = stream.try_clone() else {
+        return;
+    };
+    for line in BufReader::new(stream).lines() {
+        let Ok(line) = line else { return };
+        let (tx, rx) = mpsc::channel();
+        let request = line.clone();
+        let posted = weak.upgrade_in_event_loop(move |w| {
+            let _ = tx.send(answer(&w, &request));
+        });
+        let reply = match posted {
+            Ok(()) => rx
+                .recv_timeout(Duration::from_secs(10))
+                .unwrap_or_else(|_| json!({"error": "the event loop did not answer"})),
+            Err(e) => json!({"error": e.to_string()}),
+        };
+        if writeln!(out, "{reply}").is_err() {
+            return;
+        }
+    }
+}
+
+fn answer(window: &AppWindow, request: &str) -> Value {
+    let (verb, rest) = request.split_once(' ').unwrap_or((request, ""));
+    let scale = window.window().scale_factor();
+    match verb {
+        "ping" => json!("ok"),
+        "window" => {
+            let size = window.window().size();
+            json!({"w": size.width, "h": size.height, "scale": scale})
+        }
+        "locate" => Value::Array(
+            ElementHandle::find_by_accessible_label(window, rest)
+                .map(|e| describe(&e, scale))
+                .collect(),
+        ),
+        "locate-id" => {
+            let (needle, qualified) = (rest.to_string(), format!("::{rest}"));
+            query(window, scale, move |e| {
+                e.id()
+                    .is_some_and(|id| id == needle.as_str() || id.ends_with(&qualified))
+                    || e.type_name().is_some_and(|t| t == needle.as_str())
+            })
+        }
+        "ids" => {
+            let needle = rest.to_string();
+            query(window, scale, move |e| {
+                e.id()
+                    .is_some_and(|id| !id.is_empty() && id.contains(needle.as_str()))
+            })
+        }
+        "labels" => {
+            let needle = rest.to_string();
+            query(window, scale, move |e| {
+                e.accessible_label()
+                    .is_some_and(|l| !l.is_empty() && l.contains(needle.as_str()))
+            })
+        }
+        "canvas" => {
+            let size = window.get_canvas().size();
+            json!({"w": size.width, "h": size.height})
+        }
+        _ => json!({"error": format!("unknown request {verb:?}")}),
+    }
+}
+
+/// Every drawn element the predicate accepts, described, in tree order.
+fn query(
+    window: &AppWindow,
+    scale: f32,
+    predicate: impl Fn(&ElementHandle) -> bool + 'static,
+) -> Value {
+    window
+        .root_element()
+        .query_descendants()
+        .match_predicate(predicate)
+        .find_all()
+        .iter()
+        .map(|e| describe(e, scale))
+        .collect()
+}
+
+fn describe(e: &ElementHandle, scale: f32) -> Value {
+    let at = e.absolute_position();
+    let size = e.size();
+    json!({
+        "label": e.accessible_label().map(|s| s.to_string()),
+        "id": e.id().map(|s| s.to_string()),
+        "type": e.type_name().map(|s| s.to_string()),
+        "role": e.accessible_role().map(|r| format!("{r:?}")),
+        "x": (at.x * scale).round(),
+        "y": (at.y * scale).round(),
+        "w": (size.width * scale).round(),
+        "h": (size.height * scale).round(),
+        "opacity": e.computed_opacity(),
+        "enabled": e.accessible_enabled(),
+        "checked": e.accessible_checked(),
+        "value": e.accessible_value().map(|s| s.to_string()),
+    })
+}

ui/dr-ui/src/lib.rs

@@ -20,6 +20,8 @@
 //! in `ui/` names an operation or knows a shader exists (FR-DEV-3a).
 
 mod activity;
+#[cfg(all(feature = "automation", unix))]
+mod automation;
 mod bursts;
 mod collections_ui;
 #[cfg(test)]
@@ -1361,6 +1363,9 @@ pub fn run(paths: Vec<PathBuf>) -> Result<()> {
         launch_began.elapsed().as_millis()
     );
 
+    #[cfg(all(feature = "automation", unix))]
+    automation::attach(&window);
+
     window.run()?;
     Ok(())
 }

ui/dr-ui/ui/adjust.slint

@@ -1285,6 +1285,12 @@ export component AdjustPanel inherits Rectangle {
                 width: 44px;
                 height: 20px;
                 clicked => { Adjustments.reset-all(); }
+                // Named apart from the groups' "Reset" buttons below it: this
+                // one empties the whole panel, and a reader moving through
+                // the column has to be able to tell the two apart.
+                accessible-role: button;
+                accessible-label: "Reset all adjustments";
+                accessible-action-default => { Adjustments.reset-all(); }
                 Label {
                     text: "reset";
                     emphasised: reset.has-hover;

ui/dr-ui/ui/collections.slint

@@ -579,6 +579,10 @@ export component CollectionsPanel inherits Rectangle {
                             : (add-touch.has-hover ? Theme.hover : transparent);
                 border-radius: Theme.radius-sm;
 
+                accessible-role: button;
+                accessible-label: "New collection";
+                accessible-action-default => { root.new-collection(); }
+
                 Icon {
                     name: "plus";
                     ink: Theme.ink-dim;

ui/dr-ui/ui/library.slint

@@ -1210,6 +1210,21 @@ export component StarStrip inherits Rectangle {
             width: root.star;
             height: root.star;
 
+            // Each star is a button that sets its rating, and says so. The
+            // strip is only in the tree while it is drawn — on the cell under
+            // the pointer, or on a rated one — which is also when it can be
+            // pressed.
+            accessible-role: button;
+            accessible-label: n == 1 ? "1 star" : n + " stars";
+            accessible-enabled: root.interactive;
+            accessible-checkable: true;
+            accessible-checked: root.rating >= n;
+            accessible-action-default => {
+                if (root.interactive) {
+                    root.rate(root.rating == n ? 0 : n);
+                }
+            }
+
             Icon {
                 // Solid versus outline: the shape says it, not the colour.
                 name: root.rating >= n ? "star" : "star-outline";
@@ -3730,6 +3745,14 @@ export component LibraryGrid inherits Rectangle {
                 //            photographs file in one gesture.
                 //   manual:  collections
                 for cell[i] in root.cells: DragArea {
+                    // A cell is its photograph, named by its file: the name
+                    // printed under the thumbnail, and the one a screen
+                    // reader or the manual's recording scripts ask for.
+                    accessible-role: list-item;
+                    accessible-label: cell.name;
+                    accessible-item-selectable: true;
+                    accessible-item-selected: cell.selected;
+
                     // Cells are positioned at their **absolute** place in the
                     // library, not their index in the loaded window: the window
                     // starts at `offset`, so a cell drawn at window-index 0 belongs