dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Let the manual's scripts find a control by its name

Browse Source 
Every scene in tools/manual aimed at window pixels written in by hand, so
a panel that gained a row moved every slider under it and the recording
went on dragging where the slider used to be. The develop column has
already moved that way (Compose now sits above Adjust), and nothing said.

A build with the `automation` feature listens on the Unix socket named
by DR_AUTOMATION and answers where an element is: by its accessible
label, the name a screen reader reads, or by its markup id for the few
things that are not controls (the canvas, the crop rectangle). It uses
Slint's element queries, which need the compiler's debug tables, so the
feature also turns those on in build.rs. It only answers questions; the
input is still xdotool's real pointer. No default build has the feature,
and one that has it listens only when the variable is set.

drive.py gains click-on, drag-on, hold-on, wait-for, wait-gone, labels
and ids. The grid's cells are now named by their file, each rating star
by its value, the sidebar's + as "New collection", and the Adjust
heading's reset as "Reset all adjustments" - controls a screen reader
could not reach before either.
This commit is contained in:
dtourolle
2026-09-25 07:26:36 -04:00
parent b480de5bff
commit a6ea6ba83f
12 changed files with 569 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Cargo.lock
Generated
+13
View File
@@ -1704,6 +1704,7 @@ dependencies = [
"dr-types",
"dr-xmp",
"env_logger",
"i-slint-backend-testing",
"jni 0.22.4",
"log",
"ndk-context",
@@ -2781,6 +2782,18 @@ dependencies = [
"i-slint-renderer-skia",
]
[[package]]
name = "i-slint-backend-testing"
version = "1.17.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "521e901e3d47ab829c0ef500c63155776208707cd93259e6a7803ed627fa2786"
dependencies = [
"cfg_aliases",
"i-slint-common",
"i-slint-core",
"vtable",
]
[[package]]
name = "i-slint-backend-winit"
version = "1.17.1"
apps/darkroom-desktop/Cargo.toml
+3
View File
@@ -25,3 +25,6 @@ winresource = "0.1"
[features]
default = []
# The manual's recording hook (dr-ui's `automation`); tools/manual/record.sh
# builds with it, nothing else does.
automation = ["dr-ui/automation"]
docs/dev/traceability.md
+54 -54
View File
File diff suppressed because one or more lines are too long
docs/gestures.md
+26 -26
View File
@@ -440,7 +440,7 @@ The right match confidence is a property of your library, not of the model. "Wha
Touch has no ctrl, so without a mode there is no way to select a second photograph — the first tap would open it. The hold is the fast way in and the button is the one that can be found.
<sub>`ui/dr-ui/ui/library.slint:1683`</sub>
<sub>`ui/dr-ui/ui/library.slint:1698`</sub>
### Add or remove one photograph
@@ -450,7 +450,7 @@ Touch has no ctrl, so without a mode there is no way to select a second photogra
While selecting, a tap never opens. That is the whole point of the mode: one meaning per gesture at a time. Press Done to get tap-to-open back.
<sub>`ui/dr-ui/ui/library.slint:1693`</sub>
<sub>`ui/dr-ui/ui/library.slint:1708`</sub>
### Leave selecting
@@ -459,7 +459,7 @@ While selecting, a tap never opens. That is the whole point of the mode: one mea
- **Keyboard** — `Escape`, or `Back`; an open sheet closes first
- **See it** — [in the manual](manual/README.md#selecting-several)
<sub>`ui/dr-ui/ui/library.slint:1702`</sub>
<sub>`ui/dr-ui/ui/library.slint:1717`</sub>
### Pick a photograph up to drag it
@@ -469,7 +469,7 @@ While selecting, a tap never opens. That is the whole point of the mode: one mea
A finger on a photograph might be starting a scroll, and for the first half-second the grid assumes it is. Holding says otherwise, and the ring is the grid saying it heard — from there the drag cannot be lost to a scroll. A mouse never waits: the cursor is precise enough that a sideways drag is unambiguous from the first pixel.
<sub>`ui/dr-ui/ui/library.slint:1733`</sub>
<sub>`ui/dr-ui/ui/library.slint:1748`</sub>
### Select a range
@@ -480,7 +480,7 @@ A finger on a photograph might be starting a scroll, and for the first half-seco
This replaced a double tap, which had no visible state and could take forty photographs by accident. The run is resolved by the catalog rather than by what is on screen, so the grid can scroll between the two taps — the ranges that hurt on a tablet are longer than a screenful, which is exactly where a finger sweep runs out.
<sub>`ui/dr-ui/ui/library.slint:1799`</sub>
<sub>`ui/dr-ui/ui/library.slint:1814`</sub>
### Take the blinks out of a burst
@@ -490,7 +490,7 @@ This replaced a double tap, which had no visible state and could take forty phot
Face indexing reads each face's eyes. The chip drops frames where the chosen people are caught blinking, and leaves sunglasses and eyes it could not read alone.
<sub>`ui/dr-ui/ui/library.slint:2516`</sub>
<sub>`ui/dr-ui/ui/library.slint:2531`</sub>
### Find photographs with two people in them
@@ -500,7 +500,7 @@ Face indexing reads each face's eyes. The chip drops frames where the chosen peo
"Any of them" is a union and "all of them" is an intersection. The tray is where both terms and the choice between them live, because a filter belongs on the filter bar.
<sub>`ui/dr-ui/ui/library.slint:2546`</sub>
<sub>`ui/dr-ui/ui/library.slint:2561`</sub>
### Show only photographs with one colour label
@@ -510,7 +510,7 @@ Face indexing reads each face's eyes. The chip drops frames where the chosen peo
Each chip is the label's mark and its name, so the one you want is found by reading it; tap the lit chip again to show every label.
<sub>`ui/dr-ui/ui/library.slint:2670`</sub>
<sub>`ui/dr-ui/ui/library.slint:2685`</sub>
### Export the selection as the last export was
@@ -521,7 +521,7 @@ Each chip is the label's mark and its name, so the one you want is found by read
Lightroom's and darktable's chords. Every export runs on the saved defaults, so the plain chord opens them beside an Export button and the shifted one skips straight to exporting.
<sub>`ui/dr-ui/ui/library.slint:3153`</sub>
<sub>`ui/dr-ui/ui/library.slint:3168`</sub>
### Paste copied settings onto the selection
@@ -530,7 +530,7 @@ Lightroom's and darktable's chords. Every export runs on the saved defaults, so
- **Keyboard** — `Ctrl+V`
- **See it** — [in the manual](manual/README.md#copying-settings)
<sub>`ui/dr-ui/ui/library.slint:3177`</sub>
<sub>`ui/dr-ui/ui/library.slint:3192`</sub>
### Keyword the selection
@@ -540,7 +540,7 @@ Lightroom's and darktable's chords. Every export runs on the saved defaults, so
Lightroom's keywording chord. The sheet opens with its field ready for typing, so the keys that judge in the grid are out of the way until it closes.
<sub>`ui/dr-ui/ui/library.slint:3206`</sub>
<sub>`ui/dr-ui/ui/library.slint:3221`</sub>
### Show only photographs with some number of stars
@@ -551,7 +551,7 @@ Lightroom's keywording chord. The sheet opens with its field ready for typing, s
The chips say "this many or more". A range with a ceiling — the twos and threes still to be decided — is the keyboard's alone, and the bar says so in words while it holds.
<sub>`ui/dr-ui/ui/library.slint:3240`</sub>
<sub>`ui/dr-ui/ui/library.slint:3255`</sub>
### Give photographs a colour label
@@ -562,7 +562,7 @@ The chips say "this many or more". A range with a ceiling — the twos and three
Lightroom's keys, so hands that learned them there need not learn them again. Purple has no key there either, and is on the bar. Every mark carries its label's initial, so the label is read without telling the colours apart.
<sub>`ui/dr-ui/ui/library.slint:3290`</sub>
<sub>`ui/dr-ui/ui/library.slint:3305`</sub>
### Pick or reject a photograph
@@ -572,7 +572,7 @@ Lightroom's keys, so hands that learned them there need not learn them again. Pu
The keys every culling tool uses, so muscle memory built elsewhere works here.
<sub>`ui/dr-ui/ui/library.slint:3314`</sub>
<sub>`ui/dr-ui/ui/library.slint:3329`</sub>
### Move photographs to the trash
@@ -582,7 +582,7 @@ The keys every culling tool uses, so muscle memory built elsewhere works here.
The bin acts on one photograph, so a stray click cannot trash a selection; the key acts on the selection because that is what every file manager's Delete does. Both are undone from the trash view.
<sub>`ui/dr-ui/ui/library.slint:3341`</sub>
<sub>`ui/dr-ui/ui/library.slint:3356`</sub>
### Open this list
@@ -590,7 +590,7 @@ The bin acts on one photograph, so a stray click cannot trash a selection; the k
- **Pointer** — Press Help in the header, and Done to put it away
- **Keyboard** — `F1`, and `Escape` to put it away
<sub>`ui/dr-ui/ui/library.slint:3366`</sub>
<sub>`ui/dr-ui/ui/library.slint:3381`</sub>
### Rename the collection the grid is showing
@@ -598,7 +598,7 @@ The bin acts on one photograph, so a stray click cannot trash a selection; the k
- **Pointer** — Double-click it in the sidebar
- **Keyboard** — `F2`
<sub>`ui/dr-ui/ui/library.slint:3374`</sub>
<sub>`ui/dr-ui/ui/library.slint:3389`</sub>
### Move through the grid
@@ -608,7 +608,7 @@ The bin acts on one photograph, so a stray click cannot trash a selection; the k
The cursor selects what it lands on, so walking and judging are one hand's work.
<sub>`ui/dr-ui/ui/library.slint:3394`</sub>
<sub>`ui/dr-ui/ui/library.slint:3409`</sub>
### Resize the thumbnails
@@ -619,7 +619,7 @@ The cursor selects what it lands on, so walking and judging are one hand's work.
There is no wheel on a tablet, so without the pinch the cell size could only be changed by a control a finger cannot reach.
<sub>`ui/dr-ui/ui/library.slint:3521`</sub>
<sub>`ui/dr-ui/ui/library.slint:3536`</sub>
### File photographs in a collection
@@ -629,7 +629,7 @@ There is no wheel on a tablet, so without the pinch the cell size could only be
The selection is what the drag carries, which is why selecting several is worth the mode: forty photographs file in one gesture.
<sub>`ui/dr-ui/ui/library.slint:3720`</sub>
<sub>`ui/dr-ui/ui/library.slint:3735`</sub>
### Open a photograph
@@ -640,7 +640,7 @@ The selection is what the drag carries, which is why selecting several is worth
A tap opens; a tap that *moved* does not. Travel is what separates a deliberate tap from a hand brushing past, and it is the only thing that does: the two are the same length. An earlier version required the finger to dwell 120 ms instead, and that rejected ordinary taps — a real tap is often quicker than a brush.
<sub>`ui/dr-ui/ui/library.slint:4017`</sub>
<sub>`ui/dr-ui/ui/library.slint:4040`</sub>
### Rate a photograph without opening it
@@ -651,7 +651,7 @@ A tap opens; a tap that *moved* does not. Travel is what separates a deliberate
A star has to take the press without it also reaching the cell, or every rating throws the user into develop.
<sub>`ui/dr-ui/ui/library.slint:4140`</sub>
<sub>`ui/dr-ui/ui/library.slint:4163`</sub>
### Choose the frame a folded burst shows
@@ -661,7 +661,7 @@ A star has to take the press without it also reaching the cell, or every rating
A folded burst draws its earliest frame, which is a fact about the clock and not a judgement about the photograph — nothing in this application ranks a frame (FR-CULL-5). But the point of a burst is that one of the twelve is better than the other eleven, and the photographer is the only one who knows which. So the choice is offered on the frames themselves, while they are open and side by side, which is the one moment the alternatives are on screen to be compared.
<sub>`ui/dr-ui/ui/library.slint:4273`</sub>
<sub>`ui/dr-ui/ui/library.slint:4296`</sub>
### Drop the selection but keep selecting
@@ -672,7 +672,7 @@ A folded burst draws its earliest frame, which is a fact about the clock and not
Distinct from Done, which leaves the mode entirely. Clearing keeps it, so the next selection can start straight away.
<sub>`ui/dr-ui/ui/library.slint:4950`</sub>
<sub>`ui/dr-ui/ui/library.slint:4973`</sub>
### Select everything the grid is showing
@@ -683,7 +683,7 @@ Distinct from Done, which leaves the mode entirely. Clearing keeps it, so the ne
A scoped grid of two hundred frames is two hundred taps otherwise, and "all of them, except those three" is a far more common shape than the taps it took to say it.
<sub>`ui/dr-ui/ui/library.slint:4969`</sub>
<sub>`ui/dr-ui/ui/library.slint:4992`</sub>
### Take photographs out of a collection
@@ -693,4 +693,4 @@ A scoped grid of two hundred frames is two hundred taps otherwise, and "all of t
The badge on a cell says a photograph is filed in three collections and never which. This is the sheet that names them, and the only way out of one the grid is not currently scoped to.
<sub>`ui/dr-ui/ui/library.slint:5150`</sub>
<sub>`ui/dr-ui/ui/library.slint:5173`</sub>
tools/manual/drive.py
+234 -14
View File
@@ -12,6 +12,31 @@
drive.py rec OUT.mp4 / cut start and stop a recording of the window
drive.py where the pointer, in window coordinates
By name — the accessible label a screen reader reads, answered by the app
itself when it is built with `--features automation` (see record.sh and
ui/dr-ui/src/automation.rs):
drive.py labels [TEXT] every labelled element on screen (containing TEXT)
drive.py ids [TEXT] every element with a markup id (containing TEXT)
drive.py locate NAME what NAME finds, as JSON
drive.py click-on NAME [button]
drive.py drag-on NAME DX DY [steps] from the element's centre
drive.py hold-on NAME SECONDS
drive.py wait-for NAME [timeout] until it is on screen
drive.py wait-gone NAME [timeout] until it is not
A NAME is an accessible label, exactly: `Exposure`, `Select`, `_MG_8393`.
Suffixes narrow it — `@ROLE` takes only elements of that role (`Button`,
`Slider`, `ListItem`, `Text`...), `#N` the N-th match in tree order from 0,
`#-1` the last (a sheet is drawn after what it covers). A leading `id:`
names an element by its id in the markup or its component type instead
(`id:canvas-image`, `id:move-area`, `id:Timeline`), for the few things a
scene aims at that are not controls. Only what Slint draws is answered:
nothing hidden, nothing scrolled out of its list.
The input itself is still xdotool — a real pointer, pressed and held — so a
scene exercises exactly what a hand would. The hook only says where to aim.
Everything is in *window* pixels, at scale 1, with the window at the origin
of a 1920×1200 Xvfb on `DR_DISPLAY` (`:7`). The app gets its own XDG
profile under `DR_HOME` (`/var/tmp/dr-manual`), so nothing here touches the
@@ -23,7 +48,10 @@ xdotool's `click` is sometimes dropped; and under XWayland at scale 2 a
pointer warp lands at twice the coordinate asked for, which is why this runs
on Xvfb rather than the desktop.
"""
import json
import os
import re
import socket
import subprocess
import sys
import time
@@ -31,16 +59,27 @@ import time
HOME = os.environ.get('DR_HOME', '/var/tmp/dr-manual')
DISPLAY = os.environ.get('DR_DISPLAY', ':7')
BIN = os.environ.get('DR_BIN', 'target/release/darkroom-desktop')
ENV = dict(
os.environ,
XDG_CONFIG_HOME=f'{HOME}/xdg/config',
XDG_DATA_HOME=f'{HOME}/xdg/data',
XDG_STATE_HOME=f'{HOME}/xdg/state',
RUST_LOG='info',
WINIT_X11_SCALE_FACTOR='1',
DISPLAY=DISPLAY,
)
ENV.pop('WAYLAND_DISPLAY', None)
SOCKET = os.environ.get('DR_AUTOMATION', f'{HOME}/automation.sock')
def app_env(xdg=None):
"""The app's environment: its own XDG tree (`DR_HOME/xdg` unless told
otherwise), X11 at scale 1, and the automation socket."""
xdg = xdg or f'{HOME}/xdg'
env = dict(
os.environ,
XDG_CONFIG_HOME=f'{xdg}/config',
XDG_DATA_HOME=f'{xdg}/data',
XDG_STATE_HOME=f'{xdg}/state',
RUST_LOG='info',
WINIT_X11_SCALE_FACTOR='1',
DISPLAY=DISPLAY,
DR_AUTOMATION=SOCKET,
)
env.pop('WAYLAND_DISPLAY', None)
return env
os.environ['DISPLAY'] = DISPLAY
@@ -61,10 +100,10 @@ def geometry():
return px, py
def launch(args):
def launch(args, xdg=None):
os.makedirs(HOME, exist_ok=True)
log = open(f'{HOME}/app.log', 'w')
p = subprocess.Popen([BIN, *args], env=ENV, stdout=log, stderr=subprocess.STDOUT)
p = subprocess.Popen([BIN, *args], env=app_env(xdg), stdout=log, stderr=subprocess.STDOUT)
open(f'{HOME}/app.pid', 'w').write(str(p.pid))
w = ''
for _ in range(120):
@@ -83,11 +122,21 @@ def launch(args):
def stop():
"""SIGTERM to the app this profile launched, and wait for it to go.
Not `windowclose`: winit panics on that under Xvfb."""
try:
os.kill(int(open(f'{HOME}/app.pid').read()), 15)
pid = int(open(f'{HOME}/app.pid').read())
os.kill(pid, 15)
except (OSError, ValueError) as e:
print(e)
time.sleep(1)
return
for _ in range(100):
try:
os.kill(pid, 0)
time.sleep(0.1)
except OSError:
break
time.sleep(0.5)
def shot(out):
@@ -142,6 +191,153 @@ def drag_path(points, steps=20):
x('mouseup', 1)
# --- by name ----------------------------------------------------------------
class NotFound(Exception):
pass
def ask(request):
"""One request to the app's automation hook, and its JSON answer."""
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
s.settimeout(15)
s.connect(SOCKET)
s.sendall(request.encode() + b'\n')
buf = b''
while not buf.endswith(b'\n'):
chunk = s.recv(65536)
if not chunk:
break
buf += chunk
answer = json.loads(buf)
if isinstance(answer, dict) and 'error' in answer:
raise RuntimeError(f'{request}: {answer["error"]}')
return answer
def wait_ready(timeout=120):
"""Until the hook answers: the app is up and its event loop running."""
t0 = time.time()
while True:
try:
if ask('ping') == 'ok':
return
except OSError:
pass
if time.time() - t0 > timeout:
raise TimeoutError('the automation hook never answered; was the app built '
'with --features automation?')
time.sleep(0.5)
def parse_name(name):
"""`LABEL`, `id:ID`, each with optional `@ROLE` and `#N`."""
m = re.match(r'^(.*?)(?:@(\w+))?(?:#(-?\d+))?$', name, re.S)
what, role, n = m.group(1), m.group(2), m.group(3)
return what, role, int(n) if n is not None else 0
def matches(name, within=None):
"""Every on-screen element `name` names, in tree order. `within` is an
optional (x0, y0, x1, y1) the element's centre must fall inside."""
what, role, _ = parse_name(name)
if what.startswith('id:'):
hits = ask(f'locate-id {what[3:]}')
else:
hits = ask(f'locate {what}')
hits = [e for e in hits if e['w'] > 0 and e['h'] > 0 and e['opacity'] > 0]
if role:
hits = [e for e in hits if (e['role'] or '').lower() == role.lower()]
if within:
x0, y0, x1, y1 = within
hits = [e for e in hits if x0 <= e['x'] + e['w'] / 2 < x1 and y0 <= e['y'] + e['h'] / 2 < y1]
return hits
def find(name, within=None):
"""The element `name` names, or NotFound."""
index = parse_name(name)[2]
hits = matches(name, within)
if not -len(hits) <= index < len(hits):
raise NotFound(f'{name!r}: {len(hits)} on screen')
return hits[index]
def rect(name, within=None):
"""(x0, y0, x1, y1) of `name`."""
e = find(name, within)
return e['x'], e['y'], e['x'] + e['w'], e['y'] + e['h']
def point(name, fx=0.5, fy=0.5, within=None):
"""A point inside `name`, as fractions of its width and height."""
e = find(name, within)
return int(e['x'] + e['w'] * float(fx)), int(e['y'] + e['h'] * float(fy))
def centre(name, within=None):
return point(name, within=within)
def present(name, within=None):
try:
find(name, within)
return True
except NotFound:
return False
def wait_for(name, timeout=30, within=None):
t0 = time.time()
while True:
try:
return find(name, within)
except (NotFound, OSError):
if time.time() - t0 > float(timeout):
raise
time.sleep(0.25)
def wait_gone(name, timeout=30):
t0 = time.time()
while time.time() - t0 < float(timeout):
if not present(name):
return
time.sleep(0.25)
raise TimeoutError(f'{name!r} still on screen after {timeout}s')
def click_on(name, button=1, within=None):
click(*centre(name, within), button)
def drag_on(name, dx, dy, steps=20, within=None, fx=0.5, fy=0.5):
"""Drag from a point in `name` (its centre by default) by (dx, dy)."""
cx, cy = point(name, fx, fy, within)
drag(cx, cy, cx + int(dx), cy + int(dy), steps)
def hold_on(name, seconds, within=None):
x('windowfocus', '--sync', win(), check=False)
move(*centre(name, within))
time.sleep(0.15)
x('mousedown', 1)
time.sleep(float(seconds))
x('mouseup', 1)
def photo(fx=0.5, fy=0.5):
"""A point on the photograph in develop, as fractions of the picture as
drawn — `contain`-fitted into the canvas, so not of the canvas itself."""
cx0, cy0, cx1, cy1 = rect('id:canvas-image')
img = ask('canvas')
cw, ch = cx1 - cx0, cy1 - cy0
if not img['w'] or not img['h']:
return int(cx0 + cw * fx), int(cy0 + ch * fy)
s = min(cw / img['w'], ch / img['h'])
w, h = img['w'] * s, img['h'] * s
return int(cx0 + (cw - w) / 2 + w * fx), int(cy0 + (ch - h) / 2 + h * fy)
def rec_start(out):
size = x('getwindowgeometry', win()).split('Geometry: ')[1].strip()
ox, oy = geometry()
@@ -172,6 +368,14 @@ def where():
print(mx - ox, my - oy)
def print_elements(elements):
for e in elements:
name = e['label'] if e['label'] else f"id:{e['id']}" if e['id'] else f"({e['type']})"
print(f"{e['x']:5.0f} {e['y']:5.0f} {e['w']:5.0f}x{e['h']:<4.0f} "
f"{(e['role'] or ''):14} {name!r}"
+ (f" = {e['value']!r}" if e['value'] else ''))
if __name__ == '__main__':
cmd, *a = sys.argv[1:]
if cmd == 'launch':
@@ -198,5 +402,21 @@ if __name__ == '__main__':
rec_stop()
elif cmd == 'where':
where()
elif cmd == 'labels':
print_elements(ask('labels ' + (a[0] if a else '')))
elif cmd == 'ids':
print_elements(ask('ids ' + (a[0] if a else '')))
elif cmd == 'locate':
print(json.dumps(matches(a[0]), indent=1))
elif cmd == 'click-on':
click_on(*a)
elif cmd == 'drag-on':
drag_on(*a)
elif cmd == 'hold-on':
hold_on(*a)
elif cmd == 'wait-for':
print(wait_for(*a))
elif cmd == 'wait-gone':
wait_gone(*a)
else:
sys.exit(__doc__)
ui/dr-ui/Cargo.toml
+10
View File
@@ -104,6 +104,10 @@ anyhow.workspace = true
thiserror.workspace = true
log.workspace = true
pollster.workspace = true
# Slint's element queries, for the `automation` feature only. Pinned to the
# exact Slint release: it reaches into i-slint-core, which is versioned in
# lockstep and has no stable API of its own.
i-slint-backend-testing = { version = "=1.17.1", default-features = false, optional = true }
# Runtime YAML only for `live-style`; release builds read the tokens the
# Slint compiler folded in at build time and never touch style.yaml.
serde_norway = { workspace = true, optional = true }
@@ -143,6 +147,12 @@ serde_norway.workspace = true
scene-model = ["dr-segment/embedded-scene-model"]
default = []
# Answer "where is the control labelled X?" over a Unix socket named by
# `DR_AUTOMATION`, for the scripts that record the manual (src/automation.rs,
# tools/manual). Off by default and never in a shipped build: it adds Slint's
# element-query crate, and a listener nobody asked for has no place in a
# release. `tools/manual/record.sh` builds with it.
automation = ["dep:i-slint-backend-testing"]
# Debug convenience: re-read style.yaml at startup so a palette can be tuned
# without rebuilding. Costs the constant-folding of every token, so it stays
# off by default and has no business in a release build.
ui/dr-ui/build.rs
+7
View File
@@ -134,6 +134,13 @@ fn main() {
config = config.with_bundled_translations(lang);
}
// The recording hook's element queries walk a tree the compiler only
// describes when asked to (src/automation.rs). Only that build asks: the
// extra tables are dead weight to everyone else.
if std::env::var_os("CARGO_FEATURE_AUTOMATION").is_some() {
config = config.with_debug_info(true);
}
slint_build::compile_with_config(entry(&out_dir, live), config).expect("compiling app.slint");
}
ui/dr-ui/src/automation.rs
+184
View File
@@ -0,0 +1,184 @@
//! Where a control is, asked by its name — for the scripts that record the
//! manual, and for nobody else.
//!
//! `tools/manual/drive.py` moves a real pointer over the window with xdotool,
//! and until this existed it aimed at pixel coordinates written into
//! `scenes.py` by hand. A panel that grew a row moved every slider below it,
//! and the recording went on clicking where the slider used to be: no error,
//! just a picture of the wrong thing. This answers "where is *Exposure*?" from
//! the element tree itself, by the same accessible label a screen reader reads
//! (NFR-A11Y-2), so a script that names controls survives a layout change the
//! way a screen-reader user does.
//!
//! **Absent from every normal build.** The module exists only under the
//! `automation` cargo feature, which no default build enables and
//! `tools/manual/record.sh` does; and even in a build that has it, nothing
//! listens unless `DR_AUTOMATION` names a socket path. It only reads the tree
//! and answers: the input itself still arrives as ordinary pointer and key
//! events from outside, so a scene exercises exactly what a person's hand
//! would, gesture recognisers and all.
//!
//! ## The protocol
//!
//! One request per line on a Unix socket, one JSON line back:
//!
//! - `ping` — `"ok"`.
//! - `locate LABEL` — every element whose accessible label is exactly
//! `LABEL`: `[{"label","role","x","y","w","h","opacity","enabled",
//! "checked","value"}, …]`, in physical window pixels, tree order.
//! - `labels [SUBSTRING]` — every labelled element, or those whose label
//! contains `SUBSTRING`; the same objects. For finding what a scene can
//! name.
//! - `locate-id ID` — the same, for elements by their id in the markup
//! (`canvas-image`, or qualified, `CropOverlay::move-area`) or by
//! component type (`Timeline`). For the few things a scene has to aim at
//! that are not controls — the photograph, the crop rectangle — and so
//! have no accessible name to give.
//! - `ids [SUBSTRING]` — every element with an id, for finding those.
//! - `window` — `{"w","h","scale"}`.
//! - `canvas` — `{"w","h"}` of the develop canvas's current image, so a
//! script can place a point on the photograph within `canvas-image`.
//!
//! Only elements Slint considers visible are answered — not hidden, not
//! clipped away by a scrolled list — so "on screen" is the default.
//!
//! Waiting is the client's job (poll `locate`), which keeps this stateless.
use std::io::{BufRead, BufReader, Write};
use std::os::unix::net::{UnixListener, UnixStream};
use std::sync::mpsc;
use std::time::Duration;
use i_slint_backend_testing::{ElementHandle, ElementRoot as _};
use serde_json::{json, Value};
use slint::ComponentHandle as _;
use crate::AppWindow;
/// Listen on `DR_AUTOMATION` if it is set. Called once, before the event loop.
pub(crate) fn attach(window: &AppWindow) {
let Some(path) = std::env::var_os("DR_AUTOMATION") else {
return;
};
let _ = std::fs::remove_file(&path);
let listener = match UnixListener::bind(&path) {
Ok(l) => l,
Err(e) => {
log::warn!("automation: cannot listen on {path:?}: {e}");
return;
}
};
log::info!("automation: listening on {path:?}");
let weak = window.as_weak();
std::thread::Builder::new()
.name("automation".into())
.spawn(move || {
for stream in listener.incoming().flatten() {
let weak = weak.clone();
std::thread::spawn(move || serve(stream, weak));
}
})
.ok();
}
fn serve(stream: UnixStream, weak: slint::Weak<AppWindow>) {
let Ok(mut out) = stream.try_clone() else {
return;
};
for line in BufReader::new(stream).lines() {
let Ok(line) = line else { return };
let (tx, rx) = mpsc::channel();
let request = line.clone();
let posted = weak.upgrade_in_event_loop(move |w| {
let _ = tx.send(answer(&w, &request));
});
let reply = match posted {
Ok(()) => rx
.recv_timeout(Duration::from_secs(10))
.unwrap_or_else(|_| json!({"error": "the event loop did not answer"})),
Err(e) => json!({"error": e.to_string()}),
};
if writeln!(out, "{reply}").is_err() {
return;
}
}
}
fn answer(window: &AppWindow, request: &str) -> Value {
let (verb, rest) = request.split_once(' ').unwrap_or((request, ""));
let scale = window.window().scale_factor();
match verb {
"ping" => json!("ok"),
"window" => {
let size = window.window().size();
json!({"w": size.width, "h": size.height, "scale": scale})
}
"locate" => Value::Array(
ElementHandle::find_by_accessible_label(window, rest)
.map(|e| describe(&e, scale))
.collect(),
),
"locate-id" => {
let (needle, qualified) = (rest.to_string(), format!("::{rest}"));
query(window, scale, move |e| {
e.id()
.is_some_and(|id| id == needle.as_str() || id.ends_with(&qualified))
|| e.type_name().is_some_and(|t| t == needle.as_str())
})
}
"ids" => {
let needle = rest.to_string();
query(window, scale, move |e| {
e.id()
.is_some_and(|id| !id.is_empty() && id.contains(needle.as_str()))
})
}
"labels" => {
let needle = rest.to_string();
query(window, scale, move |e| {
e.accessible_label()
.is_some_and(|l| !l.is_empty() && l.contains(needle.as_str()))
})
}
"canvas" => {
let size = window.get_canvas().size();
json!({"w": size.width, "h": size.height})
}
_ => json!({"error": format!("unknown request {verb:?}")}),
}
}
/// Every drawn element the predicate accepts, described, in tree order.
fn query(
window: &AppWindow,
scale: f32,
predicate: impl Fn(&ElementHandle) -> bool + 'static,
) -> Value {
window
.root_element()
.query_descendants()
.match_predicate(predicate)
.find_all()
.iter()
.map(|e| describe(e, scale))
.collect()
}
fn describe(e: &ElementHandle, scale: f32) -> Value {
let at = e.absolute_position();
let size = e.size();
json!({
"label": e.accessible_label().map(|s| s.to_string()),
"id": e.id().map(|s| s.to_string()),
"type": e.type_name().map(|s| s.to_string()),
"role": e.accessible_role().map(|r| format!("{r:?}")),
"x": (at.x * scale).round(),
"y": (at.y * scale).round(),
"w": (size.width * scale).round(),
"h": (size.height * scale).round(),
"opacity": e.computed_opacity(),
"enabled": e.accessible_enabled(),
"checked": e.accessible_checked(),
"value": e.accessible_value().map(|s| s.to_string()),
})
}
ui/dr-ui/src/lib.rs
+5
View File
@@ -20,6 +20,8 @@
//! in `ui/` names an operation or knows a shader exists (FR-DEV-3a).
mod activity;
#[cfg(all(feature = "automation", unix))]
mod automation;
mod bursts;
mod collections_ui;
#[cfg(test)]
@@ -1361,6 +1363,9 @@ pub fn run(paths: Vec<PathBuf>) -> Result<()> {
launch_began.elapsed().as_millis()
);
#[cfg(all(feature = "automation", unix))]
automation::attach(&window);
window.run()?;
Ok(())
}
ui/dr-ui/ui/adjust.slint
+6
View File
@@ -1285,6 +1285,12 @@ export component AdjustPanel inherits Rectangle {
width: 44px;
height: 20px;
clicked => { Adjustments.reset-all(); }
// Named apart from the groups' "Reset" buttons below it: this
// one empties the whole panel, and a reader moving through
// the column has to be able to tell the two apart.
accessible-role: button;
accessible-label: "Reset all adjustments";
accessible-action-default => { Adjustments.reset-all(); }
Label {
text: "reset";
emphasised: reset.has-hover;
ui/dr-ui/ui/collections.slint
+4
View File
@@ -579,6 +579,10 @@ export component CollectionsPanel inherits Rectangle {
: (add-touch.has-hover ? Theme.hover : transparent);
border-radius: Theme.radius-sm;
accessible-role: button;
accessible-label: "New collection";
accessible-action-default => { root.new-collection(); }
Icon {
name: "plus";
ink: Theme.ink-dim;
ui/dr-ui/ui/library.slint
+23
View File
@@ -1210,6 +1210,21 @@ export component StarStrip inherits Rectangle {
width: root.star;
height: root.star;
// Each star is a button that sets its rating, and says so. The
// strip is only in the tree while it is drawn — on the cell under
// the pointer, or on a rated one — which is also when it can be
// pressed.
accessible-role: button;
accessible-label: n == 1 ? "1 star" : n + " stars";
accessible-enabled: root.interactive;
accessible-checkable: true;
accessible-checked: root.rating >= n;
accessible-action-default => {
if (root.interactive) {
root.rate(root.rating == n ? 0 : n);
}
}
Icon {
// Solid versus outline: the shape says it, not the colour.
name: root.rating >= n ? "star" : "star-outline";
@@ -3730,6 +3745,14 @@ export component LibraryGrid inherits Rectangle {
// photographs file in one gesture.
// manual: collections
for cell[i] in root.cells: DragArea {
// A cell is its photograph, named by its file: the name
// printed under the thumbnail, and the one a screen
// reader or the manual's recording scripts ask for.
accessible-role: list-item;
accessible-label: cell.name;
accessible-item-selectable: true;
accessible-item-selected: cell.selected;
// Cells are positioned at their **absolute** place in the
// library, not their index in the loaded window: the window
// starts at `offset`, so a cell drawn at window-index 0 belongs