dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Find the subjects without stopping the window

Browse Source 
"Find subjects" took the UI thread for two thirds of a second on a 22 MP
frame — a proxy render, a readback and a YOLO pass through `ort` — and for
that time the interface was simply gone. The panel apologised for it rather
than hiding it: a "Looking…" label, and a 16 ms `single_shot` so the label
reached the screen before the freeze began, with a comment saying the obvious
fix needed the develop session restructured and was not being taken.

The obstacle was never `Send`. `DevelopSession` is `Send` — the device, the
source texture and the passes all are. What cannot go to a worker is the
`Rc<RefCell<Option<DevelopSession>>>` that every callback in the window
reaches through, and the window has to keep reaching through it while the work
runs. Handing the session over would freeze the interface exactly as
thoroughly as blocking on it did.

So the work takes a copy of what it needs instead. A `SegmentationJob` is the
device, the demosaiced source behind an `Arc`, and the name of the session
that asked. Taking one is two `Arc` bumps; running one is 495 ms on this
desktop; none of it touches the session, and there is deliberately no
`&mut DevelopSession` in scope for a caller to hold across it. The proxy
render travels with it rather than staying behind — a `GpuContext` and a
texture handle are both `Send`, and the model was never the only expensive
half. So does building the mask rasteriser, which is a shader compile:
adopting the result was costing 23 ms, a dropped frame on the one redraw the
user is waiting for, and the rasteriser is needed exactly when the subjects
arrive and never before. What is left on the UI thread is a microsecond.

The answer comes back through a channel a `slint::Timer` polls, which is the
shape `apply_when_ready` already uses for a sidecar fetch.

**A result can outlive the photograph it describes.** Two thirds of a second
is long enough to press the button, think better of it and swipe to the next
frame — and the result landing then would fill the panel with subjects that
are not in the picture, drawing outlines around a dog two photographs back.
Nothing downstream can tell: the masks rasterise and the overlay draws either
way. So every session is minted with an id, a job carries the id it was taken
from, and `delivery` compares the two before anything is applied. An id
rather than a counter beside the session slot, because that slot is written
from four places in `lib.rs` and the fifth would be the one that forgot.

A discard touches nothing on the way out. `segmenting` belongs to whichever
photograph is open now, which may well have a run of its own going, and
clearing it would re-enable a button that is correctly insensitive.

One run at a time, and abandonment is what stops that being a trap. A job
left over from a photograph the user has left is displaced rather than waited
for — otherwise the next frame's "Find subjects" would do nothing for the
length of a run nobody wants, which is the wait this exists to remove. `ort`
offers no way into the inference, so abandoning is checked at the seams there
are: before the job starts, and between the readback and the model. Abandoned
early it costs nothing, abandoned mid-inference it costs the run it was
already committed to, and either way the answer is dropped at the channel.

`DevelopSession::segment` survives as a test-only convenience. Left public it
is precisely the shape that put two thirds of a second on the UI thread in the
first place, and the next caller would reach for it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dtourolle
2026-08-22 12:26:11 +02:00
co-authored by Claude Opus 5
parent 586698db00
commit c0e1179936
4 changed files with 612 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ui/dr-ui/src/develop.rs
+317 -69
View File
@@ -9,6 +9,9 @@
//! declared [`ParamKind`], not from which parameter it is (ARCH §4.3), so a
//! new operation appears in the panel with no change here (FR-DEV-3c).
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;
use dr_decode::RawImage;
use dr_gpu::{
AdjustPass, DemosaicedImage, Demosaicer, GpuContext, Histogram, HistogramPass, MaskPass,
@@ -33,7 +36,185 @@ use crate::ParamRow;
/// milliseconds and its field a few megabytes.
const SEGMENT_PROXY_EDGE: u32 = 1600;
/// Which photograph a piece of background work was started for.
///
/// Minted per session, never reused, and carried by the work rather than
/// looked up when it finishes. A segmentation takes most of a second, so the
/// user can be two frames further on by the time one lands, and the answer to
/// "is this still wanted" has to be decided from what the work *was* rather
/// than from what happens to be open.
///
/// The alternative — a counter beside the session slot, bumped on every open —
/// is written from four places in `lib.rs` and would apply one photograph's
/// subjects to another the first time somebody added a fifth and forgot.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct SessionId(u64);
impl SessionId {
pub(crate) fn next() -> Self {
static NEXT: AtomicU64 = AtomicU64::new(1);
Self(NEXT.fetch_add(1, Ordering::Relaxed))
}
}
/// Says that nobody is waiting for a job's answer any more.
///
/// Not a cancellation in the sense of stopping the work: the model is one
/// opaque call of about half a second and `ort` offers no way in. This is
/// checked at the seams there are — before the job starts, and again between
/// the proxy readback and the inference — so a job abandoned while the user
/// was still paging usually costs nothing, and one abandoned mid-inference
/// costs only the run it was already committed to.
///
/// What it buys in every case is that the next photograph's segmentation is
/// the only one anybody is waiting on.
#[derive(Debug, Clone, Default)]
pub struct Abandon(Arc<AtomicBool>);
impl Abandon {
pub fn now(&self) {
self.0.store(true, Ordering::Relaxed);
}
pub fn asked(&self) -> bool {
self.0.load(Ordering::Relaxed)
}
}
/// What a finished [`SegmentationJob`] hands back.
///
/// The rasteriser travels with the subjects because it is needed the instant
/// they arrive and nowhere before. Building it is a shader compile — 23 ms on
/// a desktop, and compiling shaders is among the slowest things a mobile
/// driver does — so building it on adoption put a dropped frame on the one
/// redraw the user is waiting for. Here it is on the thread that was waiting
/// anyway.
///
/// `None` where the device has no mask rasteriser at all: the session keeps
/// the photograph and loses local adjustments, which is the same bargain the
/// histogram makes.
pub struct Segmented {
seg: Segmentation,
masks: Option<MaskPass>,
}
/// TRACES: FR-DEV-3
/// A segmentation lifted out of the session that asked for it.
///
/// [`DevelopSession`] cannot go to a worker. Not because of what it holds —
/// the device, the source texture and the passes are all `Send` — but because
/// it lives behind one `Rc<RefCell<Option<…>>>` that every callback in the
/// window reaches through, and the window has to keep reaching through it
/// while the work runs. Handing the session over would freeze the interface
/// exactly as thoroughly as blocking on it did.
///
/// So the work takes a copy of the two things it needs. The device is `Arc`s,
/// the source is shared rather than copied, and the answer comes back as plain
/// data.
pub struct SegmentationJob {
ctx: GpuContext,
source: Arc<DemosaicedImage>,
session: SessionId,
abandon: Abandon,
}
impl SegmentationJob {
/// The photograph this was started for.
pub fn session(&self) -> SessionId {
self.session
}
/// The handle that tells this job its answer is no longer wanted.
pub fn abandon(&self) -> Abandon {
self.abandon.clone()
}
/// TRACES: FR-DEV-3
/// Find the subjects. **Blocking, and roughly two thirds of a second.**
///
/// `Ok(None)` means abandoned rather than found-nothing: an image with no
/// recognisable subject in it still comes back as `Ok(Some(_))` with an
/// empty instance list, and the panel says so.
///
/// There is deliberately no `&mut DevelopSession` in scope here. That is
/// the whole point of the split — a caller cannot accidentally hold the
/// session across the half second, because it was never given one.
pub fn run(&self, options: &segmentation::Options) -> Result<Option<Segmented>, String> {
if self.abandon.asked() {
return Ok(None);
}
// Timed and logged because this is the feature's largest cost and the
// split between the two halves decides where any further work goes. A
// number from the device it actually runs on beats an estimate from
// the desktop.
let started = std::time::Instant::now();
let (rgb, rw, rh) = self.neutral_proxy(SEGMENT_PROXY_EDGE)?;
let proxied = started.elapsed();
// The one seam inside the run. Past here the model owns the thread
// until it is done.
if self.abandon.asked() {
return Ok(None);
}
let seg = segmentation::compute(&self.ctx, &rgb, rw, rh, options)?;
log::info!(
"segmented {rw}×{rh}: {} subject(s), proxy {:.0} ms, total {:.0} ms",
seg.instances().len(),
proxied.as_secs_f32() * 1000.0,
started.elapsed().as_secs_f32() * 1000.0,
);
let masks = MaskPass::new(&self.ctx)
.inspect_err(|e| log::warn!("no mask rasteriser on this device: {e}"))
.ok();
Ok(Some(Segmented { seg, masks }))
}
/// Render the *unedited* image to a CPU buffer at proxy size.
///
/// The model reads the photograph as captured, not as edited: the
/// segmentation must survive an exposure change, or every slider would
/// invalidate the masks that depend on it (docs/segmentation.md §3).
///
/// A throwaway [`AdjustPass`] with a neutral graph rather than the
/// session's own — which this could not reach from here in any case, and
/// must not: reusing it would overwrite the frame the histogram reads and
/// leave the view showing an unedited image until the next redraw.
///
/// This is `export_pixels`, which is ungated: an export is not the display
/// round-trip AC-8 forbids, and neither is this.
fn neutral_proxy(&self, max_edge: u32) -> Result<(Vec<f32>, usize, usize), String> {
let (sw, sh) = self.source.size();
let scale = (max_edge as f32 / sw.max(sh) as f32).min(1.0);
let (w, h) = (
((sw as f32 * scale) as u32).max(1),
((sh as f32 * scale) as u32).max(1),
);
let neutral = EditGraph::default_chain();
let mut pass = AdjustPass::new(&self.ctx);
pass.render(&self.source, &neutral.compose(), w, h)
.map_err(|e| format!("could not render the segmentation proxy: {e}"))?;
let (rgba, pw, ph) = pass
.export_pixels()
.map_err(|e| format!("could not read the segmentation proxy: {e}"))?;
// Straight to float RGB, dropping alpha. The values stay display-
// encoded because that is what the model was trained on — one of the
// few places in this codebase where not linearising is correct.
let rgb = rgba
.chunks_exact(4)
.flat_map(|p| [p[0] as f32 / 255.0, p[1] as f32 / 255.0, p[2] as f32 / 255.0])
.collect();
Ok((rgb, pw as usize, ph as usize))
}
}
pub struct DevelopSession {
/// This session's name, for work that outlives the frame it started on.
id: SessionId,
/// Kept so the session can build GPU resources after construction.
///
/// The distance fields behind a subject mask are made when a layer is
@@ -50,7 +231,7 @@ pub struct DevelopSession {
/// a history the *call sites* had to remember would be one press of undo
/// away from wrong every time a control is added.
history: History,
demosaiced: DemosaicedImage,
demosaiced: Arc<DemosaicedImage>,
adjust: AdjustPass,
/// TRACES: FR-DSP-7
/// Optional, because a session that cannot count its frames is still a
@@ -142,10 +323,11 @@ impl DevelopSession {
graph.set_orientation(orientation);
let history = History::new(&graph);
Self {
id: SessionId::next(),
ctx: ctx.clone(),
graph,
history,
demosaiced,
demosaiced: Arc::new(demosaiced),
adjust: AdjustPass::new(ctx),
histogram: HistogramPass::new(ctx)
.inspect_err(|e| log::warn!("no histogram on this device: {e}"))
@@ -864,84 +1046,60 @@ impl DevelopSession {
// Segmentation (S15, docs/segmentation.md)
// ----------------------------------------------------------------------
/// This session's name, carried by any work started against it.
pub fn id(&self) -> SessionId {
self.id
}
/// TRACES: FR-DEV-3
/// Compute the region map this image's local masks select from.
/// Everything a segmentation needs, so it can be run somewhere else.
///
/// **Blocking, and roughly half a second.** The caller is responsible for
/// running it off the UI thread — see the worker in `lib.rs`. It is
/// exposed as a plain blocking call rather than something async because
/// what it needs is a GPU context and a CPU core, not a runtime.
pub fn segment(&mut self, ctx: &GpuContext, options: &segmentation::Options) -> Result<(), String> {
// The model reads the photograph as captured, not as edited: the
// segmentation must survive an exposure change, or every slider would
// invalidate the masks that depend on it (docs/segmentation.md §3).
// Timed and logged, because this blocks the interface and the size of
// that stall is the feature's largest open risk. A number from the
// device it actually runs on beats an estimate from the desktop.
let started = std::time::Instant::now();
let (rgb, rw, rh) = self.neutral_proxy(ctx, SEGMENT_PROXY_EDGE)?;
let proxied = started.elapsed();
let seg = segmentation::compute(ctx, &rgb, rw, rh, options)?;
log::info!(
"segmented {rw}×{rh}: {} subject(s), proxy {:.0} ms, total {:.0} ms",
seg.instances().len(),
proxied.as_secs_f32() * 1000.0,
started.elapsed().as_secs_f32() * 1000.0,
);
if self.masks.is_none() {
self.masks = MaskPass::new(ctx)
.inspect_err(|e| log::warn!("no mask rasteriser on this device: {e}"))
.ok();
/// Taking the job is cheap — two `Arc` bumps and a texture handle — and
/// nothing about the session is borrowed past the call, which is what
/// lets the window go on drawing while the answer is being found.
pub fn segmentation_job(&self) -> SegmentationJob {
SegmentationJob {
ctx: self.ctx.clone(),
source: self.demosaiced.clone(),
session: self.id,
abandon: Abandon::default(),
}
}
/// TRACES: FR-DEV-3
/// Take on a segmentation found elsewhere.
///
/// The caller is responsible for checking that this result was computed
/// for *this* session — see [`SegmentationJob::session`]. Nothing here can
/// tell one photograph's subjects from another's, and a mismatch is
/// silent: the masks would rasterise, the overlay would draw, and the
/// outlines would simply follow a subject that is not in the picture.
pub fn adopt_segmentation(&mut self, found: Segmented) {
// Kept rather than replaced where there is one already: a second
// segmentation of the same photograph would otherwise throw away a
// working rasteriser for an identical one.
self.masks = self.masks.take().or(found.masks);
// The fields themselves are built per *layer*, on demand — there are
// none yet, and building one per detected object would transform
// several megapixels for masks the user may never make.
self.segmentation = Some(seg);
self.segmentation = Some(found.seg);
self.subjects = None;
self.subject_key = 0;
Ok(())
}
/// Render the *unedited* image to a CPU buffer at proxy size.
/// Find the subjects and take them on, blocking until both are done.
///
/// Goes through a throwaway [`AdjustPass`] with a neutral graph rather
/// than the session's own. Reusing `self.adjust` would overwrite the frame
/// the histogram reads and leave the view showing an unedited image until
/// the next redraw — a visible flicker for the sake of not allocating.
///
/// This is `export_pixels`, which is ungated: an export is not the display
/// round-trip AC-8 forbids, and neither is this.
fn neutral_proxy(
&self,
ctx: &GpuContext,
max_edge: u32,
) -> Result<(Vec<f32>, usize, usize), String> {
let (sw, sh) = self.demosaiced.size();
let scale = (max_edge as f32 / sw.max(sh) as f32).min(1.0);
let (w, h) = (
((sw as f32 * scale) as u32).max(1),
((sh as f32 * scale) as u32).max(1),
);
let neutral = EditGraph::default_chain();
let mut pass = AdjustPass::new(ctx);
pass.render(&self.demosaiced, &neutral.compose(), w, h)
.map_err(|e| format!("could not render the segmentation proxy: {e}"))?;
let (rgba, pw, ph) = pass
.export_pixels()
.map_err(|e| format!("could not read the segmentation proxy: {e}"))?;
// Straight to float RGB, dropping alpha. The values stay display-
// encoded because that is what the model was trained on — one of the
// few places in this codebase where not linearising is correct.
let rgb = rgba
.chunks_exact(4)
.flat_map(|p| [p[0] as f32 / 255.0, p[1] as f32 / 255.0, p[2] as f32 / 255.0])
.collect();
Ok((rgb, pw as usize, ph as usize))
/// Test-only, and deliberately: a session-shaped blocking call is exactly
/// the shape that put two thirds of a second on the UI thread in the first
/// place, and leaving it public would invite the next caller to reach for
/// it. A test has nothing else to be doing.
#[cfg(test)]
fn segment(&mut self, options: &segmentation::Options) -> Result<(), String> {
if let Some(found) = self.segmentation_job().run(options)? {
self.adopt_segmentation(found);
}
Ok(())
}
pub fn has_segmentation(&self) -> bool {
@@ -2005,6 +2163,96 @@ mod tests {
out
}
// ----------------------------------------------------------------------
// Segmentation off the UI thread
// ----------------------------------------------------------------------
/// The property the whole arrangement rests on.
///
/// If someone puts an `Rc`, a `Cell` or a raw pipeline handle into
/// `SegmentationJob`, this stops compiling — which is the only warning
/// there would be, since the call site in `masks_ui` would then fail with
/// a lifetime error a long way from the cause.
#[test]
fn a_job_and_its_answer_can_cross_a_thread() {
fn is_send<T: Send>() {}
is_send::<SegmentationJob>();
is_send::<Segmented>();
is_send::<Abandon>();
}
/// Two sessions over the same file are still two photographs as far as a
/// late result is concerned, because opening one twice is opening it
/// twice.
#[test]
fn every_session_has_its_own_identity() {
let Some(ctx) = headless() else { return };
let rgba: Vec<u8> = (0..16 * 16).flat_map(|_| [128, 128, 128, 255]).collect();
let open = || {
DevelopSession::open_rgb(&ctx, &rgba, 16, 16, dr_types::Orientation::NORMAL)
.expect("session")
};
let (a, b) = (open(), open());
assert_ne!(a.id(), b.id());
assert_eq!(a.id(), a.id(), "and stable within one session");
}
#[test]
fn a_job_carries_the_session_it_was_taken_from() {
let Some(ctx) = headless() else { return };
let rgba: Vec<u8> = (0..16 * 16).flat_map(|_| [128, 128, 128, 255]).collect();
let session =
DevelopSession::open_rgb(&ctx, &rgba, 16, 16, dr_types::Orientation::NORMAL)
.expect("session");
assert_eq!(session.segmentation_job().session(), session.id());
}
/// Abandoning before the run reaches the proxy must cost nothing at all —
/// this is the case that fires when the user pages on while a job is still
/// waiting for a thread.
#[test]
fn an_abandoned_job_does_no_work() {
let Some(ctx) = headless() else { return };
let rgba: Vec<u8> = (0..16 * 16).flat_map(|_| [128, 128, 128, 255]).collect();
let session =
DevelopSession::open_rgb(&ctx, &rgba, 16, 16, dr_types::Orientation::NORMAL)
.expect("session");
let job = session.segmentation_job();
job.abandon().now();
let started = std::time::Instant::now();
let out = job.run(&crate::segmentation::Options::default());
assert!(
matches!(out, Ok(None)),
"abandoned is not an error and not an empty answer: {:?}",
out.map(|o| o.is_some())
);
assert!(
started.elapsed() < std::time::Duration::from_millis(50),
"it returned without loading the model"
);
}
/// A finished segmentation is adopted whole, and the session says so.
#[test]
fn adopting_a_result_gives_the_session_its_subjects() {
let Some(ctx) = headless() else { return };
let rgba: Vec<u8> = (0..100 * 100).flat_map(|_| [128, 128, 128, 255]).collect();
let mut session =
DevelopSession::open_rgb(&ctx, &rgba, 100, 100, dr_types::Orientation::NORMAL)
.expect("session");
assert!(!session.has_segmentation());
let job = session.segmentation_job();
let Ok(Some(found)) = job.run(&crate::segmentation::Options::default()) else {
eprintln!("no model; skipping");
return;
};
session.adopt_segmentation(found);
assert!(session.has_segmentation());
}
// ----------------------------------------------------------------------
// The overlay's clip rectangle
// ----------------------------------------------------------------------
@@ -2026,7 +2274,7 @@ mod tests {
DevelopSession::open_rgb(ctx, &rgba, 100, 100, dr_types::Orientation::NORMAL)
.expect("session");
session
.segment(ctx, &crate::segmentation::Options::default())
.segment(&crate::segmentation::Options::default())
.ok()?;
Some(session)
}