dtourolle
050c2c9d16
develop.rs had grown to 9,327 lines covering everything the develop session does: opening a photograph, the parameter-row and curve-widget panel model, mask viewing and editing, mask creation and the rasteriser that turns a mask stack into GPU arrays, spot repairs, scene segmentation, framing and zoom, white-balance sampling, rendering and film choice, and the undo/snapshot history. docs/dev/code-health.md CH-1 names dr-ui's lack of a view layer as the reason every feature kept landing in a handful of files; this is the first of the two pure splits it recommends as easy, no-behaviour-change wins independent of that larger rework. The boundaries follow the file's own sections (several were already marked off with comment headers) and the seams a full read turned up underneath them -- mask storage/rasterisation turned out to be a distinct concern from mask viewing and editing, and rows/tabs/curves from each other, so those split further than the headers alone suggested. Each module stays under about 1,500 lines. Struct fields and the handful of helper methods now called from a sibling module became `pub(super)`, which is strictly narrower than the whole-crate reachability a single file gave them; nothing gained visibility outside `develop`. Tests moved with the code they test, including the few cases where a helper one file's tests needed was itself only defined in another's -- those became shared fixtures in `mod.rs` alongside the `headless`/`read_back`/`grey_session` helpers that already worked that way. `mod.rs` re-exports every item `develop::` callers outside this module used before, so lib.rs, masks_ui.rs and the rest needed no changes.
835 lines
34 KiB
Rust
835 lines
34 KiB
Rust
//! Running the scene segmentation model off the UI thread, and the session
|
||
//! state that adopts what it finds (S15, docs/dev/segmentation.md).
|
||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||
use std::sync::Arc;
|
||
|
||
use dr_gpu::{AdjustPass, DemosaicedImage, GpuContext, MaskPass};
|
||
use dr_pipeline::mask::MaskSource;
|
||
use dr_pipeline::{CropRect, EditGraph};
|
||
|
||
use crate::segmentation::{self, Segmentation};
|
||
|
||
use super::session::DevelopSession;
|
||
use super::SEGMENT_PROXY_EDGE;
|
||
|
||
/// Which photograph a piece of background work was started for.
|
||
///
|
||
/// Minted per session, never reused, and carried by the work rather than
|
||
/// looked up when it finishes. A segmentation takes most of a second, so the
|
||
/// user can be two frames further on by the time one lands, and the answer to
|
||
/// "is this still wanted" has to be decided from what the work *was* rather
|
||
/// than from what happens to be open.
|
||
///
|
||
/// The alternative — a counter beside the session slot, bumped on every open —
|
||
/// is written from four places in `lib.rs` and would apply one photograph's
|
||
/// subjects to another the first time somebody added a fifth and forgot.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
pub struct SessionId(u64);
|
||
|
||
impl SessionId {
|
||
pub(crate) fn next() -> Self {
|
||
static NEXT: AtomicU64 = AtomicU64::new(1);
|
||
Self(NEXT.fetch_add(1, Ordering::Relaxed))
|
||
}
|
||
}
|
||
|
||
/// Says that nobody is waiting for a job's answer any more.
|
||
///
|
||
/// Not a cancellation in the sense of stopping the work: the model is one
|
||
/// opaque call of about half a second and `ort` offers no way in. This is
|
||
/// checked at the seams there are — before the job starts, and again between
|
||
/// the proxy readback and the inference — so a job abandoned while the user
|
||
/// was still paging usually costs nothing, and one abandoned mid-inference
|
||
/// costs only the run it was already committed to.
|
||
///
|
||
/// What it buys in every case is that the next photograph's segmentation is
|
||
/// the only one anybody is waiting on.
|
||
#[derive(Debug, Clone, Default)]
|
||
pub struct Abandon(Arc<AtomicBool>);
|
||
|
||
impl Abandon {
|
||
pub fn now(&self) {
|
||
self.0.store(true, Ordering::Relaxed);
|
||
}
|
||
|
||
pub fn asked(&self) -> bool {
|
||
self.0.load(Ordering::Relaxed)
|
||
}
|
||
}
|
||
|
||
/// What a finished [`SegmentationJob`] hands back.
|
||
///
|
||
/// The rasteriser travels with the subjects because it is needed the instant
|
||
/// they arrive and nowhere before. Building it is a shader compile — 23 ms on
|
||
/// a desktop, and compiling shaders is among the slowest things a mobile
|
||
/// driver does — so building it on adoption put a dropped frame on the one
|
||
/// redraw the user is waiting for. Here it is on the thread that was waiting
|
||
/// anyway.
|
||
///
|
||
/// `None` where the device has no mask rasteriser at all: the session keeps
|
||
/// the photograph and loses local adjustments, which is the same bargain the
|
||
/// histogram makes.
|
||
pub struct Segmented {
|
||
seg: Segmentation,
|
||
masks: Option<MaskPass>,
|
||
}
|
||
|
||
/// TRACES: FR-DEV-3
|
||
/// A segmentation lifted out of the session that asked for it.
|
||
///
|
||
/// [`DevelopSession`] cannot go to a worker. Not because of what it holds —
|
||
/// the device, the source texture and the passes are all `Send` — but because
|
||
/// it lives behind one `Rc<RefCell<Option<…>>>` that every callback in the
|
||
/// window reaches through, and the window has to keep reaching through it
|
||
/// while the work runs. Handing the session over would freeze the interface
|
||
/// exactly as thoroughly as blocking on it did.
|
||
///
|
||
/// So the work takes a copy of the two things it needs. The device is `Arc`s,
|
||
/// the source is shared rather than copied, and the answer comes back as plain
|
||
/// data.
|
||
pub struct SegmentationJob {
|
||
ctx: GpuContext,
|
||
source: Arc<DemosaicedImage>,
|
||
session: SessionId,
|
||
abandon: Abandon,
|
||
/// TRACES: FR-CULL-10
|
||
/// Confirmed faces in this photograph, **normalised to the long edge** of
|
||
/// the EXIF-upright image.
|
||
///
|
||
/// Carried rather than looked up, because the job runs on a thread with no
|
||
/// catalog in reach — the same reason it carries the pixels. Normalised
|
||
/// rather than in pixels because the proxy size is only settled inside
|
||
/// `run`.
|
||
names: Vec<crate::identity::NormalisedNamedBox>,
|
||
/// TRACES: FR-CULL-10
|
||
/// The file's EXIF turn alone, which is the space `names` is expressed in.
|
||
///
|
||
/// **Deliberately not [`SegmentationJob::orientation`].** Faces are found
|
||
/// on the thumbnail, which is stood up by the EXIF tag and knows nothing
|
||
/// about the photographer's later turns; the segmentation below composes
|
||
/// both. Using the composed one here would turn the faces twice on any
|
||
/// photograph the user has rotated, and the failure would be silent —
|
||
/// names simply landing on nobody.
|
||
exif_orientation: dr_types::Orientation,
|
||
/// TRACES: FR-DEV-3h
|
||
/// How the sensor's pixels have to be turned to be the photograph.
|
||
///
|
||
/// The file's EXIF tag and the photographer's own turns, composed into
|
||
/// one permutation by `Framing::effective_orientation`. Carried rather
|
||
/// than read from the session for the reason everything else here is: the
|
||
/// job runs on a worker and the session stays behind.
|
||
///
|
||
/// A *snapshot*, so turning the photograph while a run is in flight
|
||
/// leaves that run answering the question it was asked. The next press
|
||
/// takes the new one, and the signature says the two are different runs.
|
||
orientation: dr_types::Orientation,
|
||
}
|
||
|
||
impl SegmentationJob {
|
||
/// The photograph this was started for.
|
||
pub fn session(&self) -> SessionId {
|
||
self.session
|
||
}
|
||
|
||
/// The handle that tells this job its answer is no longer wanted.
|
||
pub fn abandon(&self) -> Abandon {
|
||
self.abandon.clone()
|
||
}
|
||
|
||
/// TRACES: FR-DEV-3
|
||
/// Find the subjects. **Blocking, and roughly two thirds of a second.**
|
||
///
|
||
/// `Ok(None)` means abandoned rather than found-nothing: an image with no
|
||
/// recognisable subject in it still comes back as `Ok(Some(_))` with an
|
||
/// empty instance list, and the panel says so.
|
||
///
|
||
/// There is deliberately no `&mut DevelopSession` in scope here. That is
|
||
/// the whole point of the split — a caller cannot accidentally hold the
|
||
/// session across the half second, because it was never given one.
|
||
pub fn run(&self, options: &segmentation::Options) -> Result<Option<Segmented>, String> {
|
||
if self.abandon.asked() {
|
||
return Ok(None);
|
||
}
|
||
|
||
// Timed and logged because this is the feature's largest cost and the
|
||
// split between the two halves decides where any further work goes. A
|
||
// number from the device it actually runs on beats an estimate from
|
||
// the desktop.
|
||
let started = std::time::Instant::now();
|
||
let (rgb, rw, rh) = self.neutral_proxy(SEGMENT_PROXY_EDGE)?;
|
||
let proxied = started.elapsed();
|
||
|
||
// The one seam inside the run. Past here the model owns the thread
|
||
// until it is done.
|
||
if self.abandon.asked() {
|
||
return Ok(None);
|
||
}
|
||
|
||
let mut seg = segmentation::compute(&self.ctx, &rgb, rw, rh, self.orientation, options)?;
|
||
|
||
// TRACES: FR-CULL-10
|
||
// Put names on the people the segmenter found.
|
||
//
|
||
// `compute` stands the frame up to detect and lays the instances back
|
||
// down, so `bbox` is in the sensor's space. The faces came off the
|
||
// thumbnail and are in the EXIF-upright one. Two different spaces, and
|
||
// on a portrait photograph they are a quarter turn apart — so the
|
||
// faces are turned down to meet the instances, through the same
|
||
// `Orientation` map every other consumer uses rather than a second
|
||
// copy of the arithmetic.
|
||
//
|
||
// The catalog normalises a face to the image's **long edge**, where
|
||
// `ShownRect` is normalised per axis; the conversions either side of
|
||
// the turn are that difference and nothing more.
|
||
if !self.names.is_empty() {
|
||
let long_edge = rw.max(rh) as f32;
|
||
let (dw, dh) = self.exif_orientation.oriented_size(rw as u32, rh as u32);
|
||
let (dw, dh) = (dw as f32, dh as f32);
|
||
|
||
let boxes: Vec<dr_face::NamedFace<'_>> = self
|
||
.names
|
||
.iter()
|
||
.map(|(x, y, w, h, name)| {
|
||
let shown = dr_types::ShownRect {
|
||
x: x * long_edge / dw,
|
||
y: y * long_edge / dh,
|
||
width: w * long_edge / dw,
|
||
height: h * long_edge / dh,
|
||
};
|
||
let stored = self.exif_orientation.into_stored_rect(shown);
|
||
dr_face::NamedFace {
|
||
bbox: (
|
||
stored.x * rw as f32,
|
||
stored.y * rh as f32,
|
||
(stored.x + stored.width) * rw as f32,
|
||
(stored.y + stored.height) * rh as f32,
|
||
),
|
||
name,
|
||
}
|
||
})
|
||
.collect();
|
||
|
||
let named = seg.apply_names(&boxes);
|
||
if named > 0 {
|
||
log::info!("named {named} segmented region(s) from known faces");
|
||
}
|
||
}
|
||
|
||
log::info!(
|
||
"segmented {rw}×{rh}: {} subject(s), proxy {:.0} ms, total {:.0} ms",
|
||
seg.instances().len(),
|
||
proxied.as_secs_f32() * 1000.0,
|
||
started.elapsed().as_secs_f32() * 1000.0,
|
||
);
|
||
|
||
let masks = MaskPass::new(&self.ctx)
|
||
.inspect_err(|e| log::warn!("no mask rasteriser on this device: {e}"))
|
||
.ok();
|
||
Ok(Some(Segmented { seg, masks }))
|
||
}
|
||
|
||
/// Render the *unedited* image to a CPU buffer at proxy size.
|
||
///
|
||
/// The model reads the photograph as captured, not as edited: the
|
||
/// segmentation must survive an exposure change, or every slider would
|
||
/// invalidate the masks that depend on it (docs/dev/segmentation.md §3).
|
||
///
|
||
/// **Still the sensor's orientation, deliberately.** Standing the picture
|
||
/// up is what `segmentation::compute` does to the buffer this returns,
|
||
/// and it is done there rather than here because a mask has to come back
|
||
/// in this space: the generated shader samples the mask array at `uv_src`,
|
||
/// after the framing map. Rendering an upright proxy would put every mask
|
||
/// a quarter turn away from the subject it was drawn around — a wrong
|
||
/// mask rather than a weak one, and nothing would announce it.
|
||
///
|
||
/// A throwaway [`AdjustPass`] with a neutral graph rather than the
|
||
/// session's own — which this could not reach from here in any case, and
|
||
/// must not: reusing it would overwrite the frame the histogram reads and
|
||
/// leave the view showing an unedited image until the next redraw.
|
||
///
|
||
/// This is `export_pixels`, which is ungated: an export is not the display
|
||
/// round-trip AC-8 forbids, and neither is this.
|
||
fn neutral_proxy(&self, max_edge: u32) -> Result<(Vec<f32>, usize, usize), String> {
|
||
let (sw, sh) = self.source.size();
|
||
let scale = (max_edge as f32 / sw.max(sh) as f32).min(1.0);
|
||
let (w, h) = (
|
||
((sw as f32 * scale) as u32).max(1),
|
||
((sh as f32 * scale) as u32).max(1),
|
||
);
|
||
|
||
let neutral = EditGraph::default_chain();
|
||
let mut pass = AdjustPass::new(&self.ctx);
|
||
pass.render(&self.source, &neutral.compose(), w, h)
|
||
.map_err(|e| format!("could not render the segmentation proxy: {e}"))?;
|
||
let (rgba, pw, ph) = pass
|
||
.export_pixels()
|
||
.map_err(|e| format!("could not read the segmentation proxy: {e}"))?;
|
||
|
||
// Straight to float RGB, dropping alpha. The values stay display-
|
||
// encoded because that is what the model was trained on — one of the
|
||
// few places in this codebase where not linearising is correct.
|
||
let rgb = rgba
|
||
.chunks_exact(4)
|
||
.flat_map(|p| {
|
||
[
|
||
p[0] as f32 / 255.0,
|
||
p[1] as f32 / 255.0,
|
||
p[2] as f32 / 255.0,
|
||
]
|
||
})
|
||
.collect();
|
||
Ok((rgb, pw as usize, ph as usize))
|
||
}
|
||
}
|
||
|
||
/// Longest edge the refine crop is rendered at.
|
||
///
|
||
/// Matches [`dr_segment::semantic::INPUT_EDGE`] rather than exceeding it: the
|
||
/// model's own input is still fixed at 640x640, so rendering the crop larger
|
||
/// only gets downsampled again inside the model's letterbox. The resolution
|
||
/// win is entirely from *what fills the window* — a padded crop around one
|
||
/// subject rather than the whole frame — not from feeding the model more
|
||
/// pixels than it has ever read.
|
||
const REFINE_EDGE: u32 = 640;
|
||
|
||
/// How much of the box's own size is added on each side before cropping.
|
||
///
|
||
/// Context for the model to place the subject's edge against, and slack for
|
||
/// a box that under-ran the subject slightly on the first pass. Not so much
|
||
/// that a second subject standing nearby gets pulled into the same window.
|
||
const REFINE_PADDING: f32 = 0.25;
|
||
|
||
/// TRACES: FR-DEV-3
|
||
/// A refine pass lifted out of the session that asked for it.
|
||
///
|
||
/// The same split as [`SegmentationJob`], for the same reason — see its
|
||
/// docs. This one answers for a single already-detected instance rather than
|
||
/// the whole frame: it re-runs the model over a padded crop of just that
|
||
/// subject's box, so the subject reaches the model at its own size instead
|
||
/// of squeezed into the model's fixed window alongside everything else in
|
||
/// the photograph.
|
||
pub struct RefineJob {
|
||
ctx: GpuContext,
|
||
source: Arc<DemosaicedImage>,
|
||
session: SessionId,
|
||
abandon: Abandon,
|
||
/// Which instance this answers for. A snapshot of what it needs from the
|
||
/// segmentation, taken when the job was built — the same reason
|
||
/// `SegmentationJob` carries a proxy render rather than the session.
|
||
index: usize,
|
||
class_name: Arc<str>,
|
||
bbox: (f32, f32, f32, f32),
|
||
proxy: (usize, usize),
|
||
/// The same permutation, for the same reason — see
|
||
/// [`SegmentationJob::orientation`]. A refine pass that read the crop
|
||
/// sideways would hand back a worse mask than the one it was asked to
|
||
/// improve, on the subject the photographer had just pointed at.
|
||
orientation: dr_types::Orientation,
|
||
}
|
||
|
||
impl RefineJob {
|
||
pub fn session(&self) -> SessionId {
|
||
self.session
|
||
}
|
||
|
||
pub fn abandon(&self) -> Abandon {
|
||
self.abandon.clone()
|
||
}
|
||
|
||
/// TRACES: FR-DEV-3
|
||
/// Re-detect this one subject at higher effective resolution. **Blocking.**
|
||
///
|
||
/// `Ok(None)` covers both "abandoned" and "the model found nothing of the
|
||
/// same class in the crop" — a photographer pressing refine on a subject
|
||
/// that the padded window no longer contains is a possible outcome, not
|
||
/// a bug, and the caller treats it as "kept what was there" either way.
|
||
pub fn run(&self) -> Result<Option<RefinedInstance>, String> {
|
||
if self.abandon.asked() {
|
||
return Ok(None);
|
||
}
|
||
|
||
let (px0, py0, px1, py1) = self.bbox;
|
||
let (pw, ph) = (self.proxy.0 as f32, self.proxy.1 as f32);
|
||
if pw <= 0.0 || ph <= 0.0 {
|
||
return Ok(None);
|
||
}
|
||
|
||
let (bw, bh) = ((px1 - px0).max(1.0), (py1 - py0).max(1.0));
|
||
let (padx, pady) = (bw * REFINE_PADDING, bh * REFINE_PADDING);
|
||
let x0 = (px0 - padx).max(0.0);
|
||
let y0 = (py0 - pady).max(0.0);
|
||
let x1 = (px1 + padx).min(pw);
|
||
let y1 = (py1 + pady).min(ph);
|
||
if x1 <= x0 || y1 <= y0 {
|
||
return Ok(None);
|
||
}
|
||
let (cw_px, ch_px) = (x1 - x0, y1 - y0);
|
||
|
||
let view = CropRect {
|
||
x: x0 / pw,
|
||
y: y0 / ph,
|
||
width: cw_px / pw,
|
||
height: ch_px / ph,
|
||
};
|
||
|
||
// Aspect-matched render target, capped against blowing a tiny box up
|
||
// absurdly far past what the source ever had to offer.
|
||
let scale = (REFINE_EDGE as f32 / cw_px.max(ch_px)).min(4.0);
|
||
let (tw, th) = (
|
||
(cw_px * scale).round().max(1.0) as u32,
|
||
(ch_px * scale).round().max(1.0) as u32,
|
||
);
|
||
|
||
if self.abandon.asked() {
|
||
return Ok(None);
|
||
}
|
||
let (rgb, rw, rh) = self.render_view(view, tw, th)?;
|
||
if self.abandon.asked() {
|
||
return Ok(None);
|
||
}
|
||
|
||
// Stood up before the model reads it and laid back down after, the
|
||
// same way the whole-frame pass does it — `segmentation::upright` is
|
||
// the one place that permutation is written. A crop rendered in
|
||
// sensor space is exactly as sideways as the frame it came from.
|
||
let (upright, uw, uh) = segmentation::upright(&rgb, rw, rh, self.orientation);
|
||
|
||
let mut model = dr_segment::SemanticModel::embedded().map_err(|e| e.to_string())?;
|
||
let options = dr_segment::SemanticOptions {
|
||
tiling: dr_segment::Tiling::Whole,
|
||
..dr_segment::SemanticOptions::default()
|
||
};
|
||
let found = model
|
||
.detect(&upright, uw, uh, &options)
|
||
.map_err(|e| e.to_string())?;
|
||
|
||
// The crop was built around one subject, so the right answer among
|
||
// whatever the model found in it is the same class closest to the
|
||
// window's centre — not merely the highest score, which a second,
|
||
// unrelated instance caught in the padding could win.
|
||
//
|
||
// Measured in the upright frame, which is where the model's boxes are.
|
||
// The centre is the centre either way; the distances are not, once the
|
||
// window is not square.
|
||
let (cx, cy) = (uw as f32 * 0.5, uh as f32 * 0.5);
|
||
let best = found
|
||
.into_iter()
|
||
.filter(|i| i.class_name.as_ref() == self.class_name.as_ref())
|
||
.min_by(|a, b| centre_distance(a, cx, cy).total_cmp(¢re_distance(b, cx, cy)));
|
||
|
||
let Some(instance) = best else {
|
||
return Ok(None);
|
||
};
|
||
|
||
// Back into the crop's own sensor-space pixels, so everything below
|
||
// this line measures in the space `self.bbox` and `self.proxy` are in.
|
||
let (crop_mask, crop_bbox) =
|
||
segmentation::lay_down(&instance.mask, instance.bbox, uw, uh, self.orientation);
|
||
|
||
// Downsampled back onto the shared proxy grid like every other
|
||
// instance's mask is, but built from a sharper source than the
|
||
// whole-frame pass ever saw for this subject.
|
||
let mask = paste_into_proxy(&crop_mask, rw, rh, self.proxy, (x0, y0), (cw_px, ch_px));
|
||
let bbox = (
|
||
x0 + crop_bbox.0 / scale,
|
||
y0 + crop_bbox.1 / scale,
|
||
x0 + crop_bbox.2 / scale,
|
||
y0 + crop_bbox.3 / scale,
|
||
);
|
||
|
||
Ok(Some(RefinedInstance {
|
||
index: self.index,
|
||
summary: segmentation::InstanceSummary {
|
||
class_name: instance.class_name,
|
||
score: instance.score,
|
||
mask,
|
||
bbox,
|
||
},
|
||
}))
|
||
}
|
||
|
||
/// Render the *unedited* image, showing only `view`, at `(width, height)`.
|
||
///
|
||
/// The same neutral, as-captured render [`SegmentationJob::neutral_proxy`]
|
||
/// uses — a refine pass must read the same kind of pixels the first pass
|
||
/// did, or a subject would gain or lose an edge depending on which pass
|
||
/// found it. `view` is framing's ephemeral viewport (`Framing::set_view`),
|
||
/// the mechanism the on-screen zoom already uses to render a region at
|
||
/// more than proxy resolution — not the crop tool's own persisted
|
||
/// rectangle, and nothing here touches that.
|
||
fn render_view(
|
||
&self,
|
||
view: CropRect,
|
||
width: u32,
|
||
height: u32,
|
||
) -> Result<(Vec<f32>, usize, usize), String> {
|
||
let mut neutral = EditGraph::default_chain();
|
||
neutral.framing_mut().set_view(view);
|
||
|
||
let mut pass = AdjustPass::new(&self.ctx);
|
||
pass.render(&self.source, &neutral.compose(), width, height)
|
||
.map_err(|e| format!("could not render the refine crop: {e}"))?;
|
||
let (rgba, pw, ph) = pass
|
||
.export_pixels()
|
||
.map_err(|e| format!("could not read the refine crop: {e}"))?;
|
||
|
||
let rgb = rgba
|
||
.chunks_exact(4)
|
||
.flat_map(|p| {
|
||
[
|
||
p[0] as f32 / 255.0,
|
||
p[1] as f32 / 255.0,
|
||
p[2] as f32 / 255.0,
|
||
]
|
||
})
|
||
.collect();
|
||
Ok((rgb, pw as usize, ph as usize))
|
||
}
|
||
}
|
||
|
||
/// What a finished [`RefineJob`] hands back: a replacement for one instance
|
||
/// in the segmentation it was run against.
|
||
pub struct RefinedInstance {
|
||
index: usize,
|
||
summary: segmentation::InstanceSummary,
|
||
}
|
||
|
||
fn centre_distance(instance: &dr_segment::Instance, cx: f32, cy: f32) -> f32 {
|
||
let (x0, y0, x1, y1) = instance.bbox;
|
||
let (ix, iy) = ((x0 + x1) * 0.5, (y0 + y1) * 0.5);
|
||
((ix - cx).powi(2) + (iy - cy).powi(2)).sqrt()
|
||
}
|
||
|
||
/// Sample a crop-local mask back onto its footprint in the shared proxy grid.
|
||
///
|
||
/// Bilinear, the same as every other resampling in this mask pipeline
|
||
/// (`dr_segment::semantic`'s own prototype sampling, the letterbox that feeds
|
||
/// it) — correct whether the crop was rendered denser than the proxy (the
|
||
/// common case this feature exists for) or coarser than it (a subject large
|
||
/// enough that refining it buys little, which still renders a sensible if
|
||
/// unremarkable answer rather than a distorted one).
|
||
fn paste_into_proxy(
|
||
crop_mask: &[f32],
|
||
crop_w: usize,
|
||
crop_h: usize,
|
||
proxy: (usize, usize),
|
||
origin_px: (f32, f32),
|
||
extent_px: (f32, f32),
|
||
) -> Vec<u8> {
|
||
let (pw, ph) = proxy;
|
||
let mut out = vec![0u8; pw * ph];
|
||
let (ew, eh) = extent_px;
|
||
if crop_w == 0 || crop_h == 0 || pw == 0 || ph == 0 || ew <= 0.0 || eh <= 0.0 {
|
||
return out;
|
||
}
|
||
|
||
let (ox, oy) = origin_px;
|
||
let px0 = ox.floor().max(0.0) as usize;
|
||
let py0 = oy.floor().max(0.0) as usize;
|
||
let px1 = ((ox + ew).ceil() as usize).min(pw);
|
||
let py1 = ((oy + eh).ceil() as usize).min(ph);
|
||
|
||
for py in py0..py1 {
|
||
let gy = (py as f32 + 0.5 - oy) / eh * crop_h as f32;
|
||
if gy < 0.0 || gy >= crop_h as f32 {
|
||
continue;
|
||
}
|
||
for px in px0..px1 {
|
||
let gx = (px as f32 + 0.5 - ox) / ew * crop_w as f32;
|
||
if gx < 0.0 || gx >= crop_w as f32 {
|
||
continue;
|
||
}
|
||
let v = bilinear_sample(crop_mask, crop_w, crop_h, gx, gy);
|
||
out[py * pw + px] = (v.clamp(0.0, 1.0) * 255.0).round() as u8;
|
||
}
|
||
}
|
||
out
|
||
}
|
||
|
||
fn bilinear_sample(mask: &[f32], w: usize, h: usize, x: f32, y: f32) -> f32 {
|
||
let (fx0, fy0) = (x.floor(), y.floor());
|
||
let (fx, fy) = (x - fx0, y - fy0);
|
||
let x0 = (fx0 as isize).clamp(0, w as isize - 1) as usize;
|
||
let y0 = (fy0 as isize).clamp(0, h as isize - 1) as usize;
|
||
let x1 = (x0 + 1).min(w - 1);
|
||
let y1 = (y0 + 1).min(h - 1);
|
||
let at = |x: usize, y: usize| mask[y * w + x];
|
||
let top = at(x0, y0) * (1.0 - fx) + at(x1, y0) * fx;
|
||
let bot = at(x0, y1) * (1.0 - fx) + at(x1, y1) * fx;
|
||
top * (1.0 - fy) + bot * fy
|
||
}
|
||
|
||
impl DevelopSession {
|
||
// ----------------------------------------------------------------------
|
||
// Segmentation (S15, docs/dev/segmentation.md)
|
||
// ----------------------------------------------------------------------
|
||
|
||
/// This session's name, carried by any work started against it.
|
||
pub fn id(&self) -> SessionId {
|
||
self.id
|
||
}
|
||
|
||
/// TRACES: FR-DEV-3
|
||
/// Everything a segmentation needs, so it can be run somewhere else.
|
||
///
|
||
/// Taking the job is cheap — two `Arc` bumps and a texture handle — and
|
||
/// nothing about the session is borrowed past the call, which is what
|
||
/// lets the window go on drawing while the answer is being found.
|
||
pub fn segmentation_job(&self) -> SegmentationJob {
|
||
SegmentationJob {
|
||
ctx: self.ctx.clone(),
|
||
source: self.demosaiced.clone(),
|
||
session: self.id,
|
||
abandon: Abandon::default(),
|
||
names: self.face_names.clone(),
|
||
exif_orientation: self.orientation,
|
||
orientation: self.graph.framing().effective_orientation(),
|
||
}
|
||
}
|
||
|
||
/// TRACES: FR-CULL-10 | FR-DEV-3
|
||
/// The confirmed faces in this photograph, for naming segmented regions.
|
||
///
|
||
/// Set once when the image opens, because that is the only moment the
|
||
/// catalog and the image id are both in reach — the develop session
|
||
/// deliberately knows nothing about either, and every segmentation run
|
||
/// after this point picks the names up for free.
|
||
///
|
||
/// Boxes are normalised to the long edge, as the catalog stores them, so
|
||
/// they survive whatever proxy size a run settles on.
|
||
pub fn set_face_names(&mut self, names: Vec<crate::identity::NormalisedNamedBox>) {
|
||
self.face_names = names;
|
||
}
|
||
|
||
/// TRACES: FR-DEV-3
|
||
/// Take on a segmentation found elsewhere.
|
||
///
|
||
/// The caller is responsible for checking that this result was computed
|
||
/// for *this* session — see [`SegmentationJob::session`]. Nothing here can
|
||
/// tell one photograph's subjects from another's, and a mismatch is
|
||
/// silent: the masks would rasterise, the overlay would draw, and the
|
||
/// outlines would simply follow a subject that is not in the picture.
|
||
pub fn adopt_segmentation(&mut self, found: Segmented) {
|
||
// Kept rather than replaced where there is one already: a second
|
||
// segmentation of the same photograph would otherwise throw away a
|
||
// working rasteriser for an identical one.
|
||
self.masks = self.masks.take().or(found.masks);
|
||
|
||
// The fields themselves are built per *layer*, on demand — there are
|
||
// none yet, and building one per detected object would transform
|
||
// several megapixels for masks the user may never make.
|
||
self.segmentation = Some(found.seg);
|
||
self.subjects = None;
|
||
self.subject_key = 0;
|
||
}
|
||
|
||
/// TRACES: FR-DEV-3
|
||
/// Everything a refine pass needs for one already-detected instance, so
|
||
/// it can be run somewhere else — see [`Self::segmentation_job`] for why
|
||
/// the split exists.
|
||
///
|
||
/// `None` where there is nothing to refine: no segmentation yet, or an
|
||
/// index the panel offered a button for a moment ago but the segmentation
|
||
/// underneath has since changed.
|
||
pub fn refine_job(&self, index: usize) -> Option<RefineJob> {
|
||
let seg = self.segmentation.as_ref()?;
|
||
let instance = seg.instances().get(index)?;
|
||
Some(RefineJob {
|
||
ctx: self.ctx.clone(),
|
||
source: self.demosaiced.clone(),
|
||
session: self.id,
|
||
abandon: Abandon::default(),
|
||
index,
|
||
class_name: instance.class_name.clone(),
|
||
bbox: instance.bbox,
|
||
proxy: seg.proxy_size(),
|
||
orientation: self.graph.framing().effective_orientation(),
|
||
})
|
||
}
|
||
|
||
/// Take on a refined instance found elsewhere.
|
||
///
|
||
/// Same caution as [`Self::adopt_segmentation`]: the caller checks the
|
||
/// job's session before handing back its answer, not this. Every mask
|
||
/// layer pointing at this instance's index reads it fresh next redraw —
|
||
/// `subject_key` is reset because the pixels changed under an index
|
||
/// `subject_signature` has no way to know changed, unlike a morphology
|
||
/// slider it does track.
|
||
pub fn adopt_refined(&mut self, refined: RefinedInstance) {
|
||
if let Some(seg) = self.segmentation.as_mut() {
|
||
seg.replace_instance(refined.index, refined.summary);
|
||
}
|
||
self.subjects = None;
|
||
self.subject_key = 0;
|
||
}
|
||
|
||
/// The mask layer's original detection index, for the refine button —
|
||
/// `None` for a layer that is not a subject at all (a gradient has no
|
||
/// instance to re-detect).
|
||
pub fn subject_instance_index(&self, id: &str) -> Option<usize> {
|
||
match &self.graph.masks().get(id)?.base().source {
|
||
MaskSource::Subject { index, .. } => Some(*index as usize),
|
||
_ => None,
|
||
}
|
||
}
|
||
|
||
/// Find the subjects and take them on, blocking until both are done.
|
||
///
|
||
/// Test-only, and deliberately: a session-shaped blocking call is exactly
|
||
/// the shape that put two thirds of a second on the UI thread in the first
|
||
/// place, and leaving it public would invite the next caller to reach for
|
||
/// it. A test has nothing else to be doing.
|
||
#[cfg(test)]
|
||
pub(super) fn segment(&mut self, options: &segmentation::Options) -> Result<(), String> {
|
||
if let Some(found) = self.segmentation_job().run(options)? {
|
||
self.adopt_segmentation(found);
|
||
}
|
||
Ok(())
|
||
}
|
||
|
||
pub fn has_segmentation(&self) -> bool {
|
||
self.segmentation.is_some()
|
||
}
|
||
|
||
/// The subjects the model recognised, as `(label, confidence)`.
|
||
///
|
||
/// Confidence is shown rather than hidden because the detector is offered
|
||
/// as a shortcut, not as an authority: a 0.42 "dog" is worth listing and
|
||
/// worth flagging, and a list that presented it identically to a 0.95 one
|
||
/// would make the tool look wrong when the guess was merely weak.
|
||
pub fn detected_subjects(&self) -> Vec<(String, f32)> {
|
||
self.segmentation
|
||
.as_ref()
|
||
.map(|s| {
|
||
s.instances()
|
||
.iter()
|
||
.map(|i| (i.class_name.to_string(), i.score))
|
||
.collect()
|
||
})
|
||
.unwrap_or_default()
|
||
}
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
use crate::develop::test_support::*;
|
||
|
||
// ----------------------------------------------------------------------
|
||
// Segmentation off the UI thread
|
||
// ----------------------------------------------------------------------
|
||
|
||
/// The property the whole arrangement rests on.
|
||
///
|
||
/// If someone puts an `Rc`, a `Cell` or a raw pipeline handle into
|
||
/// `SegmentationJob`, this stops compiling — which is the only warning
|
||
/// there would be, since the call site in `masks_ui` would then fail with
|
||
/// a lifetime error a long way from the cause.
|
||
#[test]
|
||
fn a_job_and_its_answer_can_cross_a_thread() {
|
||
fn is_send<T: Send>() {}
|
||
is_send::<SegmentationJob>();
|
||
is_send::<Segmented>();
|
||
is_send::<Abandon>();
|
||
}
|
||
|
||
/// Two sessions over the same file are still two photographs as far as a
|
||
/// late result is concerned, because opening one twice is opening it
|
||
/// twice.
|
||
#[test]
|
||
fn every_session_has_its_own_identity() {
|
||
let Some(ctx) = headless() else { return };
|
||
let rgba: Vec<u8> = (0..16 * 16).flat_map(|_| [128, 128, 128, 255]).collect();
|
||
let open = || {
|
||
DevelopSession::open_rgb(&ctx, &rgba, 16, 16, dr_types::Orientation::NORMAL)
|
||
.expect("session")
|
||
};
|
||
let (a, b) = (open(), open());
|
||
assert_ne!(a.id(), b.id());
|
||
assert_eq!(a.id(), a.id(), "and stable within one session");
|
||
}
|
||
|
||
#[test]
|
||
fn a_job_carries_the_session_it_was_taken_from() {
|
||
let Some(ctx) = headless() else { return };
|
||
let rgba: Vec<u8> = (0..16 * 16).flat_map(|_| [128, 128, 128, 255]).collect();
|
||
let session = DevelopSession::open_rgb(&ctx, &rgba, 16, 16, dr_types::Orientation::NORMAL)
|
||
.expect("session");
|
||
assert_eq!(session.segmentation_job().session(), session.id());
|
||
}
|
||
|
||
/// Abandoning before the run reaches the proxy must cost nothing at all —
|
||
/// this is the case that fires when the user pages on while a job is still
|
||
/// waiting for a thread.
|
||
#[test]
|
||
fn an_abandoned_job_does_no_work() {
|
||
let Some(ctx) = headless() else { return };
|
||
let rgba: Vec<u8> = (0..16 * 16).flat_map(|_| [128, 128, 128, 255]).collect();
|
||
let session = DevelopSession::open_rgb(&ctx, &rgba, 16, 16, dr_types::Orientation::NORMAL)
|
||
.expect("session");
|
||
|
||
let job = session.segmentation_job();
|
||
job.abandon().now();
|
||
|
||
let started = std::time::Instant::now();
|
||
let out = job.run(&crate::segmentation::Options::default());
|
||
assert!(
|
||
matches!(out, Ok(None)),
|
||
"abandoned is not an error and not an empty answer: {:?}",
|
||
out.map(|o| o.is_some())
|
||
);
|
||
assert!(
|
||
started.elapsed() < std::time::Duration::from_millis(50),
|
||
"it returned without loading the model"
|
||
);
|
||
}
|
||
|
||
/// A finished segmentation is adopted whole, and the session says so.
|
||
#[test]
|
||
fn adopting_a_result_gives_the_session_its_subjects() {
|
||
let Some(ctx) = headless() else { return };
|
||
let rgba: Vec<u8> = (0..100 * 100).flat_map(|_| [128, 128, 128, 255]).collect();
|
||
let mut session =
|
||
DevelopSession::open_rgb(&ctx, &rgba, 100, 100, dr_types::Orientation::NORMAL)
|
||
.expect("session");
|
||
assert!(!session.has_segmentation());
|
||
|
||
let job = session.segmentation_job();
|
||
let Ok(Some(found)) = job.run(&crate::segmentation::Options::default()) else {
|
||
eprintln!("no model; skipping");
|
||
return;
|
||
};
|
||
session.adopt_segmentation(found);
|
||
assert!(session.has_segmentation());
|
||
}
|
||
|
||
/// TRACES: FR-DEV-3
|
||
/// A refine crop's mask, pasted back onto the proxy grid it replaces,
|
||
/// must land exactly where the crop was — not shifted by the origin, not
|
||
/// scaled onto the wrong footprint.
|
||
#[test]
|
||
fn a_refined_mask_pastes_back_at_the_crops_own_position() {
|
||
// A crop entirely on (1.0), covering proxy pixels 2..6 in x and
|
||
// 2..6 in y of an 8x8 proxy — everywhere inside that box must read
|
||
// back as fully covered, everywhere outside as untouched.
|
||
let crop = vec![1.0f32; 4 * 4];
|
||
let mask = paste_into_proxy(&crop, 4, 4, (8, 8), (2.0, 2.0), (4.0, 4.0));
|
||
|
||
assert_eq!(mask[2 * 8 + 2], 255, "top-left corner of the box");
|
||
assert_eq!(mask[5 * 8 + 5], 255, "bottom-right corner of the box");
|
||
assert_eq!(mask[0], 0, "outside the box, untouched");
|
||
assert_eq!(mask[7 * 8 + 7], 0, "outside the box, untouched");
|
||
}
|
||
|
||
#[test]
|
||
fn bilinear_sample_averages_its_four_neighbours() {
|
||
// Two rows, black then white: the exact midpoint reads as grey.
|
||
let mask = [0.0f32, 0.0, 1.0, 1.0];
|
||
let v = bilinear_sample(&mask, 2, 2, 0.5, 0.5);
|
||
assert!(
|
||
(v - 0.5).abs() < 1e-6,
|
||
"expected the midpoint grey, got {v}"
|
||
);
|
||
}
|
||
}
|