Mask the subject the model found, not the regions underneath it

The watershed hierarchy does not survive a photograph, so local masking
stops depending on it. A layer can now be one recognised object, and the
object's own coverage is the mask.

`Options::watershed` defaults off. It costs ~80 ms plus a full-resolution
readback to produce a ladder that collapses, and paying that on every
photograph buys a control that misleads. Kept switchable rather than
deleted: the passes and the hierarchy are correct in themselves and it is
the merge criterion that fails, which is a change to one function.

Masks now rasterise in **source** space at proxy resolution and are sampled
by the composed shader after the framing map. That fixes a real bug: they
were rasterised in output space, so zooming slid the photograph underneath
a mask that stayed pinned to the viewport, and cropping moved every
adjustment to a different part of the picture. Doing it this way also
leaves the framing map in exactly one place — a second copy in the mask
shader would have been a second thing to keep in step, failing only when
straightened.

A subject is stored as identity, not pixels: the mask is megabytes and is
reproducible by running the same model over the same image, so the sidecar
carries the index, the class and the score, and the session carries the
pixels. The class is there to be checked — if instance 3 comes back a "car"
where it was a "dog", something changed and the layer is stale rather than
silently masking the wrong thing.

The overlay now draws instances and is transparent everywhere else. The
region version covered every pixel and so hid the photograph it was drawn
over; the question it exists to answer is whether an outline follows the
subject, which you can only answer by seeing both.

`examples/local.rs` is the worked example: subject in colour with the rest
monochrome, and the subject lifted out of its background. Run on a 5472x3648
CR2 it finds two people and two cars, and the colour-pop keeps her hat and
hair while the wall and grass behind go grey.
This commit is contained in:
2026-08-22 08:39:17 +02:00
parent b1433ad4a9
commit ee10097435
13 changed files with 963 additions and 101 deletions
+70 -11
View File
@@ -55,6 +55,8 @@ pub struct DevelopSession {
segmentation: Option<Segmentation>,
/// Rasterises the mask layers. Built lazily for the same reason.
masks: Option<MaskPass>,
/// The recognised objects' coverage, on the GPU.
subjects: Option<dr_gpu::SubjectMasks>,
/// Which layer the develop panel is editing, if any.
///
/// This is what lets one panel serve both scopes: with a layer selected,
@@ -122,6 +124,7 @@ impl DevelopSession {
.ok(),
segmentation: None,
masks: None,
subjects: None,
active_mask: None,
show_overlay: false,
}
@@ -601,17 +604,37 @@ impl DevelopSession {
/// `self.adjust` mutably while holding `self.masks` immutably. Those are
/// disjoint fields and the borrow checker will allow it — but only when
/// each is reached directly rather than through a method taking `self`.
fn rasterise_masks(&mut self, w: u32, h: u32) -> bool {
fn rasterise_masks(&mut self) -> bool {
if self.graph.masks().is_neutral() {
return false;
}
let labels = self.segmentation.as_ref().and_then(|s| s.labels());
// **Source space, at the segmentation's proxy size** — not the
// viewport's. The generated shader samples this after the framing map,
// so a mask drawn here stays on the photograph through a zoom, a pan
// and a crop. Rasterising at viewport size, as this first did, pinned
// the mask to the screen instead: zooming slid the picture underneath
// one that stayed put.
//
// It also means the array does not reallocate when the window
// resizes, and does not need redrawing when the view moves.
let Some(seg) = self.segmentation.as_ref() else {
return false;
};
let (pw, ph) = seg.proxy_size();
let labels = seg.labels();
let subjects = self.subjects.as_ref();
let Some(pass) = self.masks.as_mut() else {
return false;
};
pass.render(self.graph.masks(), labels, w, h)
.inspect_err(|e| log::warn!("mask rasterisation failed: {e}"))
.is_ok()
pass.render(
self.graph.masks(),
labels,
subjects,
pw as u32,
ph as u32,
)
.inspect_err(|e| log::warn!("mask rasterisation failed: {e}"))
.is_ok()
}
// ----------------------------------------------------------------------
@@ -637,6 +660,19 @@ impl DevelopSession {
.inspect_err(|e| log::warn!("no mask rasteriser on this device: {e}"))
.ok();
}
let (pw, ph) = seg.proxy_size();
let alphas: Vec<&[u8]> = (0..seg.instances().len())
.filter_map(|i| seg.instance_mask(i))
.collect();
self.subjects = if alphas.is_empty() {
None
} else {
dr_gpu::SubjectMasks::upload(ctx, &alphas, pw as u32, ph as u32)
.inspect_err(|e| log::warn!("could not upload the subject masks: {e}"))
.ok()
};
self.segmentation = Some(seg);
Ok(())
}
@@ -688,6 +724,11 @@ impl DevelopSession {
self.segmentation.as_ref().map_or(0, |s| s.level())
}
/// Whether a region hierarchy exists to click into.
pub fn has_regions(&self) -> bool {
self.segmentation.as_ref().is_some_and(|s| s.has_regions())
}
pub fn segmentation_region_count(&self) -> usize {
self.segmentation.as_ref().map_or(0, |s| s.region_count())
}
@@ -821,6 +862,15 @@ impl DevelopSession {
///
/// Returns the layer that now holds the selection.
pub fn select_region_at(&mut self, x: f32, y: f32, add: bool) -> Option<String> {
// With no region hierarchy — the default — a click means "the object
// under the cursor", which is the interaction the model can actually
// support. `add` has no meaning for a whole object, so it is ignored
// rather than quietly doing something else.
if self.segmentation.as_ref().is_some_and(|s| !s.has_regions()) {
let index = self.segmentation.as_ref()?.instance_at(x, y)?;
return self.add_subject_mask(index);
}
let seg = self.segmentation.as_ref()?;
let picked = seg.regions_at(x, y);
if picked.is_empty() {
@@ -875,19 +925,28 @@ impl DevelopSession {
}
/// Add a layer selecting one detected subject.
///
/// The instance's own coverage is the mask, rather than the watershed
/// regions it overlaps. Snapping to regions was the original design and
/// it is not currently worth doing: the hierarchy those ids index into
/// collapses on a photograph (docs/segmentation.md §15), so snapping
/// would trade the model's approximately-right outline for a
/// confidently-wrong one.
pub fn add_subject_mask(&mut self, index: usize) -> Option<String> {
let seg = self.segmentation.as_ref()?;
let instance = seg.instances().get(index)?;
let (signature, level) = (seg.signature(), seg.level());
let (name, ids) = (instance.class_name.to_string(), instance.regions.clone());
let signature = seg.signature();
let name = instance.class_name.to_string();
let score = instance.score;
let id = self.graph.masks().next_id();
let mut layer = MaskLayer::new(
id.clone(),
MaskSource::Regions {
MaskSource::Subject {
signature,
level,
ids,
index: index as u32,
class: name.clone(),
score,
},
);
layer.name = name;
@@ -1024,7 +1083,7 @@ impl DevelopSession {
// Rasterise the masks first: the shader addresses array slices by
// index, so the array has to describe *this* stack before it is bound.
let masks = self
.rasterise_masks(w, h)
.rasterise_masks()
.then(|| self.masks.as_ref().and_then(|p| p.array()))
.flatten();
let texture = self