Mask the subject the model found, not the regions underneath it
The watershed hierarchy does not survive a photograph, so local masking stops depending on it. A layer can now be one recognised object, and the object's own coverage is the mask. `Options::watershed` defaults off. It costs ~80 ms plus a full-resolution readback to produce a ladder that collapses, and paying that on every photograph buys a control that misleads. Kept switchable rather than deleted: the passes and the hierarchy are correct in themselves and it is the merge criterion that fails, which is a change to one function. Masks now rasterise in **source** space at proxy resolution and are sampled by the composed shader after the framing map. That fixes a real bug: they were rasterised in output space, so zooming slid the photograph underneath a mask that stayed pinned to the viewport, and cropping moved every adjustment to a different part of the picture. Doing it this way also leaves the framing map in exactly one place — a second copy in the mask shader would have been a second thing to keep in step, failing only when straightened. A subject is stored as identity, not pixels: the mask is megabytes and is reproducible by running the same model over the same image, so the sidecar carries the index, the class and the score, and the session carries the pixels. The class is there to be checked — if instance 3 comes back a "car" where it was a "dog", something changed and the layer is stale rather than silently masking the wrong thing. The overlay now draws instances and is transparent everywhere else. The region version covered every pixel and so hid the photograph it was drawn over; the question it exists to answer is whether an outline follows the subject, which you can only answer by seeing both. `examples/local.rs` is the worked example: subject in colour with the rest monochrome, and the subject lifted out of its background. Run on a 5472x3648 CR2 it finds two people and two cars, and the colour-pop keeps her hat and hair while the wall and grass behind go grey.
This commit is contained in:
+70
-11
@@ -55,6 +55,8 @@ pub struct DevelopSession {
|
||||
segmentation: Option<Segmentation>,
|
||||
/// Rasterises the mask layers. Built lazily for the same reason.
|
||||
masks: Option<MaskPass>,
|
||||
/// The recognised objects' coverage, on the GPU.
|
||||
subjects: Option<dr_gpu::SubjectMasks>,
|
||||
/// Which layer the develop panel is editing, if any.
|
||||
///
|
||||
/// This is what lets one panel serve both scopes: with a layer selected,
|
||||
@@ -122,6 +124,7 @@ impl DevelopSession {
|
||||
.ok(),
|
||||
segmentation: None,
|
||||
masks: None,
|
||||
subjects: None,
|
||||
active_mask: None,
|
||||
show_overlay: false,
|
||||
}
|
||||
@@ -601,17 +604,37 @@ impl DevelopSession {
|
||||
/// `self.adjust` mutably while holding `self.masks` immutably. Those are
|
||||
/// disjoint fields and the borrow checker will allow it — but only when
|
||||
/// each is reached directly rather than through a method taking `self`.
|
||||
fn rasterise_masks(&mut self, w: u32, h: u32) -> bool {
|
||||
fn rasterise_masks(&mut self) -> bool {
|
||||
if self.graph.masks().is_neutral() {
|
||||
return false;
|
||||
}
|
||||
let labels = self.segmentation.as_ref().and_then(|s| s.labels());
|
||||
// **Source space, at the segmentation's proxy size** — not the
|
||||
// viewport's. The generated shader samples this after the framing map,
|
||||
// so a mask drawn here stays on the photograph through a zoom, a pan
|
||||
// and a crop. Rasterising at viewport size, as this first did, pinned
|
||||
// the mask to the screen instead: zooming slid the picture underneath
|
||||
// one that stayed put.
|
||||
//
|
||||
// It also means the array does not reallocate when the window
|
||||
// resizes, and does not need redrawing when the view moves.
|
||||
let Some(seg) = self.segmentation.as_ref() else {
|
||||
return false;
|
||||
};
|
||||
let (pw, ph) = seg.proxy_size();
|
||||
let labels = seg.labels();
|
||||
let subjects = self.subjects.as_ref();
|
||||
let Some(pass) = self.masks.as_mut() else {
|
||||
return false;
|
||||
};
|
||||
pass.render(self.graph.masks(), labels, w, h)
|
||||
.inspect_err(|e| log::warn!("mask rasterisation failed: {e}"))
|
||||
.is_ok()
|
||||
pass.render(
|
||||
self.graph.masks(),
|
||||
labels,
|
||||
subjects,
|
||||
pw as u32,
|
||||
ph as u32,
|
||||
)
|
||||
.inspect_err(|e| log::warn!("mask rasterisation failed: {e}"))
|
||||
.is_ok()
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------
|
||||
@@ -637,6 +660,19 @@ impl DevelopSession {
|
||||
.inspect_err(|e| log::warn!("no mask rasteriser on this device: {e}"))
|
||||
.ok();
|
||||
}
|
||||
|
||||
let (pw, ph) = seg.proxy_size();
|
||||
let alphas: Vec<&[u8]> = (0..seg.instances().len())
|
||||
.filter_map(|i| seg.instance_mask(i))
|
||||
.collect();
|
||||
self.subjects = if alphas.is_empty() {
|
||||
None
|
||||
} else {
|
||||
dr_gpu::SubjectMasks::upload(ctx, &alphas, pw as u32, ph as u32)
|
||||
.inspect_err(|e| log::warn!("could not upload the subject masks: {e}"))
|
||||
.ok()
|
||||
};
|
||||
|
||||
self.segmentation = Some(seg);
|
||||
Ok(())
|
||||
}
|
||||
@@ -688,6 +724,11 @@ impl DevelopSession {
|
||||
self.segmentation.as_ref().map_or(0, |s| s.level())
|
||||
}
|
||||
|
||||
/// Whether a region hierarchy exists to click into.
|
||||
pub fn has_regions(&self) -> bool {
|
||||
self.segmentation.as_ref().is_some_and(|s| s.has_regions())
|
||||
}
|
||||
|
||||
pub fn segmentation_region_count(&self) -> usize {
|
||||
self.segmentation.as_ref().map_or(0, |s| s.region_count())
|
||||
}
|
||||
@@ -821,6 +862,15 @@ impl DevelopSession {
|
||||
///
|
||||
/// Returns the layer that now holds the selection.
|
||||
pub fn select_region_at(&mut self, x: f32, y: f32, add: bool) -> Option<String> {
|
||||
// With no region hierarchy — the default — a click means "the object
|
||||
// under the cursor", which is the interaction the model can actually
|
||||
// support. `add` has no meaning for a whole object, so it is ignored
|
||||
// rather than quietly doing something else.
|
||||
if self.segmentation.as_ref().is_some_and(|s| !s.has_regions()) {
|
||||
let index = self.segmentation.as_ref()?.instance_at(x, y)?;
|
||||
return self.add_subject_mask(index);
|
||||
}
|
||||
|
||||
let seg = self.segmentation.as_ref()?;
|
||||
let picked = seg.regions_at(x, y);
|
||||
if picked.is_empty() {
|
||||
@@ -875,19 +925,28 @@ impl DevelopSession {
|
||||
}
|
||||
|
||||
/// Add a layer selecting one detected subject.
|
||||
///
|
||||
/// The instance's own coverage is the mask, rather than the watershed
|
||||
/// regions it overlaps. Snapping to regions was the original design and
|
||||
/// it is not currently worth doing: the hierarchy those ids index into
|
||||
/// collapses on a photograph (docs/segmentation.md §15), so snapping
|
||||
/// would trade the model's approximately-right outline for a
|
||||
/// confidently-wrong one.
|
||||
pub fn add_subject_mask(&mut self, index: usize) -> Option<String> {
|
||||
let seg = self.segmentation.as_ref()?;
|
||||
let instance = seg.instances().get(index)?;
|
||||
let (signature, level) = (seg.signature(), seg.level());
|
||||
let (name, ids) = (instance.class_name.to_string(), instance.regions.clone());
|
||||
let signature = seg.signature();
|
||||
let name = instance.class_name.to_string();
|
||||
let score = instance.score;
|
||||
|
||||
let id = self.graph.masks().next_id();
|
||||
let mut layer = MaskLayer::new(
|
||||
id.clone(),
|
||||
MaskSource::Regions {
|
||||
MaskSource::Subject {
|
||||
signature,
|
||||
level,
|
||||
ids,
|
||||
index: index as u32,
|
||||
class: name.clone(),
|
||||
score,
|
||||
},
|
||||
);
|
||||
layer.name = name;
|
||||
@@ -1024,7 +1083,7 @@ impl DevelopSession {
|
||||
// Rasterise the masks first: the shader addresses array slices by
|
||||
// index, so the array has to describe *this* stack before it is bound.
|
||||
let masks = self
|
||||
.rasterise_masks(w, h)
|
||||
.rasterise_masks()
|
||||
.then(|| self.masks.as_ref().and_then(|p| p.array()))
|
||||
.flatten();
|
||||
let texture = self
|
||||
|
||||
Reference in New Issue
Block a user