Mask the subject the model found, not the regions underneath it
The watershed hierarchy does not survive a photograph, so local masking stops depending on it. A layer can now be one recognised object, and the object's own coverage is the mask. `Options::watershed` defaults off. It costs ~80 ms plus a full-resolution readback to produce a ladder that collapses, and paying that on every photograph buys a control that misleads. Kept switchable rather than deleted: the passes and the hierarchy are correct in themselves and it is the merge criterion that fails, which is a change to one function. Masks now rasterise in **source** space at proxy resolution and are sampled by the composed shader after the framing map. That fixes a real bug: they were rasterised in output space, so zooming slid the photograph underneath a mask that stayed pinned to the viewport, and cropping moved every adjustment to a different part of the picture. Doing it this way also leaves the framing map in exactly one place — a second copy in the mask shader would have been a second thing to keep in step, failing only when straightened. A subject is stored as identity, not pixels: the mask is megabytes and is reproducible by running the same model over the same image, so the sidecar carries the index, the class and the score, and the session carries the pixels. The class is there to be checked — if instance 3 comes back a "car" where it was a "dog", something changed and the layer is stale rather than silently masking the wrong thing. The overlay now draws instances and is transparent everywhere else. The region version covered every pixel and so hid the photograph it was drawn over; the question it exists to answer is whether an outline follows the subject, which you can only answer by seeing both. `examples/local.rs` is the worked example: subject in colour with the rest monochrome, and the subject lifted out of its background. Run on a 5472x3648 CR2 it finds two people and two cars, and the colour-pop keeps her hat and hair while the wall and grass behind go grey.
This commit is contained in:
@@ -44,6 +44,40 @@ use crate::ops;
|
||||
/// be a literal in one.
|
||||
pub const DEFAULT_FEATHER: f32 = 0.004;
|
||||
|
||||
/// Bilinear sampling of one slice of the mask array, in **source** space.
|
||||
///
|
||||
/// Hand-rolled rather than done with a sampler, matching how the source
|
||||
/// texture is read: adding a sampler would change a bind group layout every
|
||||
/// pass shares.
|
||||
///
|
||||
/// Bilinear rather than a straight load because the array is at proxy
|
||||
/// resolution and the view may be zoomed well past it. A nearest-neighbour
|
||||
/// mask shows as visible stair-stepping along the edge of the adjustment at
|
||||
/// 100%, which is exactly where a mask is judged.
|
||||
pub(crate) const MASK_SAMPLER: crate::operation::Helper = crate::operation::Helper {
|
||||
name: "sample_mask",
|
||||
source: "fn sample_mask(uv: vec2<f32>, layer: i32) -> f32 {
|
||||
let dims = vec2<f32>(textureDimensions(masks));
|
||||
let last = vec2<i32>(dims) - vec2<i32>(1);
|
||||
|
||||
// Sample positions are texel centres, so the half-texel offset is what
|
||||
// keeps the interpolated edge where the rasteriser drew it.
|
||||
let t = uv * dims - vec2<f32>(0.5);
|
||||
let base = vec2<i32>(floor(t));
|
||||
let f = fract(t);
|
||||
|
||||
let p0 = clamp(base, vec2<i32>(0), last);
|
||||
let p1 = clamp(base + vec2<i32>(1), vec2<i32>(0), last);
|
||||
|
||||
let a = textureLoad(masks, vec2<i32>(p0.x, p0.y), layer, 0).r;
|
||||
let b = textureLoad(masks, vec2<i32>(p1.x, p0.y), layer, 0).r;
|
||||
let c = textureLoad(masks, vec2<i32>(p0.x, p1.y), layer, 0).r;
|
||||
let d = textureLoad(masks, vec2<i32>(p1.x, p1.y), layer, 0).r;
|
||||
|
||||
return mix(mix(a, b, f.x), mix(c, d, f.x), f.y);
|
||||
}",
|
||||
};
|
||||
|
||||
/// Per-layer uniforms the generated shader reads: `invert`, then `opacity`.
|
||||
pub const LAYER_UNIFORM_FIELDS: usize = 2;
|
||||
|
||||
@@ -215,6 +249,38 @@ pub enum MaskSource {
|
||||
ids: Vec<u32>,
|
||||
},
|
||||
|
||||
/// One object the model recognised, used as the mask directly.
|
||||
///
|
||||
/// **The primary way a local adjustment is made.** The watershed hierarchy
|
||||
/// this crate was first built around does not survive a photograph: its
|
||||
/// saddles are near zero almost everywhere, so a global cut collapses the
|
||||
/// frame into one region plus noise (docs/segmentation.md §15). A model
|
||||
/// instance is a whole object, found as one thing, and needs no ladder.
|
||||
///
|
||||
/// The trade is that the boundary is the model's — a quarter-resolution
|
||||
/// sigmoid — rather than the image's own gradient. That is what the edge
|
||||
/// treatment on [`MaskLayer`] is for: the mask arrives approximately
|
||||
/// right and soft, and dilation, erosion and a chosen falloff are how it
|
||||
/// is made to fit.
|
||||
///
|
||||
/// Stored as *identity*, not as pixels. The mask itself is several
|
||||
/// megabytes and is reproducible by running the same model over the same
|
||||
/// image, so the sidecar carries what is needed to find it again and the
|
||||
/// session carries the pixels.
|
||||
Subject {
|
||||
/// Which segmentation run produced it, so a layer can tell whether
|
||||
/// the index below still means what it meant.
|
||||
signature: u64,
|
||||
/// Position in that run's detection list, strongest first.
|
||||
index: u32,
|
||||
/// The class name, for display and as a sanity check on re-detection:
|
||||
/// if instance 3 is a "car" where it was a "dog", the model or the
|
||||
/// image changed and the layer should be treated as stale rather than
|
||||
/// silently masking something else.
|
||||
class: String,
|
||||
score: f32,
|
||||
},
|
||||
|
||||
/// A linear gradient — the graduated-filter mask.
|
||||
///
|
||||
/// Geometry is in **normalised output coordinates**, so it survives a crop
|
||||
@@ -247,6 +313,7 @@ impl MaskSource {
|
||||
pub fn kind(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Regions { .. } => "regions",
|
||||
Self::Subject { .. } => "subject",
|
||||
Self::Linear { .. } => "linear",
|
||||
Self::Radial { .. } => "radial",
|
||||
}
|
||||
@@ -405,7 +472,15 @@ impl MaskLayer {
|
||||
/// Applying it anyway would produce a confidently wrong mask, so callers
|
||||
/// should offer to recompute rather than render it.
|
||||
pub fn is_stale(&self, current: u64) -> bool {
|
||||
matches!(self.source, MaskSource::Regions { signature, .. } if signature != current)
|
||||
match self.source {
|
||||
MaskSource::Regions { signature, .. } | MaskSource::Subject { signature, .. } => {
|
||||
signature != current
|
||||
}
|
||||
// A gradient is geometry in normalised coordinates. It means the
|
||||
// same thing whatever was or was not detected, so nothing about a
|
||||
// new run can invalidate it.
|
||||
MaskSource::Linear { .. } | MaskSource::Radial { .. } => false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn descriptors(&self) -> Vec<&'static OpDescriptor> {
|
||||
@@ -599,6 +674,10 @@ pub(crate) fn compose_layers(stack: &MaskStack) -> LayerShader {
|
||||
helpers: Vec::new(),
|
||||
};
|
||||
|
||||
if stack.active().next().is_some() {
|
||||
out.helpers.push(MASK_SAMPLER);
|
||||
}
|
||||
|
||||
for (slot, layer) in stack.active().enumerate() {
|
||||
let prefix = format!("mask{slot}");
|
||||
|
||||
@@ -616,9 +695,21 @@ pub(crate) fn compose_layers(stack: &MaskStack) -> LayerShader {
|
||||
layer.source.kind()
|
||||
);
|
||||
let _ = writeln!(out.body, " {{");
|
||||
// **`uv_src`, not `gid.xy`.** The mask array is rasterised in *source*
|
||||
// space, and `uv_src` is the source position this output pixel came
|
||||
// from — after the crop, the zoom, the pan, the straightening and the
|
||||
// flips. Sampling by output pixel instead, as this once did, pins the
|
||||
// mask to the viewport: zooming in slides the photograph under a mask
|
||||
// that stays where it was, and cropping moves the adjustment to a
|
||||
// different part of the picture.
|
||||
//
|
||||
// Doing it this way also means the framing map exists in exactly one
|
||||
// place. A second copy here would be a second thing to keep in step
|
||||
// with `Framing::wgsl_prologue`, and the failure would be a mask that
|
||||
// is subtly wrong only when straightened.
|
||||
let _ = writeln!(
|
||||
out.body,
|
||||
" var m = textureLoad(masks, vec2<i32>(gid.xy), {slot}, 0).r;"
|
||||
" var m = sample_mask(uv_src, {slot});"
|
||||
);
|
||||
let _ = writeln!(
|
||||
out.body,
|
||||
@@ -763,8 +854,8 @@ mod tests {
|
||||
stack.push(lit_layer("m2", -1.0));
|
||||
|
||||
let shader = compose_layers(&stack);
|
||||
assert!(shader.body.contains("textureLoad(masks, vec2<i32>(gid.xy), 0, 0)"));
|
||||
assert!(shader.body.contains("textureLoad(masks, vec2<i32>(gid.xy), 1, 0)"));
|
||||
assert!(shader.body.contains("sample_mask(uv_src, 0)"));
|
||||
assert!(shader.body.contains("sample_mask(uv_src, 1)"));
|
||||
assert!(shader.body.contains("u.mask0_opacity"));
|
||||
assert!(shader.body.contains("u.mask1_opacity"));
|
||||
}
|
||||
@@ -781,10 +872,10 @@ mod tests {
|
||||
|
||||
let shader = compose_layers(&stack);
|
||||
assert!(
|
||||
shader.body.contains("gid.xy), 0, 0"),
|
||||
shader.body.contains("sample_mask(uv_src, 0)"),
|
||||
"the one active layer must use slot 0, not slot 1"
|
||||
);
|
||||
assert!(!shader.body.contains("gid.xy), 1, 0"));
|
||||
assert!(!shader.body.contains("sample_mask(uv_src, 1)"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
Reference in New Issue
Block a user