Measure the distance to the edge, and get four controls for one transform
Feathering, growing, shrinking, closing and opening are the same number read differently. With the signed distance from the boundary in hand, dilation is the set where d >= -r, erosion where d >= +r, and a feather of any shape is a function of d. So the field is computed once and the controls are arithmetic on it. The **field** is what reaches the GPU, not a finished alpha, and that is the point: growing a mask or changing its falloff then costs a uniform upload and no recomputation, which is what makes them live controls rather than ones that stall on every drag. Only closing and opening rebuild, because after the first threshold the shape has changed and the old distances describe the old one. Exact Euclidean, via Felzenszwalb's separable transform — not a chamfer approximation, which leaves a mask visibly octagonal once grown more than a few pixels. A test asserts the diagonal is √2 rather than 1 or 2. It runs on the CPU, which ARCH §5.4 forbids for masks. The rule is about brush lag — a stroke rasterised per frame — and this is a different operation: once per mask edit, on input the model already produced here, producing a field the GPU then samples for free. What it buys is exact determinism, which matters because masks reach the sidecar as indices and a field that varied by vendor would mean a mask meaning one thing on the desktop and another on the phone. The half-pixel in `signed_distance` is not a detail, and a test caught it. Measuring to the nearest opposite pixel *centre* puts the smallest magnitude at 1 either side, so the boundary is nowhere and **eroding by less than a pixel removes nothing**. A control whose first notch does nothing is a broken control. Half a pixel off each side puts the boundary where it physically is, and eroding by 1 takes exactly the outermost ring. Every falloff curve is 0.5 at the boundary by construction, asserted for all five: changing the curve should change how the transition looks and never where it sits.
This commit is contained in:
@@ -30,10 +30,10 @@ use dr_gpu::{
|
||||
AdjustPass, DemosaicedImage, Demosaicer, GpuContext, MaskPass, SubjectMasks,
|
||||
};
|
||||
use dr_pipeline::descriptor::ParamId;
|
||||
use dr_pipeline::mask::{MaskLayer, MaskSource, MaskStack};
|
||||
use dr_pipeline::mask::{MaskLayer, MaskSource, MaskStack, Morphology};
|
||||
use dr_pipeline::operation::compose_full;
|
||||
use dr_pipeline::{ops, EditGraph, Framing};
|
||||
use dr_segment::{SemanticModel, SemanticOptions};
|
||||
use dr_segment::{SemanticModel, SemanticOptions, Shaped};
|
||||
use dr_types::ColourSpace;
|
||||
|
||||
/// Longest edge the mask and the model work at.
|
||||
@@ -127,7 +127,6 @@ fn main() {
|
||||
.iter()
|
||||
.map(|&v| (v.clamp(0.0, 1.0) * 255.0).round() as u8)
|
||||
.collect();
|
||||
let subjects = SubjectMasks::upload(&ctx, &[&alpha], pw, ph).expect("upload subject");
|
||||
|
||||
let (ow, oh) = fit(sw, sh, OUT);
|
||||
let mut masks = MaskPass::new(&ctx).expect("mask pass");
|
||||
@@ -153,7 +152,7 @@ fn main() {
|
||||
drain.feather = 0.02;
|
||||
pop.push(drain);
|
||||
|
||||
render_stack(&ctx, &source, &mut masks, &mut adjust, &pop, &subjects, pw, ph, ow, oh);
|
||||
render_stack(&ctx, &source, &mut masks, &mut adjust, &pop, &alpha, pw, ph, ow, oh);
|
||||
write(&format!("{prefix}-colour-pop.ppm"), &adjust);
|
||||
|
||||
// ---- 2. lift the subject out of its background ------------------------
|
||||
@@ -171,15 +170,39 @@ fn main() {
|
||||
darker.feather = 0.03;
|
||||
lift.push(darker);
|
||||
|
||||
render_stack(&ctx, &source, &mut masks, &mut adjust, &lift, &subjects, pw, ph, ow, oh);
|
||||
render_stack(&ctx, &source, &mut masks, &mut adjust, &lift, &alpha, pw, ph, ow, oh);
|
||||
write(&format!("{prefix}-subject-lift.ppm"), &adjust);
|
||||
|
||||
// ---- 3. the same edit, grown and shrunk -------------------------------
|
||||
//
|
||||
// The model's outline is approximately right and slightly soft, so the
|
||||
// everyday correction is to move it: grow to catch a halo the detector
|
||||
// stopped short of, shrink to pull off one it caught. Both are a threshold
|
||||
// of the distance field, which is why they cost a uniform.
|
||||
for (name, morphology, radius) in [
|
||||
("grown", Morphology::Dilate, 0.012),
|
||||
("shrunk", Morphology::Erode, 0.012),
|
||||
] {
|
||||
let mut stack = MaskStack::new();
|
||||
let mut layer = subject_layer("m1", index, subject);
|
||||
layer.invert = true;
|
||||
layer.set_param("saturation", ParamId("saturation"), -100.0);
|
||||
layer.feather = 0.004;
|
||||
layer.morphology = morphology;
|
||||
layer.morph_radius = radius;
|
||||
stack.push(layer);
|
||||
|
||||
render_stack(&ctx, &source, &mut masks, &mut adjust, &stack, &alpha, pw, ph, ow, oh);
|
||||
write(&format!("{prefix}-{name}.ppm"), &adjust);
|
||||
}
|
||||
|
||||
// ---- the mask itself, to check the outline ----------------------------
|
||||
write_mask(&format!("{prefix}-mask.ppm"), &alpha, pw, ph);
|
||||
|
||||
println!("\nwrote {prefix}-original.ppm");
|
||||
println!(" {prefix}-colour-pop.ppm");
|
||||
println!(" {prefix}-subject-lift.ppm");
|
||||
println!(" {prefix}-grown.ppm, {prefix}-shrunk.ppm");
|
||||
println!(" {prefix}-mask.ppm");
|
||||
}
|
||||
|
||||
@@ -221,18 +244,44 @@ fn render_stack(
|
||||
masks: &mut MaskPass,
|
||||
adjust: &mut AdjustPass,
|
||||
stack: &MaskStack,
|
||||
subjects: &SubjectMasks,
|
||||
coverage: &[u8],
|
||||
pw: u32,
|
||||
ph: u32,
|
||||
ow: u32,
|
||||
oh: u32,
|
||||
) {
|
||||
let _ = ctx;
|
||||
// One signed distance field per active layer, in that order — the order
|
||||
// the rasteriser indexes them by. Built here rather than once up front
|
||||
// because a compound morphology rebuilds the field, so it belongs to the
|
||||
// layer that shaped it rather than to the object.
|
||||
let fields: Vec<Vec<f32>> = stack
|
||||
.active()
|
||||
.map(|layer| {
|
||||
Shaped::build(
|
||||
coverage,
|
||||
pw as usize,
|
||||
ph as usize,
|
||||
128,
|
||||
match layer.morphology {
|
||||
Morphology::None => dr_segment::Morphology::None,
|
||||
Morphology::Dilate => dr_segment::Morphology::Dilate,
|
||||
Morphology::Erode => dr_segment::Morphology::Erode,
|
||||
Morphology::Close => dr_segment::Morphology::Close,
|
||||
Morphology::Open => dr_segment::Morphology::Open,
|
||||
},
|
||||
layer.morph_radius * pw.min(ph) as f32,
|
||||
)
|
||||
.distance
|
||||
})
|
||||
.collect();
|
||||
let refs: Vec<&[f32]> = fields.iter().map(|f| f.as_slice()).collect();
|
||||
let subjects = SubjectMasks::upload(ctx, &refs, pw, ph).expect("upload fields");
|
||||
|
||||
// Rasterised at *proxy* size in source space, then sampled by the shader
|
||||
// after the framing map — which is what makes one mask correct at every
|
||||
// output size, zoom and crop.
|
||||
let array = masks
|
||||
.render(stack, None, Some(subjects), pw, ph)
|
||||
.render(stack, None, Some(&subjects), pw, ph)
|
||||
.expect("rasterise masks");
|
||||
|
||||
let shader = compose_full(&ops::chain(), &Framing::new(), ColourSpace::Srgb, stack);
|
||||
|
||||
+96
-40
@@ -43,7 +43,9 @@ struct MaskParams {
|
||||
mode: u32,
|
||||
region_count: u32,
|
||||
feather: f32,
|
||||
_pad0: f32,
|
||||
/// Which falloff curve a subject layer uses. Kept in step with the
|
||||
/// `switch` in `mask.wgsl` by `falloff_code`.
|
||||
falloff: u32,
|
||||
|
||||
centre: [f32; 2],
|
||||
axis: [f32; 2],
|
||||
@@ -110,10 +112,17 @@ impl LabelField {
|
||||
}
|
||||
}
|
||||
|
||||
/// The recognised objects' coverage, resident on the GPU.
|
||||
/// Signed distance fields for the subject layers, resident on the GPU.
|
||||
///
|
||||
/// Uploaded once per segmentation, indexed exactly as the detection list is,
|
||||
/// so a layer storing "instance 3" finds instance 3 here.
|
||||
/// **One per active layer, in that order** — not one per detected object. Two
|
||||
/// layers can mask the same subject with different morphology, and closing or
|
||||
/// opening rebuilds the field rather than offsetting it, so the field belongs
|
||||
/// to the layer that shaped it.
|
||||
///
|
||||
/// `R32Float`, because the values are signed distances in pixels and the
|
||||
/// controls read them at sub-pixel precision. That is four bytes a pixel:
|
||||
/// ~7 MB per layer at a 1600 px proxy, which is the price of making grow,
|
||||
/// shrink and feather cost nothing per frame.
|
||||
pub struct SubjectMasks {
|
||||
views: Vec<wgpu::TextureView>,
|
||||
width: u32,
|
||||
@@ -121,33 +130,28 @@ pub struct SubjectMasks {
|
||||
}
|
||||
|
||||
impl SubjectMasks {
|
||||
/// Upload one `R8Unorm` texture per instance.
|
||||
///
|
||||
/// A byte per pixel, which is what the model's coverage was quantised to
|
||||
/// on the way out of inference: 256 levels is finer than any edge a person
|
||||
/// can see, and four bytes would make a handful of objects most of a
|
||||
/// hundred megabytes for one photograph.
|
||||
/// Upload one distance field per active subject layer.
|
||||
pub fn upload(
|
||||
ctx: &GpuContext,
|
||||
masks: &[&[u8]],
|
||||
fields: &[&[f32]],
|
||||
width: u32,
|
||||
height: u32,
|
||||
) -> Result<Self, GpuError> {
|
||||
let expected = (width * height) as usize;
|
||||
let mut views = Vec::with_capacity(masks.len());
|
||||
let mut views = Vec::with_capacity(fields.len());
|
||||
|
||||
for (i, mask) in masks.iter().enumerate() {
|
||||
if mask.len() != expected {
|
||||
for (i, field) in fields.iter().enumerate() {
|
||||
if field.len() != expected {
|
||||
return Err(GpuError::InvalidMask(format!(
|
||||
"subject {i} mask is {} bytes, expected {width}x{height}",
|
||||
mask.len()
|
||||
"subject field {i} is {} values, expected {width}x{height}",
|
||||
field.len()
|
||||
)));
|
||||
}
|
||||
|
||||
let texture = ctx.device.create_texture_with_data(
|
||||
&ctx.queue,
|
||||
&wgpu::TextureDescriptor {
|
||||
label: Some("subject-mask"),
|
||||
label: Some("subject-distance"),
|
||||
size: wgpu::Extent3d {
|
||||
width,
|
||||
height,
|
||||
@@ -156,12 +160,12 @@ impl SubjectMasks {
|
||||
mip_level_count: 1,
|
||||
sample_count: 1,
|
||||
dimension: wgpu::TextureDimension::D2,
|
||||
format: wgpu::TextureFormat::R8Unorm,
|
||||
format: wgpu::TextureFormat::R32Float,
|
||||
usage: wgpu::TextureUsages::TEXTURE_BINDING,
|
||||
view_formats: &[],
|
||||
},
|
||||
wgpu::util::TextureDataOrder::LayerMajor,
|
||||
mask,
|
||||
bytemuck::cast_slice(field),
|
||||
);
|
||||
views.push(texture.create_view(&wgpu::TextureViewDescriptor::default()));
|
||||
}
|
||||
@@ -190,7 +194,7 @@ impl SubjectMasks {
|
||||
}
|
||||
}
|
||||
|
||||
/// The rasterised masks for one edit.
|
||||
/// The rasterised masks for one edit./// The rasterised masks for one edit.
|
||||
pub struct MaskArray {
|
||||
texture: wgpu::Texture,
|
||||
view: wgpu::TextureView,
|
||||
@@ -266,7 +270,10 @@ impl MaskPass {
|
||||
binding: 3,
|
||||
visibility: wgpu::ShaderStages::FRAGMENT,
|
||||
ty: wgpu::BindingType::Texture {
|
||||
sample_type: wgpu::TextureSampleType::Float { filterable: true },
|
||||
// `filterable: false`: R32Float cannot be filtered
|
||||
// without an optional feature, and the shader loads
|
||||
// texels and interpolates them itself anyway.
|
||||
sample_type: wgpu::TextureSampleType::Float { filterable: false },
|
||||
view_dimension: wgpu::TextureViewDimension::D2,
|
||||
multisampled: false,
|
||||
},
|
||||
@@ -312,7 +319,9 @@ impl MaskPass {
|
||||
}
|
||||
|
||||
let placeholder = LabelField::upload(ctx, &[0], 1, 1, 0)?;
|
||||
let empty_subject = SubjectMasks::upload(ctx, &[&[0u8][..]], 1, 1)?;
|
||||
// Everywhere outside, so a layer that somehow reaches this masks
|
||||
// nothing rather than everything.
|
||||
let empty_subject = SubjectMasks::upload(ctx, &[&[-1.0f32][..]], 1, 1)?;
|
||||
|
||||
Ok(Self {
|
||||
ctx: ctx.clone(),
|
||||
@@ -367,20 +376,20 @@ impl MaskPass {
|
||||
// same reason a region layer without a segmentation is: an absent
|
||||
// mask that defaults to "everything" would apply the adjustment to
|
||||
// the whole photograph, which is a much louder failure than none.
|
||||
// Indexed by *slot*, not by the instance the layer names: the
|
||||
// fields are built per layer, in this same order, because two
|
||||
// layers over one subject can carry different morphology.
|
||||
let subject = match &layer.source {
|
||||
MaskSource::Subject { index, .. } => {
|
||||
match subjects.filter(|s| (*index as usize) < s.len()) {
|
||||
Some(s) => (s, *index as usize),
|
||||
None => {
|
||||
log::warn!(
|
||||
"mask layer {} names subject {index}, which this segmentation \
|
||||
does not have; skipping",
|
||||
layer.id
|
||||
);
|
||||
continue;
|
||||
}
|
||||
MaskSource::Subject { .. } => match subjects.filter(|s| slot < s.len()) {
|
||||
Some(s) => (s, slot),
|
||||
None => {
|
||||
log::warn!(
|
||||
"mask layer {} has no distance field; skipping",
|
||||
layer.id
|
||||
);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
},
|
||||
_ => (&self.empty_subject, 0),
|
||||
};
|
||||
|
||||
@@ -418,7 +427,7 @@ impl MaskPass {
|
||||
mode: MODE_REGIONS,
|
||||
region_count: field.region_count,
|
||||
feather: 0.0,
|
||||
_pad0: 0.0,
|
||||
falloff: 0,
|
||||
centre: [0.5, 0.5],
|
||||
axis: [1.0, 0.0],
|
||||
softness: 0.0,
|
||||
@@ -431,11 +440,24 @@ impl MaskPass {
|
||||
// already a soft sigmoid, so zero means "use the edge the model
|
||||
// drew" rather than "hard edge" — the one place in this shader
|
||||
// where zero softness is not a step.
|
||||
MaskSource::Subject { .. } => MaskParams {
|
||||
mode: MODE_SUBJECT,
|
||||
softness: layer.feather.clamp(0.0, 0.5),
|
||||
..base
|
||||
},
|
||||
// Feather and morphology are in fractions of the frame's shorter
|
||||
// edge; the field is in proxy pixels. Converting here keeps the
|
||||
// stored edit resolution-independent while the shader works in the
|
||||
// units its texture is actually measured in.
|
||||
MaskSource::Subject { .. } => {
|
||||
let short = field_short_edge(width, height);
|
||||
MaskParams {
|
||||
mode: MODE_SUBJECT,
|
||||
// `softness` is the feather half-width in pixels.
|
||||
softness: (layer.feather * short).max(0.0),
|
||||
// `angle` carries the morphology offset — reused rather
|
||||
// than padded, since a subject layer has no ellipse to
|
||||
// rotate.
|
||||
angle: morph_offset(layer) * short,
|
||||
falloff: falloff_code(layer.falloff),
|
||||
..base
|
||||
}
|
||||
}
|
||||
MaskSource::Regions { .. } => MaskParams {
|
||||
// A pixel of softening at the proxy-to-output ratio, so the
|
||||
// edge is equally soft whatever size the render is.
|
||||
@@ -619,6 +641,40 @@ impl MaskPass {
|
||||
}
|
||||
}
|
||||
|
||||
/// The shorter edge of the space the mask is rasterised in.
|
||||
///
|
||||
/// Feather and morphology are stored as fractions of it, so the same edit is
|
||||
/// the same edge whether it renders to a viewport or to a 24 MP export.
|
||||
fn field_short_edge(width: u32, height: u32) -> f32 {
|
||||
width.min(height).max(1) as f32
|
||||
}
|
||||
|
||||
/// How far the boundary moves, in fractions of the shorter edge.
|
||||
///
|
||||
/// Zero for closing and opening: those are folded into the field itself when
|
||||
/// it is built, because their second half acts on a shape the original field
|
||||
/// does not describe.
|
||||
fn morph_offset(layer: &dr_pipeline::mask::MaskLayer) -> f32 {
|
||||
use dr_pipeline::mask::Morphology;
|
||||
match layer.morphology {
|
||||
Morphology::Dilate => layer.morph_radius,
|
||||
Morphology::Erode => -layer.morph_radius,
|
||||
Morphology::None | Morphology::Close | Morphology::Open => 0.0,
|
||||
}
|
||||
}
|
||||
|
||||
/// Kept in step with the `switch` in `mask.wgsl`.
|
||||
fn falloff_code(falloff: dr_pipeline::mask::Falloff) -> u32 {
|
||||
use dr_pipeline::mask::Falloff;
|
||||
match falloff {
|
||||
Falloff::Hard => 0,
|
||||
Falloff::Linear => 1,
|
||||
Falloff::Smooth => 2,
|
||||
Falloff::Gaussian => 3,
|
||||
Falloff::Exponential => 4,
|
||||
}
|
||||
}
|
||||
|
||||
fn uniform_entry(binding: u32) -> wgpu::BindGroupLayoutEntry {
|
||||
wgpu::BindGroupLayoutEntry {
|
||||
binding,
|
||||
|
||||
@@ -35,7 +35,9 @@ struct MaskParams {
|
||||
region_count: u32,
|
||||
// Softening applied to a region mask, in output pixels.
|
||||
feather: f32,
|
||||
_pad0: f32,
|
||||
// 0 hard, 1 linear, 2 smooth, 3 gaussian, 4 exponential. Kept in step with
|
||||
// `falloff_code` on the Rust side.
|
||||
falloff: u32,
|
||||
|
||||
// Geometry, in normalised output coordinates. Meaning depends on `mode`.
|
||||
centre: vec2<f32>,
|
||||
@@ -57,9 +59,14 @@ struct MaskParams {
|
||||
// One entry per region: non-zero if the region is in this mask. Small — a few
|
||||
// thousand bytes — which is what makes changing a selection cheap.
|
||||
@group(0) @binding(2) var<storage, read> selected: array<u32>;
|
||||
// One recognised object's coverage, for a subject mask. A 1x1 placeholder
|
||||
// when the layer is not one — the binding is fixed, and a second pipeline
|
||||
// differing only in what it ignores would be worse than a wasted texel.
|
||||
// The **signed distance** from one subject's boundary, in proxy pixels:
|
||||
// positive inside, negative outside. A 1x1 placeholder when the layer is not a
|
||||
// subject — the binding is fixed, and a second pipeline differing only in what
|
||||
// it ignores would be worse than a wasted texel.
|
||||
//
|
||||
// A distance field rather than a finished alpha is what makes growing,
|
||||
// shrinking and feathering free: each is arithmetic on this, so a slider moves
|
||||
// a uniform instead of rebuilding a mask.
|
||||
@group(0) @binding(3) var subject: texture_2d<f32>;
|
||||
|
||||
// A full-screen triangle rather than a quad: three vertices instead of six,
|
||||
@@ -142,12 +149,12 @@ fn radial_mask(uv: vec2<f32>) -> f32 {
|
||||
return 1.0 - smoothstep(1.0 - edge, 1.0, r);
|
||||
}
|
||||
|
||||
// The model's coverage for one object, resampled to the mask's own grid.
|
||||
// Coverage for one object, from its distance field.
|
||||
//
|
||||
// Bilinear, unlike the region lookup above: this is a *quantity*, not a name,
|
||||
// so the value between two samples is meaningful. The model's own mask is a
|
||||
// quarter-resolution sigmoid, and interpolating it is what stops the outline
|
||||
// stair-stepping in blocks of four.
|
||||
// Bilinear on the *distance*, which is the reason this is a distance field at
|
||||
// all: distance varies smoothly across the boundary where coverage does not,
|
||||
// so interpolating it gives a clean sub-pixel edge even though the model's
|
||||
// own mask was quarter-resolution.
|
||||
fn subject_mask(uv: vec2<f32>) -> f32 {
|
||||
let dims = vec2<f32>(textureDimensions(subject));
|
||||
let last = vec2<i32>(dims) - vec2<i32>(1);
|
||||
@@ -164,15 +171,34 @@ fn subject_mask(uv: vec2<f32>) -> f32 {
|
||||
let c = textureLoad(subject, vec2<i32>(p0.x, p1.y), 0).r;
|
||||
let d = textureLoad(subject, vec2<i32>(p1.x, p1.y), 0).r;
|
||||
|
||||
let cov = mix(mix(a, b, f.x), mix(c, d, f.x), f.y);
|
||||
// `angle` carries the morphology offset in pixels: positive grows the
|
||||
// mask, negative shrinks it. Adding it before the falloff is what makes
|
||||
// dilation move the boundary rather than merely brighten the edge.
|
||||
let dist = mix(mix(a, b, f.x), mix(c, d, f.x), f.y) + p.angle;
|
||||
|
||||
// `softness` carries the layer's feather here. Zero gives the model's own
|
||||
// soft edge untouched, which is a perfectly good mask edge and a better
|
||||
// default than imposing a ramp on top of one that already exists.
|
||||
// `softness` is the feather half-width, also in pixels.
|
||||
if (p.softness <= 0.0) {
|
||||
return cov;
|
||||
return select(0.0, 1.0, dist >= 0.0);
|
||||
}
|
||||
let t_norm = dist / p.softness;
|
||||
|
||||
// Every curve is 0.5 at the boundary, so changing the falloff changes how
|
||||
// the transition looks and never where it sits.
|
||||
switch p.falloff {
|
||||
case 0u: { return select(0.0, 1.0, dist >= 0.0); }
|
||||
case 1u: { return clamp(t_norm * 0.5 + 0.5, 0.0, 1.0); }
|
||||
case 3u: { return 1.0 / (1.0 + exp(-3.0 * t_norm)); }
|
||||
case 4u: {
|
||||
if (t_norm >= 0.0) {
|
||||
return 1.0 - 0.5 * exp(-3.0 * t_norm);
|
||||
}
|
||||
return 0.5 * exp(3.0 * t_norm);
|
||||
}
|
||||
default: {
|
||||
let x = clamp(t_norm * 0.5 + 0.5, 0.0, 1.0);
|
||||
return x * x * (3.0 - 2.0 * x);
|
||||
}
|
||||
}
|
||||
return smoothstep(0.5 - p.softness, 0.5 + p.softness, cov);
|
||||
}
|
||||
|
||||
@fragment
|
||||
|
||||
Reference in New Issue
Block a user