Record what the spec got wrong about the model that exists
docs/segmentation.md §4 priced arm B as costing a C dependency under the NDK and treated that as most of the difference between the arms. It is not a cost that has to be paid: `ort`'s `alternative-backend` disables its linking entirely and `ort-tract` supplies the API from tract, which is pure Rust. D13's "largest exception the policy would tolerate" turns out not to be needed, and the answer generalises to the face pipeline — so D13's runtime half is now answered and only its licensing half is open. Three findings contradict §4 outright and are recorded as F4-F6 rather than quietly designed around. There is no ADE20K-trained YOLO, so the shipped vocabulary selects subjects and not stuff — "select the sky" comes from the watershed or from nowhere. It is instance segmentation, so it partitions nothing and two people come back as two instances. And tract cannot parse a dynamic-shape export, which fixes the input at 640 square and makes tiling the only route to more semantic resolution. Arm C ships, but §8's criteria are not what decided it, and saying so matters more than claiming the process worked. §8 asked for a two- interaction margin over arm A on a traced corpus. That comparison was never run: F4 and F5 changed what the arms are, and a model that recognises subjects but has no word for sky cannot be a selection tool alone, while a watershed cannot tell a person from the wall behind them. They stopped being candidates and became complements. What is *not* done is written down as plainly: the 24-image corpus is untraced, so M1-M4 have no numbers and "this feels right" has not become one. M5 is answered on one device only, and region ids now reach the sidecar — so a cross-vendor divergence would mean a mask written on the desktop meaning something else on Android. F3 stands.
This commit is contained in:
@@ -535,4 +535,182 @@ mod tests {
|
||||
stronger.prior.strength = 0.1;
|
||||
assert_eq!(tuning_hash(&base), tuning_hash(&stronger));
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// On a device, end to end
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
/// A bright disc on a dark ground: one unambiguous boundary, which is the
|
||||
/// backlit-silhouette control case from docs/segmentation.md §9.
|
||||
fn disc(size: u32) -> Vec<u8> {
|
||||
let r = size as f32 * 0.28;
|
||||
let c = size as f32 / 2.0;
|
||||
let mut out = Vec::with_capacity((size * size) as usize * 4);
|
||||
for y in 0..size {
|
||||
for x in 0..size {
|
||||
let d = ((x as f32 + 0.5 - c).powi(2) + (y as f32 + 0.5 - c).powi(2)).sqrt();
|
||||
let v = if d < r { 230 } else { 30 };
|
||||
out.extend_from_slice(&[v, v, v, 255]);
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn context() -> Option<dr_gpu::GpuContext> {
|
||||
pollster::block_on(dr_gpu::GpuContext::new_headless()).ok()
|
||||
}
|
||||
|
||||
/// The whole arm-A chain on a real adapter: watershed, region graph, merge
|
||||
/// tree, and a click landing on the disc rather than on the ground.
|
||||
///
|
||||
/// The CPU tests above use hand-built fields, which cannot catch a
|
||||
/// watershed that produces nothing, a proxy scaled the wrong way, or a
|
||||
/// click transformed into the wrong pixel.
|
||||
#[test]
|
||||
fn a_real_segmentation_separates_the_disc_from_the_ground() {
|
||||
let Some(ctx) = context() else {
|
||||
eprintln!("no adapter; skipping");
|
||||
return;
|
||||
};
|
||||
|
||||
const SIZE: u32 = 256;
|
||||
let rgba = disc(SIZE);
|
||||
let source = dr_gpu::DemosaicedImage::from_rgba8(&ctx, &rgba, SIZE, SIZE).expect("upload");
|
||||
let rgb: Vec<f32> = rgba
|
||||
.chunks_exact(4)
|
||||
.flat_map(|p| [p[0] as f32 / 255.0, p[1] as f32 / 255.0, p[2] as f32 / 255.0])
|
||||
.collect();
|
||||
|
||||
// Arm A alone. The model has no COCO class for "grey disc", so running
|
||||
// it here would cost half a second to contribute nothing — and the
|
||||
// point of this test is the watershed half.
|
||||
let options = Options {
|
||||
semantic: false,
|
||||
..Options::default()
|
||||
};
|
||||
|
||||
let mut seg = compute(&ctx, &source, &rgb, SIZE as usize, SIZE as usize, &options)
|
||||
.expect("segmentation");
|
||||
assert!(seg.region_count() > 1, "a boundary should make regions");
|
||||
|
||||
// Coarse enough that the disc is one region rather than several.
|
||||
seg.set_level(4);
|
||||
|
||||
let centre = seg.group_at(0.5, 0.5).expect("centre is in frame");
|
||||
let corner = seg.group_at(0.04, 0.04).expect("corner is in frame");
|
||||
assert_ne!(
|
||||
centre, corner,
|
||||
"the disc and the ground must not be one group"
|
||||
);
|
||||
|
||||
// And the selection must be the disc, not the whole frame: a mask that
|
||||
// covers everything is the failure mode a "they differ" assertion on
|
||||
// its own would not catch.
|
||||
let selected = seg.regions_at(0.5, 0.5);
|
||||
assert!(!selected.is_empty());
|
||||
assert!(
|
||||
selected.len() < seg.region_count(),
|
||||
"selecting the disc should not select every region"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_overlay_matches_the_segmentation_it_describes() {
|
||||
let Some(ctx) = context() else {
|
||||
eprintln!("no adapter; skipping");
|
||||
return;
|
||||
};
|
||||
|
||||
const SIZE: u32 = 128;
|
||||
let rgba = disc(SIZE);
|
||||
let source = dr_gpu::DemosaicedImage::from_rgba8(&ctx, &rgba, SIZE, SIZE).expect("upload");
|
||||
let rgb: Vec<f32> = rgba
|
||||
.chunks_exact(4)
|
||||
.flat_map(|p| [p[0] as f32 / 255.0, p[1] as f32 / 255.0, p[2] as f32 / 255.0])
|
||||
.collect();
|
||||
|
||||
let options = Options {
|
||||
semantic: false,
|
||||
..Options::default()
|
||||
};
|
||||
let mut seg = compute(&ctx, &source, &rgb, SIZE as usize, SIZE as usize, &options)
|
||||
.expect("segmentation");
|
||||
seg.set_level(4);
|
||||
|
||||
let (pixels, w, h) = seg.overlay_rgba();
|
||||
assert_eq!(pixels.len(), (w * h) as usize * 4);
|
||||
|
||||
// The disc's centre and the ground must be drawn differently, or the
|
||||
// overlay is not showing the thing it claims to show.
|
||||
let at = |x: u32, y: u32| {
|
||||
let p = ((y * w + x) * 4) as usize;
|
||||
[pixels[p], pixels[p + 1], pixels[p + 2]]
|
||||
};
|
||||
assert_ne!(at(w / 2, h / 2), at(2, 2));
|
||||
}
|
||||
|
||||
/// The signature has to change when a mask's ids would mean something
|
||||
/// different, and this is the case that actually happens: the same image
|
||||
/// segmented at another proxy size.
|
||||
#[test]
|
||||
fn re_segmenting_at_another_size_invalidates_stored_ids() {
|
||||
let Some(ctx) = context() else {
|
||||
eprintln!("no adapter; skipping");
|
||||
return;
|
||||
};
|
||||
|
||||
const SIZE: u32 = 192;
|
||||
let rgba = disc(SIZE);
|
||||
let source = dr_gpu::DemosaicedImage::from_rgba8(&ctx, &rgba, SIZE, SIZE).expect("upload");
|
||||
let rgb: Vec<f32> = rgba
|
||||
.chunks_exact(4)
|
||||
.flat_map(|p| [p[0] as f32 / 255.0, p[1] as f32 / 255.0, p[2] as f32 / 255.0])
|
||||
.collect();
|
||||
|
||||
let mut coarse = Options {
|
||||
semantic: false,
|
||||
..Options::default()
|
||||
};
|
||||
coarse.segment.max_edge = 96;
|
||||
|
||||
let mut fine = coarse;
|
||||
fine.segment.max_edge = 192;
|
||||
|
||||
let a = compute(&ctx, &source, &rgb, SIZE as usize, SIZE as usize, &coarse).expect("a");
|
||||
let b = compute(&ctx, &source, &rgb, SIZE as usize, SIZE as usize, &fine).expect("b");
|
||||
assert_ne!(
|
||||
a.signature(),
|
||||
b.signature(),
|
||||
"ids from one proxy must not be read as ids from another"
|
||||
);
|
||||
}
|
||||
|
||||
/// Run-to-run stability on one device, which is what lets a selection be
|
||||
/// stored and reopened at all (M5).
|
||||
#[test]
|
||||
fn the_same_image_segments_identically_twice() {
|
||||
let Some(ctx) = context() else {
|
||||
eprintln!("no adapter; skipping");
|
||||
return;
|
||||
};
|
||||
|
||||
const SIZE: u32 = 128;
|
||||
let rgba = disc(SIZE);
|
||||
let source = dr_gpu::DemosaicedImage::from_rgba8(&ctx, &rgba, SIZE, SIZE).expect("upload");
|
||||
let rgb: Vec<f32> = rgba
|
||||
.chunks_exact(4)
|
||||
.flat_map(|p| [p[0] as f32 / 255.0, p[1] as f32 / 255.0, p[2] as f32 / 255.0])
|
||||
.collect();
|
||||
let options = Options {
|
||||
semantic: false,
|
||||
..Options::default()
|
||||
};
|
||||
|
||||
let a = compute(&ctx, &source, &rgb, SIZE as usize, SIZE as usize, &options).expect("a");
|
||||
let b = compute(&ctx, &source, &rgb, SIZE as usize, SIZE as usize, &options).expect("b");
|
||||
|
||||
assert_eq!(a.signature(), b.signature());
|
||||
assert_eq!(a.region_count(), b.region_count());
|
||||
assert_eq!(a.regions_at(0.5, 0.5), b.regions_at(0.5, 0.5));
|
||||
}
|
||||
}
|
||||
|
||||
+74
-64
@@ -96,6 +96,10 @@ component MaskEntry inherits Rectangle {
|
||||
}
|
||||
|
||||
Caption {
|
||||
// Elided rather than wrapped: this sits beside two fixed
|
||||
// targets in a row, and wrapping would give the rows of a
|
||||
// stack different heights for no gain.
|
||||
overflow: elide;
|
||||
// Three states worth distinguishing, and each has a
|
||||
// different remedy: stale needs the segmentation re-run,
|
||||
// unadjusted needs a slider moved, and the ordinary case
|
||||
@@ -194,6 +198,13 @@ export component MaskPanel inherits Rectangle {
|
||||
|
||||
background: Theme.surface;
|
||||
|
||||
// **Flat, deliberately.** An earlier version grouped each state into its
|
||||
// own `if ...: VerticalLayout { ... }`. The nested layout under-reported
|
||||
// its height, so the divider and the mask row below it were placed over
|
||||
// the "Find regions" button — the sort of fault that is invisible in the
|
||||
// source and obvious the moment anyone looks at the panel. Conditioning
|
||||
// each child on its own keeps every element a direct child of one layout,
|
||||
// which is the layout that measures them.
|
||||
VerticalLayout {
|
||||
padding: Theme.gap;
|
||||
spacing: Theme.gap-sm;
|
||||
@@ -207,73 +218,70 @@ export component MaskPanel inherits Rectangle {
|
||||
|
||||
if !root.enabled: Caption { text: "No image"; }
|
||||
|
||||
// --- the region map ------------------------------------------------
|
||||
// --- the region map ---------------------------------------------
|
||||
//
|
||||
// Segmentation is a thing the user starts. Half a second of watershed
|
||||
// and inference is too long to spend on every photograph opened, and
|
||||
// most edits never need one.
|
||||
if root.enabled && !root.segmented: VerticalLayout {
|
||||
if root.enabled && !root.segmented: Caption {
|
||||
text: "Find the regions in this photograph, so a mask can snap to them.";
|
||||
// Wrapped, or this sentence's single-line width becomes the
|
||||
// panel's minimum and levers the fixed 280px develop column open,
|
||||
// taking every other panel's controls off the right edge with it.
|
||||
wrap: word-wrap;
|
||||
}
|
||||
|
||||
if root.enabled && !root.segmented: Button {
|
||||
text: root.segmenting ? "Finding regions…" : "Find regions";
|
||||
enabled: !root.segmenting;
|
||||
primary: true;
|
||||
clicked => { root.segment(); }
|
||||
}
|
||||
|
||||
// The overlay the whole map is judged by, and the picking mode that
|
||||
// makes a click mean "select" instead of "pan".
|
||||
if root.enabled && root.segmented: HorizontalLayout {
|
||||
spacing: Theme.gap-sm;
|
||||
|
||||
Caption {
|
||||
text: "Find the regions in this photograph, so a mask can snap to them.";
|
||||
Button {
|
||||
text: "Overlay";
|
||||
active: root.overlay;
|
||||
clicked => { root.overlay-toggled(!root.overlay); }
|
||||
}
|
||||
|
||||
Button {
|
||||
text: root.segmenting ? "Finding regions…" : "Find regions";
|
||||
enabled: !root.segmenting;
|
||||
primary: true;
|
||||
clicked => { root.segment(); }
|
||||
text: "Select";
|
||||
active: root.picking;
|
||||
clicked => { root.picking-toggled(!root.picking); }
|
||||
}
|
||||
}
|
||||
|
||||
if root.enabled && root.segmented && root.picking: Caption {
|
||||
text: "Click the photograph to select a region. Shift-click to add or remove.";
|
||||
wrap: word-wrap;
|
||||
}
|
||||
|
||||
// Granularity, labelled by what it does rather than by its number:
|
||||
// "detail" is what a photographer is choosing between, where "300
|
||||
// regions" is an implementation detail they would have to learn.
|
||||
if root.enabled && root.segmented: SliderRow {
|
||||
label: "Detail";
|
||||
value: root.level;
|
||||
default-value: 300;
|
||||
minimum: 8;
|
||||
maximum: 2000;
|
||||
changed(v) => { root.level-changed(v); }
|
||||
reset => { root.level-changed(300); }
|
||||
}
|
||||
|
||||
// --- what the model found ----------------------------------------
|
||||
if root.enabled && root.segmented && root.subjects.length > 0: Caption {
|
||||
text: "Recognised";
|
||||
}
|
||||
|
||||
if root.enabled && root.segmented: VerticalLayout {
|
||||
spacing: Theme.gap-sm;
|
||||
|
||||
HorizontalLayout {
|
||||
spacing: Theme.gap-sm;
|
||||
|
||||
// The overlay the whole map is judged by. Without it the
|
||||
// granularity control is a number with no visible meaning.
|
||||
Button {
|
||||
text: "Overlay";
|
||||
active: root.overlay;
|
||||
clicked => { root.overlay-toggled(!root.overlay); }
|
||||
}
|
||||
|
||||
Button {
|
||||
text: "Select";
|
||||
active: root.picking;
|
||||
clicked => { root.picking-toggled(!root.picking); }
|
||||
}
|
||||
}
|
||||
|
||||
if root.picking: Caption {
|
||||
text: "Click the photograph to select a region. Shift-click to add or remove.";
|
||||
}
|
||||
|
||||
// Granularity. Labelled by what it does rather than by its number:
|
||||
// "detail" is what a photographer is choosing between, where "300
|
||||
// regions" is an implementation detail they would have to learn.
|
||||
SliderRow {
|
||||
label: "Detail";
|
||||
value: root.level;
|
||||
default-value: 300;
|
||||
minimum: 8;
|
||||
maximum: 2000;
|
||||
changed(v) => { root.level-changed(v); }
|
||||
reset => { root.level-changed(300); }
|
||||
}
|
||||
}
|
||||
|
||||
// --- what the model found -------------------------------------------
|
||||
if root.enabled && root.segmented && root.subjects.length > 0: VerticalLayout {
|
||||
spacing: 2px;
|
||||
padding-top: Theme.gap-sm;
|
||||
|
||||
Caption { text: "Recognised"; }
|
||||
|
||||
for subject[i] in root.subjects: subject-row := TouchArea {
|
||||
spacing: 0px;
|
||||
for subject in root.subjects: subject-row := TouchArea {
|
||||
height: Theme.touch-target;
|
||||
mouse-cursor: pointer;
|
||||
clicked => { root.add-subject(subject.index); }
|
||||
@@ -285,6 +293,7 @@ export component MaskPanel inherits Rectangle {
|
||||
text: subject.label;
|
||||
emphasised: subject-row.has-hover;
|
||||
horizontal-stretch: 1;
|
||||
overflow: elide;
|
||||
}
|
||||
Value { text: round(subject.score * 100) + "%"; }
|
||||
}
|
||||
@@ -296,15 +305,15 @@ export component MaskPanel inherits Rectangle {
|
||||
background: Theme.rule;
|
||||
}
|
||||
|
||||
// --- the stack -------------------------------------------------------
|
||||
// --- the stack -----------------------------------------------------
|
||||
Caption { text: "Masks"; }
|
||||
|
||||
// Gradients need no segmentation, so they are offered whether or not
|
||||
// one has been computed — a graduated filter over a sky is a local
|
||||
// adjustment that never needed to know what a sky is.
|
||||
HorizontalLayout {
|
||||
spacing: Theme.gap-sm;
|
||||
|
||||
Caption { text: "Masks"; horizontal-stretch: 1; }
|
||||
|
||||
// Gradients need no segmentation, so they are offered whether or
|
||||
// not one has been computed — a graduated filter over a sky is a
|
||||
// local adjustment that never needed to know what a sky is.
|
||||
Button {
|
||||
text: "Linear";
|
||||
enabled: root.enabled;
|
||||
@@ -319,6 +328,7 @@ export component MaskPanel inherits Rectangle {
|
||||
|
||||
if root.enabled && root.masks.length == 0: Caption {
|
||||
text: "No local adjustments. Select a region, or add a gradient.";
|
||||
wrap: word-wrap;
|
||||
}
|
||||
|
||||
for mask in root.masks: MaskEntry {
|
||||
@@ -331,14 +341,14 @@ export component MaskPanel inherits Rectangle {
|
||||
opacity-changed(v) => { root.mask-opacity-changed(mask.id, v); }
|
||||
}
|
||||
|
||||
// The one thing the panel has to say about the adjust panel below
|
||||
// it, because otherwise selecting a layer silently changes what those
|
||||
// sliders mean — the single most confusing thing a scoped panel can
|
||||
// do.
|
||||
// The one thing this panel has to say about the adjust panel below it,
|
||||
// because otherwise selecting a layer silently changes what those
|
||||
// sliders mean — the most confusing thing a scoped panel can do.
|
||||
if root.enabled && root.masks.length > 0: Caption {
|
||||
text: root.editing-mask
|
||||
? "The controls below adjust the selected mask."
|
||||
: "The controls below adjust the whole photograph.";
|
||||
wrap: word-wrap;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user