Weigh the categories in the same precompute, and mask with them

The scene model shipped with a decoder and no caller. This runs it.

## Beside the instance pass, not instead of it

`compute` now does both on the same upright frame and lays both back down
the same way, so instance masks and category masks index into one grid —
the sensor's. A failure in the scene half is logged and dropped rather than
propagated: no scene model is an ordinary state, and a photograph that can
still be masked by subject should not become unopenable because the
categories are missing.

Categories under half a percent of the frame never reach the cache. A
control that does nothing when moved is worse than an absent one, and each
one it skips is a proxy-sized buffer not allocated.

## The shader needed nothing

A category reaches `dr-gpu` as a soft coverage buffer at proxy resolution,
turned into a distance field — which is exactly what a subject is. So they
share `MODE_SUBJECT`. That is not a shortcut taken for speed: the shader has
no way to tell them apart and no reason to want one. What differs is only
which model produced the coverage, and that has already happened by then.

Feather, falloff, dilation and erosion therefore work on a category on the
day it arrives, because they were never subject-specific.

## Where the weights come from

`scene-model` compiles the graph in and the desktop app takes it; Android
leaves it off and reads the copy `install_bundled_models` unpacks, because
24 MB of constant is worth avoiding in a mobile install and not worth the
plumbing to avoid on a desktop one. Embedded is tried first — a build that
has the weights compiled in should not be silently overridden by a stale
file in a data directory.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-30 14:02:22 +02:00
co-authored by Claude Opus 5
parent 485297f0c6
commit 763dfd353a
5 changed files with 235 additions and 3 deletions
+136
View File
@@ -36,6 +36,23 @@ use dr_gpu::GpuContext;
use dr_pipeline::mask::segmentation_signature;
use dr_types::Orientation;
/// One photographic category, over the whole frame.
///
/// The counterpart to [`InstanceSummary`] and deliberately thinner: a category
/// has no box, because it is not one object in one place — sky is wherever the
/// sky is, in as many disconnected pieces as the frame has windows.
#[derive(Debug, Clone)]
pub struct CategorySummary {
pub name: std::sync::Arc<str>,
/// Fraction of the frame this category covers, for ordering the list and
/// for hiding a category that would give the user a control that does
/// nothing.
pub coverage: f32,
/// Coverage at proxy resolution, quantised to a byte — the same
/// representation, and for the same reasons, as `InstanceSummary::mask`.
pub mask: Vec<u8>,
}
/// One recognised object.
#[derive(Debug, Clone)]
pub struct InstanceSummary {
@@ -61,6 +78,14 @@ pub struct InstanceSummary {
/// One image's recognised objects, ready to mask.
pub struct Segmentation {
instances: Vec<InstanceSummary>,
/// What the scene model made of the same frame, empty when no scene model
/// could be found.
///
/// Empty is an ordinary state, not a failure: a build without the weights
/// compiled in and without them installed simply offers no categories, the
/// same way a missing face model turns face indexing off rather than
/// stopping the app.
categories: Vec<CategorySummary>,
/// Identifies this run, so a stored layer can tell whether the index it
/// holds still means what it meant.
signature: u64,
@@ -90,6 +115,21 @@ impl Segmentation {
self.instances.get(index).map(|i| i.mask.as_slice())
}
pub fn categories(&self) -> &[CategorySummary] {
&self.categories
}
/// One category's coverage, at [`Self::proxy_size`].
///
/// By name, matching `MaskSource::Category`. A linear scan because there
/// are eight of them and a map would be more machinery than lookup.
pub fn category_mask(&self, name: &str) -> Option<&[u8]> {
self.categories
.iter()
.find(|c| &*c.name == name)
.map(|c| c.mask.as_slice())
}
/// Replace one instance in place, keeping every other index and the
/// signature unchanged.
///
@@ -309,8 +349,21 @@ pub fn compute(
^ orientation_key(orientation),
);
// The scene pass, on the same upright frame and laid back down the same
// way. Failures here are logged and dropped rather than propagated: no
// scene model is an ordinary state, and a photograph that can be masked by
// subject should not become unopenable because the categories are absent.
let categories = match scene_categories(&stood_up, uw, uh, orientation) {
Ok(c) => c,
Err(e) => {
log::info!("no scene categories for this frame: {e}");
Vec::new()
}
};
Ok(Segmentation {
instances,
categories,
signature,
// **Sensor space, not the model's.** `lay_down` put every mask back,
// so the grid a stored layer indexes into is the one it always was —
@@ -475,6 +528,86 @@ fn detect(
.map_err(|e| e.to_string())
}
/// Weigh the photographic categories, if this build can find a scene model.
///
/// Separate from [`detect`] rather than folded into it because the two are
/// independent: a build with no scene model still segments subjects, and a
/// frame with no recognisable subject still has sky. Neither failure should
/// take the other down.
fn scene_categories(
rgb: &[f32],
width: usize,
height: usize,
orientation: Orientation,
) -> Result<Vec<CategorySummary>, String> {
let mut model = load_scene_model()?;
let scene = model
.analyse(rgb, width, height)
.map_err(|e| e.to_string())?;
let mut out = Vec::new();
for (index, name) in scene.categories().iter().enumerate() {
let coverage = scene.coverage(index);
// A category the model barely saw is not worth a mask buffer the size
// of the proxy, and offering it in the list would be offering a
// control that does nothing when moved. The threshold is the one the
// example prints against.
if coverage < 0.005 {
continue;
}
let Some(weights) = scene.rasterise(index, width, height) else {
continue;
};
// Back into sensor space, exactly as an instance mask is: the grid a
// stored layer indexes into has to be the sensor's whatever the model
// was shown. `lay_down` wants a box too, so it gets the whole frame —
// a category has no meaningful extent.
let (mask, _) = lay_down(
&weights,
(0.0, 0.0, width as f32, height as f32),
width,
height,
orientation,
);
out.push(CategorySummary {
name: name.clone(),
coverage,
mask: quantise(&mask),
});
}
// Largest first, which is the order the list is worth reading in.
out.sort_by(|a, b| b.coverage.total_cmp(&a.coverage));
Ok(out)
}
/// Find a scene model: compiled in if this build has one, installed otherwise.
///
/// The order matters. A build with the weights compiled in should not be
/// silently overridden by a stale file in a data directory, and a build
/// without them has nothing to fall back *from* — so "embedded, then
/// installed" is the only ordering that is not surprising either way.
fn load_scene_model() -> Result<dr_segment::SceneModel, String> {
#[cfg(feature = "scene-model")]
{
return dr_segment::SceneModel::embedded().map_err(|e| e.to_string());
}
#[cfg(not(feature = "scene-model"))]
{
// Account-independent, like `shared_face_models_dir` and for the same
// reason: this runs on a worker with no session in hand. Android
// unpacks the APK's copy to exactly this directory before any store
// opens.
let dir = crate::library::shared_face_models_dir();
dr_segment::SceneModel::from_path(
dir.join("yolo26s-sem-ade20k.onnx"),
dir.join("yolo26s-sem-ade20k.classes.json"),
dir.join("categories.txt"),
)
.map_err(|e| e.to_string())
}
}
/// The model's soft coverage, to a byte per pixel.
///
/// Rounded rather than truncated, so a coverage of exactly 0.5 lands on the
@@ -519,6 +652,7 @@ mod tests {
bbox: (4.0, 0.0, 8.0, h as f32),
},
],
categories: Vec::new(),
signature: 1,
proxy: (w, h),
}
@@ -555,6 +689,7 @@ mod tests {
fn a_click_on_nothing_selects_nothing() {
let seg = Segmentation {
instances: Vec::new(),
categories: Vec::new(),
signature: 1,
proxy: (4, 4),
};
@@ -582,6 +717,7 @@ mod tests {
fn the_overlay_is_transparent_where_nothing_was_found() {
let seg = Segmentation {
instances: Vec::new(),
categories: Vec::new(),
signature: 1,
proxy: (4, 4),
};