Decode the scene model into per-category weights

The weights landed last commit with nothing to read them. This is the
decoder, and the shape of it follows from one property worth stating
before the code: the categories must partition the image.

## Why a partition, and not a mask per category

The scene tab applies one grade to every pixel of a category — lift the
sky, desaturate foliage — and both grades meet at the horizon. If each
category carried an independent mask, feathering them outward would make
the boundary band belong to both, so both grades would land there and
every horizon would acquire a visible seam. Feathering has to *blend*
there, not accumulate.

So `marginalise` takes one softmax over all 150 channels and sums within
each category. Grouping cannot change a total of one, so the listed
categories plus the unlisted remainder sum to one at every pixel, by
construction rather than by normalising afterwards. `parse_categories`
refuses a descriptor that claims a class twice, because that is the one
input that would quietly make the property untrue.

## The descriptor is data, and hand-written

`models/scene/categories.txt` groups ADE20K's 150 classes into the eight
a photographer would recognise. It is a file rather than a table in Rust
for the reason `models/LICENCE.md` predicted — a vocabulary is model
metadata — and it is line-oriented with comments rather than JSON like
the `.classes.json` beside it, because that file is generated and this
one is argued. Why `swimming pool` is water and not architecture belongs
next to the line that says so.

Classes are named, not indexed. An index is silently wrong after a
re-export; a name is loudly wrong, and the loader refuses one the model
does not have.

## Resolution, kept visible

`Scene` holds the native 80×80 logit grid and resamples on demand rather
than upsampling once at load. The coarseness is real — it is what the
graph produces — and a type that hides it behind an early resize invites
callers to expect detail that was never there. `rasterise` is where the
letterbox inverse lives, once.

`Letterbox` and `Window` become `pub(crate)` and `to_proto` generalises
to `to_grid`, because both dense outputs this crate reads are an even
fraction of the same letterboxed square and differ only in the divisor.

## Verified by looking, which is the only way this gets verified

`examples/scene.rs` writes the photograph dimmed outside each category. A
transposed axis or an off-by-one in the inverse produces perfectly
plausible weights over slightly the wrong pixels, and no unit test
catches that. On an indoor frame the person mask lands on the person,
including the outstretched arm, and sky reads ~5% against a bright
ceiling.

It doubles as the benchmark, because every timing quoted while this model
was chosen came off a laptop compiling other things and none of them
belong in a document.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-30 10:48:37 +02:00
co-authored by Claude Opus 5
parent 79b7f54e04
commit 8df6000e4b
8 changed files with 922 additions and 56 deletions
+30 -13
View File
@@ -446,16 +446,16 @@ fn decode(
/// A source-space rectangle fed through one inference.
#[derive(Debug, Clone, Copy)]
struct Window {
x: f32,
y: f32,
w: f32,
h: f32,
pub(crate) struct Window {
pub(crate) x: f32,
pub(crate) y: f32,
pub(crate) w: f32,
pub(crate) h: f32,
}
/// The scale-and-pad that fits an arbitrary rectangle into the square input.
#[derive(Debug, Clone, Copy)]
struct Letterbox {
pub(crate) struct Letterbox {
/// Input pixels per source pixel.
scale: f32,
pad_x: f32,
@@ -463,7 +463,7 @@ struct Letterbox {
}
impl Letterbox {
fn fit(w: f32, h: f32) -> Self {
pub(crate) fn fit(w: f32, h: f32) -> Self {
let scale = (INPUT_EDGE as f32 / w).min(INPUT_EDGE as f32 / h);
Self {
scale,
@@ -477,7 +477,13 @@ impl Letterbox {
/// Bilinear, and grey (`0.5`) in the padding — the value the network sees
/// least as an edge, where black would draw a hard border across the frame
/// and invite a detection along it.
fn sample(&self, rgb: &[f32], width: usize, height: usize, window: &Window) -> Array4<f32> {
pub(crate) fn sample(
&self,
rgb: &[f32],
width: usize,
height: usize,
window: &Window,
) -> Array4<f32> {
let mut input = Array4::<f32>::from_elem((1, 3, INPUT_EDGE, INPUT_EDGE), 0.5);
for iy in 0..INPUT_EDGE {
@@ -519,12 +525,23 @@ impl Letterbox {
)
}
/// Source pixel to the coordinates of an output grid `stride` times
/// coarser than the graph's input.
///
/// Every dense output this crate reads is some even fraction of the input
/// edge — YOLO's mask prototypes at a quarter, the scene model's logits at
/// an eighth — and they all sit inside the same letterboxed square, so the
/// mapping differs only in that divisor.
pub(crate) fn to_grid(self, sx: f32, sy: f32, w: &Window, stride: f32) -> (f32, f32) {
(
((sx - w.x) * self.scale + self.pad_x) / stride,
((sy - w.y) * self.scale + self.pad_y) / stride,
)
}
/// Source pixel to prototype-grid coordinates.
fn to_proto(self, sx: f32, sy: f32, w: &Window) -> (f32, f32) {
(
((sx - w.x) * self.scale + self.pad_x) / PROTO_STRIDE as f32,
((sy - w.y) * self.scale + self.pad_y) / PROTO_STRIDE as f32,
)
self.to_grid(sx, sy, w, PROTO_STRIDE as f32)
}
}
@@ -648,7 +665,7 @@ fn steps(extent: f32, edge: f32, stride: f32) -> usize {
}
/// Point `ort` at tract, exactly once per process.
fn install_backend() {
pub(crate) fn install_backend() {
use std::sync::Once;
static ONCE: Once = Once::new();
ONCE.call_once(|| {