Show the detector the photograph, not the sensor's scanlines
🐳 Android image / Build and push (push) Successful in 1s
Build and test / android-image (push) Successful in 1s
Build and test / Desktop (Linux) (push) Successful in 19m7s
Build and test / Layer separation (push) Successful in 25s
Traceability / Requirement traces (push) Successful in 23s
Build and test / Android (aarch64) (push) Failing after 33m10s
🐳 Android image / Build and push (push) Successful in 1s
Build and test / android-image (push) Successful in 1s
Build and test / Desktop (Linux) (push) Successful in 19m7s
Build and test / Layer separation (push) Successful in 25s
Traceability / Requirement traces (push) Successful in 23s
Build and test / Android (aarch64) (push) Failing after 33m10s
"Find subjects" was handed the proxy in the sensor's own orientation, so every frame shot on a body held sideways reached the model lying on its side — and a model trained on upright photographs is very bad at those. Measured end to end on a 22 MP frame of two people and a dog: `person 0.36` and nothing else, against `dog 0.82, person 0.61, person 0.49` for the same pixels stood up. Nothing failed; the panel simply offered one poor subject where there were three good ones. The orientation was never dropped on purpose. The proxy is deliberately rendered through a *neutral* graph — the detection has to survive an exposure change, or every slider would invalidate the masks built on it — and neutral took the file's orientation with it along with everything else. Landscape frames were unaffected, which is why it stood for as long as it did. The turn is `Orientation::source_pixel`, the same function the grid's thumbnails already go through, so the detector and the thumbnailer now agree about which way is up rather than holding two opinions. What it is turned by is `Framing::effective_orientation` — the file's EXIF tag and the photographer's own rotations composed into one permutation, by the group law rather than by adding the turns, which is a distinction `Framing` already had to make and had already tested. Rotating the picture and pressing the button again therefore does what it looks like it does. The proxy stays in sensor space and the masks come back into it. That is not a detail to be tidied later: the generated shader samples the mask array at `uv_src`, *after* the framing map, so a mask stored upright would sit a quarter turn off the subject it was drawn around. That is a wrong mask rather than a weak one, and nothing announces it. So the picture is stood up for the model and laid back down for everything else, and `upright`/`lay_down` are returned as a pair because calling one and forgetting the other is silent. Both directions are the one function: `upright` gathers through `source_pixel` and `lay_down` scatters through it. A quarter turn is a bijection of the pixel grid, so the round trip is exact — no filter, no resampling, and no hole to fill — and an inverse written out by hand would be a second thing to keep in step, whose way of being wrong is a mask mirrored about the wrong axis, which still looks like a mask. The orientation joins the confidence and the tiling flag in the segmentation signature, and for the same reason: turning the photograph changes what the model recognises, so two runs either side of a rotation are different instance lists. Two that happened to come out the same length would otherwise share a signature and a stored layer would be silently re-indexed from one into the other. The refine pass had it too — it re-runs the model over a crop rendered in the same sensor space — so it makes the same turn, and would otherwise have handed back a worse mask than the one it was asked to improve, on the subject the photographer had just pointed at. `dr-gpu`'s `local` example is fixed with it. It exists to be the shipping path with pictures attached, and a diagnostic that reproduces the bug it is meant to catch is a trap for whoever reads it next. Seven tests. The round trip is the identity over all eight EXIF tags on a non-square asymmetric grid; a turn carries whole pixels rather than shearing the channels apart; a sideways frame reaches the model upright; a box comes back in sensor pixels, worked out by hand for the one turn a portrait frame actually writes; a restored box still reads low-to-high for every tag, since the rest of the pipeline takes `x1 - x0` without checking the sign; and the eight tags cannot collapse into one signature key. The existing composition test now runs against `effective_orientation` itself, over all 8 x 16 baseline-and-user pairs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+50
-8
@@ -116,6 +116,18 @@ pub struct SegmentationJob {
|
||||
source: Arc<DemosaicedImage>,
|
||||
session: SessionId,
|
||||
abandon: Abandon,
|
||||
/// TRACES: FR-DEV-3h
|
||||
/// How the sensor's pixels have to be turned to be the photograph.
|
||||
///
|
||||
/// The file's EXIF tag and the photographer's own turns, composed into
|
||||
/// one permutation by `Framing::effective_orientation`. Carried rather
|
||||
/// than read from the session for the reason everything else here is: the
|
||||
/// job runs on a worker and the session stays behind.
|
||||
///
|
||||
/// A *snapshot*, so turning the photograph while a run is in flight
|
||||
/// leaves that run answering the question it was asked. The next press
|
||||
/// takes the new one, and the signature says the two are different runs.
|
||||
orientation: dr_types::Orientation,
|
||||
}
|
||||
|
||||
impl SegmentationJob {
|
||||
@@ -158,7 +170,7 @@ impl SegmentationJob {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let seg = segmentation::compute(&self.ctx, &rgb, rw, rh, options)?;
|
||||
let seg = segmentation::compute(&self.ctx, &rgb, rw, rh, self.orientation, options)?;
|
||||
log::info!(
|
||||
"segmented {rw}×{rh}: {} subject(s), proxy {:.0} ms, total {:.0} ms",
|
||||
seg.instances().len(),
|
||||
@@ -178,6 +190,14 @@ impl SegmentationJob {
|
||||
/// segmentation must survive an exposure change, or every slider would
|
||||
/// invalidate the masks that depend on it (docs/segmentation.md §3).
|
||||
///
|
||||
/// **Still the sensor's orientation, deliberately.** Standing the picture
|
||||
/// up is what `segmentation::compute` does to the buffer this returns,
|
||||
/// and it is done there rather than here because a mask has to come back
|
||||
/// in this space: the generated shader samples the mask array at `uv_src`,
|
||||
/// after the framing map. Rendering an upright proxy would put every mask
|
||||
/// a quarter turn away from the subject it was drawn around — a wrong
|
||||
/// mask rather than a weak one, and nothing would announce it.
|
||||
///
|
||||
/// A throwaway [`AdjustPass`] with a neutral graph rather than the
|
||||
/// session's own — which this could not reach from here in any case, and
|
||||
/// must not: reusing it would overwrite the frame the histogram reads and
|
||||
@@ -256,6 +276,11 @@ pub struct RefineJob {
|
||||
class_name: Arc<str>,
|
||||
bbox: (f32, f32, f32, f32),
|
||||
proxy: (usize, usize),
|
||||
/// The same permutation, for the same reason — see
|
||||
/// [`SegmentationJob::orientation`]. A refine pass that read the crop
|
||||
/// sideways would hand back a worse mask than the one it was asked to
|
||||
/// improve, on the subject the photographer had just pointed at.
|
||||
orientation: dr_types::Orientation,
|
||||
}
|
||||
|
||||
impl RefineJob {
|
||||
@@ -319,20 +344,30 @@ impl RefineJob {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
// Stood up before the model reads it and laid back down after, the
|
||||
// same way the whole-frame pass does it — `segmentation::upright` is
|
||||
// the one place that permutation is written. A crop rendered in
|
||||
// sensor space is exactly as sideways as the frame it came from.
|
||||
let (upright, uw, uh) = segmentation::upright(&rgb, rw, rh, self.orientation);
|
||||
|
||||
let mut model = dr_segment::SemanticModel::embedded().map_err(|e| e.to_string())?;
|
||||
let options = dr_segment::SemanticOptions {
|
||||
tiling: dr_segment::Tiling::Whole,
|
||||
..dr_segment::SemanticOptions::default()
|
||||
};
|
||||
let found = model
|
||||
.detect(&rgb, rw, rh, &options)
|
||||
.detect(&upright, uw, uh, &options)
|
||||
.map_err(|e| e.to_string())?;
|
||||
|
||||
// The crop was built around one subject, so the right answer among
|
||||
// whatever the model found in it is the same class closest to the
|
||||
// window's centre — not merely the highest score, which a second,
|
||||
// unrelated instance caught in the padding could win.
|
||||
let (cx, cy) = (rw as f32 * 0.5, rh as f32 * 0.5);
|
||||
//
|
||||
// Measured in the upright frame, which is where the model's boxes are.
|
||||
// The centre is the centre either way; the distances are not, once the
|
||||
// window is not square.
|
||||
let (cx, cy) = (uw as f32 * 0.5, uh as f32 * 0.5);
|
||||
let best = found
|
||||
.into_iter()
|
||||
.filter(|i| i.class_name.as_ref() == self.class_name.as_ref())
|
||||
@@ -342,15 +377,20 @@ impl RefineJob {
|
||||
return Ok(None);
|
||||
};
|
||||
|
||||
// Back into the crop's own sensor-space pixels, so everything below
|
||||
// this line measures in the space `self.bbox` and `self.proxy` are in.
|
||||
let (crop_mask, crop_bbox) =
|
||||
segmentation::lay_down(&instance.mask, instance.bbox, uw, uh, self.orientation);
|
||||
|
||||
// Downsampled back onto the shared proxy grid like every other
|
||||
// instance's mask is, but built from a sharper source than the
|
||||
// whole-frame pass ever saw for this subject.
|
||||
let mask = paste_into_proxy(&instance.mask, rw, rh, self.proxy, (x0, y0), (cw_px, ch_px));
|
||||
let mask = paste_into_proxy(&crop_mask, rw, rh, self.proxy, (x0, y0), (cw_px, ch_px));
|
||||
let bbox = (
|
||||
x0 + instance.bbox.0 / scale,
|
||||
y0 + instance.bbox.1 / scale,
|
||||
x0 + instance.bbox.2 / scale,
|
||||
y0 + instance.bbox.3 / scale,
|
||||
x0 + crop_bbox.0 / scale,
|
||||
y0 + crop_bbox.1 / scale,
|
||||
x0 + crop_bbox.2 / scale,
|
||||
y0 + crop_bbox.3 / scale,
|
||||
);
|
||||
|
||||
Ok(Some(RefinedInstance {
|
||||
@@ -1639,6 +1679,7 @@ impl DevelopSession {
|
||||
source: self.demosaiced.clone(),
|
||||
session: self.id,
|
||||
abandon: Abandon::default(),
|
||||
orientation: self.graph.framing().effective_orientation(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1684,6 +1725,7 @@ impl DevelopSession {
|
||||
class_name: instance.class_name.clone(),
|
||||
bbox: instance.bbox,
|
||||
proxy: seg.proxy_size(),
|
||||
orientation: self.graph.framing().effective_orientation(),
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user