dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Leave a panorama frame out without leaving the page

Browse Source 
A frame that did not fit ended the job with its name, and the only way
on was Back, a smaller selection and every frame read, demosaiced and
searched for keypoints again. Each row on the page now has a box. An
unticked frame is left out and the rest are solved again from what the
first pass measured.

dr_pano::align is split for it: match_pairs does the matching and the
pairwise RANSAC once over every frame (about 5 s for twelve), and solve
takes a subset of the frames and uses only the links among them (about
0.1 s). Solving a subset by re-aligning it also moved every RANSAC seed,
which are keyed on frame position, and on the fixture set that was
enough to lose a marginal link and strand a neighbour of the frame left
out.

A frame that cannot be placed no longer stops the job either. Its row
names it and Merge stays off until it is unticked; the headless example
leaves such frames out the same way, and takes --leave-out N to try it.
This commit is contained in:
dtourolle
2026-09-27 16:29:39 -04:00
parent e98b1def98
commit 621a6b8313
9 changed files with 616 additions and 202 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ui/dr-ui/src/merge.rs
+290 -117
View File
@@ -14,8 +14,10 @@
//! (FR-MRG-2).
//! 2. **Proxies.** Each frame is demosaiced and rendered through the tap
//! at proxy size, upright; the keypoint detector reads that.
//! 3. **Alignment.** `dr_pano::align`. A frame it could not place stops
//! the job with the frame named (FR-MRG-5).
//! 3. **Alignment.** `dr_pano::match_pairs` once, then `dr_pano::solve`
//! over the frames kept — again whenever the page leaves one out. A
//! frame it could not place is named (FR-MRG-5), and the merge waits
//! until it is left out.
//! 4. **Gain.** One scalar per frame from the proxies' overlaps, so the
//! stop of exposure drift a hand-held sweep collects does not band.
//! 5. **Merge.** `dr_gpu::MergePass`, chunk by chunk, into a DNG written
@@ -124,7 +126,7 @@ impl MergeRequest {
/// What the photographer decides once the alignment is shown (FR-MRG-1:
/// never automatic).
#[derive(Debug, Clone, Copy, PartialEq)]
#[derive(Debug, Clone, PartialEq)]
pub enum Decision {
Merge {
projection: Option<Projection>,
@@ -142,6 +144,17 @@ pub enum Decision {
projection: Option<Projection>,
fill: Option<FillSettings>,
},
/// Align again over these frames — one flag per input frame, `true` to
/// keep it — and report, drawn as `projection` and `fill` say. The
/// page's checkboxes: a frame that does not fit is left out without
/// leaving the page, and the frames are not read, demosaiced or
/// searched for keypoints again; only the matching, the solve and the
/// gains run over the ones kept.
Frames {
keep: Vec<bool>,
projection: Option<Projection>,
fill: Option<FillSettings>,
},
Abandon,
}
@@ -248,8 +261,10 @@ pub struct AlignmentReport {
pub width: u32,
pub height: u32,
/// Per frame, in input order: yaw and pitch in degrees if aligned, or
/// why not.
/// why not — "left out" for a frame the photographer unticked.
pub frames: Vec<Result<(f64, f64), String>>,
/// Per frame, in input order: whether it is in this alignment.
pub included: Vec<bool>,
pub links: usize,
/// The aligned set drawn on the suggested surface at proxy resolution,
/// `(width, height, rgba)` — what the photographer confirms.
@@ -260,6 +275,19 @@ pub struct AlignmentReport {
pub filler: Result<String, String>,
}
impl AlignmentReport {
/// Whether every frame kept was placed, and there are two of them: the
/// only alignment the job will merge.
pub fn is_complete(&self) -> bool {
self.included.iter().filter(|k| **k).count() >= 2
&& self
.frames
.iter()
.zip(&self.included)
.all(|(f, &kept)| !kept || f.is_ok())
}
}
/// Run the whole job on the calling thread, reporting on `events`.
///
/// After `Aligned` the job waits on `decision` — the photographer's
@@ -378,45 +406,17 @@ fn run_inner(
}
log::info!("merge: features in {:?}", t.elapsed());
// 3. Alignment.
let t = Instant::now();
// 3–4. Alignment and gains, over the frames the photographer keeps.
// Everything up to the pairs — each frame read, demosaiced, searched for
// keypoints, and every pair matched — is done once; unticking a frame
// on the page solves again from the pairs already measured.
progress("Aligning", 0, 1);
let alignment =
dr_pano::align(&features, &dr_pano::AlignOptions::default()).map_err(|e| e.to_string())?;
log::info!(
"merge: aligned in {:?}, focal {:.1} px, rms {:.2} px, {} links",
t.elapsed(),
alignment.focal,
alignment.rms_px,
alignment.links.len()
);
// The geometry at full resolution: the proxy's long edge against the
// frame's.
let proxy_long = proxies[0].width.max(proxies[0].height) as f64;
let full_long = frames[0].size.0.max(frames[0].size.1) as f64;
let focal_full = alignment.focal * full_long / proxy_long;
let cameras = Cameras {
rotations: alignment
.rotations
.iter()
.map(|r| r.unwrap_or(dr_pano::linalg::Mat3::IDENTITY))
.collect(),
focal: focal_full,
};
let frame_size = (frames[0].size.0 as f64, frames[0].size.1 as f64);
let (hfov, vfov) = field_of_view(&cameras, frame_size);
let projection = request
.projection
.unwrap_or_else(|| Projection::suggest(hfov, vfov));
let suggested = projection;
// 4. Gains, before the report so the preview shows them.
let gains = if alignment.is_complete() {
gains(&proxies, &alignment)
} else {
vec![1.0; frames.len()]
};
let t = Instant::now();
let pairs = dr_pano::match_pairs(&features, &dr_pano::AlignOptions::default())
.map_err(|e| e.to_string())?;
log::info!("merge: pairs matched in {:?}", t.elapsed());
let mut included = vec![true; frames.len()];
let mut solved = solve(&frames, &pairs, &proxies, &included, request.projection)?;
// The filler, loaded once if the device has it. Its absence is a
// report, not a failure: the crop is always there.
let mut filler: Option<dr_pano::migan::MiGan> = None;
@@ -431,79 +431,94 @@ fn run_inner(
Err(e) => Err(format!("the border-fill model could not be loaded: {e}")),
},
};
let first_raw = &frames[0].raw;
let mut report =
|projection: Projection, fill: Option<FillSettings>| -> Result<AlignmentReport, String> {
let bounds = projection::bounds(projection, focal_full, &cameras, frame_size)
.ok_or("the frames project nowhere")?;
let mut filled = false;
let preview = match alignment.is_complete() {
false => None,
true => {
let (w, h, mut rgb, mut known) = preview_planes(
&colour, &proxies, &alignment, &gains, projection, first_raw, 1600,
);
if let Some(settings) = fill {
if let Some(model) = filler.as_mut() {
progress("Filling the preview", 0, 1);
// The knobs at the preview's scale.
let here = settings.at(w as f64 / bounds.width());
dr_pano::fill::erode(&mut known, w, h, here.erosion);
match dr_pano::fill_border(
&mut rgb,
w,
h,
&known,
model,
here.params,
&mut |_, _| {},
) {
Ok(_) => filled = true,
Err(e) => log::warn!("preview fill: {e}"),
}
let has_filler = filler.is_some();
let mut report = |s: &Solved,
included: &[bool],
projection: Projection,
fill: Option<FillSettings>|
-> Result<AlignmentReport, String> {
let bounds = projection::bounds(projection, s.focal_full, &s.cameras, s.frame_size)
.ok_or("the frames project nowhere")?;
let proxies: Vec<&Gray> = s.keep.iter().map(|&k| &proxies[k]).collect();
let colour: Vec<&[f32]> = s.keep.iter().map(|&k| colour[k].as_slice()).collect();
let first_raw = &frames[s.keep[0]].raw;
let mut filled = false;
let preview = match s.alignment.is_complete() {
false => None,
true => {
let (w, h, mut rgb, mut known) = preview_planes(
&colour,
&proxies,
&s.alignment,
&s.gains,
projection,
first_raw,
1600,
);
if let Some(settings) = fill {
if let Some(model) = filler.as_mut() {
progress("Filling the preview", 0, 1);
// The knobs at the preview's scale.
let here = settings.at(w as f64 / bounds.width());
dr_pano::fill::erode(&mut known, w, h, here.erosion);
match dr_pano::fill_border(
&mut rgb,
w,
h,
&known,
model,
here.params,
&mut |_, _| {},
) {
Ok(_) => filled = true,
Err(e) => log::warn!("preview fill: {e}"),
}
}
Some(preview_rgba(
&rgb,
w,
h,
&alignment,
projection,
&bounds_of(&proxies, &alignment, projection),
&proxies,
))
}
};
Ok(AlignmentReport {
focal_mm: focal_full * 36.0 / full_long,
rms_px: alignment.rms_px,
projection,
width: bounds.width().ceil() as u32,
height: bounds.height().ceil() as u32,
frames: describe(&alignment),
links: alignment.links.len(),
preview,
filled,
filler: filler_status.clone(),
})
Some(preview_rgba(
&rgb,
w,
h,
&s.alignment,
projection,
&bounds_of(&proxies, &s.alignment, projection),
&proxies,
))
}
};
// The solve's frames back in input order, the unticked ones named.
let mut rows: Vec<Result<(f64, f64), String>> =
vec![Err("left out".to_string()); included.len()];
for (&k, row) in s.keep.iter().zip(describe(&s.alignment)) {
rows[k] = row;
}
Ok(AlignmentReport {
focal_mm: s.focal_full * 36.0 / s.full_long,
rms_px: s.alignment.rms_px,
projection,
width: bounds.width().ceil() as u32,
height: bounds.height().ceil() as u32,
frames: rows,
included: included.to_vec(),
links: s.alignment.links.len(),
preview,
filled,
filler: filler_status.clone(),
})
};
progress("Drawing the preview", 0, 1);
let _ = events.send(MergeEvent::Aligned(report(suggested, None)?));
if !alignment.is_complete() {
let names: Vec<String> = alignment
.unaligned
.iter()
.map(|(k, why)| format!("{}: {why}", request.frames[*k].name))
.collect();
return Err(format!(
"not every frame could be placed — {}",
names.join("; ")
));
}
let _ = events.send(MergeEvent::Aligned(report(
&solved,
&included,
solved.suggested,
None,
)?));
// Never automatic (FR-MRG-1): nothing is written until the alignment
// has been seen and confirmed. Polled, so a cancel while waiting is
// seen within a moment.
// seen within a moment. A frame that could not be placed no longer
// ends the job (FR-MRG-5 names it on its row): the page leaves it out
// and the job aligns again without it.
let (projection, fill) = loop {
if cancel.is_cancelled() {
return Ok(None);
@@ -512,23 +527,76 @@ fn run_inner(
Ok(Decision::Merge {
projection: p,
fill,
}) => break (p.unwrap_or(suggested), fill.filter(|_| filler.is_some())),
}) => {
// The page's button is off until the alignment is
// complete; a press that raced a re-solve is dropped.
if !solved.alignment.is_complete() {
continue;
}
break (p.unwrap_or(solved.suggested), fill.filter(|_| has_filler));
}
Ok(Decision::Preview {
projection: p,
fill,
}) => {
let _ = events.send(MergeEvent::Aligned(report(p.unwrap_or(suggested), fill)?));
let _ = events.send(MergeEvent::Aligned(report(
&solved,
&included,
p.unwrap_or(solved.suggested),
fill,
)?));
}
Ok(Decision::Frames {
keep,
projection: p,
fill,
}) => {
// Two frames or it is not a panorama; the page does not
// offer fewer, and a malformed list is not acted on.
if keep.len() != frames.len() || keep.iter().filter(|k| **k).count() < 2 {
continue;
}
progress("Aligning", 0, 1);
included = keep;
solved = solve(&frames, &pairs, &proxies, &included, request.projection)?;
progress("Drawing the preview", 0, 1);
let _ = events.send(MergeEvent::Aligned(report(
&solved,
&included,
p.unwrap_or(solved.suggested),
fill,
)?));
}
Ok(Decision::Abandon) => return Ok(None),
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => continue,
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => return Ok(None),
}
};
// From here on only the kept frames exist, indexed as the alignment
// indexes them.
let Solved {
keep,
cameras,
focal_full,
full_long,
frame_size,
gains,
..
} = solved;
let first_name = request.frames[keep[0]].name.clone();
let frames: Vec<Loaded> = frames
.into_iter()
.enumerate()
.filter(|(k, _)| included[*k])
.map(|(_, f)| f)
.collect();
let first_raw = &frames[0].raw;
let bounds = projection::bounds(projection, focal_full, &cameras, frame_size)
.ok_or("the frames project nowhere")?;
// 5. Merge, into a DNG beside the first frame.
let name = format!("{}-pano.dng", stem(Path::new(&request.frames[0].name)));
let name = format!("{}-pano.dng", stem(Path::new(&first_name)));
let (out_path, staged) = match &request.destination {
MergeDestination::Local(dir) => (unused_name(dir, &name), false),
MergeDestination::Outbox { outbox, remote_dir } => {
@@ -943,6 +1011,83 @@ fn camera_proxy(
))
}
/// An alignment over some of the frames, and what follows from it.
struct Solved {
/// The input frames solved, in input order: the alignment's frame `i`
/// is input frame `keep[i]`.
keep: Vec<usize>,
alignment: Alignment,
/// At full resolution.
cameras: Cameras,
focal_full: f64,
/// The first kept frame's long edge and upright size, full resolution.
full_long: f64,
frame_size: (f64, f64),
/// The projection the field of view suggests, or the request's.
suggested: Projection,
/// One per kept frame.
gains: Vec<f32>,
}
/// Align the frames `included` keeps, from the pairs already measured,
/// and take the geometry to full resolution and the gains from it.
fn solve(
frames: &[Loaded],
pairs: &dr_pano::Pairs,
proxies: &[Gray],
included: &[bool],
projection: Option<Projection>,
) -> Result<Solved, String> {
let keep: Vec<usize> = (0..frames.len()).filter(|&k| included[k]).collect();
let t = Instant::now();
let alignment = dr_pano::solve(pairs, included, &dr_pano::AlignOptions::default())
.map_err(|e| e.to_string())?;
log::info!(
"merge: {} frames aligned in {:?}, focal {:.1} px, rms {:.2} px, {} links",
keep.len(),
t.elapsed(),
alignment.focal,
alignment.rms_px,
alignment.links.len()
);
// The geometry at full resolution: the proxy's long edge against the
// frame's.
let first = &frames[keep[0]];
let proxy_long = proxies[keep[0]].width.max(proxies[keep[0]].height) as f64;
let full_long = first.size.0.max(first.size.1) as f64;
let focal_full = alignment.focal * full_long / proxy_long;
let cameras = Cameras {
rotations: alignment
.rotations
.iter()
.map(|r| r.unwrap_or(dr_pano::linalg::Mat3::IDENTITY))
.collect(),
focal: focal_full,
};
let frame_size = (first.size.0 as f64, first.size.1 as f64);
let (hfov, vfov) = field_of_view(&cameras, frame_size);
let suggested = projection.unwrap_or_else(|| Projection::suggest(hfov, vfov));
// Gains, before the report so the preview shows them.
let gains = if alignment.is_complete() {
let kept: Vec<&Gray> = keep.iter().map(|&k| &proxies[k]).collect();
gains(&kept, &alignment)
} else {
vec![1.0; keep.len()]
};
Ok(Solved {
keep,
alignment,
cameras,
focal_full,
full_long,
frame_size,
suggested,
gains,
})
}
/// The aligned set on its surface, in colour, for the page.
///
/// A quick look, not the pipeline: the first frame's white balance and
@@ -954,8 +1099,8 @@ fn camera_proxy(
/// known)`. Display-ish space — the first frame's white balance and matrix,
/// a gamma — which is also what the border filler was trained on.
fn preview_planes(
colour: &[Vec<f32>],
proxies: &[Gray],
colour: &[&[f32]],
proxies: &[&Gray],
alignment: &Alignment,
gains: &[f32],
projection: Projection,
@@ -1010,7 +1155,7 @@ fn preview_planes(
}
fn bounds_of(
proxies: &[Gray],
proxies: &[&Gray],
alignment: &Alignment,
projection: Projection,
) -> projection::Bounds {
@@ -1039,7 +1184,7 @@ fn preview_rgba(
alignment: &Alignment,
projection: Projection,
bounds: &projection::Bounds,
proxies: &[Gray],
proxies: &[&Gray],
) -> (u32, u32, Vec<u8>) {
let cameras = alignment.cameras();
let (fw, fh) = (proxies[0].width as f64, proxies[0].height as f64);
@@ -1232,7 +1377,7 @@ fn describe(a: &Alignment) -> Vec<Result<(f64, f64), String>> {
/// least squares with the reference frame held at 1. Grey here is the
/// proxy's gamma-encoded value raised back to linear, so the gain is a
/// linear multiplier as the shader applies it.
fn gains(proxies: &[Gray], a: &Alignment) -> Vec<f32> {
fn gains(proxies: &[&Gray], a: &Alignment) -> Vec<f32> {
let n = proxies.len();
let cameras = a.cameras();
let (w, h) = (proxies[0].width as f64, proxies[0].height as f64);
@@ -1254,8 +1399,8 @@ fn gains(proxies: &[Gray], a: &Alignment) -> Vec<f32> {
if qx < 0.0 || qy < 0.0 || qx >= w - 1.0 || qy >= h - 1.0 {
continue;
}
sum_i += linear(&proxies[l.i], x, y);
sum_j += linear(&proxies[l.j], qx as usize, qy as usize);
sum_i += linear(proxies[l.i], x, y);
sum_j += linear(proxies[l.j], qx as usize, qy as usize);
count += 1;
}
}
@@ -1395,6 +1540,34 @@ mod tests {
assert_eq!(grey_if_blown([0.5, 0.8, 0.4], wb), [0.5, 0.8, 0.4]);
}
#[test]
fn a_frame_left_out_does_not_hold_the_merge_back() {
let report =
|frames: Vec<Result<(f64, f64), String>>, included: Vec<bool>| AlignmentReport {
focal_mm: 50.0,
rms_px: 1.0,
projection: Projection::Cylindrical,
width: 1,
height: 1,
frames,
included,
links: 1,
preview: None,
filled: false,
filler: Err(String::new()),
};
let placed = || Ok((0.0, 0.0));
let lost = || Err("no consistent overlap with any other frame".to_string());
// A frame that could not be placed holds the merge back while it
// is kept, and not once it is left out.
assert!(!report(vec![placed(), placed(), lost()], vec![true; 3]).is_complete());
assert!(report(vec![placed(), placed(), lost()], vec![true, true, false]).is_complete());
// Two frames or it is not a panorama.
assert!(
!report(vec![placed(), placed(), placed()], vec![true, false, false]).is_complete()
);
}
#[test]
fn a_worker_that_hangs_up_mid_job_is_reported_as_gone() {
// The failure the page could not see: a panic drops the sender with