Thumbnail a composite during the merge, as develop first shows it
The DNG a merge writes has no embedded preview, and an embedded preview is all the grid's thumbnail path reads, so a composite stood in the grid as a blank cell until it was opened. Reading an 800 MB file back to make one would cost what the merge already has in hand. The bands are box-reduced as they are written, after the border fill, into a copy 4096 pixels long. That copy is written as a linear DNG in memory with the composite's own profile, header and crop, and opened through open_session, the function develop opens every file with: the same decode, the default graph and view transform, the as-shot white balance and the conversion to the display's space. The grid and large thumbnails are rendered from that session, staged beside the payload before the rename releases it to a drain, and put in the store under the file id once the upload has learned it. Until then the grid draws them from memory, so the cell is not blank while the file is uploading. A test develops a synthetic composite both ways, the whole file as develop opens it and the merge's reduced copy, and holds the thumbnail's mean, 95th and 99.5th luma percentiles to within 3-4 levels of develop's render; the naive balanced-and-gamma picture the merge's preview draws misses by 13.
This commit is contained in:
@@ -0,0 +1,409 @@
|
||||
//! TRACES: FR-MRG-6
|
||||
//! The composite's thumbnails, made by the merge from its own final image.
|
||||
//!
|
||||
//! # Why the merge makes them
|
||||
//!
|
||||
//! A linear DNG as the merge writes it carries no embedded preview, and an
|
||||
//! embedded preview is all the grid's ordinary thumbnail path reads — so a
|
||||
//! composite sat in the grid as a blank cell until someone opened it. Making
|
||||
//! the thumbnail afterwards means reading back a file of 800 MB; the merge
|
||||
//! has every pixel of it in hand as the bands go by.
|
||||
//!
|
||||
//! # From the final image, through develop
|
||||
//!
|
||||
//! The bands are box-reduced as they are written — after the border fill,
|
||||
//! so the thumbnail shows what the file holds — into a copy
|
||||
//! [`SOURCE_EDGE`] pixels long. That copy is written as a linear DNG in
|
||||
//! memory with the composite's own profile, header and crop, and opened
|
||||
//! exactly as develop opens a file on first sight (`crate::open_session`):
|
||||
//! the same decode, the same default graph with its default view transform
|
||||
//! and tone mapping, the file's as-shot white balance and the conversion
|
||||
//! from the working space to the display's. The thumbnail is therefore the
|
||||
//! photograph the photographer will see on opening it, not a second,
|
||||
//! plausible conversion — the merge's own quick look ([`crate::merge`]'s
|
||||
//! `Look`) is exactly the naive linear-to-gamma picture this is not.
|
||||
//!
|
||||
//! The reduction is develop's too, in effect: a composite past one texture
|
||||
//! is developed from a box-reduced copy (`DevelopSession::open_owned`), and
|
||||
//! this is the same box at a coarser step.
|
||||
|
||||
use std::io::Cursor;
|
||||
|
||||
use dr_thumbs::{ThumbSize, Thumbnail};
|
||||
|
||||
/// The long edge of the reduced copy the thumbnails are developed from:
|
||||
/// twice the largest thumbnail, so the develop render resamples down rather
|
||||
/// than up, and small enough to cost the merge a few tens of megabytes.
|
||||
pub const SOURCE_EDGE: u32 = 4096;
|
||||
|
||||
/// The composite box-reduced as its bands go by.
|
||||
pub struct Reduced {
|
||||
width: u32,
|
||||
height: u32,
|
||||
factor: u32,
|
||||
/// Per reduced pixel, per channel: the sum of the full-resolution
|
||||
/// samples that fall in its box.
|
||||
sums: Vec<u32>,
|
||||
}
|
||||
|
||||
impl Reduced {
|
||||
/// A reduction of a `width × height` composite to at most `long_edge`
|
||||
/// pixels on its longer side, by a whole factor.
|
||||
pub fn new(width: u32, height: u32, long_edge: u32) -> Self {
|
||||
let factor = width.max(height).div_ceil(long_edge.max(1)).max(1);
|
||||
let (w, h) = (width.div_ceil(factor), height.div_ceil(factor));
|
||||
Reduced {
|
||||
width,
|
||||
height,
|
||||
factor,
|
||||
sums: vec![0; w as usize * h as usize * 3],
|
||||
}
|
||||
}
|
||||
|
||||
/// The reduced size.
|
||||
pub fn size(&self) -> (u32, u32) {
|
||||
(
|
||||
self.width.div_ceil(self.factor),
|
||||
self.height.div_ceil(self.factor),
|
||||
)
|
||||
}
|
||||
|
||||
/// The whole factor each side is divided by.
|
||||
pub fn factor(&self) -> u32 {
|
||||
self.factor
|
||||
}
|
||||
|
||||
/// Fold in `rows` rows of the composite starting at `first_row`,
|
||||
/// interleaved RGB at the full width.
|
||||
pub fn push(&mut self, first_row: u32, rows: u32, rgb: &[u16]) {
|
||||
let (rw, _) = self.size();
|
||||
let (w, f) = (self.width as usize, self.factor as usize);
|
||||
for r in 0..rows as usize {
|
||||
let y = first_row as usize + r;
|
||||
if y >= self.height as usize {
|
||||
break;
|
||||
}
|
||||
let row = &rgb[r * w * 3..(r + 1) * w * 3];
|
||||
let out = (y / f) * rw as usize * 3;
|
||||
for (x, px) in row.chunks_exact(3).enumerate() {
|
||||
let o = out + (x / f) * 3;
|
||||
self.sums[o] += u32::from(px[0]);
|
||||
self.sums[o + 1] += u32::from(px[1]);
|
||||
self.sums[o + 2] += u32::from(px[2]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The averages, as the DNG's `u16` samples. A box cut short by the
|
||||
/// edge of the composite is the mean of what it holds.
|
||||
pub fn finish(self) -> (u32, u32, Vec<u16>) {
|
||||
let (rw, rh) = self.size();
|
||||
let f = self.factor;
|
||||
let mut out = vec![0u16; self.sums.len()];
|
||||
for y in 0..rh {
|
||||
let bh = f.min(self.height - y * f);
|
||||
for x in 0..rw {
|
||||
let bw = f.min(self.width - x * f);
|
||||
let n = bh * bw;
|
||||
let i = (y * rw + x) as usize * 3;
|
||||
for c in 0..3 {
|
||||
out[i + c] = ((self.sums[i + c] + n / 2) / n) as u16;
|
||||
}
|
||||
}
|
||||
}
|
||||
(rw, rh, out)
|
||||
}
|
||||
}
|
||||
|
||||
/// Which size classes a composite of this shape is thumbnailed at.
|
||||
pub fn classes(width: u32, height: u32) -> Vec<ThumbSize> {
|
||||
let _ = (width, height);
|
||||
vec![ThumbSize::Grid, ThumbSize::Large]
|
||||
}
|
||||
|
||||
/// Develop the reduced composite as develop would open the file, and render
|
||||
/// it at each of `classes`.
|
||||
///
|
||||
/// `crop` is the rectangle the file opens on, in the composite's pixels;
|
||||
/// it is carried to the reduced copy so the thumbnail is of the picture
|
||||
/// and not of the border around it.
|
||||
pub fn render(
|
||||
ctx: &dr_gpu::GpuContext,
|
||||
reduced: Reduced,
|
||||
profile: &dr_export::DngProfile,
|
||||
header: &dr_export::SourceMetadata,
|
||||
crop: Option<dr_export::Rect>,
|
||||
decoder: &dyn dr_decode::Decoder,
|
||||
classes: &[ThumbSize],
|
||||
) -> Result<Vec<(ThumbSize, Thumbnail)>, String> {
|
||||
let f = reduced.factor();
|
||||
let (w, h, samples) = reduced.finish();
|
||||
let crop = crop.map(|r| {
|
||||
let x = (r.x / f).min(w.saturating_sub(1));
|
||||
let y = (r.y / f).min(h.saturating_sub(1));
|
||||
dr_export::Rect {
|
||||
x,
|
||||
y,
|
||||
width: (r.width / f).clamp(1, w - x),
|
||||
height: (r.height / f).clamp(1, h - y),
|
||||
}
|
||||
});
|
||||
let mut file = Cursor::new(Vec::new());
|
||||
let mut once = Some(samples);
|
||||
dr_export::write_linear_dng(
|
||||
&mut file,
|
||||
w,
|
||||
h,
|
||||
h,
|
||||
profile,
|
||||
Some(header),
|
||||
|_, buf| {
|
||||
buf.extend_from_slice(&once.take().unwrap_or_default());
|
||||
Ok(())
|
||||
},
|
||||
|| crop,
|
||||
)
|
||||
.map_err(|e| e.to_string())?;
|
||||
let bytes = file.into_inner();
|
||||
|
||||
// The file as develop reads it on first open: its header, then the
|
||||
// pixels, through the one function every develop session starts from.
|
||||
let meta = decoder.metadata(&bytes).map_err(|e| e.to_string())?;
|
||||
let mut session = crate::open_session(ctx, decoder, &bytes, &meta)?;
|
||||
let mut out = Vec::with_capacity(classes.len());
|
||||
for &class in classes {
|
||||
let (tw, th, rgba) = session.render_thumbnail(class.edge())?;
|
||||
let jpeg = dr_thumbs::encode_rgba(tw, th, &rgba).map_err(|e| e.to_string())?;
|
||||
out.push((
|
||||
class,
|
||||
Thumbnail {
|
||||
width: tw,
|
||||
height: th,
|
||||
bytes: jpeg,
|
||||
},
|
||||
));
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_reduction_averages_its_boxes_and_the_ragged_edge() {
|
||||
// 5 × 3 by 2: boxes of 2 × 2, then 1 × 2 at the right, 2 × 1 and
|
||||
// 1 × 1 along the bottom.
|
||||
let mut r = Reduced::new(5, 3, 3);
|
||||
assert_eq!(r.factor(), 2);
|
||||
assert_eq!(r.size(), (3, 2));
|
||||
let px = |x: u16, y: u16| [x * 10 + y, 0, 1000];
|
||||
let rows: Vec<u16> = (0..3)
|
||||
.flat_map(|y| (0..5).flat_map(move |x| px(x, y)))
|
||||
.collect();
|
||||
// In two bands, as the merge hands them over.
|
||||
r.push(0, 2, &rows[..5 * 2 * 3]);
|
||||
r.push(2, 1, &rows[5 * 2 * 3..]);
|
||||
let (w, h, out) = r.finish();
|
||||
assert_eq!((w, h), (3, 2));
|
||||
let at = |x: usize, y: usize| out[(y * 3 + x) * 3];
|
||||
// (0,0),(1,0),(0,1),(1,1) → 0, 10, 1, 11: mean 5.5, rounded.
|
||||
assert_eq!(at(0, 0), 6);
|
||||
// The right-hand column is one pixel wide: (4,0) and (4,1).
|
||||
assert_eq!(at(2, 0), (40 + 41 + 1) / 2);
|
||||
// The bottom-right is a single pixel.
|
||||
assert_eq!(at(2, 1), 42);
|
||||
assert!(out.chunks_exact(3).all(|p| p[2] == 1000));
|
||||
}
|
||||
|
||||
/// A synthetic composite in camera space: a sky that brightens to a
|
||||
/// clipped band, green ground with texture, and a blown sun.
|
||||
fn composite(width: u32, height: u32, white: u32) -> Vec<u16> {
|
||||
let mut out = Vec::with_capacity((width * height * 3) as usize);
|
||||
for y in 0..height {
|
||||
for x in 0..width {
|
||||
let fy = y as f32 / height as f32;
|
||||
let fx = x as f32 / width as f32;
|
||||
let tex = ((x / 7 + y / 5) % 2) as f32 * 0.03;
|
||||
let cam = if fy < 0.45 {
|
||||
// Sky: blue-ish in camera space, brighter toward the top
|
||||
// and toward the right, clipping in the top corner.
|
||||
let b = (0.25 + 0.9 * (1.0 - fy / 0.45) * fx).min(1.2);
|
||||
[b * 0.45, b * 0.8, b]
|
||||
} else {
|
||||
let g = 0.08 + 0.25 * fx + tex;
|
||||
[g * 0.6, g, g * 0.4]
|
||||
};
|
||||
let sun = ((fx - 0.8).powi(2) + (fy - 0.2).powi(2)).sqrt() < 0.04;
|
||||
let cam = if sun { [1.0, 1.0, 1.0] } else { cam };
|
||||
for c in cam {
|
||||
out.push(((c * white as f32).round() as u32).min(white) as u16);
|
||||
}
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn profile(white: u32) -> dr_export::DngProfile {
|
||||
// A Canon 6D's D65 matrix and as-shot neutral, as the fixture's
|
||||
// composite carries them.
|
||||
dr_export::DngProfile {
|
||||
unique_model: "Canon EOS 6D".into(),
|
||||
calibrations: vec![(
|
||||
21,
|
||||
[
|
||||
[0.7034, -0.0804, -0.1014],
|
||||
[-0.4420, 1.2564, 0.2058],
|
||||
[-0.0851, 0.1994, 0.5758],
|
||||
],
|
||||
)],
|
||||
as_shot_neutral: [0.4985, 1.0, 0.6505],
|
||||
white_level: white,
|
||||
}
|
||||
}
|
||||
|
||||
/// Luma, in 0..255, for each pixel of an RGBA buffer.
|
||||
fn luma(rgba: &[u8]) -> Vec<f32> {
|
||||
rgba.chunks_exact(4)
|
||||
.map(|p| 0.2126 * p[0] as f32 + 0.7152 * p[1] as f32 + 0.0722 * p[2] as f32)
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn stats(mut l: Vec<f32>) -> (f32, f32, f32) {
|
||||
let mean = l.iter().sum::<f32>() / l.len() as f32;
|
||||
l.sort_by(f32::total_cmp);
|
||||
let at = |q: f32| l[((l.len() - 1) as f32 * q).round() as usize];
|
||||
(mean, at(0.95), at(0.995))
|
||||
}
|
||||
|
||||
/// FR-MRG-6's acceptance: the thumbnail made during the merge is the
|
||||
/// picture develop shows on first opening the composite — its overall
|
||||
/// level and its highlights — and not a naive conversion of the linear
|
||||
/// data, which comes out a stop and more away on both.
|
||||
#[test]
|
||||
fn the_thumbnail_is_what_develop_shows_on_first_open() {
|
||||
let Some(ctx) = crate::develop::test_support::headless() else {
|
||||
eprintln!("no GPU; skipping");
|
||||
return;
|
||||
};
|
||||
let (width, height, white) = (3000u32, 760u32, 13_023u32);
|
||||
let pixels = composite(width, height, white);
|
||||
let profile = profile(white);
|
||||
let header = dr_export::SourceMetadata {
|
||||
make: Some("Canon".into()),
|
||||
model: Some("Canon EOS 6D".into()),
|
||||
captured_at: Some(1_600_000_000),
|
||||
..Default::default()
|
||||
};
|
||||
let crop = dr_export::Rect {
|
||||
x: 40,
|
||||
y: 30,
|
||||
width: 2900,
|
||||
height: 700,
|
||||
};
|
||||
|
||||
// Develop's side: the whole composite written as the merge writes
|
||||
// it, opened as develop opens it, rendered for the grid.
|
||||
let rows_per_strip = 256u32;
|
||||
let mut file = Cursor::new(Vec::new());
|
||||
dr_export::write_linear_dng(
|
||||
&mut file,
|
||||
width,
|
||||
height,
|
||||
rows_per_strip,
|
||||
&profile,
|
||||
Some(&header),
|
||||
|k, buf| {
|
||||
let from = k * (rows_per_strip * width * 3) as usize;
|
||||
let to = (from + (rows_per_strip * width * 3) as usize).min(pixels.len());
|
||||
buf.extend_from_slice(&pixels[from..to]);
|
||||
Ok(())
|
||||
},
|
||||
|| Some(crop),
|
||||
)
|
||||
.unwrap();
|
||||
let bytes = file.into_inner();
|
||||
let decoder = dr_decode::default();
|
||||
let meta = decoder.metadata(&bytes).unwrap();
|
||||
let mut developed = crate::open_session(&ctx, decoder, &bytes, &meta).unwrap();
|
||||
|
||||
// The merge's side: the same bands, reduced as they pass.
|
||||
let mut reduced = Reduced::new(width, height, 1500);
|
||||
for first in (0..height).step_by(rows_per_strip as usize) {
|
||||
let rows = rows_per_strip.min(height - first);
|
||||
let from = (first * width * 3) as usize;
|
||||
reduced.push(
|
||||
first,
|
||||
rows,
|
||||
&pixels[from..from + (rows * width * 3) as usize],
|
||||
);
|
||||
}
|
||||
let thumbs = render(
|
||||
&ctx,
|
||||
reduced,
|
||||
&profile,
|
||||
&header,
|
||||
Some(crop),
|
||||
decoder,
|
||||
&[ThumbSize::Grid, ThumbSize::Large],
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
// The naive picture: the linear camera values balanced and gamma
|
||||
// encoded, with no matrix, no curve and no highlight handling — what
|
||||
// the merge's preview draws.
|
||||
let naive = {
|
||||
let mut l = Vec::new();
|
||||
for y in (crop.y..crop.y + crop.height).step_by(4) {
|
||||
for x in (crop.x..crop.x + crop.width).step_by(4) {
|
||||
let i = ((y * width + x) * 3) as usize;
|
||||
let v = |c: usize| {
|
||||
let lin = pixels[i + c] as f32 / white as f32 / profile.as_shot_neutral[c];
|
||||
lin.clamp(0.0, 1.0).powf(1.0 / 2.2) * 255.0
|
||||
};
|
||||
l.push(0.2126 * v(0) + 0.7152 * v(1) + 0.0722 * v(2));
|
||||
}
|
||||
}
|
||||
stats(l)
|
||||
};
|
||||
|
||||
for (class, thumb) in thumbs {
|
||||
let (tw, th, ours) = dr_thumbs::decode_rgba(&thumb.bytes).unwrap();
|
||||
let (dw, dh, theirs) = developed.render_thumbnail(class.edge()).unwrap();
|
||||
// Through the store's encoding too: both are JPEGs in the store,
|
||||
// and it is the encoding, not the development, that rounds a
|
||||
// blown sun off to 247.
|
||||
let theirs = {
|
||||
let jpeg = dr_thumbs::encode_rgba(dw, dh, &theirs).unwrap();
|
||||
dr_thumbs::decode_rgba(&jpeg).unwrap().2
|
||||
};
|
||||
// The picture, not the border: the crop's shape at the class's
|
||||
// size, give or take the reduction's rounding.
|
||||
assert!(
|
||||
(tw as i64 - dw as i64).abs() <= 2 && (th as i64 - dh as i64).abs() <= 2,
|
||||
"{class:?}: {tw}×{th} against develop's {dw}×{dh}"
|
||||
);
|
||||
let (m0, p95_0, p995_0) = stats(luma(&ours));
|
||||
let (m1, p95_1, p995_1) = stats(luma(&theirs));
|
||||
eprintln!(
|
||||
"{class:?}: mean {m0:.1}/{m1:.1}, p95 {p95_0:.1}/{p95_1:.1}, \
|
||||
p99.5 {p995_0:.1}/{p995_1:.1}; naive {naive:?}"
|
||||
);
|
||||
assert!((m0 - m1).abs() <= 3.0, "{class:?} mean {m0} vs {m1}");
|
||||
// And the tolerance means something: the conversion this
|
||||
// replaces is nowhere near it.
|
||||
assert!(
|
||||
(naive.0 - m1).abs() > 12.0 || (naive.1 - p95_1).abs() > 12.0,
|
||||
"a naive conversion ({naive:?}) would pass as well"
|
||||
);
|
||||
assert!(
|
||||
(p95_0 - p95_1).abs() <= 4.0,
|
||||
"{class:?} p95 {p95_0} vs {p95_1}"
|
||||
);
|
||||
assert!(
|
||||
(p995_0 - p995_1).abs() <= 4.0,
|
||||
"{class:?} p99.5 {p995_0} vs {p995_1}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user