dtourolle
48c4b403d2
The scan reads the first 256 KB of a file for its metadata. A camera writes its IFDs at the front, so that is the whole structure; the linear DNG a merge writes puts its first IFD after the pixels, and rawler, given the head alone, finds no decoder in it. The composite was catalogued without a date and sorted to the very end of the grid, after every dated photograph — which is where a panorama merged on the tablet went unfound. dr-decode's own TIFF reader now reads through a head and a tail at a known offset; trailing_ifd says where the tail starts and metadata_split reads the two together. The scan, when the head fails and points beyond itself, fetches from the IFD to the end — kilobytes — and dates the file from both. Tested against the writer's own output.
352 lines
13 KiB
Rust
352 lines
13 KiB
Rust
//! TRACES: FR-MRG-3
|
||
//! A linear DNG: the container a merge writes its composite into.
|
||
//!
|
||
//! Decided by S15.1 (2026-09-19): rawler reads back a `LinearRaw` DNG the
|
||
//! application writes, so a composite re-enters the library as
|
||
//! `Format::Dng` through the decoder every camera DNG uses. What is written
|
||
//! is a RAW in every sense a warp can preserve — camera-linear `u16`
|
||
//! samples at the first source's own scale, its matrices, illuminants,
|
||
//! as-shot neutral and body name — so the panorama is developed afterwards
|
||
//! as one photograph, from the sensor's numbers.
|
||
//!
|
||
//! # Streamed, not buffered
|
||
//!
|
||
//! The composite is larger than any single photograph the pipeline renders
|
||
//! and larger than the tablet's memory (FR-MRG-11), so the writer never
|
||
//! holds it. Strips are pulled from the caller one at a time through a
|
||
//! closure, in order, and written as they arrive; the caller renders a band
|
||
//! of chunks, hands over its rows, and moves on.
|
||
//!
|
||
//! # Why the `tiff` crate after all
|
||
//!
|
||
//! S15.1's spike hand-rolled its IFD because the crate's encoder fixes
|
||
//! `PhotometricInterpretation` to RGB when the image is opened. It does — but
|
||
//! a directory is a map and a later `write_tag` on the same tag replaces the
|
||
//! earlier, so `LinearRaw` goes in over the top and everything else the
|
||
//! crate does (strips, offsets, sub-IFDs, the EXIF block `encode.rs` already
|
||
//! knows how to write) is kept.
|
||
|
||
use std::io::{Seek, Write};
|
||
|
||
use tiff::encoder::{colortype, DirectoryEncoder, SRational, TiffEncoder, TiffKind, TiffValue};
|
||
use tiff::tags::Tag;
|
||
|
||
use crate::encode::{sub_directories, tag_metadata, Ascii, Rationals};
|
||
use crate::{ExportError, SourceMetadata};
|
||
|
||
/// What the DNG says about the camera that "took" the composite: the first
|
||
/// source's profile, carried across so the composite develops through it.
|
||
#[derive(Debug, Clone, PartialEq)]
|
||
pub struct DngProfile {
|
||
/// `UniqueCameraModel`, the name the profile database matches on.
|
||
pub unique_model: String,
|
||
/// `(CalibrationIlluminant, ColorMatrix)`: the EXIF light-source code and
|
||
/// the XYZ → camera matrix measured under it. One or two.
|
||
pub calibrations: Vec<(u16, [[f32; 3]; 3])>,
|
||
/// `AsShotNeutral`, camera RGB of the scene's white.
|
||
pub as_shot_neutral: [f32; 3],
|
||
/// `WhiteLevel`: the sample value that is clipping. The first source's
|
||
/// white minus its black, since the samples are black-subtracted.
|
||
pub white_level: u32,
|
||
}
|
||
|
||
/// Write a linear DNG, pulling `rows_per_strip`-row strips from `strips`.
|
||
///
|
||
/// Each call to `strips` receives the strip index and a buffer to fill with
|
||
/// `width × rows × 3` interleaved RGB `u16` samples (the last strip may be
|
||
/// shorter). `source` supplies the `Make`, `Model`, dates and EXIF block
|
||
/// exactly as an export does (FR-EXP-8 sanitising already applied by the
|
||
/// caller).
|
||
///
|
||
/// `PhotometricInterpretation = LinearRaw`, `DNGVersion 1.4`, uncompressed,
|
||
/// `Orientation = 1` — the composite is written upright (panorama.md §8).
|
||
///
|
||
/// `crop` is asked once every strip is in, and its answer — the largest
|
||
/// rectangle the frames covered, found while the strips went by
|
||
/// (`Inscribed`) — becomes `DefaultCropOrigin`/`DefaultCropSize`
|
||
/// (FR-MRG-4): the file opens on the picture, and the border is still in it.
|
||
// Eight arguments, and each is a different thing: the sink, three
|
||
// dimensions, the profile, the header, the strip source and the crop. A
|
||
// struct for them would be a struct with one caller.
|
||
#[allow(clippy::too_many_arguments)]
|
||
pub fn write_linear_dng<W, F, C>(
|
||
out: W,
|
||
width: u32,
|
||
height: u32,
|
||
rows_per_strip: u32,
|
||
profile: &DngProfile,
|
||
source: Option<&SourceMetadata>,
|
||
mut strips: F,
|
||
crop: C,
|
||
) -> Result<(), ExportError>
|
||
where
|
||
W: Write + Seek,
|
||
F: FnMut(usize, &mut Vec<u16>) -> Result<(), ExportError>,
|
||
C: FnOnce() -> Option<crate::Rect>,
|
||
{
|
||
let enc = |e: tiff::TiffError| ExportError::Encode(e.to_string());
|
||
let mut encoder = TiffEncoder::new(out).map_err(enc)?;
|
||
let sub = sub_directories(&mut encoder, source, width, height)?;
|
||
let mut image = encoder
|
||
.new_image::<colortype::RGB16>(width, height)
|
||
.map_err(enc)?;
|
||
image.rows_per_strip(rows_per_strip.max(1)).map_err(enc)?;
|
||
tag_metadata(image.encoder(), source, &sub)?;
|
||
tag_dng(image.encoder(), profile).map_err(enc)?;
|
||
|
||
let rows = rows_per_strip.max(1);
|
||
let strip_count = height.div_ceil(rows) as usize;
|
||
let mut buf: Vec<u16> = Vec::with_capacity((width * rows * 3) as usize);
|
||
for k in 0..strip_count {
|
||
buf.clear();
|
||
strips(k, &mut buf)?;
|
||
let expected_rows = rows.min(height - k as u32 * rows);
|
||
let expected = (width * expected_rows * 3) as usize;
|
||
if buf.len() != expected {
|
||
return Err(ExportError::Encode(format!(
|
||
"strip {k} has {} samples, expected {expected}",
|
||
buf.len()
|
||
)));
|
||
}
|
||
image.write_strip(&buf).map_err(enc)?;
|
||
}
|
||
if let Some(r) = crop().filter(|r| r.width > 0 && r.height > 0) {
|
||
let r = crate::Rect {
|
||
x: r.x.min(width - 1),
|
||
y: r.y.min(height - 1),
|
||
width: r.width.min(width - r.x.min(width - 1)),
|
||
height: r.height.min(height - r.y.min(height - 1)),
|
||
};
|
||
image
|
||
.encoder()
|
||
.write_tag(Tag::Unknown(tag::DEFAULT_CROP_ORIGIN), &[r.x, r.y][..])
|
||
.map_err(enc)?;
|
||
image
|
||
.encoder()
|
||
.write_tag(
|
||
Tag::Unknown(tag::DEFAULT_CROP_SIZE),
|
||
&[r.width, r.height][..],
|
||
)
|
||
.map_err(enc)?;
|
||
}
|
||
image.finish().map_err(enc)
|
||
}
|
||
|
||
/// The tags that make a TIFF a DNG, and a linear one.
|
||
fn tag_dng<W, K>(dir: &mut DirectoryEncoder<'_, W, K>, profile: &DngProfile) -> tiff::TiffResult<()>
|
||
where
|
||
W: Write + Seek,
|
||
K: TiffKind,
|
||
{
|
||
// Over the top of what `new_image` wrote: this is the whole trick.
|
||
dir.write_tag(Tag::PhotometricInterpretation, LINEAR_RAW)?;
|
||
dir.write_tag(Tag::Orientation, 1u16)?;
|
||
dir.write_tag(Tag::Unknown(tag::DNG_VERSION), &[1u8, 4, 0, 0][..])?;
|
||
dir.write_tag(Tag::Unknown(tag::DNG_BACKWARD_VERSION), &[1u8, 4, 0, 0][..])?;
|
||
dir.write_tag(
|
||
Tag::Unknown(tag::UNIQUE_CAMERA_MODEL),
|
||
Ascii(&profile.unique_model),
|
||
)?;
|
||
dir.write_tag(
|
||
Tag::Unknown(tag::WHITE_LEVEL),
|
||
&[profile.white_level; 3][..],
|
||
)?;
|
||
dir.write_tag(Tag::Unknown(tag::BLACK_LEVEL), &[0u32; 3][..])?;
|
||
|
||
for (slot, (illuminant, matrix)) in profile.calibrations.iter().take(2).enumerate() {
|
||
let (ill_tag, mat_tag) = if slot == 0 {
|
||
(tag::CALIBRATION_ILLUMINANT_1, tag::COLOR_MATRIX_1)
|
||
} else {
|
||
(tag::CALIBRATION_ILLUMINANT_2, tag::COLOR_MATRIX_2)
|
||
};
|
||
dir.write_tag(Tag::Unknown(ill_tag), *illuminant)?;
|
||
let flat: Vec<SRational> = matrix
|
||
.iter()
|
||
.flatten()
|
||
.map(|&v| SRational {
|
||
n: (v * 10_000.0).round() as i32,
|
||
d: 10_000,
|
||
})
|
||
.collect();
|
||
dir.write_tag(Tag::Unknown(mat_tag), SRationals(&flat))?;
|
||
}
|
||
|
||
let neutral: Vec<(u32, u32)> = profile
|
||
.as_shot_neutral
|
||
.iter()
|
||
.map(|&v| ((v.max(0.0) * 1_000_000.0).round() as u32, 1_000_000))
|
||
.collect();
|
||
dir.write_tag(Tag::Unknown(tag::AS_SHOT_NEUTRAL), Rationals(&neutral))?;
|
||
Ok(())
|
||
}
|
||
|
||
/// `PhotometricInterpretation` for demosaiced, un-rendered sensor data.
|
||
const LINEAR_RAW: u16 = 34892;
|
||
|
||
/// DNG tag numbers the `tiff` crate has no names for.
|
||
mod tag {
|
||
pub const DNG_VERSION: u16 = 50706;
|
||
pub const DNG_BACKWARD_VERSION: u16 = 50707;
|
||
pub const UNIQUE_CAMERA_MODEL: u16 = 50708;
|
||
pub const BLACK_LEVEL: u16 = 50714;
|
||
pub const WHITE_LEVEL: u16 = 50717;
|
||
pub const DEFAULT_CROP_ORIGIN: u16 = 50719;
|
||
pub const DEFAULT_CROP_SIZE: u16 = 50720;
|
||
pub const COLOR_MATRIX_1: u16 = 50721;
|
||
pub const COLOR_MATRIX_2: u16 = 50722;
|
||
pub const AS_SHOT_NEUTRAL: u16 = 50728;
|
||
pub const CALIBRATION_ILLUMINANT_1: u16 = 50778;
|
||
pub const CALIBRATION_ILLUMINANT_2: u16 = 50779;
|
||
}
|
||
|
||
/// A run of `SRATIONAL`s, as `encode::Rationals` is for `RATIONAL`.
|
||
struct SRationals<'a>(&'a [SRational]);
|
||
|
||
impl TiffValue for SRationals<'_> {
|
||
const BYTE_LEN: u8 = 8;
|
||
const FIELD_TYPE: tiff::tags::Type = tiff::tags::Type::SRATIONAL;
|
||
|
||
fn count(&self) -> usize {
|
||
self.0.len()
|
||
}
|
||
|
||
fn data(&self) -> std::borrow::Cow<'_, [u8]> {
|
||
let mut out = Vec::with_capacity(self.0.len() * 8);
|
||
for r in self.0 {
|
||
out.extend_from_slice(&r.n.to_ne_bytes());
|
||
out.extend_from_slice(&r.d.to_ne_bytes());
|
||
}
|
||
std::borrow::Cow::Owned(out)
|
||
}
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
|
||
fn profile() -> DngProfile {
|
||
DngProfile {
|
||
unique_model: "Canon EOS 6D".into(),
|
||
calibrations: vec![
|
||
(17, [[0.8, -0.2, 0.1], [-0.3, 1.1, 0.2], [0.0, -0.1, 0.9]]),
|
||
(21, [[0.7, -0.1, 0.0], [-0.2, 1.0, 0.1], [0.0, -0.2, 0.8]]),
|
||
],
|
||
as_shot_neutral: [0.5, 1.0, 0.6],
|
||
white_level: 13_023,
|
||
}
|
||
}
|
||
|
||
fn write(width: u32, height: u32, rows: u32) -> Vec<u8> {
|
||
let mut bytes = std::io::Cursor::new(Vec::new());
|
||
let source = SourceMetadata {
|
||
make: Some("Canon".into()),
|
||
model: Some("Canon EOS 6D".into()),
|
||
captured_at: Some(1_754_398_664),
|
||
captured_offset: Some(120),
|
||
..Default::default()
|
||
};
|
||
write_linear_dng(
|
||
&mut bytes,
|
||
width,
|
||
height,
|
||
rows,
|
||
&profile(),
|
||
Some(&source),
|
||
|k, buf| {
|
||
let first = k as u32 * rows;
|
||
let n = rows.min(height - first);
|
||
for y in first..first + n {
|
||
for x in 0..width {
|
||
buf.extend([(x + y * width) as u16, 1000, 2000]);
|
||
}
|
||
}
|
||
Ok(())
|
||
},
|
||
|| {
|
||
Some(crate::Rect {
|
||
x: 2,
|
||
y: 1,
|
||
width: 15,
|
||
height: 10,
|
||
})
|
||
},
|
||
)
|
||
.expect("written");
|
||
bytes.into_inner()
|
||
}
|
||
|
||
#[test]
|
||
fn rawler_reads_it_back_as_linear_raw() {
|
||
let bytes = write(20, 13, 4);
|
||
let source = rawler::rawsource::RawSource::new_from_slice(&bytes);
|
||
let decoder = rawler::get_decoder(&source).expect("a DNG");
|
||
let image = decoder
|
||
.raw_image(&source, &Default::default(), false)
|
||
.expect("decodes");
|
||
assert_eq!((image.width, image.height, image.cpp), (20, 13, 3));
|
||
assert_eq!(image.whitelevel.0[0], 13_023);
|
||
// Pixel (3, 2) is (3 + 2·20, 1000, 2000) — samples in order, strips
|
||
// joined without a seam.
|
||
let rawler::RawImageData::Integer(data) = &image.data else {
|
||
panic!("integer samples")
|
||
};
|
||
let i = (2 * 20 + 3) * 3;
|
||
assert_eq!(&data[i..i + 3], &[43, 1000, 2000]);
|
||
// Last row, from the short final strip.
|
||
let i = (12 * 20 + 19) * 3;
|
||
assert_eq!(data[i], (19 + 12 * 20) as u16);
|
||
// The profile came through as the camera's.
|
||
assert!(!image.camera.color_matrix.is_empty());
|
||
assert_eq!(image.model, "Canon EOS 6D");
|
||
// The default crop is what the decoder reports as the picture.
|
||
let crop = image.crop_area.expect("a crop");
|
||
assert_eq!((crop.p.x, crop.p.y, crop.d.w, crop.d.h), (2, 1, 15, 10));
|
||
}
|
||
|
||
#[test]
|
||
fn the_catalog_reads_the_date_from_a_head_and_a_tail() {
|
||
// TRACES: FR-CAT-5
|
||
// The IFDs follow the pixels, so a scan that has the first bytes of
|
||
// the file has a pointer into nothing; rawler finds no decoder in
|
||
// that, and the composite would sit undated at the end of the grid.
|
||
// The scan's second range — from the first IFD to the end — with
|
||
// the head is enough to date it, and to name the camera.
|
||
let bytes = write(640, 400, 64);
|
||
let head = &bytes[..4096];
|
||
assert!(
|
||
dr_decode::metadata(head).is_err(),
|
||
"the head alone must not read"
|
||
);
|
||
let at = dr_decode::trailing_ifd(head).expect("the IFD is beyond the head");
|
||
assert!(at as usize > head.len());
|
||
let tail = &bytes[at as usize..];
|
||
assert!(tail.len() < 4096, "the tail is the IFD, not the pixels");
|
||
let md = dr_decode::metadata_split(head, tail, at).expect("read from two ranges");
|
||
assert_eq!(md.captured_at, Some(1_754_398_664));
|
||
assert_eq!(md.captured_offset, Some(120));
|
||
assert_eq!(md.model.as_deref(), Some("Canon EOS 6D"));
|
||
// A head that holds everything is not a trailing-IFD file.
|
||
assert_eq!(dr_decode::trailing_ifd(&bytes), None);
|
||
}
|
||
|
||
#[test]
|
||
fn a_strip_of_the_wrong_length_is_refused() {
|
||
let mut bytes = std::io::Cursor::new(Vec::new());
|
||
let err = write_linear_dng(
|
||
&mut bytes,
|
||
8,
|
||
8,
|
||
8,
|
||
&profile(),
|
||
None,
|
||
|_, buf| {
|
||
buf.extend([0u16; 10]);
|
||
Ok(())
|
||
},
|
||
|| None,
|
||
)
|
||
.unwrap_err();
|
||
assert!(matches!(err, ExportError::Encode(_)));
|
||
}
|
||
}
|