dtourolle/DarkRoom
1
0
Fork 0
Code Issues 39 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Render each body through the profile its own files describe

Browse Source 
Colour came from whichever matrix rawler happened to key `D65`, the second
one was discarded, and the rendering was left linear. That is the dcraw
default, and FR-DEV-3e names it as the reason people abandon a converter in
the first hour: correct in the abstract, flat and poor on skin in practice.

The decoder now builds a camera profile.

- `ColorMatrix1/2` and `CalibrationIlluminant1/2`. rawler surfaces these as
  an illuminant-keyed map — for DNGs from the tags, and for native formats
  from its own camera database — so a Canon CR2 arrives with a tungsten
  matrix and a daylight matrix exactly as an Adobe DNG of the same frame
  would. Dual-illuminant support is therefore not a DNG feature here.

- `ForwardMatrix1/2`, read straight from the root IFD, because rawler parses
  them and never surfaces them. Where a file carries both, they replace the
  inverted colour matrix: the same relationship measured in the direction
  rendering actually wants, rather than an inversion that amplifies the
  measurement error exactly where skin lives.

- `AsShotNeutral`, used to estimate what the scene was lit by and to
  interpolate between the two calibrations in mireds. The estimate is
  circular — the temperature needs a matrix and the matrix needs the
  temperature — so it is a fixed point, three rounds, as Adobe's SDK does it.

Bodies calibrated at neither D65 nor A stopped rendering uncalibrated as a
side effect: a Phase One IQ3 carries D55 and D75 and used to get no matrix
at all.

And a base curve, applied per channel in camera RGB between the last
adjustment and the conversion out of camera space — a toe, a steep midtone
and a shoulder, which is the difference between a photograph and a scan of
one. It is not an edit: no slider, nothing in the sidecar, because it
belongs to the body rather than to anything anyone decided, and a sidecar is
shared between bodies. It is not a develop node either, and `ops/README.md`
now records why. It evaluates on the tone curve's own spline rather than a
second copy, so a profile author placing a control point and a photographer
dragging one mean the same thing by it.

The curves are data. `core/dr-decode/profiles/base_curves.yaml` ships inside
the binary as a floor and is superseded by any copy on disk carrying a
higher `version:`, so a body can be added and distributed without a release
— and, under the GPL, contributed. The comparison runs both ways: a stale
pack cannot hold an upgraded binary back at last year's rendering.

Canon EOS 6D and R6, Nikon Z 6 and D750, Sony A7 III and Fujifilm X-T3 ship
with their own curves. Every other body gets a conservative default, which
is much closer to right than the identity is for any of them. A JPEG gets
none — it has already been rendered once, by the camera.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
dtourolle
2026-08-22 14:37:33 +02:00
co-authored by Claude Opus 5
parent 586698db00
commit 743fefe7f1
12 changed files with 2396 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/dr-decode/Cargo.toml
+7
View File
@@ -8,6 +8,13 @@ license.workspace = true
[dependencies]
dr-types.workspace = true
rawler.workspace = true
# The camera profile database is data, not code (FR-DEV-3e): a YAML file that
# ships with the binary and is superseded by a newer one on disk. serde_norway
# is the workspace's YAML crate — the fork still receiving releases — and it is
# already in the tree for `dr-pipeline`'s node declarations and `dr-ui`'s style
# tokens. Pure Rust, so it costs nothing under the Android NDK.
serde = { workspace = true }
serde_norway.workspace = true
zune-jpeg.workspace = true
thiserror.workspace = true
log.workspace = true
core/dr-decode/profiles/base_curves.yaml
+160
View File
@@ -0,0 +1,160 @@
# DarkRoom camera base curves (FR-DEV-3e).
#
# ---------------------------------------------------------------------------
# Adding a body is editing this file. It is not a code change.
# ---------------------------------------------------------------------------
#
# The copy you are reading is compiled into the binary as a floor. At startup
# `dr_decode::base_curve::load` also looks for `base_curves.yaml` in:
#
# 1. $DARKROOM_PROFILES/ (set it while you are tuning)
# 2. $XDG_DATA_HOME/darkroom/profiles/
# or $HOME/.local/share/darkroom/profiles/
#
# and uses the first one it finds *whose `version:` is higher than this one's*.
# So: bump `version`, drop the file in that directory, restart. A body added
# this afternoon renders correctly this afternoon, with no release and no
# rebuild — which is what the requirement asks for, and what makes these
# contributable under the GPL.
#
# The version check runs both ways on purpose. A file older than the built-in
# copy is ignored with a log line, so upgrading DarkRoom cannot silently lose
# curves to a pack somebody downloaded a year ago.
#
# ---------------------------------------------------------------------------
# What the numbers mean
# ---------------------------------------------------------------------------
#
# Five `[x, y]` control points on a monotone spline (Fritsch-Carlson, the same
# one the tone curve widget draws). Both axes are **linear**:
#
# x scene-referred camera RGB after white balance, 1.0 = sensor saturation
# y display-referred linear; the sRGB transfer function is applied later,
# at the end of the shader, so do not pre-apply a gamma here
#
# The identity is y = x, and it is what an unrecognised body gets if `default:`
# is removed. It is also the wrong answer for almost every photograph: linear
# scene data has middle grey at about 13% and a camera JPEG puts it near 18%,
# so an uncurved render is roughly half a stop dark through the midtones and
# has no highlight rolloff at all.
#
# A curve that works has three parts, and it is worth naming them because they
# are what you are actually tuning:
#
# the toe the first span, slope near or below 1. Deep shadows stay
# deep. Lift it and blacks go milky; crush it and shadow
# detail the sensor recorded disappears.
# the midtones the middle spans, slope well above 1. This is the contrast
# and the brightness people read as "the camera's look".
# the shoulder the last span, slope well below 1. Highlights compress
# toward white instead of arriving there and clipping. It is
# the difference between a rolled-off sky and a white hole.
#
# Two invariants are enforced in code and tested, so a mistake here fails the
# build rather than the photograph: x must strictly increase, y must not
# decrease, and everything must lie inside the unit square.
#
# ---------------------------------------------------------------------------
# Honesty about these values
# ---------------------------------------------------------------------------
#
# These are hand-tuned shapes, not measurements. They encode what every camera
# JPEG rendering has in common — the toe/midtone/shoulder structure above —
# plus each maker's well-known house differences: Canon's gentler shoulder and
# warmer-reading midtones, Nikon's slightly higher midtone contrast, Sony's
# flatter and more conservative default, Fujifilm's markedly contrastier
# Provia-derived rendering.
#
# FR-DEV-3e's acceptance criterion is subjective comparison against each body's
# own JPEG, and meeting it properly needs a frame from that body in front of
# you. Where that has not been done, the entry is still much closer to right
# than the identity — which is the bar these have to clear, and do.
version: 1
# The rendering for a body with no entry of its own.
#
# **Deliberately not the identity.** The failure this requirement exists to fix
# is the flat render, and a conservative curve is far closer to right for every
# body than no curve is for any of them. It is gentler than the per-body
# entries below — a shallower midtone and an earlier, softer shoulder — because
# it has to be safe on a sensor nobody has looked at, and the cost of being too
# tame is a photograph that wants a little contrast rather than one that has
# lost its highlights.
default:
points:
- [0.00, 0.000]
- [0.04, 0.043]
- [0.13, 0.175]
- [0.45, 0.690]
- [1.00, 1.000]
bodies:
# Canon. A soft toe and a long, gradual shoulder — the reason Canon files
# are described as forgiving in highlights and a little low in contrast
# straight out of camera.
- make: Canon
model: EOS 6D
points:
- [0.00, 0.000]
- [0.04, 0.045]
- [0.13, 0.190]
- [0.45, 0.720]
- [1.00, 1.000]
- make: Canon
model: EOS R6
points:
- [0.00, 0.000]
- [0.04, 0.044]
- [0.13, 0.195]
- [0.45, 0.730]
- [1.00, 1.000]
# Nikon. A slightly deeper toe and more midtone slope than Canon, which is
# the "punchier out of camera" difference people describe between the two.
- make: Nikon
model: Z 6
points:
- [0.00, 0.000]
- [0.04, 0.038]
- [0.13, 0.200]
- [0.46, 0.750]
- [1.00, 1.000]
- make: Nikon
model: D750
points:
- [0.00, 0.000]
- [0.04, 0.039]
- [0.13, 0.198]
- [0.46, 0.745]
- [1.00, 1.000]
# Sony. The flattest default of the four, and intentionally so — Sony's own
# rendering leaves more headroom than it uses, which is why Sony files are
# the ones people describe as needing the most work.
- make: Sony
model: ILCE-7M3
points:
- [0.00, 0.000]
- [0.04, 0.048]
- [0.13, 0.185]
- [0.44, 0.700]
- [1.00, 1.000]
# Fujifilm. Provia, the default film simulation: a firm toe, the steepest
# midtones here, and a hard shoulder. It is the most distinctive rendering of
# the four and the one where a flat render looks most obviously wrong.
#
# This entry does *not* read the in-RAF film simulation tag — that is
# FR-DEV-3f, and until it lands every Fujifilm file gets the Provia shape
# whatever the camera was set to.
- make: Fujifilm
model: X-T3
points:
- [0.00, 0.000]
- [0.045, 0.040]
- [0.14, 0.215]
- [0.47, 0.775]
- [1.00, 1.000]
core/dr-decode/src/base_curve.rs
+763
View File
@@ -0,0 +1,763 @@
//! TRACES: FR-DEV-3e
//! Base curves — the per-body rendering that turns a correct exposure into a
//! photograph.
//!
//! # What this is for
//!
//! A camera matrix gets the *colours* right and leaves the picture flat. Sensor
//! data is scene-referred and very nearly linear; a print, a screen and a
//! camera's own JPEG are none of those things. Rendering linear data straight
//! out is the dcraw default, and FR-DEV-3e names it precisely: "the flat,
//! poor-skin-tone rendering characteristic of dcraw defaults, which is the
//! documented reason people abandon darktable in the first hour."
//!
//! The fix is a tone curve applied as part of *reading* the file rather than as
//! an edit — a toe, a steep midtone, and a shoulder that rolls highlights off
//! instead of clipping them. Every raw converter has one. Adobe calls it the
//! camera profile's tone curve, darktable calls it the base curve, and the name
//! here follows darktable's because the placement does too: it runs in camera
//! RGB, after white balance and the user's adjustments, immediately before the
//! conversion out to a working space.
//!
//! # Why it is not an edit
//!
//! It never reaches the sidecar and there is no slider for it, for the same
//! reason the EXIF orientation is not an edit (FR-DEV-3h): it is a property of
//! the body that took the frame, not of what anyone decided about the frame.
//! Sidecars are shared between devices and bodies (FR-NC-9), and one camera's
//! rendering must not follow an edit onto another camera's file.
//!
//! # Why it is data
//!
//! FR-DEV-3e requires the profile database to be "versioned independently of
//! the app binary so bodies and curves can be added without a release — and,
//! under D8's GPLv3, contributed by users". So the curves live in
//! `profiles/base_curves.yaml`, a file that is compiled in as a floor and
//! *overridden* by a copy on disk carrying a higher `version:`. Adding a body
//! is adding ten numbers to a YAML file; shipping that body to users is
//! publishing the file. Neither is a code change and neither needs a release.
//!
//! See [`load`] for the search path and [`Curves::body`] for the matching.
use std::path::{Path, PathBuf};
use std::sync::OnceLock;
/// How many control points a base curve has.
///
/// Five, which is not a coincidence: it is what the tone curve widget uses
/// (`dr_pipeline::ops::curve::POINTS`), so the shader evaluates a profile's
/// curve and a photographer's curve through exactly the same spline. A profile
/// author and a photographer dragging a point mean the same thing by it, and
/// the generated shader carries one implementation rather than two that could
/// disagree.
pub const POINTS: usize = 5;
/// TRACES: FR-DEV-3e
/// A base curve: five points on a monotone spline through the unit square.
///
/// `xs` is scene-linear camera RGB, normalised so that 1.0 is the sensor's
/// saturation point. `ys` is display-referred linear — *not* gamma-encoded,
/// because the sRGB transfer function is applied at the very end of the
/// generated shader and applying it twice would wash the image out.
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct BaseCurve {
pub xs: [f32; POINTS],
pub ys: [f32; POINTS],
}
impl BaseCurve {
/// The curve that does nothing — the identity diagonal.
///
/// What an unrecognised body gets if the database carries no default, and
/// what a JPEG gets always: an already-rendered image must not be rendered
/// a second time.
pub const IDENTITY: Self = Self {
xs: [0.0, 0.25, 0.5, 0.75, 1.0],
ys: [0.0, 0.25, 0.5, 0.75, 1.0],
};
/// Whether this curve would leave the image alone.
///
/// The shader is told to skip the stage entirely when it would, so an
/// unprofiled body costs a branch that is uniform across the dispatch
/// rather than a spline evaluation per channel per pixel.
pub fn is_identity(&self) -> bool {
self.xs
.iter()
.zip(self.ys.iter())
.all(|(x, y)| (x - y).abs() < 1e-6)
}
/// Build from raw pairs, rejecting anything that is not a curve.
///
/// A profile file is data a user may have edited, so this is the boundary
/// where "ten numbers" becomes "a curve": the x coordinates must increase,
/// the y coordinates must not decrease, and both must lie in the unit
/// square. A non-monotone x sends the spline's span search backwards and
/// divides by a negative width; a decreasing y inverts tones locally,
/// which reads as a dark halo through smooth gradients rather than as a
/// bad profile.
///
/// Endpoints are not forced to (0,0) and (1,1). A curve that lifts black
/// slightly, or that places the shoulder below white, is a legitimate
/// rendering choice and several bodies make it.
pub fn from_points(points: &[[f32; 2]]) -> Option<Self> {
if points.len() != POINTS {
return None;
}
let mut xs = [0.0f32; POINTS];
let mut ys = [0.0f32; POINTS];
for (i, p) in points.iter().enumerate() {
if !p[0].is_finite() || !p[1].is_finite() {
return None;
}
if !(0.0..=1.0).contains(&p[0]) || !(0.0..=1.0).contains(&p[1]) {
return None;
}
xs[i] = p[0];
ys[i] = p[1];
}
for i in 1..POINTS {
// Strictly increasing in x — the spline divides by the span width.
if xs[i] <= xs[i - 1] {
return None;
}
// Non-decreasing in y. Flat is allowed: a curve that holds a
// highlight range at white is clipping deliberately.
if ys[i] < ys[i - 1] {
return None;
}
}
Some(Self { xs, ys })
}
}
/// One body's entry in the database.
#[derive(Debug, Clone, PartialEq)]
pub struct BodyCurve {
/// The manufacturer, as the file writes it — "Canon", "NIKON CORPORATION".
pub make: String,
/// The model, as the file writes it — "EOS 6D", "ILCE-7M3".
pub model: String,
pub curve: BaseCurve,
}
/// TRACES: FR-DEV-3e
/// The base curve database.
///
/// Versioned as a whole rather than per body, because that is the unit a user
/// downloads and the unit that has to beat the built-in copy. See [`load`].
#[derive(Debug, Clone, PartialEq)]
pub struct Curves {
version: u32,
default: Option<BaseCurve>,
bodies: Vec<BodyCurve>,
}
impl Curves {
/// TRACES: FR-DEV-3e
/// The curve to render a frame from this body with.
///
/// Falls back, in order, to the database's `default:` and then to the
/// identity. **The default is deliberately not the identity**: an
/// unrecognised body rendered flat is the failure this requirement exists
/// to prevent, and a gentle, conservative curve is much closer to right for
/// every body than no curve is for any of them. A body with its own entry
/// gets that instead.
///
/// # What "this body" has to survive
///
/// The same camera names itself three ways depending on which program last
/// touched the file. A native NEF says make "NIKON CORPORATION", model
/// "NIKON Z 6"; rawler's own database cleans that to "Nikon" and "Z 6"; an
/// Adobe-converted DNG keeps the uncleaned pair. A database that had to
/// spell every variant would go stale the first time a maker changed its
/// mind about its own name, so the matching does the folding instead:
///
/// - Case, punctuation and runs of whitespace are flattened, so
/// "ILCE-7M3", "ILCE 7M3" and "ilce-7m3" are one body.
/// - The make is compared on its **first word only**. Every maker's
/// trailing corporate boilerplate — "CORPORATION", "IMAGING CORP" — is
/// noise, and no two camera manufacturers share a first word.
/// - The model is tried both as written and with a leading copy of the
/// make removed, which is what lets one "Canon"/"EOS 6D" entry cover
/// "Canon EOS 6D" as well.
pub fn body(&self, make: &str, model: &str) -> BaseCurve {
let (make, model) = (make_key(make), normalise(model));
// The model with a leading copy of the maker's name removed.
let bare = model.strip_prefix(&format!("{make} ")).unwrap_or(&model);
self.bodies
.iter()
.find(|b| {
let entry_model = normalise(&b.model);
make_key(&b.make) == make && (entry_model == model || entry_model == bare)
})
.map(|b| b.curve)
.or(self.default)
.unwrap_or(BaseCurve::IDENTITY)
}
/// The database version. Higher wins; see [`load`].
pub fn version(&self) -> u32 {
self.version
}
/// How many bodies have their own curve, excluding the default.
pub fn len(&self) -> usize {
self.bodies.len()
}
pub fn is_empty(&self) -> bool {
self.bodies.is_empty()
}
/// Parse a database from YAML.
///
/// Entries that are not curves are dropped with a warning rather than
/// failing the parse. A user-contributed file with one bad body should
/// cost that body's rendering, not every body's — and the alternative is an
/// application that will not open a photograph because somebody typed a
/// comma.
pub fn parse(yaml: &str) -> Result<Self, String> {
let file: File = serde_norway::from_str(yaml).map_err(|e| e.to_string())?;
let default = file.default.and_then(|d| {
BaseCurve::from_points(&d.points).or_else(|| {
log::warn!("base curves: the default entry is not a monotone curve; ignoring it");
None
})
});
let bodies = file
.bodies
.into_iter()
.filter_map(|b| match BaseCurve::from_points(&b.points) {
Some(curve) => Some(BodyCurve {
make: b.make,
model: b.model,
curve,
}),
None => {
log::warn!(
"base curves: {} {} is not a monotone curve; ignoring it",
b.make,
b.model
);
None
}
})
.collect();
Ok(Self {
version: file.version,
default,
bodies,
})
}
}
/// The copy that ships inside the binary.
///
/// A floor, not the answer: [`load`] prefers a newer file on disk. Compiled in
/// so that a fresh install with no profile directory — and every Android build,
/// where there is no such directory to speak of — still renders properly.
const BUILT_IN: &str = include_str!("../profiles/base_curves.yaml");
/// TRACES: FR-DEV-3e
/// The base curve database, loaded once.
///
/// # The search path, and why it is a version comparison
///
/// 1. `$DARKROOM_PROFILES`, a directory, when set. The escape hatch: a profile
/// author iterating on a curve points this at their working copy and does
/// not have to install anything.
/// 2. `$XDG_DATA_HOME/darkroom/profiles/`, else `$HOME/.local/share/darkroom/profiles/`.
/// The same base directory the catalog uses, chosen there for the same
/// reason — it is data, not cache, and must survive a storage sweep.
/// 3. The copy compiled into the binary.
///
/// The first file that parses *and carries a higher `version:` than the
/// built-in copy* wins. The version check is the whole mechanism the
/// requirement asks for, and it runs in both directions:
///
/// - A downloaded pack at version 7 supersedes a binary shipping version 3, so
/// a body added after the release renders correctly with no release.
/// - A stale pack at version 2 does **not** supersede a binary shipping version
/// 3, so upgrading the application cannot silently lose curves to a file
/// somebody downloaded a year ago and forgot.
///
/// Failures are warnings, never errors. A malformed profile file must cost the
/// user their curves, not their photographs.
pub fn load() -> &'static Curves {
static LOADED: OnceLock<Curves> = OnceLock::new();
LOADED.get_or_init(|| {
let built_in = Curves::parse(BUILT_IN).unwrap_or_else(|e| {
// Unreachable in a build that ran its tests — `the_shipped_database_parses`
// asserts exactly this — but a panic here would mean an
// application that cannot open a photograph because of a typo in a
// data file, which is never the right trade.
log::error!("base curves: the built-in database does not parse: {e}");
Curves {
version: 0,
default: None,
bodies: Vec::new(),
}
});
choose(built_in, &search_path())
})
}
/// The version comparison, separated from where the directories come from.
///
/// Split out so it can be tested against real files in a real directory
/// without the process-wide `OnceLock` and the environment `load` reads. The
/// rule this implements is the whole of what FR-DEV-3e asks for, so it is
/// worth being able to state it as a test rather than as a comment.
fn choose(built_in: Curves, dirs: &[PathBuf]) -> Curves {
for dir in dirs {
let path = dir.join("base_curves.yaml");
let Ok(text) = std::fs::read_to_string(&path) else {
continue;
};
match Curves::parse(&text) {
Ok(external) if external.version > built_in.version => {
log::info!(
"base curves: using {} (version {}, {} bodies) over the built-in version {}",
path.display(),
external.version,
external.len(),
built_in.version
);
return external;
}
Ok(external) => log::info!(
"base curves: ignoring {} at version {}; the built-in database is version {}",
path.display(),
external.version,
built_in.version
),
Err(e) => log::warn!("base curves: {} does not parse: {e}", path.display()),
}
}
built_in
}
/// TRACES: FR-DEV-3e
/// The curve for a body, from the loaded database.
///
/// The one call site the decoder needs; everything above is reachable for
/// tests and for a future profile editor.
pub fn for_body(make: &str, model: &str) -> BaseCurve {
load().body(make, model)
}
/// Directories that may hold a `base_curves.yaml`, most specific first.
fn search_path() -> Vec<PathBuf> {
let mut dirs = Vec::new();
if let Some(explicit) = std::env::var_os("DARKROOM_PROFILES") {
dirs.push(PathBuf::from(explicit));
}
// The same resolution `dr_ui::library::catalog_path` uses, and for the
// same reason: this is data a user may have installed, not a cache. It is
// duplicated rather than shared because `dr-decode` sits far below the UI
// and must not acquire a dependency on it to find a directory.
let base = std::env::var_os("XDG_DATA_HOME")
.map(PathBuf::from)
.or_else(|| std::env::var_os("HOME").map(|h| Path::new(&h).join(".local/share")));
if let Some(base) = base {
dirs.push(base.join("darkroom").join("profiles"));
}
dirs
}
/// A manufacturer's first word, folded.
///
/// "NIKON CORPORATION", "Nikon" and "nikon" all become `NIKON`. The corporate
/// suffixes are not information — they appear or not depending on whether the
/// file went through a DNG converter — and no two camera manufacturers share a
/// first word, so nothing is lost by dropping them.
fn make_key(s: &str) -> String {
normalise(s)
.split(' ')
.next()
.unwrap_or_default()
.to_string()
}
/// Fold a make or model into something two files can agree on.
///
/// Upper-cased, with every run of non-alphanumeric characters collapsed to one
/// space and the ends trimmed, so that "ILCE-7M3", "ILCE 7M3" and "ilce-7m3"
/// become one.
fn normalise(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut pending_space = false;
for c in s.chars() {
if c.is_ascii_alphanumeric() {
if pending_space && !out.is_empty() {
out.push(' ');
}
pending_space = false;
out.push(c.to_ascii_uppercase());
} else {
pending_space = true;
}
}
out
}
// ---- The on-disk shape, kept apart from the in-memory one ----------------
//
// Deliberately separate types. The file is data a user edits and is allowed to
// be wrong; `Curves` is a parsed database whose every entry is known to be a
// monotone curve. Deriving `Deserialize` on `BaseCurve` directly would delete
// that boundary and let an unchecked five-point array reach the shader.
//
// Unknown fields are **accepted**, which is not laziness. The database is
// versioned independently of the binary and moves in both directions: a pack
// published after this release may carry keys this build has never heard of —
// a hue twist, a look table (FR-DEV-3f) — and it must still deliver its curves
// to an older DarkRoom rather than failing to parse and leaving every body
// flat. `deny_unknown_fields` would trade that for a diagnostic nobody needs.
#[derive(serde::Deserialize)]
struct File {
version: u32,
#[serde(default)]
default: Option<Entry>,
#[serde(default)]
bodies: Vec<BodyEntry>,
}
#[derive(serde::Deserialize)]
struct Entry {
points: Vec<[f32; 2]>,
}
#[derive(serde::Deserialize)]
struct BodyEntry {
make: String,
model: String,
points: Vec<[f32; 2]>,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_shipped_database_parses_and_carries_a_default() {
// The one test that must never be allowed to fail quietly: `load`
// degrades to an empty database rather than panicking, so without this
// a typo in the YAML would ship as "every photograph renders flat"
// rather than as a build failure.
let curves = Curves::parse(BUILT_IN).expect("the shipped database parses");
assert!(curves.version() >= 1);
assert!(!curves.is_empty(), "the database ships bodies");
assert!(
!curves.body("Nobody", "Nothing").is_identity(),
"an unknown body must still get the default rendering"
);
}
#[test]
fn every_shipped_curve_lifts_the_midtones_and_rolls_the_highlights() {
// What makes a base curve a base curve rather than a decoration. If a
// shipped curve failed either half it would be a worse rendering than
// the flat one it replaced, which is the one outcome forbidden.
let curves = Curves::parse(BUILT_IN).expect("parses");
let all = curves
.bodies
.iter()
.map(|b| (format!("{} {}", b.make, b.model), b.curve))
.chain(curves.default.map(|c| ("default".to_string(), c)));
for (name, curve) in all {
// The midtone point sits above the diagonal: a linear midtone is
// roughly a stop and a half darker than any camera renders it.
let mid = 2;
assert!(
curve.ys[mid] > curve.xs[mid],
"{name} does not lift its midtones ({} -> {})",
curve.xs[mid],
curve.ys[mid]
);
// And the last span is shallower than the one before it, which is
// what a shoulder *is*. Without one the curve clips highlights
// harder than the linear rendering did.
let slope = |i: usize| (curve.ys[i + 1] - curve.ys[i]) / (curve.xs[i + 1] - curve.xs[i]);
assert!(
slope(POINTS - 2) < slope(POINTS - 3),
"{name} has no highlight shoulder"
);
}
}
#[test]
fn a_curve_that_is_not_monotone_is_refused() {
// The profile file is user-editable, so this is a real boundary and
// not a formality. A decreasing y inverts tones locally and shows up
// as a dark halo in a gradient, which reads as a rendering fault
// rather than as a bad profile.
assert_eq!(
BaseCurve::from_points(&[
[0.0, 0.0],
[0.25, 0.4],
[0.5, 0.3],
[0.75, 0.8],
[1.0, 1.0]
]),
None
);
}
#[test]
fn a_curve_whose_x_does_not_advance_is_refused() {
// The spline divides by the span width; a repeated x is a division by
// zero in the shader, which is a NaN pixel rather than an error.
assert_eq!(
BaseCurve::from_points(&[
[0.0, 0.0],
[0.25, 0.3],
[0.25, 0.5],
[0.75, 0.8],
[1.0, 1.0]
]),
None
);
}
#[test]
fn a_curve_of_the_wrong_length_is_refused() {
assert_eq!(BaseCurve::from_points(&[[0.0, 0.0], [1.0, 1.0]]), None);
}
#[test]
fn values_outside_the_unit_square_are_refused() {
// The shader clamps its output at the very end anyway, but a control
// point above 1.0 would put the shoulder outside the range the curve
// is defined over and silently flatten everything below it.
assert_eq!(
BaseCurve::from_points(&[
[0.0, 0.0],
[0.25, 0.3],
[0.5, 1.4],
[0.75, 1.5],
[1.0, 1.6]
]),
None
);
}
#[test]
fn a_body_with_its_own_entry_beats_the_default() {
let curves = Curves::parse(
"version: 2
default:
points: [[0.0, 0.0], [0.25, 0.3], [0.5, 0.6], [0.75, 0.85], [1.0, 1.0]]
bodies:
- make: Canon
model: EOS 6D
points: [[0.0, 0.0], [0.25, 0.35], [0.5, 0.7], [0.75, 0.9], [1.0, 1.0]]
",
)
.expect("parses");
assert_eq!(curves.body("Canon", "EOS 6D").ys[1], 0.35);
assert_eq!(curves.body("Canon", "EOS 5D").ys[1], 0.30);
}
#[test]
fn the_make_may_be_repeated_in_the_model() {
// Canon writes "Canon" as the make and "Canon EOS 6D" as the model;
// rawler's cleaned strings drop the repetition and both reach here.
// One entry has to cover both or half the files on a card miss.
let curves = Curves::parse(
"version: 1
bodies:
- make: Canon
model: EOS 6D
points: [[0.0, 0.0], [0.25, 0.35], [0.5, 0.7], [0.75, 0.9], [1.0, 1.0]]
",
)
.expect("parses");
assert_eq!(curves.body("Canon", "Canon EOS 6D").ys[1], 0.35);
assert_eq!(curves.body("Canon", "EOS 6D").ys[1], 0.35);
assert_eq!(curves.body("CANON", "eos 6d").ys[1], 0.35);
}
#[test]
fn a_corporate_suffix_does_not_hide_a_body() {
// The same Z 6 arrives as "Nikon"/"Z 6" from rawler's camera database
// and as "NIKON CORPORATION"/"NIKON Z 6" from a DNG converted out of
// the same file. Both must find the entry, or converting a file to
// DNG would silently change how it renders.
let curves = Curves::parse(
"version: 1
bodies:
- make: Nikon
model: Z 6
points: [[0.0, 0.0], [0.25, 0.35], [0.5, 0.7], [0.75, 0.9], [1.0, 1.0]]
",
)
.expect("parses");
assert_eq!(curves.body("Nikon", "Z 6").ys[1], 0.35);
assert_eq!(curves.body("NIKON CORPORATION", "NIKON Z 6").ys[1], 0.35);
}
#[test]
fn punctuation_and_spacing_do_not_decide_whether_a_body_is_known() {
let curves = Curves::parse(
"version: 1
bodies:
- make: Sony
model: ILCE-7M3
points: [[0.0, 0.0], [0.25, 0.35], [0.5, 0.7], [0.75, 0.9], [1.0, 1.0]]
",
)
.expect("parses");
assert_eq!(curves.body("SONY", "ILCE 7M3").ys[1], 0.35);
assert_eq!(curves.body("sony", "ilce-7m3").ys[1], 0.35);
}
#[test]
fn one_bad_entry_does_not_cost_the_rest() {
// A user-contributed file with one typo should cost that body's
// rendering, not every body's.
let curves = Curves::parse(
"version: 1
bodies:
- make: Broken
model: Body
points: [[0.0, 0.0], [0.25, 0.9], [0.5, 0.1], [0.75, 0.9], [1.0, 1.0]]
- make: Canon
model: EOS 6D
points: [[0.0, 0.0], [0.25, 0.35], [0.5, 0.7], [0.75, 0.9], [1.0, 1.0]]
",
)
.expect("parses");
assert_eq!(curves.len(), 1);
assert_eq!(curves.body("Canon", "EOS 6D").ys[1], 0.35);
assert!(curves.body("Broken", "Body").is_identity());
}
#[test]
fn a_pack_from_the_future_still_delivers_its_curves() {
// The database is versioned independently of the binary, so a pack
// published after this build may carry keys this build has never heard
// of. It must still hand over the curves it does understand — failing
// the parse would leave every body flat, which is the exact failure
// FR-DEV-3e exists to prevent, delivered by the mechanism meant to
// prevent it.
let curves = Curves::parse(
"version: 9
look_table: ambitious
bodies:
- make: Canon
model: EOS 6D
hue_twist: [1, 2, 3]
points: [[0.0, 0.0], [0.25, 0.35], [0.5, 0.7], [0.75, 0.9], [1.0, 1.0]]
",
)
.expect("an unfamiliar key must not fail the parse");
assert_eq!(curves.version(), 9);
assert_eq!(curves.body("Canon", "EOS 6D").ys[1], 0.35);
}
#[test]
fn an_unknown_body_with_no_default_gets_the_identity() {
// Graceful fallback, stated as a property: never worse than a flat
// render, and never a curve tuned for somebody else's sensor when the
// database declines to offer one.
let curves = Curves::parse("version: 1\nbodies: []\n").expect("parses");
assert!(curves.body("Nobody", "Nothing").is_identity());
}
/// A directory holding one `base_curves.yaml`, unique to the caller.
fn a_pack_dir(name: &str, yaml: &str) -> PathBuf {
let dir = std::env::temp_dir().join(format!("darkroom-base-curves-{name}"));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).expect("a writable temp directory");
std::fs::write(dir.join("base_curves.yaml"), yaml).expect("write");
dir
}
const A_CANON_ENTRY: &str = "bodies:
- make: Canon
model: EOS 6D
points: [[0.0, 0.0], [0.25, 0.42], [0.5, 0.7], [0.75, 0.9], [1.0, 1.0]]
";
#[test]
fn a_newer_pack_on_disk_supersedes_the_built_in_database() {
// **This is the requirement.** FR-DEV-3e asks for a profile database
// versioned independently of the app binary "so bodies and curves can
// be added without a release". A file with a higher version, dropped
// in the profile directory, is what that means in practice.
let built_in = Curves::parse(BUILT_IN).expect("parses");
let newer = format!("version: {}\n{A_CANON_ENTRY}", built_in.version() + 1);
let dir = a_pack_dir("newer", &newer);
let chosen = choose(built_in.clone(), &[dir]);
assert_eq!(chosen.version(), built_in.version() + 1);
assert_eq!(chosen.body("Canon", "EOS 6D").ys[1], 0.42);
}
#[test]
fn a_stale_pack_does_not_survive_an_upgrade() {
// The other direction, and the one that protects the user. Somebody
// downloads a pack, a release later ships better curves for the same
// bodies, and the forgotten file must not quietly hold the application
// back at last year's rendering.
let built_in = Curves::parse(BUILT_IN).expect("parses");
let stale = format!("version: {}\n{A_CANON_ENTRY}", built_in.version());
let dir = a_pack_dir("stale", &stale);
let chosen = choose(built_in.clone(), &[dir]);
assert_eq!(chosen.version(), built_in.version());
assert_ne!(
chosen.body("Canon", "EOS 6D").ys[1],
0.42,
"an equal version must not displace the built-in database"
);
}
#[test]
fn a_broken_pack_costs_the_curves_and_not_the_photographs() {
// A malformed profile file must degrade to the built-in database, not
// to an error. The user came here to look at a photograph.
let built_in = Curves::parse(BUILT_IN).expect("parses");
let dir = a_pack_dir("broken", "version: [this is not a number\n");
let chosen = choose(built_in.clone(), &[dir]);
assert_eq!(chosen.version(), built_in.version());
assert_eq!(chosen.len(), built_in.len());
}
#[test]
fn a_directory_with_no_pack_in_it_is_simply_skipped() {
// The ordinary case on every machine: the search path exists, the file
// does not. It must not be a warning, an error, or a slow path.
let built_in = Curves::parse(BUILT_IN).expect("parses");
let missing = std::env::temp_dir().join("darkroom-base-curves-nothing-here");
let _ = std::fs::remove_dir_all(&missing);
assert_eq!(choose(built_in.clone(), &[missing]), built_in);
}
#[test]
fn the_identity_is_recognised_as_doing_nothing() {
assert!(BaseCurve::IDENTITY.is_identity());
assert!(!Curves::parse(BUILT_IN)
.expect("parses")
.body("Canon", "EOS 6D")
.is_identity());
}
}
core/dr-decode/src/lib.rs
+52 -66
View File
@@ -12,10 +12,13 @@
//! Fusing them would force a full decode where a header read suffices, which
//! is exactly why Lightroom stalls ~2 s per image during culling.
pub mod base_curve;
mod error;
mod locate;
mod preview;
pub mod profile;
pub use base_curve::BaseCurve;
pub use error::DecodeError;
pub use locate::{
defects, is_complete_jpeg, locate_preview, BadLine, BadPixel, Defects, PreviewLocation,
@@ -25,6 +28,7 @@ pub use preview::{
decode_jpeg, extract_embedded_preview, extract_preview, Preview, PreviewSize,
PREVIEW_PROBE_BYTES,
};
pub use profile::CameraProfile;
use dr_types::{Format, Orientation};
@@ -89,7 +93,24 @@ pub struct RawImage {
/// `None` where the body is unknown to the decoder, in which case the
/// pipeline falls back to identity and the result is uncalibrated rather
/// than wrong-by-a-guess.
///
/// Where the body carries two calibrations this is already *interpolated*
/// for the light the frame was shot under; see [`profile::CameraProfile`].
pub color_matrix: Option<[f32; 9]>,
/// TRACES: FR-DEV-3e
/// The per-body rendering curve, the other half of the camera profile.
///
/// The matrix above decides what the colours *are*; this decides what the
/// picture looks like. Carried on the decoded image rather than looked up
/// downstream because this is the only point in the system that knows
/// which body took the frame, and because it is not an edit: it belongs to
/// the file in the same way the masked-photosite crop does, and must never
/// reach a sidecar (FR-NC-9).
///
/// [`BaseCurve::IDENTITY`] for an unknown body with no default in the
/// database, which renders exactly as this decoder did before profiles
/// existed.
pub base_curve: BaseCurve,
/// The usable region of `data`, excluding masked and border photosites.
pub crop: CropRect,
}
@@ -421,15 +442,38 @@ pub fn decode(bytes: &[u8]) -> Result<RawImage, DecodeError> {
let source = RawSource::new_from_slice(bytes);
let decoder =
rawler::get_decoder(&source).map_err(|e| DecodeError::Unsupported(e.to_string()))?;
// Read before decoding, while the decoder is still the cheapest thing in
// the room. These are the DNG tags rawler parses into its IFD and then
// never surfaces — `ForwardMatrix1/2` above all — and they are empty for
// every non-DNG file, which is not a failure (FR-DEV-3e).
let dng = profile::read_dng_matrices(decoder.as_ref());
let image = decoder
.raw_image(&source, &Default::default(), false)
.map_err(|e| DecodeError::Decode(e.to_string()))?;
// Both derived before the match below moves `image.data`, and from the
// same matrix: the balance and the conversion must agree about which white
// is neutral or the frame carries a cast that looks like a decode fault.
let color_matrix = cam_to_srgb(&image);
let wb_coeffs = sane_wb(image.wb_coeffs, xyz_to_cam_of(&image).as_ref());
// The camera profile, and both of the things derived from it, are built
// before the match below moves `image.data`.
//
// They come from *one* profile deliberately: the balance and the
// conversion must agree about which white is neutral, or the frame carries
// a cast that looks like a decode fault. That agreement used to be
// maintained by hand — two functions reading the same illuminant key — and
// is now structural, because there is only one interpolated matrix and
// both callers ask the same object for it.
let profile = profile::CameraProfile::extract(&image, &dng);
let color_matrix = profile.as_ref().and_then(|p| p.cam_to_srgb());
let wb_coeffs = sane_wb(image.wb_coeffs, profile.as_ref().map(|p| p.xyz_to_cam()).as_ref());
// The rendering half of the profile (FR-DEV-3e). rawler's cleaned strings
// are preferred where it has them — they are what the shipped database is
// written against — and the matching folds the variants either way, so a
// DNG naming the same body differently still finds its curve.
let base_curve = base_curve::for_body(
image.camera.clean_make.as_str(),
image.camera.clean_model.as_str(),
);
let data = match image.data {
rawler::RawImageData::Integer(v) => v,
@@ -498,68 +542,10 @@ pub fn decode(bytes: &[u8]) -> Result<RawImage, DecodeError> {
.unwrap_or(u16::MAX),
wb_coeffs,
color_matrix,
base_curve,
})
}
/// TRACES: FR-DEV-3e
/// Compose the camera→sRGB-linear matrix from rawler's XYZ→camera.
///
/// **Two rawler traps this avoids**, both measured on a Canon 6D CR2
/// (2026-08-09):
///
/// 1. `RawImage::xyz_to_cam` is **all zeros** — it carries an upstream
/// deprecation note and 0.7.2 no longer fills it. The live data is
/// `color_matrix`, keyed by illuminant. Reading the old field silently
/// yields no colour transform at all.
/// 2. `cam_to_xyz_normalized()` divides each of four rows by its own sum, and
/// the fourth row (emerald/white, unused on any Bayer body) sums to zero.
/// Every element came back `NaN`. Inverting the 3×3 ourselves avoids the
/// fourth channel entirely.
fn cam_to_srgb(image: &rawler::RawImage) -> Option<[f32; 9]> {
use rawler::imgop::xyz::Illuminant;
// Prefer D65 — it matches sRGB's white point, so no chromatic adaptation
// is needed. Illuminant A (tungsten) is a distant fallback for bodies
// that ship only one matrix; adapting it properly is a v0.2 colour-
// management concern (ARCH §5.2), not something to fake here.
let flat = image
.color_matrix
.get(&Illuminant::D65)
.or_else(|| image.color_matrix.get(&Illuminant::A))?;
if flat.len() < 9 {
return None;
}
let xyz_to_cam: [[f32; 3]; 3] = [
[flat[0], flat[1], flat[2]],
[flat[3], flat[4], flat[5]],
[flat[6], flat[7], flat[8]],
];
cam_to_srgb_from(&xyz_to_cam)
}
/// The camera's XYZ→camera matrix, as rawler holds it.
///
/// Split out so the white-balance fallback and the colour matrix read the same
/// data through the same illuminant preference; two readers disagreeing about
/// which matrix a body uses would balance to one white and convert from
/// another.
fn xyz_to_cam_of(image: &rawler::RawImage) -> Option<[[f32; 3]; 3]> {
use rawler::imgop::xyz::Illuminant;
let flat = image
.color_matrix
.get(&Illuminant::D65)
.or_else(|| image.color_matrix.get(&Illuminant::A))?;
if flat.len() < 9 {
return None;
}
Some([
[flat[0], flat[1], flat[2]],
[flat[3], flat[4], flat[5]],
[flat[6], flat[7], flat[8]],
])
}
/// The matrix maths, split out so it can be tested without a RAW file.
//
// The constants below are quoted at their published precision rather than
@@ -567,7 +553,7 @@ fn xyz_to_cam_of(image: &rawler::RawImage) -> Option<[[f32; 3]; 3]> {
// a linter makes it harder to check against the specification, and the
// rounding happens identically either way.
#[allow(clippy::excessive_precision)]
fn cam_to_srgb_from(xyz_to_cam: &[[f32; 3]; 3]) -> Option<[f32; 9]> {
pub(crate) fn cam_to_srgb_from(xyz_to_cam: &[[f32; 3]; 3]) -> Option<[f32; 9]> {
// XYZ (D65) → linear sRGB, the standard primaries.
const XYZ_TO_SRGB: [[f32; 3]; 3] = [
[3.2404542, -1.5371385, -0.4985314],
@@ -722,7 +708,7 @@ pub fn daylight_wb(xyz_to_cam: &[[f32; 3]; 3]) -> Option<[f32; 3]> {
}
/// Invert a 3×3 matrix, or `None` if it is singular.
fn invert3(m: &[[f32; 3]; 3]) -> Option<[[f32; 3]; 3]> {
pub(crate) fn invert3(m: &[[f32; 3]; 3]) -> Option<[[f32; 3]; 3]> {
let det = m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1])
- m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0])
+ m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0]);
core/dr-decode/src/profile.rs
+1081
View File
File diff suppressed because it is too large Load Diff
core/dr-gpu/src/adjust.rs
+27 -1
View File
@@ -32,6 +32,16 @@ use crate::{DemosaicedImage, GpuContext, GpuError};
/// reads them.
const RESERVED_FIELDS: usize = dr_pipeline::RESERVED_UNIFORM_FIELDS;
/// TRACES: FR-DEV-3e
/// The two crates must agree on how many points a base curve has.
///
/// `dr-decode` reads them from the profile database and `dr-pipeline` declares
/// the uniform slots; this file is the only place the two meet, and it packs
/// them by index. A disagreement would not fail to compile — it would upload a
/// curve with a point missing or a stale float in it, which renders as a
/// plausible-looking wrong tone response. Cheaper to catch here, at build time.
const _: () = assert!(dr_decode::base_curve::POINTS == dr_pipeline::BASE_CURVE_POINTS);
/// Runs composed operation chains against demosaiced images.
pub struct AdjustPass {
ctx: GpuContext,
@@ -332,6 +342,18 @@ impl AdjustPass {
// runs. See `DemosaicedImage::is_non_linear`.
let non_linear = if source.is_non_linear() { 1.0 } else { 0.0 };
uniforms[12..16].copy_from_slice(&[wb[0], wb[1], wb[2], non_linear]);
// TRACES: FR-DEV-3e
// The camera profile's base curve, packed the way the generated block
// declares it: four x, four y, then the fifth point and the flag. The
// flag is what lets one compiled shader serve a profiled body and an
// unprofiled one, so the pipeline cache is not split in two by which
// camera took the frame.
let curve = source.base_curve();
let on = if curve.is_identity() { 0.0 } else { 1.0 };
let b = dr_pipeline::BASE_CURVE_UNIFORM_OFFSET;
uniforms[b..b + 4].copy_from_slice(&curve.xs[0..4]);
uniforms[b + 4..b + 8].copy_from_slice(&curve.ys[0..4]);
uniforms[b + 8..b + 12].copy_from_slice(&[curve.xs[4], curve.ys[4], on, 0.0]);
let params_buf = self
.ctx
@@ -512,7 +534,7 @@ fn numbered(src: &str) -> String {
#[cfg(test)]
mod tests {
use super::*;
use dr_decode::{CfaPattern, CropRect, RawImage};
use dr_decode::{BaseCurve, CfaPattern, CropRect, RawImage};
use dr_pipeline::ops::{colour_mixer, exposure, saturation};
use dr_pipeline::EditGraph;
@@ -546,6 +568,7 @@ mod tests {
// Identity, so the test reasons about the operations alone
// rather than about a camera's colour response.
color_matrix: Some([1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]),
base_curve: BaseCurve::IDENTITY,
crop: CropRect {
x: 0,
y: 0,
@@ -719,6 +742,7 @@ mod tests {
white_level: 16383,
wb_coeffs: [1.0, 1.0, 1.0, 1.0],
color_matrix: Some([1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]),
base_curve: BaseCurve::IDENTITY,
crop: CropRect {
x: 0,
y: 0,
@@ -1141,6 +1165,7 @@ mod tests {
white_level: 16383,
wb_coeffs: [1.0, 1.0, 1.0, 1.0],
color_matrix: Some([1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]),
base_curve: BaseCurve::IDENTITY,
crop: CropRect {
x: 0,
y: 0,
@@ -1240,6 +1265,7 @@ mod tests {
white_level: 16383,
wb_coeffs: [1.0, 1.0, 1.0, 1.0],
color_matrix: Some([1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]),
base_curve: BaseCurve::IDENTITY,
crop: CropRect {
x: 0,
y: 0,
core/dr-gpu/src/demosaic.rs
+34 -1
View File
@@ -10,7 +10,7 @@
//! pass over this texture; it does not re-demosaic, which is what keeps the
//! interaction budget (NFR-P9) reachable on a 24 MP file.
use dr_decode::{CfaPattern, RawImage};
use dr_decode::{BaseCurve, CfaPattern, RawImage};
use wgpu::util::DeviceExt;
use crate::{GpuContext, GpuError};
@@ -83,6 +83,16 @@ pub struct DemosaicedImage {
color_matrix: [f32; 9],
/// As-shot white balance, the neutral starting point for the WB control.
as_shot_wb: [f32; 3],
/// TRACES: FR-DEV-3e
/// The camera profile's rendering curve, carried through for the adjust
/// pass exactly as `color_matrix` is.
///
/// It rides on the image rather than on the edit graph because it is not
/// an edit: it belongs to the body that took the frame, the way the
/// masked-photosite crop and the EXIF orientation do, and a sidecar shared
/// between two bodies must never carry one body's rendering onto the
/// other's file (FR-NC-9).
base_curve: BaseCurve,
/// Whether the texture holds gamma-encoded rather than linear values.
non_linear: bool,
}
@@ -108,6 +118,15 @@ impl DemosaicedImage {
self.color_matrix
}
/// TRACES: FR-DEV-3e
/// The camera profile's base curve, as five `(x, y)` points.
///
/// [`BaseCurve::IDENTITY`] where the body is unprofiled or the source was
/// never raw, in which case the adjust pass skips the stage entirely.
pub fn base_curve(&self) -> BaseCurve {
self.base_curve
}
/// As-shot white balance multipliers, green-normalised.
///
/// The white balance control is expressed *relative* to these, so its
@@ -220,6 +239,12 @@ impl DemosaicedImage {
height,
color_matrix: IDENTITY_3X3,
as_shot_wb: [1.0, 1.0, 1.0],
// **The identity, and this is the whole reason the field is here
// rather than resolved further down.** A JPEG has already had its
// camera's base curve baked in by the camera; applying one again
// would render the rendering, crushing the shadows and flattening
// the highlights of an image that was already finished.
base_curve: BaseCurve::IDENTITY,
non_linear: true,
})
}
@@ -495,6 +520,10 @@ impl Demosaicer {
// no colour transform rather than not at all.
color_matrix: raw.color_matrix.unwrap_or(IDENTITY_3X3),
as_shot_wb: [raw.wb_coeffs[0], raw.wb_coeffs[1], raw.wb_coeffs[2]],
// Whatever the profile database had for this body (FR-DEV-3e),
// resolved at decode because that is the only place the make and
// model are known.
base_curve: raw.base_curve,
// Sensor data is linear by construction — the demosaic shader
// normalises against black and white levels and applies no
// transfer function.
@@ -808,6 +837,7 @@ mod tests {
white_level: white,
wb_coeffs: [1.0, 1.0, 1.0, 1.0],
color_matrix: None,
base_curve: BaseCurve::IDENTITY,
crop: CropRect {
x: 0,
y: 0,
@@ -919,6 +949,7 @@ mod tests {
white_level: white,
wb_coeffs: [1.0, 1.0, 1.0, 1.0],
color_matrix: None,
base_curve: BaseCurve::IDENTITY,
crop: CropRect {
x: 0,
y: 0,
@@ -1164,6 +1195,7 @@ mod tests {
white_level: 16383,
wb_coeffs: [1.0, 1.0, 1.0, 1.0],
color_matrix: None,
base_curve: BaseCurve::IDENTITY,
crop: CropRect {
x: 0,
y: 0,
@@ -1244,6 +1276,7 @@ mod tests {
1.0,
],
color_matrix: None,
base_curve: BaseCurve::IDENTITY,
crop: CropRect {
x: 0,
y: 0,
core/dr-pipeline/ops/README.md
+38
View File
@@ -216,6 +216,44 @@ carries lens-profile coefficients that are not parameters. `distortion` and
`aberration` are `Warp`s rather than operations: they rewrite coordinates
before sampling rather than transforming a colour after it.
## What is not a node, and why
Three things act on every pixel and are deliberately not in this directory:
the as-shot white balance, the camera matrix, and the **base curve**
(FR-DEV-3e). They are emitted by [`../src/operation.rs`](../src/operation.rs)
into the composed shader's fixed preamble, around the block of nodes.
The test is not "does it transform a colour" — all three do. It is **whose
decision is it**. A node is something a photographer chose: it has parameters,
it moves off a neutral, it lands in the sidecar, it can be undone. These three
are properties of the *file*, at the same standing as the masked-photosite crop
(FR-RAW-3) and the stored orientation (FR-DEV-3h). Nobody chose the sensor's
green sensitivity or the body's rendering; they are what reading the file
correctly means.
Making the base curve a node would have said the opposite in four places at
once. It would have appeared in the develop panel as a control, so an
unprofiled body would show a slider that does nothing. Its values would have
gone into the sidecar, and sidecars are shared between devices and bodies
(FR-NC-9) — one camera's rendering would follow an edit onto another camera's
file. Its neutral would have had to be "the identity", so a profiled body would
open reporting itself modified. And there is no seam through which a node could
learn which camera took the frame: the profile arrives on the decoded image,
travels through `DemosaicedImage` beside the matrix it belongs with, and is
written into the uniform block by the same three lines in `dr-gpu` — which is
exactly the path the matrix already took, because it is exactly the same kind
of thing.
What it *does* share with the tone curve node is the spline. The composer asks
`ToneCurve` for its `curve_span`/`curve_eval` helpers rather than emitting a
second copy, so a profile author placing a control point and a photographer
dragging one mean the same thing by it.
The order still reads correctly from this directory: the base curve runs after
every node in the chain and before the conversion out of camera space. That is
the same reasoning `exposure` records under `placement:` — corrections to
capture are only meaningful on linear values, so the rendering goes last.
## Errors
The build script reports failures by naming the key you got wrong, and exits
core/dr-pipeline/src/lib.rs
+1 -1
View File
@@ -52,7 +52,7 @@ pub use history::{Edit, History};
pub use lens::{compose_warps, ComposedWarp, Warp};
pub use operation::{
compose, compose_with_framing, Affects, ComposedShader, Helper, Operation, Uniform,
RESERVED_UNIFORM_FIELDS,
BASE_CURVE_POINTS, BASE_CURVE_UNIFORM_OFFSET, RESERVED_UNIFORM_FIELDS,
};
pub use preset::{Preset, Scope};
pub use sidecar::{Sidecar, Version};
core/dr-pipeline/src/operation.rs
+194 -2
View File
@@ -145,7 +145,35 @@ pub struct ComposedShader {
///
/// WGSL requires a uniform struct to be non-empty and 16-byte aligned; these
/// are needed by every generated shader in any case.
const BASE_UNIFORM_FIELDS: usize = 16;
///
/// Twelve of the twenty-eight are the camera profile's base curve
/// ([`BASE_CURVE_UNIFORM_FIELDS`]); the rest are the matrix, the as-shot
/// balance and framing's own block.
const BASE_UNIFORM_FIELDS: usize = 16 + BASE_CURVE_UNIFORM_FIELDS;
/// TRACES: FR-DEV-3e
/// Slots the base curve occupies: five `(x, y)` points and an active flag.
///
/// Twelve rather than eleven so the block stays a whole number of `vec4`s,
/// which is what std140 requires of a uniform struct's members. The spare
/// float is left zero rather than repurposed — a uniform slot that means one
/// thing today and two things next year is how a shader comes to read a
/// highlight rolloff out of a crop rectangle.
const BASE_CURVE_UNIFORM_FIELDS: usize = 12;
/// TRACES: FR-DEV-3e
/// Where the base curve's slots begin in the generated uniform block.
///
/// Exported for the same reason [`RESERVED_UNIFORM_FIELDS`] is: `dr-gpu`
/// writes these by index, and an offset computed independently at both ends is
/// an offset that will eventually disagree with itself.
pub const BASE_CURVE_UNIFORM_OFFSET: usize = 16;
/// How many control points a base curve carries.
///
/// The same five the tone curve widget has, deliberately — see the helper
/// selection in [`compose_full`].
pub const BASE_CURVE_POINTS: usize = 5;
/// Where an operation's own uniforms begin in the generated block.
///
@@ -235,10 +263,44 @@ pub fn compose_full(
\x20 // `.w` is not padding: it flags a non-linear source (1.0 for a\n\
\x20 // gamma-encoded JPEG, 0.0 for demosaiced sensor data), which the\n\
\x20 // prologue reads to decide whether to linearise.\n\
\x20 as_shot_wb: vec4<f32>,\n",
\x20 as_shot_wb: vec4<f32>,\n\
\x20 // The camera profile's base curve (FR-DEV-3e): five points on a\n\
\x20 // monotone spline, packed as x0..x3, y0..y3, then (x4, y4, on).\n\
\x20 // `.z` of the last is the flag, not padding — it is 0 for a\n\
\x20 // body with no profile and for an already-rendered source.\n\
\x20 base_curve_x: vec4<f32>,\n\
\x20 base_curve_y: vec4<f32>,\n\
\x20 base_curve_last: vec4<f32>,\n",
);
uniform_values.resize(BASE_UNIFORM_FIELDS, 0.0);
// TRACES: FR-DEV-3e
// The spline the base curve is evaluated on is the *tone curve's* spline,
// reached through the trait rather than reimplemented here.
//
// Two reasons, and the second is the one that matters. The obvious one is
// that a shader carrying two `curve_eval`s would not compile, and the
// composer's helper de-duplication is what makes both stages able to ask
// for it. The real one is that a profile author placing a control point
// and a photographer dragging one must mean the same thing by it — down to
// the Fritsch-Carlson tangent limiting, which is what decides how a
// shoulder actually rolls off. Two implementations that agreed today would
// be two that could disagree later, and the disagreement would show up as
// a body whose profile renders subtly differently from the curve someone
// drew to match it.
//
// Emitted unconditionally, unlike an operation's helpers. The base curve
// is active for every RAW frame — an unprofiled body still gets the
// database's default rendering — so making the shader's shape depend on it
// would split the pipeline cache in two for no benefit. The uniform flag
// above turns it off for the cases that are genuinely already rendered,
// and a branch on a uniform is coherent across the whole dispatch.
for h in crate::ops::ToneCurve::new().helpers() {
if matches!(h.name, "curve_span" | "curve_eval") {
helpers.push(*h);
}
}
// Framing's block follows the base one at a fixed offset, for the same
// reason: the prologue is emitted whether or not any operation is active,
// so these slots cannot be positioned by the op loop below.
@@ -419,6 +481,58 @@ fn main(@builtin(global_invocation_id) gid: vec3<u32>) {{
c = mix(c, neutral, clipped);
}}
{body}
// ==== camera profile: the base curve (FR-DEV-3e) ====
//
// Marked with `====` and not the `----` an operation block carries: this
// is not one, and the difference is what several tests count on to tell
// an edit apart from the reading of a file.
//
// The stage between demosaic and the working space that turns a correct
// exposure into a photograph. Sensor data is scene-referred and nearly
// linear; nothing anybody looks at is. Rendering it straight out is the
// dcraw default, and it is flat, dark through the midtones and clips its
// highlights instead of rolling them off.
//
// **In camera RGB, and after the adjustments**, which is a deliberate pair
// of choices:
//
// - Before the matrix, because that is where a base curve is defined and
// where every other converter applies one. The curve was tuned against
// this body's own primaries; moving it after the conversion would apply
// a Canon rendering to sRGB values and change what it does.
// - After exposure and the tonal operations, because those are corrections
// to *capture* and are only meaningful on linear values. A stop is a
// doubling; run exposure after a curve and it stops being one.
//
// Per channel rather than on luminance. It desaturates the extremes
// slightly, and that is the point — it is what makes a blown sky roll
// toward white rather than toward a saturated corner of the gamut, and it
// is what the camera's own JPEG does.
//
// The branch is on a uniform, so the whole dispatch takes the same path.
// It is off for a JPEG and any other already-rendered source, which must
// not be rendered twice, and for a body the profile database declines to
// offer any curve for at all.
if (u.base_curve_last.z > 0.5) {{
c = vec3<f32>(
curve_eval(
u.base_curve_x.x, u.base_curve_y.x, u.base_curve_x.y, u.base_curve_y.y,
u.base_curve_x.z, u.base_curve_y.z, u.base_curve_x.w, u.base_curve_y.w,
u.base_curve_last.x, u.base_curve_last.y, c.r,
),
curve_eval(
u.base_curve_x.x, u.base_curve_y.x, u.base_curve_x.y, u.base_curve_y.y,
u.base_curve_x.z, u.base_curve_y.z, u.base_curve_x.w, u.base_curve_y.w,
u.base_curve_last.x, u.base_curve_last.y, c.g,
),
curve_eval(
u.base_curve_x.x, u.base_curve_y.x, u.base_curve_x.y, u.base_curve_y.y,
u.base_curve_x.z, u.base_curve_y.z, u.base_curve_x.w, u.base_curve_y.w,
u.base_curve_last.x, u.base_curve_last.y, c.b,
),
);
}}
// Camera space -> linear sRGB. Applied after the adjustments so white
// balance and exposure act on sensor-native values, which is where they
// are physically meaningful.
@@ -960,6 +1074,84 @@ mod tests {
assert!(op < matrix, "the camera matrix must come after operations");
}
#[test]
fn the_base_curve_runs_after_the_operations_and_before_the_camera_matrix() {
// TRACES: FR-DEV-3e
// Both halves matter and for different reasons.
//
// After the operations: exposure and the tonal controls are
// corrections to capture, and they are only meaningful on linear
// values. A stop is a doubling; run exposure after a curve and it is
// not one any more, and every slider in the panel starts lying about
// what it does.
//
// Before the matrix: the curve was tuned against this body's own
// primaries. Applied after the conversion it would be a Canon
// rendering acting on sRGB values, which is a different curve.
let ops = vec![fake(&DESC_A, 2.0, false)];
let source = compose(&ops).source;
let op = source.find("---- op_a ----").expect("op present");
let curve = source
.find("if (u.base_curve_last.z > 0.5)")
.expect("base curve applied");
let matrix = source.find("u.cam_to_srgb_0").expect("matrix applied");
assert!(op < curve, "the base curve must come after the operations");
assert!(curve < matrix, "and before the camera matrix");
}
#[test]
fn the_base_curve_reaches_a_shader_with_no_operations_at_all() {
// TRACES: FR-DEV-3e
// The same property as as-shot white balance, and for the same reason:
// it is part of interpreting the file, not part of the edit. An
// unedited RAW must open looking like a photograph rather than like a
// scan of one.
let shader = compose(&[]);
assert!(shader.source.contains("u.base_curve_x"));
assert!(
shader.source.contains("fn curve_eval("),
"the spline it is evaluated on must be emitted too"
);
}
#[test]
fn the_base_curve_and_the_tone_curve_share_one_spline() {
// TRACES: FR-DEV-3e
// Two `curve_eval`s in one shader would not compile — but the reason
// the helper is *shared* rather than merely renamed is that a profile
// author placing a control point and a photographer dragging one must
// mean the same thing by it, down to the tangent limiting that decides
// how a shoulder rolls off.
let mut curve = crate::ops::ToneCurve::new();
curve.set_param(crate::ops::curve::P2_Y, 0.7);
assert!(curve.is_active(), "the fixture must actually reach the shader");
let source = compose(&[Box::new(curve)]).source;
assert_eq!(
source.matches("fn curve_eval(").count(),
1,
"the spline must be declared exactly once"
);
assert_eq!(source.matches("fn curve_span(").count(), 1);
}
#[test]
fn the_base_curve_owns_the_slots_dr_gpu_writes() {
// TRACES: FR-DEV-3e
// `dr-gpu` fills these by index. The offset is exported rather than
// recomputed there, and this asserts the exported number still points
// at the block the shader declares — the failure otherwise is a
// highlight rolloff read out of a crop rectangle, which renders as
// nonsense rather than as an error.
assert_eq!(
BASE_CURVE_UNIFORM_OFFSET + BASE_CURVE_UNIFORM_FIELDS,
BASE_UNIFORM_FIELDS,
"the base curve must be the last thing in the base block"
);
assert_eq!(BASE_CURVE_POINTS * 2 + 1, BASE_CURVE_UNIFORM_FIELDS - 1);
assert!(compose(&[]).uniforms.len() >= BASE_UNIFORM_FIELDS);
}
/// Compose with neutral framing into a chosen output space.
fn compose_to(ops: &[Box<dyn Operation>], output: ColourSpace) -> ComposedShader {
compose_with_framing(ops, &Framing::new(), output)