The scan reads the first 256 KB of a file for its metadata. A camera
writes its IFDs at the front, so that is the whole structure; the linear
DNG a merge writes puts its first IFD after the pixels, and rawler,
given the head alone, finds no decoder in it. The composite was
catalogued without a date and sorted to the very end of the grid, after
every dated photograph — which is where a panorama merged on the tablet
went unfound.
dr-decode's own TIFF reader now reads through a head and a tail at a
known offset; trailing_ifd says where the tail starts and
metadata_split reads the two together. The scan, when the head fails
and points beyond itself, fetches from the IFD to the end — kilobytes —
and dates the file from both. Tested against the writer's own output.
The Malvar "R at green in R row" kernel weights the two greens two
sites away along the row at -1 and the pair up and down the column at
+1/2. The shader had the two swapped, in the comment as well as the
code, so the transcription checked against itself. Both sum to zero
and reconstruct a flat patch exactly, which is all the tests fed it.
On an edge the correction at green sites is half strength and the
false colour doubles: 0.375 against 0.19 on a grey step, and a
blue/yellow zipper around every clipped highlight at 1:1. The other
three kernels and the CFA tables were right.
A grey vertical step now runs through the pass; the transposed kernel
fails it at 0.375.
The reference desktop's only system ONNX Runtime is Arch's
onnxruntime-opt-cuda: 1.29, built without TensorRT and against cuDNN 8
on a cuDNN 9 machine. The probe rejects both providers correctly and
the app runs on the CPU provider, which is right and not what anyone
wants. runtime/ beside the models is now searched ahead of /usr/lib,
tools/fetch-desktop-runtime.sh fills it with the four libraries from
the current onnxruntime-gpu wheel (cuDNN 9, TensorRT 10), and the
About caption lists every rung that lost and why, not only the first.
Verified: the app selects TensorRT from that directory with no
environment variable set.
The unpack list gained migan-512.onnx without its length following;
nothing on the desktop compiles that crate, and the first Android build
of 0.13.0 stopped there.
A fill that went wrong took a seven-minute merge to look at again. Now
DR_FILL_DUMP=dir makes the merge write what the filler was given, and
the fill example runs fill_border on that, or a crop of it, on the engine
and writes coarse, each band and the feathered result as PPMs — seconds
per attempt on TensorRT. Both examples take DARKROOM_ORT_DIR as the app
does, and --wait-engines lets a compiling rung finish before timing.
A Border choice beside the projection — crop to the picture, or fill it
— that redraws the preview filled so the invented pixels are seen before
they are confirmed (FR-MRG-1), greyed with the reason when the model is
not there. The job fills at half the composite's resolution in a
display-ish space (white balance, matrix, gamma; invertible) and samples
the result back into the linear DNG wherever no frame reached; the
sidecar's merge line says border filled and with which knobs.
Experimental because the fill is right in thin borders and wrong in deep
corners, where the model's Places2 prior puts clouds in sky and water
under grass; so its six knobs — working scale, edge erosion, coarse pass,
band width, mirror depth, seam feather — are sliders under the choice,
each committing a redraw, until the defaults are right.
dr_pano::fill owns everything the model does not — which tiles, what
context, how to blend — behind an Inpainter trait, and dr_pano::migan is
that trait over the shipped generator on the inference engine.
The known content is mirrored across the coverage edge into the hole and
a 256-px ring, the nearest 48 px folded, so the model interpolates between
real and mirrored sky rather than extrapolating into nothing. A coarse
pass at a quarter decides the structure with the whole border in a few
tiles; fine passes in 96-px bands from the edge outward texture it; the
seam is blended over a feather inside the real edge. Every knob is a
Params field, and an Observer hears each stage for whoever is looking at
why a fill went wrong.
A border fill acquires the filler once a tile, and each acquire hashed
the 28 MB model twice — 60 ms a tile, a third of the tile's run on a
throttled TensorRT. The Model keeps its hash from open.
Sargsyan et al., ICCV 2023; MIT code and weights (models/LICENCE.md),
exported by tools/export-migan.sh at a fixed 1×4×512×512 from the
authors' checkpoint — six operator types, 28 MB, in LFS like the rest.
The package installs it beside the scene model and the APK unpacks it
with the others.
The fused shader stored black with alpha 1 for a pixel whose source
coordinate left the frame, and the merge's warp averaged it in like any
other: a dark, badly interpolated fringe along every frame's edge, visible
as a seam wherever a frame ended and, later, as the edge the border fill
continued. The display keeps its opaque black; CameraLinear stores alpha 0
and the warp weights each sample by the alpha it interpolated, dropping a
sample that has none.
MI-GAN is plain convolutions, so every rung serves it and none needs a
special form; the role exists so resolve_model and the probe's fingerprint
know the model, and so the merge job can open it through the engine rather
than tract, which takes 7.4 s a tile for it.
A library's records are never all complete at once. A face found before
its quality was kept has no quality; one found before the eye models
existed has no reading; one adopted from a peer's shard has no crop; an
image the fast detector examined on a 1024 px proxy has boxes the current
detector would not have drawn; an image the scan stat'ed has no capture
date. On the reference library that is 17,762 faces under the bare
w600k_mbf id with no quality, no reading and no dense landmarks, 4,144 of
them without a crop, beside 12,217 images the fast detector examined and
found nothing in. Every one of those gaps was its own pass — V14's
measuring pass, §17.5's eye pass, the sweep's proxy repair, the sweep's
detector upgrade — with its own work list, its own count and its own idea
of done, and adding a per-face field meant adding a pass. There was no
pass at all for the case the library is actually in: boxes and landmarks
drawn by a weaker detector on a proxy, which every later per-face pass
would have read from.
dr_ui::repairs replaces them with one job over a registry. A Repair names
one thing a record can lack — the predicate that says which images still
owe it, the input its handler needs (a header, the original, or a native
render), the handler, and what to record for an image that can never be
done. The job unions the predicates into one work list, fetches each
image once at the most any claimant asks for, renders it at most once,
and runs every handler whose predicate that image still matches, checked
again before each because a detection writes every field a per-face
handler would fill. The registry today: face-proxy, face-quality,
face-eyes, face-crop, face-detection, face-upgrade, metadata — the last
there to say that this is not a face job. Adding a field is one entry.
A repair's predicate is the only definition of its work: the count the
settings page shows, the list the job fetches and the check before its
handler run are one predicate, so the job converges. That is why the
registry is cut to what the device can do rather than listing what it
skips — an entry is a count and a set of originals to fetch — and why an
eye reading that cannot be cut is not a criterion.
The catalog side is generic to match: record_updates writes whichever
fields a FaceUpdate carries and re-marks the image so the shards export
it; faces_needing and count_needing answer a predicate the caller
supplies, replacing the measuring pass's three special cases.
Two buttons on the settings page run the job and differ in one
predicate. "Index faces" converges on coverage: has anything examined
this image. "Re-index every face" converges on provenance: face-detection
claims every image with no marker under the chosen detector, in either
of its forms (FaceDetector::model_ids, so a desktop in f32 and a tablet
on the Hexagon do not re-index each other's work), and a marker saying a
weaker one looked is not that. An original over the fetch budget is left
exactly as it was under the re-index, where the sweep marks it examined:
a re-detection with nothing found would delete the faces, and "cannot
fetch" is not "no faces".
record_detections replaces an image's faces and carried only the user's
confirmations onto the new ones, by box overlap above 0.5 IoU. Everything
else on the old faces was dropped: the suggestions the last grouping pass
made, and the people the user had said a face was not. On the reference
library that is 13,011 suggestions and 77 rejections beside 3,778
confirmations — a re-detection of it would have been correct by
FR-CULL-12's letter, since suggestions are derived data, and would have
handed back a People screen of strangers.
Now every old face is read before the delete — box, vector, assignment,
rejections — and matched to the new faces one-to-one, best pair first. A
pair qualifies when the boxes overlap at all and either the overlap alone
says so (IoU above 0.5, the old rule) or the embeddings do (cosine above
SAME_FACE_COSINE, 0.45, the reference library's P≈0.95 line). The
embedding route claims the box a low-resolution pass drew badly enough
that overlap alone would not; the vector is also what breaks the tie in a
group photograph, where two neighbouring faces overlap both new boxes.
Overlap is required on both routes, because the same vector elsewhere in
the frame — a mirror, a print on the wall — is not the same face and must
not take its name. Onto the matched face go the assignment as it was,
confirmed or suggested with its probability, and every rejection.
The merge's match_faces still matches by overlap alone across devices; it
is the same question and is not changed here.
ONNX Runtime's default is already its fullest level. ort-tract maps any
level but disabled to tract's optimiser, whose slice pass divides by
zero inside the segmenter's graph — a panic across the C API and so an
abort, which is what stopped dr-ui's develop test. The app never asked
tract for that and does not start now.
On the tablet the engine compiled arcface for the NPU: the routing
compared the form a rung wants with the form on offer, and for the
embedder both are f32, so nothing said no. A rung now says which roles
it serves at all, and the Hexagon does not serve the embedder (§7 —
its vectors must compare across devices). Tested at the routing seam.
dr-segment's onnx_probe example still named ort-tract, which is what
stopped the workspace test build.
The strict flag refused the Hexagon over the ten quantise/dequantise
nodes at the graph's edges that QNN declines by policy, which cost
microseconds. A provider that hands real work to the CPU is slower than
the CPU floor and the timing already rejects it; the tablet measured
2.3 ms on the NPU against a 29.7 ms floor.
ONNX Runtime's errors open with a source path and a template signature;
the first 160 characters of a CUDA failure were all signature. The
reason now starts at the first word a person can act on.
XFeat's two exports are a Keypoints role now; the crate no longer names
tract, and the app compiles TensorRT engines for both ahead of the
first merge. The probe picks the smallest *detector* rather than the
smallest file: the tablet's first run chose the 112 KB eye classifier,
which has no int8 form, and reported the Hexagon as failed for want of
one.
The first int8 files found no faces at all, and for two reasons the
tool now guards against. The calibration set was landscape photographs
with no faces in them, so the score head's ranges had never seen the
face regime; the set is now proxies from the library itself. And ONNX
Runtime's strided and moving-average calibration modes both degrade
these graphs measurably (a quarter of the faces at eight images, none
at ninety-six), while driving the calibrator in chunks by hand gives
ranges identical to a single pass — so the tool does that, four images
at a time, and feeds quantize_static through its range cache.
Measured against f32 over 400 proxies (docs/inference.md §10.1): the
10g form finds every face above 32 px the f32 form finds; 500m and
2.5g find 96%, and what they lose sits at a median confidence of 0.52
against the 0.50 threshold. Shipped with the number on record.
The Android unpack list gains the three int8 files; without that the
tablet never saw them. D13's runtime half records the reopening.
tools/quantise-models.sh writes the QDQ form QNN's HTP backend takes
whole: opset 17, per-channel int8 weights, uint8 activations, ranges
from running the f32 graph over photographs fed exactly as the app
feeds them. The calibration is strided, four images at a time, because
every ONNX Runtime calibrator holds each image's whole set of
activations until it folds them — a gigabyte an image on the 10g
detector, and an OOM kill with no message when folded once at the end.
Release-time, never on the device (docs/inference.md §5): it needs
real photographs and a person reading the recall measurement that
gates whether each file is offered.
The desktop names where a package may have put libonnxruntime — an
override variable, beside the executable, the package's own library
directory, the Flatpak prefix, the system library directory — and
Android points at the APK's native library directory, which is also
what Qualcomm's DSP loader must be told for the Hexagon skel. Android
starts the engine at the end of the model unpack rather than at launch,
because the probe fingerprints the model files and a first launch has
none until then.
The About panel gains an Inference row beside Graphics, re-read every
two seconds while the probe runs and engines land, and faces.model_id
carries the detector's form: an int8 detector finds a different set of
faces and is a different population (docs/inference.md §7). A
low-memory signal drops every idle session with the GPU caches.
The APK assembly bundles ONNX Runtime and the Qualcomm HTP libraries
from Maven, fetched by tools/fetch-android-runtime.sh with their
published checksums; RUNTIME_DIR=none builds the tract-only APK, which
is a slower app and not a broken one. The desktop packages carry no
runtime yet.
Two probe fixes from the first desktop run: the floor must not be
built with CPU fallback disabled, and a versioned libonnxruntime.so is
a runtime too. On the reference desktop the probe now loads ONNX
Runtime 1.30, measures 30 ms on the CPU provider, and selects TensorRT
at 1.5 ms.
One crate names the runtime, the providers and the devices; dr-face and
dr-segment ask it for a session by role. It hands ort an API table once
per process — from a libonnxruntime it dlopens when the app names a
directory holding one, otherwise from tract — so the Rust build stays
free of C on every target and a package can install the runtime as a
file (docs/inference.md §3).
Sessions live in a registry behind a Model handle that holds the bytes,
not the session: every use refreshes a timestamp and a reaper unloads
whatever sat idle past the decay. A scan that runs the detector on each
image never lets it go idle; a click in the develop view lets the
segmenter go after thirty seconds; a handle used after that reloads,
and reloads on a higher rung if a compiled engine has landed meanwhile.
The probe walks the platform's ladder by building strict sessions and
timing them against the CPU provider, caches the choice against a
fingerprint of the runtime, driver, hardware and models, and compiles
engines for the selected rung in the background, smallest model first.
Nothing in this commit turns the native path on: the apps still run on
tract until they call init with a runtime directory.
tract runs every model on one core on every platform. Measured against
ONNX Runtime's providers on the MagicPad 2 and the reference desktop:
ORT CPU alone is 3-10x, the Hexagon at int8 runs the detectors in
1-3 ms, TensorRT is ~2x the CUDA provider. NNAPI, XNNPACK, WebGPU and
CUDA int8 were tried and excluded with the numbers that excluded them.
The spec keeps the build C-free: ort::set_api takes a table from a
dlopened runtime or from ort-tract, chosen once per process. Rungs
are chosen by building a real session, cached until an input changes,
and compiled engines are built in the background after the first
frame. The embedder stays f32 everywhere; int8 detectors are a
distinct model_id and are gated on a recall measurement.
Read and measured, not built. The bare 512 generator exports at a fixed
shape and loads under tract with nothing unsupported; at f32 on the
desktop CPU it takes 7.4 s per 512×512 tile, which puts a full-resolution
fill of the fixture's border at ten minutes. The three routes that would
make it viable are recorded, with the quarter-resolution fill the cheapest
and Hexagon int8 the one the model was designed for.
Picking a chip stored the choice for the merge and changed nothing on
screen — the chip did not even highlight, since the selected property
was never written back. Now the pick is reflected, and the job, waiting
for its decision, takes a Preview request, draws the alignment on the
chosen surface at proxy cost and reports again; the drain puts the new
picture and its size up. Auto is the surface the field of view suggests.
Also:
The largest rectangle inside the frames' coverage is found a row at a
time — a histogram of consecutive covered rows and a stack pass per row —
so the composite is never held to be measured (FR-MRG-11). It is written
as DefaultCropOrigin/DefaultCropSize (FR-MRG-4): the file opens on the
picture, the border is still in it, and resetting the crop shows it.
rawler reports the crop as the picture, which the test checks.
FR-MRG-4 records the question raised the same day — fill the border
rather than crop it — as open: a non-generative fill through the heal,
or a generative inpainter with its licence and weights. Neither decided.
A sweep whose frames overlap by more than half reads as one photograph,
and the page's job is to show frames. Each footprint is walked along its
border and drawn in amber where it lands, so twelve frames look like
twelve and a misplaced one is visible as such. The preview may take most
of the page's height rather than 320 px.
derived_from and merge are top-level sidecar fields (FR-MRG-6): one line
per source in order, and how the composite was made. A build that
predates them keeps the lines as unknown and writes them back. The job
writes the sidecar beside the composite and stages it with its own
record when the composite goes through the outbox.
DARKROOM_START_MERGE=a.CR2,b.CR2 lands on the merge page at startup with
the job running on local files, on the model of DARKROOM_START_IDENTITY,
for looking at the page where synthetic clicks do not reach it. The fetch
and the start are shared with the grid's button.
panorama.md §11 records what exists, the fixture's figures, and the six
things still open, auto-crop first.
dr_ui::merge is the orchestration with no interface in it: decode each
frame to sensor data and build its graph as a session would (orientation,
lens profile); render each through the camera-space tap at proxy size and
detect keypoints there, so the alignment is measured in the undistorted
frame the tiles are rendered in; align; solve one gain per frame from the
proxies' overlaps; draw the aligned set in colour for the page; then wait.
Nothing is written until a Decision arrives (FR-MRG-1). The merge writes
a linear DNG through the outbox with a destination record, so the drain
puts it beside its sources on a folder library and a server alike, and
the library rescans (FR-MRG-3).
merge.slint is the page, on the import page's model: the alignment
table with a failed frame named on its row and the button held off
(FR-MRG-5), the preview, the projection choice, Stop and Back. A
"Merge to panorama" button joins the grid's selection bar at two frames.
Headless, the example produces the fixture's 22 993 x 5 980 DNG in 45 s
on the reference desktop, exposures balanced across the stop of drift.
merge.wgsl warps one camera-space tile into one output chunk — output
pixel to direction (the projection maths of dr_pano::projection, verbatim),
direction to the frame's camera, camera to source pixel, bilinear by hand
from four textureLoads because rgba32float is not filterable — and adds it
into a storage-buffer accumulator weighted by its distance from the
frame's edge. A resolve pass divides by the weights and packs sixteen-bit
samples at the sensor's scale with a coverage bit.
MergePass::merge drives it: bands of rows, chunks across a band, and for
each chunk only the frames whose footprint meets it, each rendered as the
source rectangle the chunk needs and nothing more. The working set is one
chunk, one tile and one band (FR-MRG-11); the frame textures are the
caller's to cache. Feathered, not seamed; gain a scalar per frame — the
blend quality is panorama.md §10's step 5, after the path writes a file.
dr-export gains write_linear_dng — LinearRaw, DNG 1.4, u16 samples at
the sensor's scale, the body's matrices with their illuminants, the
as-shot neutral, the EXIF block an export writes — streamed strip by
strip through a closure so the composite is never held (FR-MRG-11). The
tiff crate's directory is a map, so PhotometricInterpretation is written
over what new_image set, which is the trick the S15.1 spike thought it
had to hand-roll around. The test reads the file back through rawler.
dr-decode's RawImage carries samples_per_pixel (a linear DNG is 3), the
body's profile with its calibrations mapped back to EXIF illuminant
codes, and the cleaned make and model. The GPU uploads a three-sample
image as it is, normalised by black and white like a photosite, through
a full f16 conversion — subnormals kept, because a 14-bit LSB sits at
f16's smallest normal and rounding it to zero would crush exactly the
shadows the file was written to keep.
compose_camera_linear composes the fused pass with an empty operation
list, the file's orientation as the baseline, a view rect for the tile,
and a store of rgba32float. On the GPU, render_camera_linear is the only
entry that accepts it: it fills the profile uniforms neutral — unit white
balance, identity matrix, curve off — so what lands in the texture is the
sensor's numbers after the lens warp and nothing else (FR-MRG-2). A third
bind-group layout carries the format, as the linear one does, and the
readback is generalised to any pixel width for the f32 copy.
Thirty-two bits because the composite is written back at the sensor's
scale: a 14-bit sensor has 16 384 steps to white and f16 keeps 2 048 of
them in the top octave.
Twelve real frames from the fixture set now align in 4.5 s — 4.4 s of
matching, 118 ms of bundle adjustment — where the first run took 51 s and
left the first two frames out.
The matcher computes each pair's similarity matrix once, across the
cores, with a dot product written to vectorise; both nearest-neighbour
directions read it. The frames that failed were portrait: fitted into the
landscape input they used 512 of 1024 px, and their thin overlap did not
survive at half resolution. The same weights are now exported at 768×1024
as well and the detector picks the shape by aspect. The example aligns
from embedded previews and draws the set on a cylinder; on the fixture the
sweep is 152° at a fitted 47.9 mm against the EXIF's 50, RMS 1.5 px, and
the overlaps show no ghosting.
A new crate holding the CPU half of a merge (FR-MRG-10): the grayscale
proxy with orientation, the XFeat decoder ported step for step from the
reference detectAndCompute, mutual-nearest-neighbour matching, a robust
pairwise homography with the focal length read off it, a hand-rolled
Levenberg–Marquardt bundle adjustment over every rotation and the focal,
the three output projections, and align(), which chains it all and names
the frames it could not place rather than guessing (FR-MRG-5).
Dependency-free without the xfeat feature — linalg.rs says why the dense
algebra is hand-rolled — and tested on synthetic sweeps whose answer is
known exactly. The noise test records the single-row degeneracy: one
pixel of noise is a tenth of a percent of focal, which is a uniform
stretch of the sweep, not a misalignment.
tools/onnx-probe-on-device.sh cross-builds dr-segment's onnx_probe
without the embedded segmentation model, pushes it with a model to the
attached device and times two runs. The 768×1024 XFeat export takes
~400 ms on the reference tablet's NEON cores against ~300 ms on the
desktop, with identical output ranges — inside NFR-MRG-1's 1 s per frame.
The blend half of S15.4 waits for a chunked blend to exist.
fixtures/pano/2025-08-05: _MG_8320 … 8331, one portrait hand-held sweep
at 50 mm with a stop of shutter drift and sky in every frame — the set
§3.11 is built against, with each of those facts named as the test it
is. fixtures/** is tracked in LFS like the models but with the opposite
default: CI's pulls exclude it, so a build never fetches 325 MB it does
not use.
Camera-linear u16 samples on the first source's black-subtracted scale
with its white level, never rescaled to fill 16 bits, with its body,
matrices, illuminants and as-shot neutral carried — so the panorama is
developed afterwards as one photograph from the sensor's own numbers.
The only thing a warp cannot preserve is the colour filter array, and
the clause says so.
The fused chain, as operation.rs's tests fix it, is warp → as-shot white
balance → operations → base curve → camera matrix → store. LinearWorking
stores after the matrix, so the existing linear tap carries the body's
base curve, and a composite stitched from it and developed as an
unprofiled body would render that curve twice.
FR-MRG-2 therefore stitches camera-linear RGB — after the warp, before
white balance, curve and matrix — and the composite carries the first
source's body, matrices and as-shot neutral so its own develop applies
the profile once. The composer already makes this a uniform question:
white balance, matrix and the curve flag are reserved uniforms, so the
tap is a compose entry with no operations and a render entry that fills
them neutral. panorama.md §5.1 states the shape and asks for f32 buffers.
tools/export-xfeat.sh exports the convolutional network alone at 768×1024
grayscale, on the pattern of export-seg-model.sh: thirteen standard
operator types, no dynamic axes, the keypoint decoding left to Rust.
examples/onnx_probe loads it through the ort-over-tract backend the app
ships with nothing unsupported and runs it in ~300 ms on the desktop CPU.
The weights are Apache-2.0, read from the repository's LICENSE, with no
grant on the checkpoint — recorded in models/LICENCE.md before they land,
as FR-MRG-8 asks. The probe stays: the next model will need the same
check.
A hand-rolled 64×48 LinearRaw DNG — one IFD, 16-bit RGB, DNGVersion,
ColorMatrix1, AsShotNeutral — comes back through rawler 0.7 with cpp 3,
the samples in the order written and the matrix parsed into the camera
definition; CameraProfile::extract builds a profile from it. ImageMagick
reads the same bytes.
dr_decode::decode currently accepts the file as CFA and passes three
times the samples on, so the cpp == 3 branch is the decode work FR-MRG-3
needs, and the only decode work. panorama.md §8 records the result.
A merge writes a new source file beside its sources (D18) rather than a
multi-source Version, which answers the schema question §7 had been holding
open for panorama, HDR merge and focus stacking together. The panorama is
undeferred as FR-MRG-1 … 11; the other two stay in §7 with their data model
decided.
FR-MRG-10 and 11 fix where the work runs — every per-pixel stage on the GPU,
the composite never held as one texture — because the output exceeds
max_texture_dimension_2d before it exceeds memory. panorama.md carries the
stage table, the chunked output driver, the model licences and the porting
sources. S15 gates all of it.
Coverage falls from 83.0% to 77.2%: thirteen requirements entered with no
code, and outstanding.md §11 says so.
Once every image has been through the detector and only readings are
left — the state an already-indexed library is in the day the eye models
arrive — the button reads "Read eye state" rather than promising to
index, and the coverage line says what the faces are waiting for.
dtourolle