Every face stored before its quality was kept holds a unit vector, and
V14 forgot the run marker of each image holding one so that the next
sweep would look again. Looking again meant detecting again: a whole
re-detection per image, with every suggestion on it thrown away and the
confirmations carried across by box overlap, to recover one number.
The sweep now has a measuring pass between the proxy repair and the
un-indexed images. It lists every image holding an unmeasured face,
fetches the original once, warps each stored face from the landmarks it
already has, embeds it, and writes the raw vector and its length over
the old row. Ids, boxes and identities are untouched; the marker is
re-written fresh so the sync exports the measured vectors. A face whose
landmarks no longer make a warp is dropped, as detection would have
refused to store it. `faces_unindexed` leaves those images to the
measuring pass, so the V14 deletion no longer costs a second detection.
The embedder's raw output has a length, and the length is a reading of
how recognisable the crop was: a blur, an occlusion or a hard profile
comes out short. Normalising threw it away. A short vector sits near
the middle of the sphere and matches a little of everyone, which is how
one bad crop bridges two people in a grouping pass.
So the length is kept — the store now holds the raw vector, re-normalised
on load, with the length beside it as `faces.quality` — and a face under
MIN_GALLERY_QUALITY (14) is a probe: measured against the gallery and
placed where it fits, but never what another face is measured against.
Two probes are never paired, and a probe is nobody's evidence for a
confidence. The People screen shows the number as "Quality 17.3", dimmed
below the floor.
Faces indexed before this stored unit vectors and have no reading; they
are admitted to the gallery, and schema V14 forgets the run marker of
every image holding one so the next indexing pass measures them. A
peer's unmeasured shard faces are not adopted, or a sync would write
that marker back.
Everything argued for this change so far was read out of a catalog
after the fact: crop_px across 18,671 faces the old code had already
stored. That is evidence about what the previous implementation did. It
is not evidence that the new one does better, and the difference matters
because a landmark left in the detector's coordinates, or a box filter
with an off-by-one in its source span, would both produce faces that
look entirely plausible until somebody counted the pixels behind them.
So examples/face_native.rs renders one file and indexes it twice, native
and from a 1024 proxy, changing nothing else. Fourteen originals from
the reference library, 5472x3648 CR2 and DNG:
native 9 faces, mean crop 287px
1024 proxy 5 faces, mean crop 75px
Crops 3.8x larger, and across the line that decides whether the crop is
photographed or interpolated: 75px is below ALIGNED_EDGE, so the proxy
path was upsampling into the embedder on average where this one
downsamples into it. Fourteen images and nine faces is enough to show a
direction and to catch a wrong scaling; §7b says so rather than quoting
the ratio as a library-wide figure.
It also corrects something §7b asserted two commits ago. I wrote that
detector input resolution cannot affect recall, because §4.1 letterboxes
everything to 640. Native found nine faces to the proxy's five,
including four on files where the proxy found none, so it plainly can.
The two paths differ in their resampling as well as their size, and this
experiment does not separate those, so §7b now records the result as
evidence for the double-resampling hypothesis rather than as its proof.
M4 still owns settling it.
The audit-summary test went stale when the ready/to-fetch split was
collapsed and is updated to assert the single number, including that the
old wording is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FR-CULL-8 said detection runs against the thumbnail or proxy tier and
never a full decode, and faces.md §5 said the aligned crop is sampled
from that same proxy. Both are wrong in the same place: they treat
detection and cropping as one resolution problem when they are two, with
opposite answers.
Detection does not care. §4.1 fixes the graph's input at 640x640 and
letterboxes whatever arrives, so a face filling 2% of the frame reaches
the model at 12px whether the buffer handed over is 1024px or 6000px.
Every pixel above the detector's own input is discarded before inference.
The crop cares about nothing else. §5's warp produces the fixed 112x112
ArcFace sees, so source resolution converts directly into whether those
112 pixels were photographed or interpolated. Reading crop_px across the
18,671 faces the proxy-tier implementation stored: 47.3% were upsampled
to reach the embedder, 314 of them by more than 2x, the smallest from 34
source pixels. An upsampled crop does not fail loudly -- it yields a
confident embedding of detail that was never there, and the damage
appears three stages later as clusters that will not separate.
So FR-CULL-8 now specifies four stages with the resolutions named
separately: render native through FR-EXP-9's pipeline, downscale for the
detector, map boxes and landmarks back to native, crop and align from
the native render. The affordability the old rule bought is met instead
by when the pass runs -- background, preempted, resumable -- and the
requirement says plainly what it now costs on a remote library: the
original rather than FR-NC-3's byte range, 412 GB across the reference
library's 19,107 images, so a whole-library pass is a transfer under
FR-NC-6 rather than something that may start on its own.
MIN_CROP_EDGE replaces the MIN_DETECT_EDGE this branch briefly had. Same
number, guarding the quantity that turned out to matter.
faces.md §7b records both measurements, and marks the second as
unexplained rather than dressing it as a finding. Grouped by the buffer
detection ran against, faces per image was 0.078 at 1024 or below and
1.82 at 2048 or better, controlled for file type and size. That gap is
real and reproducible and I cannot account for it, because the letterbox
above says detector input should not matter. M4 is where it gets
settled. The crop measurement does not depend on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge probability was `dr_face`'s constant and the smallest group was
a bare `< 2` in the clustering pass. Both were tuned on one library —
1,813 faces of one photographer's family — and the quantity they optimise
is a property of the population, not of the model. A household at close
family resemblance and two thousand strangers at a wedding want different
answers, and neither of them is the reference library. The doc comment
already conceded the point and pointed at `face_index --tune`; a
photographer does not have a terminal.
So they are `FaceSettings` now, saved per device beside the cache budgets
and edited from the People screen — beside the Regroup button that
applies them and the rail that shows what they did, because a value
changed three screens away from its effect is one nobody can tune.
Moving them is safe by construction, which is why nothing asks for
confirmation: a regroup writes only the suggested half, and
confirmations, names and ignores enter as anchors and come back
unchanged. The smallest-group rule is applied only to groups the system
invented — a group the user named or set aside survives it whatever its
size, because a display preference does not overrule a judgement.
**Withdrawal, without which the setting does nothing visible.** Raising
the smallest group stops the pass creating small groups; it does not
remove the ones a previous pass made, because those still hold their
suggestions, so they are not empty, so the prune leaves them. The pass
now releases every unanchored face it did not place before pruning.
And a dial you cannot see the effect of is not a dial. "What would this
do?" runs the same population through the clusterer without opening a
transaction and reports groups, faces grouped and largest group — one row
of `--tune`'s table, on the user's own library, on a worker thread. The
line leads with the group count because that is the number that says
which side of the right setting you are on: it climbs as fragments are
gathered into people and falls as separate people start being welded,
while the grouped-face count rises straight through both.
The preview parks its poll timer in a slot of its own. A preview and a
regroup are allowed to be in flight together, and sharing the sweep's
single slot would have the second to start drop the first's timer —
visible as a Regroup that finished on its worker and never said so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
assign's denominator is the identities the user has ruled on, and it
reads Cluster::person to find them. Master's "Let a name hold a group
together" widened what sets that field: a confirmation, a name, or an
ignore, where before it was a confirmation alone.
The behaviour is right either way — a named person is exactly the
identity a suggestion should be discounted against — but the module note
and faces.md §9.1 both said "a confirmation", which is now too narrow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The GPU question needed a number nobody had: how a regroup divides on
the hardware whose CPU is weakest. dr-face carries no weights and
touches no display, and dr-catalog's example needs only a catalog file,
so both run under adb shell against a copy of a real library.
On the same 18,143 faces — desktop against the tablet — scan 0.96s /
2.61s, agglomerate 1.69s / 2.16s, score 0.26s / 0.40s. The scan is half
the pass on the tablet and under a third on the desktop, because twenty
cores of AVX2 pull ahead of NEON much further than the merge engine's
single-threaded hashing does. So a GPU GEMM is worth roughly 2× a
regroup on the tablet and 1.5× here, and it is the tablet that should
decide whether it is built.
The two architectures agree exactly: the same 1,531,969 evidence pairs,
the same 2,518 groups holding the same 16,246 faces, the same
reliability table. That is a better check on the NEON kernel than the
unit test can be.
Two instruments, both read-only: the example now prints its phases, and
dr-face gains scan_bench, which needs no library at all and so can
answer "how fast is this machine" on a device with nothing on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The similarity scan picks its dot product per machine, and the NEON one
is the kernel that ships to the phone and the tablet — and the one a
desktop cargo test never executes. A wrong lane index or a mishandled
tail there is a silent wrong answer on exactly the devices nobody runs
the suite on, which is a poor place for the only untested code path.
dr-face carries no weights and touches no display, so its tests are a
plain ARM64 binary that runs under adb shell with nothing installed.
The script builds it against the SDK's newest NDK, pushes it, runs it
and cleans up. It checks for the device first, so a tablet that is not
plugged in costs a second rather than the two minutes it takes to
compile for it.
Not wired into CI, which has no device attached.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The §9 note said the scan was half a regroup and implied the
agglomeration was an irreducible sequential walk. Both halves of that
are now wrong, and the numbers are the point of the section.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scan is O(n²) dot products and nothing else, so its speed is the
face subsystem's speed — and it was running at 0.7 flops per cycle.
Two separate faults, both measured over the reference 18,143-face
library on twenty cores. It walked the whole embedding array once per
row, ~336 GB of traffic, where a column tile that fits in L2 is read
once per tile of rows: 4.64s → 2.81s. And the workspace builds for
baseline x86-64 — SSE2, no FMA — into which the portable loop was not
being vectorised at all: 2.81s → 0.86s, 195 GFLOP/s.
So the dot product is now chosen per machine. AVX2 + FMA where
is_x86_feature_detected! finds it; NEON unconditionally on aarch64,
since Advanced SIMD is in that baseline and every Android device the app
builds for has it — with the explicit vfmaq, because LLVM will not fuse
a multiply and an add without being told to. The portable loop stays as
the definition the others are tested against, and
the_fastest_kernel_agrees_with_the_portable_one is the only check the
NEON path gets on a machine that is not aarch64.
Faces::embeddings is one flat buffer rather than a Vec per face: the
pointer chase defeated both the prefetcher and the tiling, and it is
also the layout a GPU pass would want.
Behaviour is unchanged and that is checked rather than asserted — the
same 1,531,969 pairs from all three kernels, and on the real library the
same 2,518 groups holding the same 16,246 faces with the same confidence
distribution. A full regroup there goes from 10.0s to 5.9s; the rest is
the agglomeration, which is a sequential heap walk and is where the next
look should go.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The number beside a suggestion was the mean calibrated probability
between the face and the rest of its group, which measures the wrong
thing twice. It punishes coverage: a person with two hundred faces over
fifteen years is *meant* to have members a given photograph is
orthogonal to, so a correct suggestion onto a well-photographed person
scored low for being well photographed. And it never asked who else the
face might be — a face matching Anna at 0.95 and nobody else, and one
matching Anna at 0.95 and her sister at 0.93, came out identical, when
the second is the only one worth the user's attention.
dr_face::assign answers both, and multiplies them: the mean of the best
ten calibrated matches into the identity (the old mean, capped, which is
what stops coverage counting against it), times that identity's share of
the evidence against every *named* rival.
Only named people compete, and per person rather than per group. Both
halves of that had to be measured on a real 18,000-face library rather
than reasoned about. Normalising across every group made the number
useless — median suggestion 21%, four in five under half — because
clustering leaves one person spread over many groups, so a face competed
against itself; and keying rivals by group left Catherine competing with
Catherine, median 39%. Per named person: median 99.5%.
Rivals are gathered below the merge threshold, down to even odds: a
named person matching at 0.6 will never be merged into but is exactly
the competition to discount for. That would be a second similarity scan,
the expensive half of regrouping a library, so cluster_scored scans once
at the looser floor and hands the merge engine the subset at or above
the threshold — pair for pair what it would have scanned for itself,
held to that by a test.
Leave-one-out over that library's 2,702 confirmations across 54 named
people: 99.33% of faces placed on the right person against the old
mean's 99.15%, and the number shown for the right person moves from a
median of 90.4% to 99.3%. It errs low — 100% correct wherever it states
80% or more — which is the safe direction, and docs/faces.md §9.1 says
plainly that the low bands are not calibrated.
The example that measures it comes too: this is a claim about a
library's numbers, and nobody should have to take it on faith.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FR-CULL-9 was read as "no fit, no number", so every library without 200
confirmed positive pairs showed "Confidence unavailable" on every
suggestion — which is every library, until enough confirmations exist to
fit one. The confirmations are made on this screen, ranked by the number
it was withholding, so the degraded state was also the permanent one.
There has always been a curve: Calibration::default is the reference
implementation's fitted MBF sigmoid, which is what clustering already
operates at. It is a published operating point, not an invention, and
what the requirement forbids is presenting it *as though it were
measured on this library*. So the percentage is shown, and the screen
says once, above the grid, where the curve came from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Face indexing was compiled into the APK all along — dr-ui takes dr-face with
`inference` on every target, so SCRFD, alignment, MBF, calibration and
clustering were all in there. What was missing was the weights, and on Android
there was no way to supply them.
Route C (docs/faces.md §2.2) says the user obtains the model and the app loads
it. On a desktop that is a real gesture: drop two files in
~/.local/share/darkroom/models/ and indexing starts working. On Android it is
not a gesture at all. `internal_data_path` is app-private, `run-as` needs a
debuggable build, and the in-app fetch route C specifies was never built — so
the settings page reported "no face model is installed" on every launch with
nothing behind the message. Not "off until you supply weights"; off.
So the shape-fixed pair goes into LFS under the APK's assets, assemble-apk.sh
copies it into the package, and `android_main` unpacks it to the shared models
directory before anything asks whether a model is present.
Three things that are not incidental:
The models directory is now shared across accounts rather than per-account.
Weights are identified by `faces.model_id`, not by who is signed in, so two
accounts had no reason to hold two copies — and the unpack runs before any
session exists to key a per-account path off. `face_models` still prefers a
per-account directory when one is populated, so anyone mid-migration keeps the
ability to pin one library to its own pair.
The unpack writes under a temporary name and renames. `face_models` decides
availability on `is_file()` alone, so a copy truncated by the process being
killed would leave a file that passes that test and fails inside tract —
reported to the user as a broken model rather than a missing one.
assemble-apk.sh refuses an LFS pointer. At ~130 bytes it looks exactly like a
model to `cp`, and unchecked it reaches the device and fails in the graph
loader instead of telling someone to run `git lfs pull` — the same guard
dr-segment's build script applies to yolo26n-seg.onnx.
The licensing half is unchanged and recorded in §2.2a: the InsightFace grant is
research-only, this is a private repository and a self-installed build, and
these files come back out before anything is published. The weights are still
not a cargo build input — dr-face has no `models/` directory and no
`embedded-model` feature, and nothing in the build reads them. The APK assembly
step copies two files and is the only thing in the tree that knows they exist.
Verified on device: both models unpack on first launch (2524817 and 13616095
bytes) and the APK carries them at assets/models/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An image with no faces in it was indistinguishable from one that had
never been looked at, so every landscape, still life and document scan in
the library was re-detected on every pass, for ever. In a real library
that is most of it: on the 23,527-image test library, 64 of the first 110
images indexed contain no face at all.
Schema v9 adds face_index, a run marker per (image, model) carrying the
face count and the proxy edge it read. Keyed on the model, so a model
change puts every image back in the queue by itself.
That makes a coverage figure possible, which is the thing a user actually
wants to see. The audit also splits the outstanding set by whether a
proxy exists, because 23,417 awaiting a proxy and 110 ready to index are
different problems, and telling the user to run indexing again would not
fix the first.
The Identity screen gains Index faces, Stop, and the coverage line.
examples/face_index.rs is the same check and sweep without a window,
which is the right shape for an overnight pass.
Measured on the real library in release: 3.5 images/second, 110 images
and 125 faces in 30 seconds, and a second run correctly finds nothing
left to do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Also notes what building it taught the design: a split has to reject
before it confirms, or the next clustering pass undoes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FR-CULL-9 forbids thresholding a bare cosine anywhere in the subsystem,
so calibrate fits P(same person) per library and reports whether the fit
is trustworthy. Two details carry most of the weight.
The fit runs against a 200-bin histogram rather than a pair list: a
25,000-face library has ~3e8 pairs and no gradient descent is running
over that. And a fresh library has no valid calibration, because the
positives have to come from user confirmations or burst siblings --
bootstrapping them from high cosine would fit the calibration to the
belief it was supposed to test.
Clustering defends against the over-merging FR-CULL-10 warns about with
constraints rather than a better threshold: two faces in one photograph
never merge, and two groups confirmed as different people never merge.
Average link rather than single link, so one strong edge cannot weld two
families together.
Calibration is defined once, in dr-face, and dr-catalog re-exports it.
Two implementations of one probability model is exactly how a number
comes to mean the wrong thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ports the pipeline from the C++ reference in ../scene-actor-extraction
(MIT, same author). End to end on real portraits it separates identities
the way the reference's fitted calibration says it should: 0.596 between
distinct photographs of one person, 0.05 between different people, either
side of MBF's 0.267 boundary.
Three things are structural rather than incidental:
Aligned112 can only be built by align::warp, so Embedder::embed cannot be
handed an unaligned bounding-box crop. That mistake yields 512 plausible
unit-norm numbers and no error, so the type system refuses it instead.
Embedding carries its ModelId and cosine() returns None across models,
because a cross-model similarity is the one mistake that produces
plausible garbage rather than a failure.
The model-free half -- alignment, embedding arithmetic, f16 storage --
sits outside the inference feature and is covered by 11 tests that need
no weights on the machine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither InsightFace export parses as shipped -- SCRFD fails at its input
node, ArcFace at the first Conv -- which is the same wall dr-segment hit
on YOLO's dynamic export. Both load cleanly with the input dims frozen,
so the pure-Rust runtime holds for the face pipeline too.
tools/fix-face-model-shapes.sh does the freezing, and exists so the
artefact is reproducible rather than a binary someone once produced. It
takes two forms because the two graphs need different ones: ArcFace's
batch is a named dim_param, SCRFD's H and W are dynamic but unnamed.
Also notes YuNet loading with no intervention, which matters for the
licence question in faces.md 2.3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FR-CULL-8..12 specify the subsystem in terms of "a 512-dimension embedding
from a stated model" and stop there, because D13 was open. This names the
models, and grounds them in the measurements and the working C++ pipeline in
../scene-actor-extraction rather than in a literature reading.
The licensing half of D13 stays open, but with a route through it: the
InsightFace weights are non-commercial and cannot be committed, so the app
ships the code and the user fetches the model. faces.model_id already makes
that a survivable choice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dtourolle
