dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dtourolle/scene-actor-extraction:e5885977dfd04741a0d1bb7aaf0270d9c40a2fa8
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
..
compare: dtourolle/scene-actor-extraction:feature/quality-knee
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
54 Commits
e5885977df .. feature/quality-knee
Author SHA1 Message Date
dtourolle 889018aa34 docs(VR-012): the quality knee, and what it retires 
Sharpness is not a sufficient statistic for identity loss. Six cells at
effectively identical measured sharpness span 15.3% to 91.0% TPI,
ordered entirely by source size, because a scalar keyed on
high-frequency energy cannot separate attenuated high frequencies from
destroyed spatial sampling. AR-028's "kept separate, not collapsed into
one scalar" now rests on a measurement rather than an argument -- and
the reasoning it used to rest on, that the aligned crop is
scale-normalised so a measure there cannot re-measure size, was wrong
and is corrected in place.

Variance of Laplacian -- the most widely used blur metric there is -- is
anti-predictive at fixed degradation on all three blur families. The
decile it calls sharpest is 2.6x less identifiable than the decile it
calls blurriest, monotone across ten bins, because within a cell its
residual variance is native contrast rather than detail, and hard
shadows and JPEG ringing raise it while making a face harder to match.
Gating on it would preferentially discard the more identifiable faces.

Blur breaks confidence, not identity: rank-1 holds at 80.2% where TPI is
15.3%, and FPI never left 0.1% in any of the 108 cells. Degradation
produces abstention, never a wrong name. That is also why sharpness
fails as a compute gate -- even a visually destroyed face stays 46.9%
identifiable, so a gate discards recoverable evidence at three times the
cost of the free size filter. Discount, do not gate; the rule AR-028
already stated now has evidence for why it is right rather than merely
cautious.

Records the shape a discount must have (flat, then a cliff between sigma
2 and 3), that its cost scales with proximity to the decision boundary
rather than with blur, and that the pose half of VR-012 has not been
run.

TRACES: VR-012, AR-028, AR-029, AR-030 | SR-002
 2026-07-31 22:29:47 +02:00
dtourolle 26de01b2e3 study(VR-012): quality knee over a joint size x blur grid, three blur families 
Extends the VR-005 protocol -- hold out one mugshot per actor, degrade
only the probe, match against a gallery held at native resolution,
decide through the Platt calibration -- from one axis to two, over 1670
actors rather than 100.

Joint rather than separable, because the interaction is the question: a
16 px face upscaled to 112 has already lost its high frequencies, so
further blur costs it almost nothing, while the same blur at full
resolution is expensive. Sweeping the axes independently would measure
each with the other implicitly at its best and miss that entirely.

Three blur families, compared at matched per-axis PSF spread rather than
at equal raw parameter. Optical defocus is a uniform disc whose transfer
function is a jinc with exact zeros, not a Gaussian that merely rolls
off, and it is also how a face ends up large and useless -- the case a
size filter cannot catch. Sweeping Gaussian alone, as the first version
did, understates real lens blur by a factor of five in error rate.

Every candidate measure is scored on every degraded crop and the
candidates are ranked by how well each predicts the pipeline's actual
decision, not by how smooth its synthetic ladder looks. Both a pooled
and a within-cell AUC are reported: they answer different questions and
the candidates rank differently under each.

Runs through sae_embed throughout. Stages gains optional engine paths so
the same study can drive a TRT build, which is what makes the full grid
five minutes rather than four and a half hours.

TRACES: VR-012, AR-028, AR-029 | SR-002
 2026-07-31 22:28:06 +02:00
dtourolle 41d30395da fix(ort): CUDA detection was gated behind the TensorRT-EP build flag 
detect_ort_provider() only tested for CUDAExecutionProvider inside
#ifdef SAE_ORT_WITH_TRT_EP, so any build that did not also opt into the
TensorRT execution provider could never select CUDA and fell straight
through to the CPU. The two are independent: the TRT EP needs the
headers and profile plumbing and is rightly an opt-in, CUDA is a plain
ORT provider and is not.

The failure is silent rather than loud, which is why it survived --
inference runs on the CPU and every answer is still correct, just far
slower. Measured on the VR-012 study: 0.32 s/crop against 0.0021 s/crop
once a GPU backend is actually used, with the card sitting at 212 MiB
and 0% utilisation throughout.

Only the TensorrtExecutionProvider line stays inside the guard.
 2026-07-31 22:25:00 +02:00
dtourolle 1ae88376e1 feat(quality): five AR-029 sharpness candidates on the aligned crop 
assess_sharpness() scores a 112x112 crop on variance-of-Laplacian, a
contrast-normalised variant, Tenengrad, a spectral high-frequency ratio
and dir_min_tenengrad, over a fixed 64x64 window on the face interior.
The window excludes the corners because studio headshots are routinely
shot at a wide aperture, and background bokeh measured over the whole
crop would drag the score down on the sharpest images in the set.

Five rather than one because AR-029's threshold has to be located, not
chosen: VR-012 ranks them by how well each predicts real identity loss.

The T1 ladders drove two corrections during development. The spectral
ratio applied its Hann window before removing the mean, so the DC term
smeared into the low-frequency bins and the "ratio" tracked absolute
brightness (a 20/255 brightening moved it 23%). And no measure taken
from the literature survived directional blur: normalising by total
energy divides out the loss being measured, so both ratio measures are
U-shaped in motion-blur length and score a 21 px smear about as sharp as
a 3 px one. dir_min_tenengrad exists to fix that -- a low-frequency
contrast denominator that blur leaves alone, and the worse of the two
Sobel axes rather than their sum.

The tests pin the disqualifying behaviours as well as the desirable
ones, so a change that makes var_laplacian contrast-free is a deliberate
act rather than an accident. They also record that every candidate falls
under downscale-upscale as well as under blur: the aligned crop is
scale-normalised geometrically, not informationally.

Exposed through sae_embed alongside the AR-030 alignment residual, so a
study scores through shipped code rather than a numpy copy -- the same
argument that already applies to the calibration.

TRACES: AR-028, AR-029 | SR-002
 2026-07-31 22:24:39 +02:00
dtourolle 5f6daefc40 Merge branch 'feature/quality-knee' into feature/opencv5 
# Conflicts:
#	docs/requirements.md
 2026-07-31 16:54:09 +02:00
dtourolle 629d698ad9 Merge branch 'feature/dump-provenance' into feature/opencv5 2026-07-31 16:53:58 +02:00
dtourolleandClaude Opus 5 71354e862a docs: VR-013 and VR-014 results; AR-002 raised to 40px 
Records study results and the requirement change that follows from them.

VR-013 measures minimum face size end to end — gallery from one recording,
probes from another — rather than by degrading an already-aligned crop. Holding
90% of the plateau needs ~50 px that way against VR-005's ~22 px, the gap being
detection and landmark error rather than the embedder. AR-002 therefore takes
40 px, not 32: VR-005 isolates the embedder and is an upper bound, and 32 admits
faces in the falling region. FPI stayed 0.0% at every scale, and the ceiling is
cross-view rather than resolution.

VR-014 exercises audio-signature offset recovery on real film audio instead of
the synthetic golden tone. Forty random in-cap offsets, every one recovered to
the nearest frame, worst error 46 ms against a 500 ms budget — and 46 ms is the
quantisation floor rather than a result, since offsets land on whole 92.88 ms
frames. The runtime/2 anchor is confirmed through head-trimmed files.

The soft spot VR-014 found is tier labelling, not accuracy: the score drops with
sub-frame misalignment, so 27 of 40 correct alignments were demoted to `loose`.
One frame of slack in the score restores all forty to `audio` with false matches
unmoved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-002, VR-005, VR-013, VR-014 | SR-002, SR-003
 2026-07-31 16:53:58 +02:00
dtourolleandClaude Opus 5 2a8ee3660b feat(audio): bind the v1 signature and validate offset recovery on real content 
sae_audio exposes the shipped signature to Python. It compiles
audio_signature.cpp directly against FFmpeg rather than linking
sae_gallery: the signature needs no model, no OpenCV and no HDF5, so a
module that dragged those in would make `import sae_audio` depend on a
GPU-capable build of a path that is pure CPU DSP.

The point of binding rather than porting is that a fingerprint is only
useful if every implementation agrees byte for byte. A numpy port would
be a third implementation, and the one nobody checks against the golden
vector.

VR-014 then recovers a known trim from real film audio rather than from
the synthetic tone: 40 random in-cap offsets, every one recovered to the
nearest frame, worst error 46 ms against a 500 ms budget — and 46 ms is
the quantisation floor, not a result, since offsets land on whole
92.88 ms frames.

The soft spot is tier labelling rather than accuracy. Sub-frame
misalignment drags the score down (0.94-0.99 near a frame boundary,
0.69-0.73 at half a frame), demoting 27 of 40 correct alignments to
`loose`. Allowing +/-1 frame of slack in the score fixes it: all 40 back
to `audio` at min 0.906, false matches unmoved at 0.12-0.16, for 81 ms
of the budget.

The module stops at the producer's edge. Sliding one signature against
another is the consumer's algorithm (server SPEC §3, and the jRay
plugin implements it), so a caller writing that slide in numpy is not
duplicating anything this repo owns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: IR-004, IR-005 | VR-014 | UT-105, UT-106, UT-107, UT-108 | SR-003
 2026-07-31 16:52:11 +02:00
dtourolleandClaude Opus 5 b4318f8d9e fix: belief accumulates across frames (lazy-OR), not once 
A track recognised on 318 of 385 frames was owned on none, so the truth file
named nobody while the matcher was accepting almost continuously.

The correlation discount was an annihilator rather than an attenuator. Weight
was 1 - P(same view), so once a track had one stored view every later frame of
that same face scored ~0.01 and the belief stopped moving. One observation just
over the accept threshold is logit(0.78) ~ 1.27, under the ownership bar — hence
recognised always, owned never.

Two changes, in the order they were found.

Correlated evidence is now attenuated by effective sample size,
n_eff = n / (1 + (n-1)·rho), each frame contributing the marginal gain. That has
the right shape at both ends: uncorrelated evidence accumulates linearly, and a
held pose converges on 1/rho rather than growing without bound. A constant floor
was tried first and rejected — it grows linearly forever, so a long shot could
out-argue genuinely varied evidence purely by lasting longer.

Combination is now weighted lazy-OR: P = 1 - (1-P_old)·(1-p)^w, stored as
log(1-P) so the update is additive and precision stays where it matters as P
approaches 1. Each frame is new evidence that this track is that actor, and the
belief is the probability that at least one sighting was right. It converges
faster than summing log-odds at the same effective count — 2.98 vs 2.53 after
two observations at p=0.78 — which is what a real clip needs.

Note that summing log-odds was already a correct sequential Bayesian update:
the matcher fits with prior 0.5, so logit(p) IS the per-frame log-likelihood
ratio and the running sum carries the prior forward. It was not wrong, it was
slow. What blocked ownership was the discount, not the combination rule.

Also fixes a real correctness bug: the observation count lived on the
discounter, which is shared by every track, so tracks pooled into one effective
sample and each was discounted by how many others happened to be on screen. It
is now a per-track parameter.

The registry's frame scope holds its lock for its lifetime and the mutex is not
recursive, so calling observe() inside a scope self-deadlocks. The pipeline
never does — separate nodes — but the test did, and hung rather than failing.
Documented at the call site.

Verified end to end: the same clip that produced zero actors now identifies
Bing Crosby and Dorothy Lamour with belief 0.97.

Suite: 96 cases, 6142 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-025 | SR-002
 2026-07-31 16:51:08 +02:00
dtourolle af5208035e docs: the RANSAC aligner was a defect, measured 
AR-005 replaced cv::estimateAffinePartial2D(..., RANSAC, 3.0) with Umeyama
least squares over all five points — the estimator InsightFace aligns with, and
so the one the ArcFace/LVFace training crops were produced by.

The first note here assumed the two agree wherever RANSAC keeps all five points,
leaving a small divergence on non-frontal faces. Measured on 400 gallery
headshots with the model held fixed, that was wrong: the crops disagree by a
median 17 source px and 83.5% embed below cos 0.99 of their Umeyama counterpart.
A 4-DoF similarity is exactly determined by two points, so every minimal sample
fits its own pair perfectly and is scored on the other three; real landmarks sit
a median 2.74 canonical px from any similarity fit, so a landmark outside the
3 px band is the common case and RANSAC returns an under-determined transform.

How much that cost in accuracy is a separate question, and the honest answer is
less than those numbers suggest. Rebuilding the full gallery moved the
intra/inter separation the AR-023 calibration is fitted from by 0.583 to 0.590:
the old warp was wrong but self-consistent, gallery and probe both went through
it, and the embedder tolerates framing variation. The sharper evidence is
duplicate detection — the rebuild dropped 1614 near-duplicates against the
original build's ~100, because unstable two-point fits gave near-identical
images visibly different vectors. That instability, not a headline accuracy
delta, is what a tracker accumulating evidence across frames was paying for.

Also records the AR-030 residual's real-data floor: on the most cooperative
images the pipeline sees, it runs a median 2.74 px, so landmark noise occupies
the first few pixels and the synthetic foreshortening ladder is optimistic about
the low end. Any discount curve has to treat that range as uninformative rather
than as mild pose, and VR-012 must set thresholds against the measured
distribution.

Tests carry the tag they verify: the residual's roll/scale invariance and
monotonicity under foreshortening are what make it a pose measure rather than a
pose-and-everything-else measure.

TRACES: AR-005, AR-030 | SR-002
 2026-07-31 16:39:12 +02:00
dtourolleandClaude Opus 5 d3ab598434 feat(artifacts): push and pull the VR-013 corpus 
The cross-source study needs two 4K recordings and a hand-sorted set of
face crops, neither of which belongs in git. Adds an xsource target to
both artifact scripts.

Push uploads the clips as-is (already compressed) and zips labelling/.
Pull fetches both and regenerates frames with ffmpeg rather than
downloading them: ~320 MB of PNG that is deterministic from the clips.
The extraction settings are pinned in the script, not left to the
caller, because the manifests key on frame filenames and on detection
order within each frame — verify_labels.py runs afterwards and fails
loudly if they drift.

Pull refuses to overwrite an existing labelling/. It is human ground
truth: somebody looked at 167 crops and placed each one, and silently
replacing that with a remote copy would destroy the expensive half of
the study.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-013
 2026-07-31 15:58:24 +02:00
dtourolleandClaude Opus 5 66c9ca0a0c refactor(VR-005): drive the study off the sae_embed bindings 
Deletes the Python ports of SCRFDDecoder, ArcFaceEmbedder, align_face,
enhance_for_retry and calibrate_gallery, and calls the shipped C++
instead. 297 lines removed, 108 added.

The ports existed because sae_embed only exposed embed(path), so a
caller could not embed a crop it had degraded. That gap is closed:
detect(), align_face(), enhance_for_retry(), embed_crop()/embed_crops()
and GalleryCalibration are bound now, so there is no longer a reason to
keep a second implementation of any of them.

The calibration is the one that mattered. A parallel copy of the sigmoid
is precisely where "always the calibrated probability, never a raw
cosine" (AR-024) breaks without anyone noticing — the copy goes on
returning plausible numbers after the original has moved. Scoring
through the binding makes the rule structural rather than remembered.

Verified against the committed run: same shape, FPI 0.0% at every size,
same operating point of 32 px. Absolute rates differ by 1-2 points
because this check sampled 100 actors / 574 crops against the original's
258 / 999, not because anything regressed.

Also: --providers and --batch are gone, since provider selection and
batching belong to the backend; embeds are chunked at its max_batch,
because the engine does not split an oversized request and a whole
gallery in one call asks CUDA for a multi-gigabyte buffer. DEDUP_SIM and
MIN_EMB_FOR_POSITIVE stay as mirrored constants — used only to report
the population the C++ fitted on, not to refit it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-005 | AR-024
 2026-07-31 15:51:50 +02:00
dtourolle 81ec77625c Merge branch 'feature/gallery-report' into feature/opencv5 2026-07-31 15:31:07 +02:00
dtourolleandClaude Opus 5 1dfd6fea11 feat: gallery build report 
GR-003 — the calibration fit already computed per-actor dedup counts, how many
actors are eligible for positive pairs, and a 200-bin histogram of the intra and
inter distributions, then discarded all of it to stderr. Nothing persisted, so
nobody could audit whether a gallery was any good.

The report is written alongside the gallery at build time. That is the right
moment: the matcher fits the same sigmoid at analysis time, but by then the
answer is per-run and nobody is looking, whereas build time is when a gallery's
quality is actually decided.

What it surfaces, in order of usefulness:
- actors with no usable image — a silent recall ceiling, since the pipeline can
  never name them and nothing else says why
- actors below the positive-pair threshold — not broken, so nothing complains;
  they just quietly weaken every threshold downstream
- near-duplicate references removed, per actor and total
- the fitted calibration AND the two distributions behind it

That last one is the point. Every threshold in the pipeline is expressed in the
probability space this sigmoid defines, so if the distributions overlap heavily
the calibration is weak and every downstream decision inherits it — while the
gallery still looks fine from the outside.

The gallery-derived prior, intra/(intra+inter), is computed and reported but the
shipped default of 0.5 is deliberately left alone. The spec records these as
disagreeing; now the real value is visible, so the decision can be made on
evidence rather than argument.

Three tests: a zero-image actor is visible in the report, an under-referenced
actor is counted, and the report round-trips through JSON.

Suite: 95 cases, 6142 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: GR-003 | SR-001
 2026-07-31 15:30:46 +02:00
dtourolleandClaude Opus 5 6aabeb9897 feat: provenance attributes on the embedding dump 
VR-010 — a dump made with one detector/embedder pair was byte-indistinguishable
from one made with another, except for the two attributes GR-004 added. Replayed
against a gallery from a different model, cosine similarities are meaningless
but look entirely plausible. The register states the principle directly: a
fixture whose provenance is unknown is worse than no fixture, because it will be
trusted.

Sixteen attributes now record everything that determines the dump's content:
detector model and thresholds, min_face_px, max_faces, cut_threshold,
dense_scale, bbox_upscale, start/end, track_assoc_min_prob, and scene_detect.

scene_detect is the one that matters most. is_scene_boundary is all-zero both
when the detector found nothing and when it never ran, and those mean completely
different things to a consumer — without the flag they are indistinguishable.

No schema_version bump: new root attributes are additive and replay.py already
reads attributes with a default, so older dumps stay readable and the committed
fixtures — which predate this — still load.

Also corrects SCHEMA.md, which claimed bbox was already mapped to original
resolution at dump time. It is not; the upscale is applied downstream in the
matcher, after the dump tap. Harmless while dense_scale is 1 and silently wrong
otherwise, so bbox_upscale is now recorded and the doc says what the code does.

Verified end to end: all sixteen attributes present and correct on a freshly
generated dump.

Suite: 92 cases, 6136 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-010, VR-001 | PR-002
 2026-07-31 15:24:06 +02:00
dtourolleandClaude Opus 5 01d7ead1e7 study(VR-013): cross-source identification probe over input resolution 
Gallery from one recording, probes from another, sweeping the probe's
input resolution end to end. VR-005 asked the same question over gallery
mugshots but degraded an already-aligned 112x112 crop with alignment held
perfect, so it isolates the embedder. Here the whole frame is downscaled
before the detector, so detection and landmark regression degrade with
it — which is most of the difference.

Corpus is two 4096x2160 clips of one shoot, four people, hand-sorted.
Ground truth is sorted by hand and gated by verify_labels.py; labels
carried down the scales geometrically by box position, never by
embedding similarity, which would keep only the faces the embedder
already gets right and drop the ones the sweep exists to find.

Findings, all scored through the production gallery sigmoid at
prob_threshold 0.754 — never a raw cosine:

- Holding 90% of the plateau needs ~50 px end to end, against VR-005's
  ~22 px. min_face_px at 40 looks right; 32 would admit faces in the
  falling region.
- FPI is 0.0% at every scale. Resolution loss goes entirely to TBI.
- The ceiling is cross-view, not resolution: everyone matches themselves
  within a recording (0.55-0.85) and collapses across two (0.14-0.45,
  threshold 0.335). Only the subject with frontal *gallery* references
  identified reliably, whatever their probe pose — so the lever is
  gallery pose coverage, not a better landmark source.
- Averaging SCRFD's overlapping detections instead of discarding them at
  NMS lifts cross-recording TPI 41% -> 49%, for one forward pass and no
  extra model.

Four identities and one shoot, so the shape is the result and the
absolute rates are not. Both clips contain all four people, so there is
no out-of-gallery class and the 10x-weighted out-of-cast misID is
untested here.

Clips, frames, hand-sorted crops and results are gitignored and belong
in the artifact registry — the sorting is human ground truth and
expensive to redo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-013 | AR-002, AR-005, AR-024
 2026-07-31 15:20:18 +02:00
dtourolleandClaude Opus 5 042e424961 study(VR-005): minimum face size from downscaled gallery mugshots 
Holds out one mugshot per actor, degrades that probe to each candidate
face size and matches it against a gallery held at native resolution,
reporting TPI/FPI per size. Replaces AR-002's 66x66 px working estimate
with a measurement. Needs no video and no ground truth beyond the
mugshot cache already on disk.

LVFace-B over 258 actors, 999 gallery embeddings, threshold 0.754:

    px    12    16    20    24    32    40   48+
   TPI   6.6% 46.5% 81.8% 93.4% 98.1% 99.2% 99.2%

FPI is 0.000 at every size — a face too small to identify degrades to
unidentified, never to a wrong name. rank-1 holds at >=99.6% from 24 px
up, so what fails first is the calibrated probability crossing
threshold, not the ranking.

Two limits on reading this. FPI grows with the number of actors
competing, so 258 understates it against a production library. And
detection and alignment run on the native image with only the resulting
112x112 crop degraded, so landmark error at small face sizes is excluded
by construction and the curve is an upper bound — VR-010 measures the
same question end to end, and lands well above these numbers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-005 | AR-002
 2026-07-31 15:17:39 +02:00
dtourolleandClaude Opus 5 9fc2763096 feat: expansion promotion gated on all three discontinuity signals 
AR-019 — promotion may only borrow same-identity evidence from a span where
identity is certain, so every discontinuity signal now clears the buffers rather
than just the histogram cut.

is_scene_boundary was already named in the gate but never set by anything, so
that half of it was dead until AR-010 gave it a producer. It now does what the
spec always said. The third signal, an identity contradiction, needs no code
here: AR-015 closes a track whose belief swapped, so it can no longer promote.

Ownership now comes from the registry rather than a second tally. TrackGallery
was computing its own plurality vote over accepted frames, which meant two
different answers to "who is this track" could coexist in one run — and the
expansion one ignored the Bayesian accumulation entirely, weighting thirty
near-identical looks the same as thirty distinct ones. The local tally survives
only as a fallback for callers with no registry attached, which is the unit
tests and the replay harness.

Suite: 92 cases, 6133 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-019, AR-010, AR-015 | SR-005
 2026-07-31 15:11:56 +02:00
dtourolleandClaude Opus 5 c843c4abe3 feat: banded admission for the per-subject embedding store 
AR-018 — an embedding joins a track's store only if its similarity to something
already there falls inside a band, rather than merely being far from the gallery.

Above the upper bound it is redundant: another look at a pose the store already
covers, teaching the annex nothing while costing a slot a novel view could have
used. Below the lower bound it is suspect: within one track every face is the
same person by construction, so an embedding unlike everything else on the track
is evidence that construction failed — a track-ID collision or a bad detection.
Admitting it is exactly how an actor's annex gets poisoned with someone else's
face.

The old gate had only the upper half of that idea, expressed as a raw cosine
against the gallery. Both bounds are now calibrated probabilities (AR-024), so
the same number means the same thing here as in association and evidence
weighting rather than three different things.

This catches track-ID collisions EARLIER than the spread gate did — at the door
rather than at promotion — so the buffer never becomes two-person in the first
place. The spread gate stays as a second line for a track that drifts gradually
instead of jumping. The existing test was asserting the mechanism rather than
the outcome, so it was rewritten to assert what actually matters: whichever gate
fires, the outsider must not reach the annex.

Rejections are counted. A store that admits nothing is as broken as one that
admits everything, and neither is visible otherwise.

Band defaults 0.90-0.95 are working values pending VR-007; the two bounds fail in
opposite directions and must be swept separately.

Suite: 92 cases, 6133 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-018, AR-024 | SR-005
 2026-07-31 15:10:31 +02:00
dtourolleandClaude Opus 5 cc1bed92d8 perf: back the CPU similarity GEMM with OpenBLAS 
The CPU path was a scalar triple loop. It is the correctness oracle for the GPU
backends, but it is also what CI runs — there is no GPU on the N100 host — and
since AR-003 removed the per-frame face cap, a crowded frame now scores many
faces against a library-scale gallery. Scoring one face against 5000 embeddings
is 2.6 MFLOP; in scalar that does not hold up (AR-027).

S(g,f) viewed as row-major [n_faces x n_gallery] is exactly query * gallery^T,
so the loop nest collapses into a single cblas_sgemm.

OpenBLAS is optional in the build: found via pkg-config, and the scalar path
remains when it is absent so no hard dependency is added and the two can be
diffed when a similarity looks wrong. The configure step warns rather than
failing, since a developer without it should still get a working tree.

The test target links it too. Without that the suite compiles the scalar
fallback while the builder image ships CBLAS, so CI would be verifying a kernel
that is not the one running in production — the same class of mistake as testing
a path the gate never executes.

Recorded as required (not optional) in the DP-007 image, for the same reason.

Suite: 92 cases, 6136 assertions, with CBLAS compiled in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-026, AR-027, DP-007 | SR-001
 2026-07-31 15:04:29 +02:00
dtourolleandClaude Opus 5 13bdc27566 feat: join the decode butterfly so scene boundaries reach the face branch 
AR-010 — is_scene_boundary had no producer: SceneDetectorFunc was a terminal
sink writing scenes.json and never annotating the frames flowing to face
detection. The flag was permanently false, so the boundary half of AR-007's
frame-dependent association was dead code that a test could still exercise
synthetically and appear to verify.

The topology already forks after decode — dense frames to TransNetV2, sampled
frames to face detection — so this is a fork-join. SceneBoundaries is the join:
the detector publishes each window's verdict with a watermark, and an annotator
on the sampled branch stamps the flag.

The watermark is the part that matters. TransNetV2 buffers 100 frames before it
can score any of them, so at any instant it has an opinion up to some time T and
none after. Without recording T a consumer cannot tell "no boundary" from "not
scored yet", and those demand opposite behaviour — treating unscored frames as
boundary-free is exactly what makes a downstream check pass while verifying
nothing.

Buffering alone does not work, which was my first attempt. Channel depth creates
lag only when the consumer is slower, and the face branch runs four orders of
magnitude faster per frame than TransNetV2 (0.01ms vs 400ms), so its channels
drain instantly and no lag accumulates. Measured: 106 of 364 frames outran the
detector. The annotator therefore waits on the watermark explicitly. The
detector signals completion so the tail cannot deadlock, and publishes from
flush_remaining too — without that the final frames arrive with no verdict.

Boundaries are deduped on publish, matching what scenes.json does at write time.
A run of adjacent high-scoring frames is one boundary, not several; leaving them
raw made this view report 357 where the file said 13. Now the two agree exactly.

Frames past the detector's last scored window remain unverified and are counted
as such rather than silently marked boundary-free.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-007, AR-010 | SR-002
 2026-07-31 14:56:38 +02:00
dtourolleandClaude Opus 5 fa1c494825 docs: AR-010 is blocked on a design decision, not an implementation gap 
Making SceneDetectorFunc a pass-through does not work. TransNetV2 buffers 100
dense frames before it can score any of them and trusts only each window's
centre, so a boundary at time T is not known until roughly 3.3s after T at
30 fps. The face pipeline runs on a parallel branch and has long since passed T.
An association hint that arrives after the association is worthless.

Three options recorded with their costs: two-pass (correct, doubles the decode
that already dominates runtime), delaying the face branch (couples the two
branches' timing, which invites heisenbugs under backpressure), or leaving it
unwired.

Leaving it unwired costs less than it looks, which is what makes this a decision
rather than a defect. The redesign made cuts and boundaries do the same thing —
both say "spatial continuity is broken, associate on embedding" — so TransNetV2
adds nothing over the histogram except on transitions the histogram cannot see:
slow dissolves and fades. That gap is real but narrow.

Where TransNetV2 still earns its cost is AR-019, whose promotion gate wants a
span free of cuts and boundaries. A late answer is fine there, because promotion
happens on track confirmation rather than per frame — so it can be wired
offline against the collected boundary list, off the hot path entirely.

Recommendation: leave the association path on is_cut alone, wire boundaries into
AR-019, and revisit if dissolve-heavy material shows association failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-010, AR-019 | SR-002
 2026-07-31 14:27:05 +02:00
dtourolleandClaude Opus 5 f99f1c5ccc feat: no fixed cap on faces per frame 
AR-003 — max_faces defaults to 0, meaning no cap. A fixed cap discards the
SMALLEST faces first, which are exactly the background cast X-Ray still credits
with scene membership, so the pipeline was systematically losing the people it
is supposed to find in crowded scenes.

This is only safe now that AR-004 landed. Previously an uncapped frame would
have pushed more work into channels that dropped on overflow, trading a visible
cap for silent loss. With backpressure the producer slows instead, so per-frame
cost is contained rather than discarded.

The matcher's kMaxFaces used to throw above 32, which made it an accidental
second cap. It sizes the similarity engine's preallocated buffer, so it bounds
memory rather than face count — the frame is now scored in batches of that size.
Memory stays bounded; faces do not.

Largest-first ordering is kept even without the cap, and the comment now says
why: the Hungarian solver tie-breaks on index order, so that ordering is
load-bearing for the replay determinism test rather than a leftover of the cap.

Verified end to end on a real clip: identical output to the capped run (385
frames, 693 faces), which is expected since that footage peaks at 4 faces per
frame — the point is the absence of a regression. The committed fixtures remain
byte-identical and valid.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-003 | SR-002
 2026-07-31 14:22:20 +02:00
dtourolleandClaude Opus 5 3605b8da78 fix: point the KPN submodule at the merged commit 
The recorded pointer was be6e922 — the backpressure fix as originally committed,
before it was rebased onto KPN master. That commit exists on no pushed branch,
so a fresh clone of this branch could not fetch the submodule at all.

Now 6595e6e, the same change on KPN master.

Worth noting for next time: rebasing a submodule commit after the superproject
has already recorded it silently invalidates the pointer. Nothing in the
superproject's status shows it, because the submodule working tree is clean and
at a valid commit — just not the one recorded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004 | SR-002
 2026-07-31 11:53:27 +02:00
dtourolleandClaude Opus 5 61e487fbee test: replay the real tracker and registry from committed fixtures 
Tier T2 — composition rather than units. The registry tests construct awkward
states directly; these feed the pieces real 480x360 footage with the cuts, gaps
and crowded frames that synthetic input does not produce.

Six cases:
- fixture integrity: exact frame and face counts, contiguous face_offset, and
  the embedder identity each dump carries (GR-004). The counts are asserted
  exactly rather than approximately, which was impossible before AR-004 — what
  a lossy run dropped depended on timing.
- determinism: replaying a fixture twice gives identical track ids and windows.
  This is the property the whole fixture strategy rests on; without it every
  golden output derived from a fixture is unreliable and the CI replay tier is
  worthless.
- every face is assigned a track, and flush leaves nothing open — a track still
  live at EOF is a window that never reaches the output.
- windows are well-formed and inside the clip. A window ends at the last
  sighting, so it can never extend past the footage that produced it.
- a longer extinction window yields fewer, longer tracks. On the sparse fixture
  (140 faces over 385 frames) that is the difference the constant actually
  makes: absorbing a gap versus splitting a window.
- the cut-heavy fixture still contains cuts. This guards the corpus, not the
  code: a regeneration that produced cut-free fixtures would leave the
  association tests passing while silently testing nothing.

Driving the functors directly rather than through a KPN network is deliberate —
no threads, no channels, no scheduling, so the same input gives the same output.

Suite: 86 cases, 6106 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004, AR-012, AR-013, VR-001, VR-002 | SR-002
 2026-07-31 11:23:17 +02:00
dtourolleandClaude Opus 5 c12838b9fd fix: dropped frames fail the run instead of printing a footer 
A drop was reported to stderr and the process exited 0, so a run that discarded
320 frames "succeeded" and produced a truth file that looked complete. The
output in that case is a claim about footage that was never analysed, and
nothing in the file says so.

Now exits 2 and says why. Distinct from 1 (node crash) because the failures are
different: a crash produced no output, a drop produced output that cannot be
trusted.

This is also the regression test for AR-004 that otherwise did not exist. The
backpressure fix is one line in the KPN submodule — easy to lose in an update —
and with data pushes blocking, a drop can no longer occur on the data path. So
any drop now means either that fix regressed or a channel was disabled mid-run,
and both are worth stopping for.

Verified: a clean run still exits 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004 | SR-002
 2026-07-31 10:45:39 +02:00
dtourolleandClaude Opus 5 d31526cfaf test: committed replay fixtures from the public-domain corpus 
Five HDF5 embedding dumps from bali/ — Road to Bali (1952) — 3.6 MB total,
generated at 5 fps with a 32 px minimum face. CI never calls a model, so
inference happens on a GPU host and CI replays these as data; everything
downstream of embedding is cheap CPU maths.

Public domain is the reason this corpus rather than a convenient one: derived
fixtures can be committed, where anything cut from a copyrighted title could not
live in the repository at all.

The set covers distinct behaviours rather than being five of the same thing:
bali_28 has 9 cuts, so it exercises shot/reverse-shot association (AR-007);
bali_46 is sparse at 140 faces over 385 frames, so it exercises gaps and
extinction (AR-013); bali_13 is the busiest at 4 faces per frame; bali_31 is
short at 29s. All five recorded zero drops.

Both pinned parameters are consequences of measurements, not defaults: 5 fps
because 1 fps over a 77s clip is 77 frames, too thin for an extinction window
measured in tens of seconds; 32 px because that is the VR-005 floor, and the
corpus is 480x360 so a stricter value would reject most of what is there.

make_fixtures.sh regenerates them. Reproducibility is the requirement — a
fixture whose provenance is unknown is worse than none, because it will be
trusted. These are byte-reproducible only because of AR-004: before node
outputs blocked rather than dropped, the same command produced different dumps
run to run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-001 | PR-002
 2026-07-31 10:41:51 +02:00
dtourolleandClaude Opus 5 09a4650fd9 feat: backpressure — the pipeline slows instead of losing frames 
Picks up the KPN fix: node data outputs block on a full channel rather than
dropping. Sentinels stay out-of-band, so EOF can always overtake a stalled data
path and the hold-and-wait deadlock that comment warns about is not reachable.

Verified on a 77s clip at 5 fps, which should yield 385 sampled frames:

  before  65 written, 320 dropped, 29s, two runs differ
  after   385 written, 0 dropped, 17s, two runs byte-identical

The determinism is the part that matters. Golden fixtures were impossible while
what got dropped depended on timing; VR-001 fixture generation is unblocked by
this, and so is the CI replay strategy that depends on it.

Faster rather than slower, which is worth recording because the intuition runs
the other way: a dropped frame has already cost its decode, and the overflow
exception cost more still.

AR-004 is not fully closed. Channel capacity remains a count of items, while a
face carries a 112x112 crop and a 512-float embedding — so a crowded frame
occupies far more memory per slot than a sparse one. Bounding by bytes in
flight is the remaining half, and it matters once max_faces is removed (AR-003).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004, VR-001 | SR-002
 2026-07-31 10:35:29 +02:00
dtourolleandClaude Opus 5 b98372bad8 docs: AR-004 is a KPN change, with the measurement behind it 
Backpressure cannot be implemented in this repository. Every node output in KPN
uses the dropping push() (pool_node.hpp:404 and :710, plus branch, fanout and
interrupt_node). A lossless push_blocking() already exists on both Channel and
OutputPort — "wait for the consumer to drain instead of dropping; the producer
just runs slower" — and nothing calls it. The fix is a per-channel policy or a
network default in KPN, and this pipeline should select lossless: a dropped
frame here does not degrade a result, it silently changes one.

Measured rather than inferred. One 77s clip at 5 fps should yield ~385 sampled
frames. On CPU it produced 49, ending at 51s, with 285 dropped at camera_pos
and 51 at face_aligner. Rebuilt with CUDA the same clip ran in 29s and reached
EOF correctly, and still dropped 320 at camera_pos, yielding 65. Faster
hardware moves where the queue backs up; it does not change what happens when
it does — which is why this is a correctness requirement rather than a
throughput one.

Raising channel capacity is therefore a stopgap: it lowers the probability of
overflow without changing the behaviour on overflow, and the failure it hides
is silent corruption of the output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-003, AR-004, VR-001 | SR-002
 2026-07-31 10:29:53 +02:00
dtourolleandClaude Opus 5 e0f9c95689 docs: minimum face size is 32px, measured (VR-005) 
Replaces the 66px working estimate with the sweep result. 258 probes degraded
to each size and matched against a native-resolution gallery: 16px 46.5% TPI,
24px 93.4%, 32px 98.1%, 40px 99.2%, flat to 112px. The knee is 24-32 and 32
sits within about a point of the ceiling.

The estimate was roughly twice too strict. At 66px a large share of usable
faces would have been discarded, and on 480x360 sources most of them — which
is exactly the resolution of the fixture corpus.

The more useful finding: false identification was 0.0 at every size, including
12px. Small faces fail by becoming unidentified, never by being attributed to
the wrong actor. That asymmetry is what makes a low threshold safe — the cost
of admitting a marginal face is a miss, not a false claim.

Caveat recorded rather than assumed: FPI grows with gallery size, so 258 actors
understates it against a full library. Treat 0.0 as an observation at this
scale, not a property.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-002, VR-005 | SR-002, PR-002
 2026-07-31 10:22:38 +02:00
dtourolleandClaude Opus 5 9e4cdc4efc docs: bali fixture corpus, and AR-004 blocks reproducible fixtures 
Records `bali/` — five ~77s clips of Road to Bali (1952) — as the fixture
source. Public domain, which is the point rather than a convenience: derived
fixtures can be committed, where anything cut from a copyrighted title could
not live in the repository at all.

Makes explicit what the tier table only implied: CI never calls a model. Not a
preference — the embedder measures ~930 ms/frame on the CPU provider, so a 77s
clip at 5 fps is six minutes of inference. Every model invocation happens
locally and CI consumes the result as data, which is what makes the T1/T2 split
load-bearing rather than stylistic.

Two properties of the corpus to design around: 480x360 puts many faces below
the AR-002 66px minimum, so generation must set and record --min-face-px; and
77s at 1 fps is too thin to exercise an extinction window measured in tens of
seconds, so fixtures want 5 fps.

The finding that matters: a trial dump produced 49 frames of an expected ~385,
stopping at 51s of 77s, with 285 frames dropped at camera_pos and 51 at
face_aligner on channel overflow. Channels drop rather than block, and what
drops depends on timing, so the same command twice can yield different dumps.
Golden fixtures cannot be built on that — AR-004 is a prerequisite for VR-001
fixtures, not just a throughput concern for crowd scenes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004, VR-001 | PR-002
 2026-07-31 10:19:10 +02:00
dtourolleandClaude Opus 5 08941540cb feat: presence windows come from registry claims (schema_version 2) 
The sink no longer reconstructs presence from per-frame detections. A reaped
track already IS a window — [first_seen, last_seen] of a track an actor owned —
so it is pushed straight to the aggregator when it dies and written out as-is.

AR-012 completed end to end. The annealing pass is deleted, not disabled:
anneal_sec existed only to bridge gaps between isolated accepted frames, and a
track that survives its own gaps leaves it nothing to do. The field is REMOVED
from the output rather than zeroed — a field naming a mechanism the pipeline no
longer has is actively misleading to anyone reading a manifest, and would
outlive everyone who remembers why it reads 0.

IR-002 — schema_version 2, matching jRay/SPEC.md JR-002. Windows become objects
carrying `belief` and `route` rather than bare float pairs, so a consumer can
caveat or filter instead of treating every window as equally certain. The new
`extraction` block carries `extinction_sec` (the successor to anneal_sec, and
what a consumer actually needs to interpret a window) and `gallery_scope` —
global vs limited being the strongest single quality signal when two manifests
compete for one cut, since identical gallery_size can mean very different
recall.

AR-016 wired: a pre-write hook flushes the registry with the last timestamp
seen, so tracks still live at EOF are emitted. A film ends with faces on screen
and those tracks have not timed out; without this the closing scene's cast is
silently dropped, which reads as a recognition miss rather than a bookkeeping
bug.

IR-003 stays In Progress deliberately: the sink now writes after the flush, but
the deferred re-identification pass (AR-020) does not exist yet, so output is
still final at EOF rather than after it.

This is a BREAKING format change and part of the coordinated SR-003 bump — it
must ship together with the jRay reader and the server's acceptance of the new
shape, not ahead of them.

Suite: 80 cases, 3250 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-012, AR-016, IR-002, IR-003 | SR-002, SR-003
 2026-07-31 10:10:51 +02:00
dtourolleandClaude Opus 5 fe29d014da feat: identity evidence reaches the registry 
Closes the link that made AR-012 inert: the tracker was maintaining registry
state, but nothing called observe(), so no belief accumulated, no track was ever
owned, and no presence claim could be emitted. Tracking worked and presence did
not.

The matcher now feeds every scored face to the registry as a calibrated
posterior plus its embedding. Deliberately every scored face, not only the ones
clearing prob_threshold: a run of near-misses for one actor is evidence, and
discarding it would leave ownership depending on the per-frame threshold this
redesign exists to stop relying on. The registry discounts for correlation and
decides ownership from the accumulated posterior (AR-025).

The registry is an optional dependency of the matcher. Without one it behaves
exactly as before, which keeps the replay harness and the unit tests working
unchanged rather than forcing every caller to construct a registry it does not
need.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-012, AR-025 | SR-002
 2026-07-31 10:05:59 +02:00
dtourolleandClaude Opus 5 be5f67fa96 feat: tracker on the registry — one pool, calibrated, frame-dependent 
Merges feature/tracker-registry. See e9aea3f for the detail; in summary the
tracker no longer owns track state, association weights embedding over position
whenever position is uninformative, and every similarity comparison is a
calibrated probability.

Four constants retired: track_max_embed_dist, cut_revive_sim,
cut_inactive_max_frames, track_max_frames_missing.

Suite: 80 cases, 3250 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-007, AR-008, AR-024 | SR-002
 2026-07-31 10:01:31 +02:00
dtourolleandClaude Opus 5 e9aea3fc41 feat: tracker owns no state; association is frame-dependent and calibrated 
Three requirements land together because they cannot be separated. The
cross-cut revival branch was the only user of cut_revive_sim, so retiring that
raw cosine forces the pool collapse, and collapsing the pool removes the only
caller of the constant. Splitting them would have produced an intermediate
commit whose only purpose was to be split.

AR-008 — FaceTrackerFunc no longer keeps its own tracks_/inactive_ maps; it
holds a shared_ptr<TrackRegistry> and operates on it directly. Two parallel
copies of track state could disagree, and every divergence would surface as a
wrong presence window with nothing to indicate it. There is now ONE candidate
pool: last_seen alone says whether IoU is meaningful. The park/revive path is
deleted outright — matching a dormant track is ordinary inter-frame
association, and continuity falls out of the embedding comparison the tracker
already did rather than being a mechanism of its own.

AR-007 — track_alpha becomes the base weight for ordinary frames only.
Association drops to embedding-only when position carries no information:
on is_cut or is_scene_boundary, because the viewpoint changed, and for a
dormant track, because time has passed since its box was last valid. The second
case matters as much as the first and had no equivalent before.

AR-024 — association cost is a calibrated probability, never a raw cosine. The
tracker takes the calibration belonging to the active embedder, the same
function object EvidenceDiscounter uses. track_max_embed_dist becomes
track_assoc_min_prob, which means the same thing for every model, gallery and
face size, where a bare cosine threshold did not.

Retired: track_max_embed_dist, cut_revive_sim, cut_inactive_max_frames, and
track_max_frames_missing — the last superseded by the registry's extinction
window. That one is worth naming: a frame count silently changed meaning with
sample_fps, so the same configuration behaved differently at 1 fps and 5 fps.
Extinction is in seconds and lives in one place.

Tests rewritten rather than deleted. The old cases asserted revival by raw
cosine; the same behaviours are now asserted through the registry — a face lost
across a cut and re-associated is the SAME track, one unbroken window, and a
face returning past the extinction window is not. Added the case AR-007 exists
for: two people swap screen positions across a cut while keeping their faces,
and identity must follow the embedding rather than the box.

Suite: 80 cases, 3250 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-007, AR-008, AR-024 | SR-002
 2026-07-31 09:58:27 +02:00
dtourolleandClaude Opus 5 b7c96641a9 docs: no exact tier — the file-hash tier was withdrawn 
A stale reference to 'the runtime/exact tiers' as the fallback for media too
short to carry an audio signature. The exact tier keyed on a file hash and was
withdrawn on legal grounds: it fingerprinted the individual copy a user holds
rather than the cut the timings describe.

The pipeline never emitted a video_hash, so nothing in the code changes — but a
spec that still names a withdrawn tier is what makes the withdrawal look like an
oversight to the next reader, which is exactly how it nearly got re-added.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-31 09:55:50 +02:00
dtourolleandClaude Opus 5 843852e19c feat: registry owns correlation discounting (AR-024, AR-025) 
Moves two responsibilities inside the registry that callers should never have
been trusted with.

AR-025 — per-frame evidence is discounted for correlation by the registry
itself, via EvidenceDiscounter. Log-odds accumulation is only valid for
independent observations, and consecutive frames of one track are anything but:
near-identical pose, lighting and expression. Accumulated naively, thirty frames
of the same face at the same angle drive the posterior to certainty on what is
effectively one measurement.

Each observation is weighted by how much it adds — a view already contributed
counts for ~nothing, a genuinely new pose counts in full. This reuses the
novelty judgement gallery expansion already makes rather than inventing a second
one. The discounter is a separate class the registry holds, so it stays testable
and swappable, but it is a constructor argument rather than an option: there is
no correct way to accumulate without it.

AR-024 — observe() takes a calibrated probability and converts to log-odds
internally. A caller can no longer hand it a raw cosine, which would have been
silently wrong rather than obviously so. Retiring the remaining raw-cosine
constants in the tracker is still open.

DeadTrack now reports effective_obs alongside observations: the raw count and
the evidence that actually counted. A large gap between them is a track the
camera stared at, and worth seeing.

Three tests, one of which is the point: two tracks given the same number of
observations at the same posterior, one repeating a single view and one seeing
eight distinct ones, must not end up equally confident. Without discounting they
would be identical.

Fixed a test that asserted a belief swap on tied evidence. A tie leaves
ownership where it is — a challenger must out-accumulate the incumbent, since
one contrary observation is noise. The original test passed only because it fed
raw log-odds directly.

Suite: 78 cases, 3245 assertions. Coverage 20/63 to 22/63.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-024, AR-025 | SR-002
 2026-07-31 09:15:44 +02:00
dtourolleandClaude Opus 5 f0c7126f80 feat: TrackRegistry — presence follows track extent 
The spine of the redesign. Presence is now the extent of a track an actor owns,
[first_seen, last_seen], rather than the subset of frames in which recognition
happened to succeed. An actor recognised only at the end of a long track is
present for all of it, which is what the scene-scoped ground truth actually
records.

AR-013 — `last_seen` as an optional carries the entire liveness state: unset
means on screen, set means went off at that timestamp and still revivable,
reaped means emitted and erased. No missing-frame counter, no expired flag. It
subsumes the tracker's existing two-pool split, so there is no separate revival
path — matching a dormant track is ordinary inter-frame association.

The asymmetry is the point: interior gaps are claimed, the trailing cool-down is
not. A face lost and re-associated within the timeout never closed its track, so
the gap is presence — someone briefly occluded has not left the scene. But a
track that dies ends at its last sighting, never at the death time. That is
precisely the over-claim the retired extinction_sec keep-alive produced, where
presence ran on into the closing credits.

AR-014 — a belief swap A→B closes the track and opens a successor at the swap
frame. Not a correction: two non-twins both clearing the threshold on one face
is not realistic, whereas a track_id carried across a viewpoint change onto a
different person is. Treating it as a swap-and-continue would emit one window
blending two people; treating it as a boundary yields two that are each right.

AR-015 — two live tracks owned by one actor means at least one is wrong, since a
person cannot be in two places at once. A reverse index catches it on the update
that causes it rather than by scanning. This makes identity a third cut
detector, independent of the histogram and TransNetV2 and firing where those
failed.

AR-016 — flush() closes tracks still live at EOF. Without it a film ending
mid-shot silently drops its closing cast, which presents as a recognition miss
rather than a bookkeeping bug.

Reaping hands the dead track to the aggregator and erases it, so the registry
holds only live tracks and its size is bounded by concurrent on-screen faces
rather than growing with the film.

Locking: a frame's association pass is atomic as a unit via FrameScope, since
per-call locking would let another thread observe a half-updated frame.
owner() reads tally and verdict under one lock — separately, a track could be
both unowned and owned within a single promotion decision. A vote for an
already-reaped track is dropped and counted, because a nonzero count means the
timeout is shorter than the matcher's lag.

11 unit tests, driven directly against the registry with no network and no
fixture — the awkward cases are constructed rather than hunted for. Suite: 75
cases, 3236 assertions. Coverage 14/63 to 20/63.

Not yet wired into FaceTrackerFunc; that is AR-007/AR-008.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-012, AR-013, AR-014, AR-015, AR-016, AR-017 | SR-002
 2026-07-31 09:08:39 +02:00
dtourolleandClaude Opus 5 b35d49c772 docs: tag the implemented core with its requirement IDs 
Adds TRACES tags to code that already satisfies a Done requirement, so coverage
reflects what exists rather than starting from zero:

AR-001 face detection, AR-005 ArcFace alignment, AR-023 calibration fit,
DP-001/DP-002 the single analysis core behind the CLI, IR-001 truth-file
emission, IR-006 the Jellyfin round trip, GR-001/GR-002 gallery build and
incremental merge, VR-001 the embedding dump, VR-002 replay through the real
nodes, VR-003 per-second scoring.

Only Done requirements are tagged. A tag on Planned work would inflate coverage
with fiction that looks plausible — the same failure family as a gate that
cannot fail, and harder to spot.

GR-005 (gallery never leaves the instance) stays untagged deliberately: it is a
prohibition satisfied by the absence of an egress path, so there is no unit that
decides it. Same shape as PR-005 in the system spec, which has no software row
for the same reason. A goal held only by prohibitions cannot be verified by
pointing at code.

Coverage 5/63 to 14/63. The three VR tags are reported as tagged-but-unexecuted
and excluded from the numerator, since their tier cannot run on the CI host —
tagging deliberately cannot raise the number on its own.

Suite still 64 cases, 3199 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-001, AR-005, AR-023, DP-001, DP-002, IR-001, IR-006, GR-001, GR-002, VR-001, VR-002, VR-003
 2026-07-30 21:20:27 +02:00
dtourolleandClaude Opus 5 62396fce75 docs: record why exclude_dirs stays unset, refresh matrix 
The tool's defaults already exclude `vendor`, which covers the submodule at
scripts/vendor/jray-project. Setting the key explicitly is a trap worth
documenting: it REPLACES the defaults rather than extending them, and matching
is on path components rather than prefixes — so ["scripts/vendor"] matches
nothing while silently dropping __pycache__, node_modules, build and the rest.
Verified: the submodule's source is not scanned, and the only vendored path in
the report is the system spec it reads for PR/SR orphan checking.

Coverage after the merges: 5/63, 0 orphans. Every tag names a real requirement,
and nothing claims a requirement that is still Planned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 21:16:15 +02:00
dtourolleandClaude Opus 5 908d166173 feat: bind galleries to the embedder that built them 
GR-004 — a gallery built with one embedding model is meaningless with another.
Cosine similarities across models are garbage but look entirely plausible, so
this fails silently and expensively; every measurement taken against a
mismatched pair would have been quietly wrong.

The stamp is the model basename plus a SHA-256 of its bytes, with embed_dim as
a cheap extra guard. The hash decides and the name explains, because neither
works alone: a name is a promise rather than a fact — models get re-exported in
place under an unchanged filename, which is exactly the case where the weights
differ and nothing else does — while a bare hash mismatch tells an operator
nothing actionable.

Mismatch is fatal in every mode with no bypass. Unstamped only warns, because
unstamped is unknown rather than known-bad, and an error firing on every legacy
gallery trains people to reach for the bypass reflexively. scripts/stamp_gallery.py
binds an existing gallery in place with no re-embedding, so the warning is a
migration step rather than a permanent state; --require-gallery-stamp promotes
it to an error once a site has migrated.

Two gaps found that would have defeated the requirement outright:

- Embedding dumps carried no stamp, so a replay — which has no live embedder —
  had nothing to check the gallery against. Dumps now carry embedder_model and
  embedder_sha256 as root attributes. Additive; schema_version stays 1. This is
  the same gap the dump audit identified independently.
- --merge produced one file holding two embedding spaces, which no later check
  can untangle. Merge paths now verify before writing.

The stamp also survives identity_matcher's calibration write-back, which would
otherwise have stripped it on the first analysis run — the check would have
worked exactly once.

Conflicts resolved additively: both branches appended a source to sae_gallery
and to the test target, and both edited the GR-004 register row.

Merged suite: 64 cases, 3199 assertions, passing on CPU with no GPU.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: GR-004, VR-001 | SR-001
 2026-07-30 19:04:07 +02:00
dtourolleandClaude Opus 5 662a469870 feat: v1 content-derived audio signature 
Implements the server spec §3 construction: a 120 s window centred on the media
midpoint, downmixed and resampled to mono 11025 Hz, a 4096/1024 Hann STFT, 32
log-spaced bands over 300-3000 Hz, one byte per frame carrying a 5-bit peak band
and a 2-bit energy class, base64 with a v1: prefix.

IR-004 — the signature itself, in src/audio_signature.{hpp,cpp}. Decode reuses
the already-linked FFmpeg libraries; libswresample was missing from ffmpeg_libs
and is added. The FFT is written out rather than taken from a library: the
output must be bit-identical against a separate C# implementation, so a
dependency whose version can change the numerics is a liability.

IR-005 — a golden fixture at tests/fixtures/audio/, verified against an
independent Python implementation producing identical bytes. FLAC rather than
WAV because 120 s of 11025 Hz PCM is 2.6 MB and does not compress in git; both
decode to identical samples. The PCM checksum is asserted separately from the
signature so a codec-level divergence is distinguishable from a DSP one.

IR-007 — media under 120 s emits no signature at all, since the centred window
underflows. The rule must be identical in both producers or signatures never
match on exactly the short items most likely to be misidentified.

IR-008 — the v1: prefix is emitted and honoured, so a future change to the DSP
chain is detectable rather than silently non-matching.

Six parameters the spec left undefined had to be pinned to reproduce a byte
stream at all: periodic Hann, band value as the mean of linear magnitudes, ties
to the lowest band, the energy-class definition and its thresholds, byte layout,
and the base64 alphabet. These are now normative in the server spec — left only
in a C++ header, the C# side would have guessed and diverged.

Not yet emitted into the truth file; that is the coordinated schema_version bump
under IR-002/IR-003.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: IR-004, IR-005, IR-007, IR-008 | SR-003
 2026-07-30 19:00:08 +02:00
dtourolle 28e3bd9496 docs: generated traceability matrix 
Committed rather than ignored, matching house precedent: coverage becomes
visible to anyone browsing the repo, and its movement over time is real history
worth having in the log.
 2026-07-30 18:59:06 +02:00
dtourolleandClaude Opus 5 d9aaf8fa4e build: consume the shared traceability tooling via submodule 
jray-project is added at scripts/vendor/jray-project and the extractor is used
from there. Only two files are repo-local: traceability.toml, which carries
everything repo-specific, and the CI workflow that invokes the vendored gate.

The extractor is deliberately NOT copied in. One implementation, parameterised
by config — a second copy would drift from the first, and the tool already
proves it works unchanged against all three registers.

Enables system_spec so PR/SR orphan checking runs: previously uncheckable,
because the system spec lived outside every component's checkout.

Gate is green at 0.0% of 63 requirements, which is correct — nothing is tagged
yet. Eleven are flagged unverifiable on this CI host and excluded from the
numerator rather than counted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 18:58:40 +02:00
dtourolleandClaude Opus 5 a2ebdc4cdd docs: GR-004 done — gallery/embedder binding 
Stamp is model basename + SHA-256 + embed_dim: the hash decides, the name
explains. A name alone is a promise rather than a fact — models get re-exported
in place under an unchanged filename, which is exactly the case where weights
differ and nothing else does. A hash alone is unactionable in an error message.

Mismatch is fatal in every mode with no bypass. Unstamped only warns, because
unstamped is unknown rather than known-bad, and an error that fires on every
legacy gallery trains people to reach for the bypass. A migration script binds
existing galleries in place with no re-embedding, so warn is not permanent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: GR-004 | SR-001
 2026-07-30 18:37:07 +02:00
Claude 7db40f430d GR-004: bind galleries to the embedder that built them 
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.

The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.

Mismatch is a hard error in every mode, with no bypass, naming both sides.

Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.

Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.

Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.

Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.

Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 18:35:46 +02:00
dtourolleandClaude Opus 5 020306c94f docs: builder images and per-backend release binaries (DP-008) 
Adds the build/deploy story that was missing: containerised builder images for
cpu / cuda / rocm, and release jobs producing prebuilt binaries so a first
install need not compile.

States explicitly that this does not reverse DP-005. That requirement rejects
Docker as a *runtime* — GPU passthrough is fragile and exists only because of
the container. Using it as a *build* environment is the opposite case, and lets
one machine produce binaries for backends it cannot itself run. Build in a
container, run natively.

Two things deliberately cannot ship, and the installer must not imply otherwise:
TensorRT engines are GPU-architecture and TRT-version specific, so
build_trt_engines.sh still runs on the target; and models are ~725 MB in LFS,
orthogonal to the binary.

The base image is chosen by the OLDEST glibc to be supported, not by
convenience — a binary built in a container runs against the host's glibc, and
getting this wrong fails at load with GLIBC_2.xx not found. Accelerator runtimes
have the same shape of problem, so each image documents its compatible
CUDA/ROCm range and the installer checks it rather than discovering a mismatch
at first inference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: DP-005, DP-007, DP-008 | PR-004
 2026-07-30 18:35:10 +02:00
dtourolleandClaude Opus 5 2919ed68d1 docs: audit findings, verification tiers, CI image, artifact storage 
Corrections from the dump audit and the completed agent work:

- AR-010 is not started, not in progress: is_scene_boundary has no producer
  anywhere. SceneDetectorFunc is a terminal sink writing scenes.json and never
  annotates the frame, so the field is permanently false and the dump column a
  constant 0. A replay test of the frame-dependent track_alpha would pass
  vacuously — the worst failure mode for a verification gate.
- T1 (functor-level) becomes the primary verification tier, not T2. KPN node
  functors are plain callables constructed outside the network, so a node is
  tested by calling operator() with hand-built inputs. That removes four
  hazards at once: fixture provenance, replay-from-frame-0, cross-test state
  leakage, and replay-harness nondeterminism. It also means a dead upstream
  producer no longer blocks testing its consumer.
- VR-010 (dump provenance) and VR-011 (replay harness rewrite) added. A dump
  made with LVFace is currently byte-indistinguishable from one made with
  w600k-R50 — the GR-004 problem again, in the dump.
- Four requirements had no verification tier at all; the traceability gate
  found them.
- DP-007: CI builder image, CPU-only, pinned by tag in the Gitea container
  registry. Corpus fixtures go to the package registry rather than LFS: LFS is
  pulled on clone and would tax every developer for data only CI reads.
- IR-004/005/007/008 marked done; the v1 DSP parameters they had to pin are
  now normative in the server spec.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-010, DP-007, IR-004, IR-005, IR-007, IR-008, VR-001, VR-010, VR-011 | SR-002, SR-003
 2026-07-30 18:33:32 +02:00
dtourolleandClaude Opus 5 45ef7c1916 Add the v1 audio signature to the pipeline (IR-004, IR-005, IR-007, IR-008) 
Implements the content-derived spectral-peak signature from
JRay-public-server/SPEC.md §3 so a truth file is self-identifying: 120 s
window centred on the media midpoint, mono at 11025 Hz, 4096/1024 Hann
STFT, 32 log-spaced bins over 300-3000 Hz, one byte per frame (5-bit peak
band + 2-bit energy class), base64, `v1:` prefix.

Audio decode is a second stream from the FFmpeg libraries the pipeline
already links for video; libswresample is added to the existing
ffmpeg_libs interface target. The FFT is written out rather than pulled
from a library for the same reason the plugin vendors one: the output has
to be bit-identical across two languages, so a dependency whose version
could change the numerics is a liability.

The server spec fixes the geometry but not enough to reproduce a byte
stream — Hann periodicity, band aggregation, the energy-class definition,
tie-breaking and the base64 alphabet are all unconstrained by it. Those
are pinned in audio_signature.hpp and mirrored in the golden fixture, so
the plugin can be implemented from the fixture alone.

IR-005: tests/fixtures/audio/ carries a deterministic 120 s tone (FLAC —
lossless, so identical PCM to the WAV make_fixture.py emits, and 3.5x
smaller in git) plus the signature it must produce, the decoded-PCM
checksum and the full parameter contract. That directory is the artefact
shared with the plugin repo; the PCM checksum is separate from the
signature so a codec-level difference is distinguishable from a DSP one.

IR-007: media under 120 s emits no signature. Same for a file with no
audio stream or one that will not open — UR-9 is an enhancement and must
never be able to break a fetch.

Verified against an independent Python reference implementation: same
bytes. All 32 bands and all 4 energy classes appear in the golden vector,
and the window-centring test wraps the fixture in 90 s of silence either
side and requires the golden value back.

Not wired into the truth-file output yet — that is the schema_version
bump under IR-002/IR-003 and is deliberately out of scope here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 18:25:37 +02:00
dtourolleandClaude Opus 5 43d2c976c3 docs: replace phased plan with a per-requirement one 
The phase structure encoded ordering assumptions that stopped being true as the
design changed, and its Phase 2 still described retuning constants that are now
withdrawn. Ordering is now derived from per-requirement dependencies instead:
anything with no unmet dependency is startable.

Carries over the TrackRegistry design (now keyed to AR-012/AR-013) and records
what was withdrawn from the old plan, including the --presence-mode flag —
comparison against old behaviour uses recorded reference output rather than a
second live code path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 17:36:09 +02:00
dtourolleandClaude Opus 5 a5299daf6e docs: software spec, requirements register, and implementation plan 
Adds the requirements baseline for the pipeline redesign:

- SPEC.md — software requirements with Current/Gap deltas per item, so the
  document doubles as a work list.
- requirements.md — stable flat IDs (AR/DP/IR/GR/VR) with parent traces,
  priorities, statuses, and a per-requirement verification plan. Replaces the
  thematic A1..E8 scheme, which had already produced an A1a and an out-of-order
  E6; IDs are now permanent and never reused.
- IMPLEMENTATION-PLAN.md — phased work.

The central change is AR-012: presence follows track extent rather than
per-frame recognition, so a window starts when an actor appears rather than
when the recogniser first succeeded. anneal_sec and extinction_sec are
withdrawn rather than retuned — a track that survives its own gaps leaves them
nothing to do.

Verification is shaped by CI running on an N100 with no dGPU: the existing
HDF5 dump makes everything downstream of embedding replayable on CPU, which
covers the bulk of the redesign.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 17:32:44 +02:00
dtourolleandClaude Opus 5 458116f118 fix(trt): drop explicit shapes for static TransNetV2; gallery over-fetch + dedup 
trtexec rejects --minShapes/--optShapes/--maxShapes for a fully static model
("Static model does not take explicit shapes"). TransNetV2's input is fixed at
1x100x27x48x3, so the shape comes from the model itself.

Gallery build now over-fetches TMDB/Wikidata candidates by a configurable
factor: near-duplicate stills (the same photo at different crops or
resolutions) are discarded after embedding, so downloading exactly
images_per_actor left actors short of that many *distinct* embeddings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 17:32:08 +02:00
dtourolle 2ea5737bbd fix(trt): read ONNX input tensor names instead of hardcoding input.1 
build_trt_engines.sh hardcoded 'input.1' for the ArcFace and SCRFD shape
profiles, which only matches arcface_w600k_{r50,mbf}. Building engines for
any other embedder failed with:

    Cannot find input tensor with name "input.1" in the network inputs!

Input names differ per model: LVFace-B_Glint360K uses 'data', arcface_r18
uses 'input', arcface_w600k_{r50,mbf} use 'input.1'. This matters now that
LVFace-B is the default embedder (src/config.hpp), so ARCFACE_MODEL=<LVFace>
is the expected path.

Read the name from each model via onnxruntime at build time.
 2026-07-30 13:32:56 +02:00
dtourolle 5d2f673a81 build: support OpenCV 5 and TensorRT 10 
OpenCV: distros (Arch/CachyOS) now ship OpenCV 5 as default. The config
package rejects a 5.x install when find_package requests 4, so probe for 5
first and fall back to 4. All components used here (core, imgproc, imgcodecs,
videoio, dnn, objdetect, highgui) exist in both.

TensorRT: nvinfer1::Dims5 was removed in TRT 10 (Dims2..Dims4 remain in
NvInferLegacyDims.h). Build the TransNetV2 rank-5 input shape via the generic
nvinfer1::Dims, which is valid on both 8.x and 10.x.
 2026-07-30 13:07:42 +02:00
102 changed files with 14897 additions and 431 deletions
Expand all files Collapse all files
.gitea/workflows/traceability-check.yml
+145
View File
@@ -0,0 +1,145 @@
name: Traceability Validation
# Mirrors JellyTau's .gitea/workflows/traceability-check.yml. The extractor is
# stdlib Python, so there is no toolchain install step and no jq.
#
# This workflow is component-agnostic: every repo-specific setting - which ID
# prefixes count, which file suffixes are source, which directories to scan,
# the threshold - lives in traceability.toml at the repo root, and the same
# extractor is shared by all three JRay components. Copying this file into
# another component needs no edits.
#
# NOTE: the runner here is an Intel N100 with no discrete GPU. This job is only
# ever static analysis of source comments plus markdown parsing, so it is cheap;
# the requirements it reports as "tagged but unexecuted" are the ones that need
# a GPU host, and they are deliberately never counted as covered.
on:
push:
branches:
- main
- master
- develop
pull_request:
branches:
- main
- master
- develop
jobs:
validate-traces:
runs-on: linux/amd64
name: Check requirement traces
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive
- name: Check Python is available
run: |
set -e
command -v python3 >/dev/null 2>&1 || {
echo "python3 is missing from the runner image."
echo "The traceability tooling is stdlib-only Python;"
echo "3.9+ with CLI flags, 3.11+ to read traceability.toml."
exit 1
}
python3 --version
# The gate's own arithmetic is the thing being trusted, so its tests run
# before it does. JellyTau's gate was believed for months while it was
# dividing by frozen literals; untested gate logic is how that happens.
- name: Test the extractor
run: python3 scripts/vendor/jray-project/scripts/traceability/test_extract_traces.py
# Threshold policy and every other repo-specific setting live in
# traceability.toml, not here, so local runs and CI runs cannot disagree
# about what "passing" means. Denominators come from docs/requirements.md
# at run time and are never hardcoded -- in this file or anywhere else.
#
# A misconfigured run (zero requirements parsed, zero files scanned) is a
# hard failure rather than a plausible-looking 0%.
- name: Traceability gate
run: sh scripts/vendor/jray-project/scripts/traceability/traceability-gate.sh
- name: Check modified files for traces
if: github.event_name == 'pull_request'
run: |
set -e
echo "Checking modified sources for TRACES tags..."
# The extensions come from the report the gate just wrote, which got
# them from traceability.toml. Restating them here would be a second
# place for the source-file definition to live, and the two would
# drift the first time a language is added.
PATTERN=$(python3 -c "
import json, re, sys
suffixes = json.load(open('traces-report.json'))['config']['sourceSuffixes']
print('(' + '|'.join(re.escape(s) + '\$' for s in suffixes) + ')')
")
echo "Source suffixes from traceability.toml: $PATTERN"
CHANGED=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD" \
| grep -E "$PATTERN" || true)
if [ -z "$CHANGED" ]; then
echo "No source files changed."
exit 0
fi
echo "Changed files:"
echo "$CHANGED" | sed 's/^/ /'
echo ""
# Advisory by design: not every file implements a requirement, and a
# tag on every function is noise that rots faster than it helps
# (CLAUDE.md: tag the unit that decides). This step exists to prompt,
# not to block. The blocking checks are in the gate step above.
#
# Piped into the loop rather than a here-string, and `case` rather
# than `[[ == ]]`, so this works under dash as well as bash. The loop
# body runs in a subshell, so misses are recorded in a file.
MISSING=$(mktemp)
echo "$CHANGED" | while IFS= read -r file; do
case "$file" in
*/test_*.py|*_test.py|*Tests.cs|tests/*|*/tests/*) continue ;;
esac
[ -f "$file" ] || continue
if ! grep -q 'TRACES:' "$file"; then
echo " no TRACES tag: $file"
echo "$file" >> "$MISSING"
fi
done
COUNT=$(wc -l < "$MISSING" | tr -d ' ')
rm -f "$MISSING"
if [ "$COUNT" -gt 0 ]; then
echo ""
echo "$COUNT changed file(s) carry no requirement tag."
echo "Format: // TRACES: AR-012, AR-013 | SR-002"
echo " (pipe separates requirement types, comma separates IDs)"
echo "A deliberate invariant exception is tagged separately:"
echo " // EXCEPTION: AR-024 <reason>"
echo "See CLAUDE.md and SPEC.md section 6."
fi
- name: Report summary
if: always()
run: |
echo "Traceability matrix: docs/traceability.md"
echo ""
head -40 docs/traceability.md || true
- name: Save reports
if: always()
uses: actions/upload-artifact@v3
with:
name: traceability-reports
path: |
traces-report.json
docs/traceability.md
retention-days: 30
.gitignore
+4
View File
@@ -19,6 +19,10 @@ compile_commands.json
# coverage). Regenerate with scripts/docs/run_holdout_all_models.py and
# scripts/docs/gallery_coverage_per_film.py.
!docs_data/*.json
# Exception: test fixtures are inputs, not build output. The audio golden
# vector (IR-005) is shared verbatim with the jRay plugin repo, so it has to be
# tracked. Regenerate the media with tests/fixtures/audio/make_fixture.py.
!tests/fixtures/**
# Video files
*.mp4
*.mkv
.gitmodules
+3
View File
@@ -2,3 +2,6 @@
path = external/KPN
url = https://gitea.tourolle.paris/dtourolle/KPN.git
branch = master
[submodule "jray-project"]
path = scripts/vendor/jray-project
url = git@gitea.tourolle.paris:dtourolle/jray-project.git
CMakeLists.txt
+56 -9
View File
@@ -18,8 +18,17 @@ endif()
add_subdirectory(external/KPN)
# OpenCV (video decode, image ops, DNN inference, face detection)
find_package(OpenCV 4 REQUIRED COMPONENTS
# Accept 4 or 5: the APIs used here are stable across both, and distros have
# begun shipping 5.x as the default (Arch/CachyOS). find_package's version
# argument is a minimum, but OpenCV's config rejects a 5.x install when 4 is
# requested, so probe for 5 first and fall back to 4.
find_package(OpenCV 5 QUIET COMPONENTS
core imgproc imgcodecs videoio dnn objdetect highgui)
if(NOT OpenCV_FOUND)
find_package(OpenCV 4 REQUIRED COMPONENTS
core imgproc imgcodecs videoio dnn objdetect highgui)
endif()
message(STATUS "OpenCV: ${OpenCV_VERSION}")
# ── Model paths ───────────────────────────────────────────────────────────────
# Defined early so the backend object libraries below can embed it.
@@ -143,6 +152,24 @@ if(SAE_GEMM_BACKEND STREQUAL "CPU")
set_target_properties(gemm_backend PROPERTIES POSITION_INDEPENDENT_CODE ON)
target_include_directories(gemm_backend PRIVATE src)
target_compile_definitions(gemm_backend PRIVATE SAE_GEMM_CPU)
# AR-026/AR-027: back the CPU path with OpenBLAS when present. Optional, so
# the build gains no hard dependency — but without it the fallback is a
# scalar loop, which does not hold up against a library-scale gallery, and
# the CPU path is exactly what CI (no GPU) and the cpu builder image use.
find_package(PkgConfig QUIET)
if(PkgConfig_FOUND)
pkg_check_modules(OPENBLAS QUIET openblas)
endif()
if(OPENBLAS_FOUND)
message(STATUS "GEMM backend: CPU + OpenBLAS ${OPENBLAS_VERSION}")
target_compile_definitions(gemm_backend PRIVATE SAE_GEMM_CBLAS)
target_include_directories(gemm_backend PRIVATE ${OPENBLAS_INCLUDE_DIRS})
target_link_libraries(gemm_backend PRIVATE ${OPENBLAS_LINK_LIBRARIES})
else()
message(WARNING "GEMM backend: CPU scalar fallback — OpenBLAS not found. "
"Correct, but slow on a large gallery (AR-027).")
endif()
elseif(SAE_GEMM_BACKEND STREQUAL "CUDA")
find_library(CUBLAS_LIB cublas
HINTS /opt/cuda/targets/x86_64-linux/lib /opt/cuda/lib64
@@ -187,23 +214,31 @@ endif()
# FFmpeg (hwaccel video decode: CUDA/VAAPI, runtime-detected + swscale colour
# conversion). Hwaccel support is built into libavcodec/libavutil; no extra
# libraries are needed here.
# libswresample is the audio side of the same dependency — downmix + resample
# for the audio signature (IR-004, src/audio_signature.cpp). Not a new project
# dependency: it ships with the libav* set already required above.
find_package(PkgConfig REQUIRED)
pkg_check_modules(AVFORMAT REQUIRED libavformat)
pkg_check_modules(AVCODEC REQUIRED libavcodec)
pkg_check_modules(AVUTIL REQUIRED libavutil)
pkg_check_modules(SWSCALE REQUIRED libswscale)
pkg_check_modules(AVFORMAT REQUIRED libavformat)
pkg_check_modules(AVCODEC REQUIRED libavcodec)
pkg_check_modules(AVUTIL REQUIRED libavutil)
pkg_check_modules(SWSCALE REQUIRED libswscale)
pkg_check_modules(SWRESAMPLE REQUIRED libswresample)
add_library(ffmpeg_libs INTERFACE)
target_compile_options(ffmpeg_libs INTERFACE
${AVFORMAT_CFLAGS_OTHER} ${AVCODEC_CFLAGS_OTHER}
${AVUTIL_CFLAGS_OTHER} ${SWSCALE_CFLAGS_OTHER})
${AVUTIL_CFLAGS_OTHER} ${SWSCALE_CFLAGS_OTHER}
${SWRESAMPLE_CFLAGS_OTHER})
target_include_directories(ffmpeg_libs INTERFACE
${AVFORMAT_INCLUDE_DIRS} ${AVCODEC_INCLUDE_DIRS}
${AVUTIL_INCLUDE_DIRS} ${SWSCALE_INCLUDE_DIRS})
${AVUTIL_INCLUDE_DIRS} ${SWSCALE_INCLUDE_DIRS}
${SWRESAMPLE_INCLUDE_DIRS})
target_link_libraries(ffmpeg_libs INTERFACE
${AVFORMAT_LIBRARIES} ${AVCODEC_LIBRARIES}
${AVUTIL_LIBRARIES} ${SWSCALE_LIBRARIES})
message(STATUS "FFmpeg: avformat=${AVFORMAT_VERSION} avcodec=${AVCODEC_VERSION}")
${AVUTIL_LIBRARIES} ${SWSCALE_LIBRARIES}
${SWRESAMPLE_LIBRARIES})
message(STATUS "FFmpeg: avformat=${AVFORMAT_VERSION} avcodec=${AVCODEC_VERSION} "
"swresample=${SWRESAMPLE_VERSION}")
# nlohmann/json (gallery + output serialisation)
include(FetchContent)
@@ -240,6 +275,8 @@ find_package(HDF5 REQUIRED COMPONENTS CXX)
add_library(sae_gallery STATIC
src/gallery/gallery_store.cpp
src/gallery/gallery_builder.cpp
src/audio_signature.cpp # IR-004 — content-derived audio signature
src/gallery/embedder_stamp.cpp # GR-004 — gallery/embedder binding
)
set_target_properties(sae_gallery PROPERTIES POSITION_INDEPENDENT_CODE ON)
target_include_directories(sae_gallery PUBLIC src ${HDF5_INCLUDE_DIRS})
@@ -271,6 +308,16 @@ target_link_libraries(sae_embed PRIVATE sae_gallery)
nanobind_add_module(sae_kpn src/kpn_bindings.cpp)
target_link_libraries(sae_kpn PRIVATE sae_gallery)
# ── sae_audio — Python module: the v1 audio signature (IR-004) ────────────────
# Compiles audio_signature.cpp directly and links only FFmpeg, rather than
# linking sae_gallery: the signature needs no model, no OpenCV and no HDF5, and
# a module that dragged all three in would make `import sae_audio` depend on a
# GPU-capable build of a repo whose audio path is pure CPU DSP. tests/ compiles
# the same source the same way, for the same reason.
nanobind_add_module(sae_audio src/audio_bindings.cpp src/audio_signature.cpp)
target_include_directories(sae_audio PRIVATE src)
target_link_libraries(sae_audio PRIVATE ffmpeg_libs)
# HDF5 already found above (before sae_gallery); vars HDF5_CXX_LIBRARIES / _INCLUDE_DIRS
# are reused by scene_analyze / dump_embeddings below.
docs/SPEC.md
+1785
View File
File diff suppressed because it is too large Load Diff
docs/index.md
+8
View File
@@ -53,6 +53,14 @@ every finding below.
A training-set effect that did not reproduce on 5 held-out films once
two methodology bugs in the comparison harness were found and fixed.
- :material-blur:{ .lg .middle } **[What does blur cost?](quality-knee.md)**
---
Sharpness is not a sufficient statistic for identity loss, blur breaks
confidence rather than ranking, and variance-of-Laplacian is
anti-predictive at fixed resolution.
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
---
docs/plan.md
+410
View File
@@ -0,0 +1,410 @@
# Implementation plan — per requirement
One entry per requirement that needs work. Requirements marked `Done` in
[`requirements.md`](requirements.md) are omitted.
**Ordering is derived from dependencies, not assigned to phases.** Each entry
lists what it depends on; anything with no unmet dependency is startable. This
replaces the earlier phase-based plan, which encoded ordering assumptions that
stopped being true as the design changed.
Verification for each requirement is specified in
[`requirements.md`](requirements.md) — this document covers *how to build it*,
not how to prove it.
---
## Startable now (no unmet dependencies)
`GR-004` · `IR-004` · `IR-005` · `IR-007` · `IR-008` · `VR-005` · `AR-011` ·
`AR-023` extension · tooling port
These touch disjoint files and can proceed concurrently.
## Blocked on the registry
Everything in `AR-007` … `AR-022` depends on `AR-012`/`AR-013` landing first,
because they all read or write track state. **This group is one coherent
refactor, not parallel work** — splitting it across concurrent efforts produces
incompatible designs in the same files.
---
# Algorithm
## AR-012, AR-013 — TrackRegistry (the spine)
**Depends on:** nothing. **Blocks:** AR-007, AR-008, AR-014 … AR-022.
Everything else in Part A waits on this, so it goes first.
### Ownership: a shared resource, not a node
The registry is **external to the dataflow network**, created in `main` and
handed to each node that needs it as `std::shared_ptr<TrackRegistry>`. Lifetime
is guaranteed by refcount rather than by the "object must outlive the node"
convention, so no ordering assumption exists between network teardown and
registry destruction.
This is idiomatic here: node functors are already constructed outside the network
and passed by reference (`main.cpp:186-207`), and KPN provides `SharedResource<T>`
for state shared across nodes (KPN SPEC §163, §445).
Not a node, because ownership is not a stage in the stream — it is state several
stages read and write, whose final answer is only known when a track dies.
Not inside `TrackGallery`, because that would couple presence to `expand_gallery`,
a switchable feature.
**The registry *is* the tracker's state.** `FaceTrackerFunc` does not keep its own
`tracks_`/`inactive_` maps and mirror them in — it operates on the registry
directly. Two parallel copies could disagree, and every divergence would surface
as wrong presence windows, silently.
### Per-track state
```
Track
first_seen : double set once, at creation
last_seen : optional<double> UNSET while on screen; set to the last
on-screen timestamp when the face is lost
actor : optional<int> set when a posterior crosses the threshold
belief : {actor_idx -> accumulated_logodds} Bayesian, not a tally
embedding : Embedding running directional mean, for association
```
`last_seen` carries the entire liveness state. Unset = on screen; set = went off
at T. No separate missing-frames counter, no expired flag — the optional *is* the
state machine, and it subsumes the current two-pool split (`tracks_` = unset,
`inactive_` = set).
### Lifecycle
```
face detected, no match → new track, first_seen = t, last_seen = unset
actor identified → update belief; set actor when threshold crossed
face lost → last_seen = t_last_on_screen (stays revivable)
face seen again, embedding match → last_seen = unset (same track continues)
tick(t), t - last_seen > timeout → emit to aggregator, DELETE the entry
```
A presence window is `[first_seen, last_seen]`. Nothing else.
**Interior gaps are claimed; the trailing cool-down is not.** A face lost at t₁
and re-acquired at t₂ within the timeout never closed its track, so the actor is
present across `[t₁, t₂]` — correct, since someone briefly occluded or off-camera
has not left the scene. But a track that dies ends at `last_seen`, not at the
moment of death. That asymmetry is what removes the old `extinction_sec`
over-claim.
**Reaping is a handoff, not a deletion into a holding pen.** The dead track goes
to the result aggregator immediately and the registry drops it, so the registry
holds only live tracks and its size is bounded by concurrent on-screen faces.
### Interface
```
TrackRegistry
tick(timestamp) ← FaceTrackerFunc, every frame
candidates() -> span<Track&> → all live tracks
create(timestamp, embedding) -> track_id
mark_seen(track_id, timestamp, embedding) → updates mean, clears last_seen
mark_lost(track_id, last_on_screen_timestamp)
on_vote(track_id, actor_idx, posterior) ← IdentityMatcherFunc
owner(track_id) -> optional<actor_idx> → TrackGallery
on_track_dead : callback(DeadTrack) → ResultSinkFunc
flush() ← at EOF
```
`candidates()` returns **one pool**; `last_seen` tells the caller whether IoU
applies. There is no separate revival path — matching a dormant track is ordinary
inter-frame association.
`tick()` advances the clock so dead tracks are reaped independently of detection
activity; without it a track only dies when some *other* face happens to appear.
### Locking
The tracker mutates registry state across a frame's association pass, so that
pass holds the lock for its duration (a `frame_scope()` handle). Every other
caller's operations must be individually atomic. A single `std::mutex` over the
whole registry is the right start — contention is a few small updates per frame
against per-frame work measured in GPU milliseconds.
Two cases constrain the API:
- `owner()` is a **read-modify-read** in disguise: `TrackGallery` calls it while
`IdentityMatcher` may be voting on the same track. Tally and verdict must be
read under one lock as a snapshot, or a track can be both unowned and owned
within a single promotion decision.
- `on_vote()` arrives downstream of the tracker's `tick()` for the same frame, so
a vote may land after the clock moved on. **Rule: a vote for a known track
always lands on its tally, regardless of clock.** Only reaping is clock-driven.
A vote for an already-reaped track is dropped and **counted** — a nonzero count
means the timeout is shorter than the matcher's lag.
`on_track_dead` fires from inside `tick()` while the frame lock is held, so the
callback must not re-enter the registry. Keep it to a push onto the aggregator's
storage.
## AR-016 — EOF flush
**Depends on:** AR-012.
`flush()` emits every still-live track through the same callback, closing at
`last_seen` if set and the final tick timestamp otherwise. Idempotent, leaving the
registry empty; the sink's `written_.exchange(true)` guard
(`result_sink_node.hpp:66`) shows the shape.
Must run on **every** termination path that produces output. Not SIGTERM during
opportunistic runs (DP-004) — those push no partial result, so there is nothing
to flush.
Without it a film ending mid-shot silently drops its closing cast, which looks
like a recognition miss rather than a bookkeeping bug.
## AR-014, AR-015 — Contradiction rules
**Depends on:** AR-012, AR-025.
| Condition | Meaning | Action |
|---|---|---|
| Belief on one track swaps A → B | `track_id` carried across a viewpoint change onto a different person | Close at `last_seen`, open a new track for B at the swap frame |
| Two **live** tracks owned by one actor | One person split in two, or an identity attached to the wrong track | Treat as a detected cut: reset affected state, re-associate on embedding |
The second makes identity a **third cut detector**, independent of histogram and
TransNetV2, firing where those failed. Detect it via a reverse index
`actor_idx → live track_ids`, so the condition is caught on the update that
causes it rather than by scanning.
Both counted and reported — the rates measure how often tracking is silently
wrong, which nothing currently reveals.
## AR-007, AR-008 — Tracker on one pool
**Depends on:** AR-012, AR-024.
`FaceTrackerFunc` is constructed with the registry and uses it as state; its
`tracks_`/`inactive_` maps and the cross-cut revival branch collapse into one
pool keyed on `last_seen`. Per frame: `tick()`, association over `candidates()`,
then `create`/`mark_seen`/`mark_lost`.
`track_alpha` becomes **frame-dependent** — normal frames use the tuned blend,
frames flagged `is_cut`/`is_scene_boundary` drop toward embedding-only.
## AR-024 — Probability space everywhere
**Depends on:** AR-023. **Blocks:** AR-007, AR-018, AR-021, AR-025.
Cuts across tracker, matcher and expansion, so it lands with the registry work
rather than after it. Retires `track_max_embed_dist`, `cut_revive_sim`,
`expand_novelty_sim`, `expand_track_spread_max`.
Enforcement is a **static grep check** for bare cosine outside a tagged
`EXCEPTION` — a unit test cannot prove absence across a codebase.
## AR-025 — Bayesian accumulation
**Depends on:** AR-023, AR-024.
Log-odds per candidate actor, added per frame. `on_vote()` is an *update*, not an
increment.
**The independence problem must be handled explicitly.** Consecutive frames are
highly correlated; naive accumulation drives the posterior to certainty on what is
effectively one observation. Preferred mitigation: update only on sufficiently
novel observations, reusing the diversity buffer's existing judgement rather than
inventing a second one. The registry should receive already-discounted evidence.
## AR-017 — Claims carry belief and route
**Depends on:** AR-012, AR-025. `DeadTrack` carries posterior plus how it was
identified (live / deferred / pooled).
## AR-018 … AR-021 — Expansion, deferred pass, clustering
**Depends on:** AR-012, AR-024, AR-026.
Ordering within the group: AR-018 (banded store) → AR-019 (annex) → AR-020 (TBI
queue + deferred pass) → AR-021 (clustering).
AR-021 needs the temporal cannot-link constraint from track extents, so it cannot
start before AR-012. The annex must be a **contiguous matrix** with promotions
appended (AR-026), not a list.
**Output timing changes:** the sink can no longer finalise at EOF — the deferred
pass runs after and may add windows (IR-003).
## AR-022 — Unidentified capture
**Depends on:** AR-020. Unidentified = TBI entries surviving the deferred pass.
Context crops opt-in behind `--dump-unidentified-crops`.
## AR-001 … AR-004 — Detection and backpressure
**Depends on:** nothing (AR-002, AR-011); AR-004 blocks AR-003.
- **AR-002** — `min_face_px` stays **40** (VR-013 measured it end to end) but must
be expressed in original resolution rather than decoded-frame space. The value
is already right in `config.hpp`; the change is the coordinate space.
- **AR-011** — feed TransNetV2 at native rate; derive the dedup window from
source fps rather than the hardcoded `0.04 s`.
- **AR-004** — backpressure. `kMaxFaces` (`identity_matcher_node.hpp:133`)
currently **throws**; channel capacities of 16 (`main.cpp:204-207`) were sized
against ≤10 faces/frame. Must block on bytes in flight, not item counts.
- **AR-003** — remove `max_faces`. **Gated on AR-004**, not a follow-up to it.
## AR-026, AR-027 — GEMM and scale
**Depends on:** nothing to start. The annex CPU loop
(`identity_matcher_node.hpp:159-162`) moves into the GEMM path.
---
# Gallery
## GR-004 — Model binding — **DONE**
**Depended on:** nothing. Landed before any measurement work, as intended.
Stamp = model basename + SHA-256 of the ONNX, written as the `/embedder` group at
build time (`gallery_builder.cpp`, `sae_gallery.save_gallery_hdf5`) and verified
at load in `scene_analyze`, `scene_preview`, the `sae_kpn` matcher binding,
`replay.py`, `optimize.py` and `movienet_eval.py`. Mismatch is a hard error naming
both sides, with no bypass. Embedding dumps carry the same stamp, since a replay
has no live embedder to check against.
Unstamped legacy galleries **warn loudly and proceed** rather than failing:
unknown is not known-bad, and hard-failing every pre-existing gallery would turn
the check into something people disable. `--require-gallery-stamp` /
`SAE_REQUIRE_GALLERY_STAMP=1` promotes that to a hard error — measurement runs
should set it. `scripts/stamp_gallery.py` re-binds an existing gallery without
re-embedding, so the warning state is cheap to leave.
Cross-model similarities are meaningless but *look* plausible — this fails
silently and expensively, and it would corrupt every measurement taken during the
rest of this work.
## GR-003 — Coverage reporting
**Depends on:** nothing. Surface what calibration already computes and discards
(`kHistBins = 200`): zero-image actors, under-referenced actors, dedup counts,
and the intra/inter PDFs.
## GR-006 … GR-008 — Provenance tiers
**Depends on:** AR-019. Tier per embedding (baked / harvested / confirmed);
harvested persisted but flagged; bell-curve outlier check
(`EXCEPTION: AR-024`).
---
# Integration
## IR-004, IR-005, IR-007, IR-008 — Audio signature
**Depends on:** nothing. **Fully independent — no existing pipeline file is
touched.** Best candidate for concurrent work.
Implement server spec §3 exactly. Audio decode is a second stream from the
already-linked FFmpeg. Media < 120 s: no signature, no offset. Emit and honour
the `v1:` prefix.
The golden-vector fixture is shared with the plugin repo and runs on CPU, so the
one place two implementations must agree bit-for-bit is verifiable in CI.
## IR-001 … IR-003 — Truth file
**Depends on:** AR-017 (belief), AR-020 (output timing).
Windows carry belief and route; `extraction.*` gains `extinction_sec` and
`gallery_scope`; `anneal_sec` removed. All breaking → **one** coordinated
`schema_version` bump with IR-004 (SR-003).
---
# Validation
## VR-005 — Minimum face size study
**Depends on:** nothing. Standalone Python, no C++ contact. **Done** — knee at
24–32 px. It measures the embedder with alignment held perfect, so it bounds the
answer from below rather than setting it; AR-002's floor comes from **VR-013**,
which sweeps input resolution end to end and lands at 40 px.
## VR-013 — Cross-source identification probe
**Depends on:** `sae_embed` exposing `detect()`, `align_face()`, `embed_crop()`
and the gallery calibration — it drives the shipped C++ rather than reimplementing
it, which is what VR-005 could not do.
Gallery from one recording, probes from another, sweeping the probe's **input
resolution before the detector**, so detection and landmark regression degrade
with the frame. `experiments/xsource/`.
**Findings.** Holding 90% of the plateau needs ~50 px end to end against VR-005's
~22 px; `min_face_px` 40 is right and 32 would admit faces in the falling region.
FPI is 0.0% at every scale — resolution loss goes entirely to TBI, never to a
wrong name. The ceiling is **cross-view, not resolution**: everyone matches
themselves within a recording (0.55–0.85) and collapses across two (0.14–0.45),
and only the subject with frontal *gallery* references identified reliably — so
the lever is gallery pose coverage (`docs/pose-expansion.md`), not a better
landmark model. Averaging SCRFD's NMS-discarded landmark estimates lifts
cross-clip TPI 41% → 49% for one forward pass.
**Open.** Four identities and one shoot, so the shape is the result and the
absolute rates are not. Both clips hold all four people, so there is no
out-of-gallery class and the 10×-weighted out-of-cast misID is untested — holding
one identity out of the gallery would fix that.
## VR-014 — Audio-signature offset recovery
**Depends on:** `sae_audio` exposing `compute_signature()` and
`signature_from_mono()` — it drives the shipped C++, as VR-013 does, so the
thing measured is the thing that ships.
`scripts/validation/test_audio_offset.py` over
`tests/fixtures/audio/bali_offset_200s.flac`: 200 s of public-domain film audio
(the same Road to Bali clips the replay fixtures use), long enough for a 120 s
window to slide past the ±600-frame search cap. The slide itself is numpy here
on purpose — matching belongs to the consumer, so writing it out keeps this a
test of the signature rather than of somebody's matcher.
**Findings.** Alignment is a solved problem here: the offset is the nearest frame
in every in-cap trial, worst error **46 ms against a 500 ms budget**, and 46 ms is
the quantisation floor — offsets are whole 92.88 ms frames, so no correct answer
can be worse. The `runtime/2` anchor's factor of two holds through real trimmed
files, and out-of-cap offsets and unrelated content are both declined.
**The score is where the slack is, and it costs a tier rather than accuracy.** It
tracks sub-frame misalignment — 0.94–0.99 near a frame boundary, 0.69–0.73 at
half a frame — so two thirds of correct alignments miss the server's 0.85 `audio`
threshold and land in `loose`. UT-108 measures the fix rather than proposing one:
±1 frame of slack in the score returns all 40 to `audio` (min 0.906) with false
matches unmoved at 0.12–0.16, costing 81 ms of the budget. See
[`SPEC.md`](SPEC.md) IR-004 — the score is normative in the server spec, so the
change is theirs to make.
**Open.** One source, one language, one era of recording. The shape (offset exact,
score set by sub-frame phase) should hold generally, but the absolute scores are
this fixture's.
## VR-001 — Dump audit
**Depends on:** nothing. Read-only investigation: confirm the HDF5 dump preserves
everything needed to reconstruct tracks deterministically, including the
park/revive path. **Prerequisite for the CI strategy**, since T2 replay is how
most of AR-007 … AR-022 is verified.
## VR-006 … VR-009
**Depends on:** their subjects landing. VR-009 (posterior calibration holds)
depends on AR-025 and is what stops the Bayesian accumulation being decoration.
---
# Withdrawn from the old plan
The phase structure, the `--presence-mode {frame,track}` flag, and "Phase 2 —
retune `anneal_sec`/`extinction_sec`". Those constants are withdrawn rather than
retuned; comparison against old behaviour uses recorded reference output instead
of a second live code path.
docs/quality-knee.md
+308
View File
@@ -0,0 +1,308 @@
# Quality knee: what does a blurred or small face cost, and can a measure predict it?
VR-012. Companion to the minimum-face-size studies VR-005 and VR-013 (see the
[requirement register](requirements.md)), which located the size floor at 40 px;
this asks the same question for **sharpness**, and asks whether any cheap
measure taken on the aligned crop can be acted on at inference.
Run by
[`scripts/validation/quality_knee.py`](https://REPOLINK/scripts/validation/quality_knee.py)
through the `sae_embed` bindings — detection, the ArcFace warp, the embedder,
the five candidate measures and the Platt calibration are all the shipped C++.
## Protocol
1670 gallery actors with 3 or more mugshots (of 2456 total), one image held out
per actor as a probe, the remaining 10326 embeddings staying in the gallery at
native resolution. Only the probe degrades — reference mugshots are clean and
the face coming out of the video is not.
Each probe passes through a **joint grid**: downscale to *S*×*S* and back to
112 (the sampling loss), then blur at level *L* in canonical pixels. Three blur
families, 36 cells each, 60120 probe-cell records per family:
| family | models | parameter |
|---|---|---|
| Gaussian | soft focus, a generic stand-in | sigma 0 … 3 |
| **Disc** | **real optical defocus** — the circle of confusion | radius 0 … 6 |
| Motion | camera pan or moving subject | length 0 … 21 px |
The three are not interchangeable, and sweeping only the first was the original
design error — one that would have produced a wrong answer, not merely an
incomplete one (Result 3). A defocused lens spreads a point into a **uniform
disc**, whose transfer function is a jinc — `2·J1(x)/x` — that crosses zero and
goes negative, annihilating whole frequency bands and returning the ones beyond
each zero phase-reversed. A Gaussian MTF is strictly positive and monotone and
does neither. More practically: defocus and motion are how a face ends up
**large and useless**, while Gaussian blur as swept here mostly co-occurs with
small faces. That difference decides whether sharpness carries anything the size
filter does not.
Families are compared at matched **per-axis PSF standard deviation** (σ for a
Gaussian, R/2 for a disc, L/√12 for a linear smear), never at equal raw
parameter, which would compare different amounts of damage.
Identification is the pipeline's own decision: per-actor best-of-N cosine →
Platt sigmoid → accept above `prob_threshold` 0.754. Never a raw cosine
(AR-024).
## Result 1 — sharpness is not a sufficient statistic
Sorting the 36 Gaussian cells by `hf_energy_ratio`, the six sigma-3 cells land
at effectively identical measured sharpness:
| size | sigma | hf_energy_ratio | TPI |
|---|---|---|---|
| 16 | 3 | 0.0003 | **15.3%** |
| 24 | 3 | 0.0003 | 63.2% |
| 32 | 3 | 0.0003 | 79.4% |
| 48 | 3 | 0.0003 | 86.6% |
| 64 | 3 | 0.0004 | 88.4% |
| 112 | 3 | 0.0005 | **91.0%** |
Same measured sharpness, a **76-point spread in identification**. It inverts
too: 16 px unblurred measures 0.0033 and scores 23.5%, while 48 px at sigma 2
measures *lower* at 0.0021 and scores 96.6%.
A canonical-frame sharpness scalar cannot separate *attenuated* high
frequencies from *destroyed* spatial sampling. Blur suppresses the high band
while preserving mid-frequency facial geometry exactly; downsampling to 16 px
destroys that geometry outright. Both look alike to any measure keyed on
high-frequency energy.
This is the measured basis for AR-028's rule that the axes are **kept separate
and not collapsed into one scalar**, and it settles the double-counting
question: size and sharpness are not redundant, and neither substitutes for the
other.
## Result 2 — blur is a cliff, and it breaks confidence, not identity
TPI % by size (rows) against Gaussian sigma (columns):
| size | 0 | 0.5 | 1 | 1.5 | 2 | 3 |
|---|---|---|---|---|---|---|
| 16 | 23.5 | 24.0 | 25.0 | 24.6 | 23.9 | 15.3 |
| 24 | 85.7 | 85.1 | 85.6 | 85.9 | 82.6 | 63.2 |
| 32 | 95.9 | 95.9 | 96.0 | 95.5 | 93.7 | 79.4 |
| 48 | 98.7 | 98.7 | 98.4 | 98.1 | 96.6 | 86.6 |
| 64 | 98.6 | 98.6 | 98.8 | 98.4 | 97.5 | 88.4 |
| 112 | 98.9 | 98.9 | 98.8 | 98.6 | 98.0 | 91.0 |
Three regimes: **sigma ≤ 1.5 is free** (every cell moves under 1.5 points, sign
flipping at random — at 16 px it slightly *improves*, smoothing upscale
artifacts); sigma 2 costs 1–3 points; the 2→3 step costs 7–19. A smooth
discount curve is therefore the wrong shape — the response is flat, then falls
off a cliff.
**The cost peaks at the size knee, not at full resolution.** Sigma 3 costs
−22.5 points at 24 px but only −7.9 at 112 px and −8.3 at 16 px. Blur has no
intrinsic cost; it costs in proportion to how close the observation already sits
to the decision boundary. At 112 px there is margin to spare, at 16 px the probe
is already below threshold, and at 24 px it sits exactly on the knee.
**What blur destroys is confidence, not ranking.** Rank-1 barely moves: 99.3% →
99.2% at 112 px across the whole sigma range. The extreme case is 16 px at sigma
3, where rank-1 is **80.2%** while TPI is **15.3%** — 65 points of probes have
the correct actor ranked first and are rejected anyway for falling under the
probability threshold.
That is why **FPI never left 0.1% in any of the 108 cells across all three
families**. Degradation produces TBI, never a wrong name. The calibration
degrades gracefully, which is what SR-002 needs.
## Result 3 — the blur *family* matters more than the blur *amount*
Comparing families by their raw parameter is meaningless — sigma, radius and
length are different units. They are matched here by the **per-axis standard
deviation of the PSF**, which puts them on one scale:
| family | per-axis σ | level giving σ = 3 px |
|---|---|---|
| Gaussian σ | σ | 3 |
| Disc radius R | R/2 | 6 |
| Motion length L | L/√12 | 10.4 |
For reference the ArcFace template places the eyes 35.2 canonical px apart, so
σ = 3 px is 9% of the inter-ocular distance.
TPI at matched severity, interpolated within each family:
| size | σ=3 Gaussian | σ=3 Motion | σ=3 **Defocus** | defocus penalty |
|---|---|---|---|---|
| 16 | 15.3 | 15.1 | 11.0 | +4.3 |
| 24 | 63.2 | 61.1 | 41.4 | +21.9 |
| 32 | 79.4 | 76.1 | 50.4 | +29.0 |
| 48 | 86.6 | 80.9 | 52.6 | +34.0 |
| 64 | 88.4 | 81.7 | 51.0 | +37.4 |
| 112 | 91.0 | 82.1 | **46.9** | **+44.0** |
**Optical defocus is up to 44 points more destructive than a Gaussian of
identical spread**, and the ordering is defocus ≫ motion > Gaussian throughout.
At σ=1 the three families are indistinguishable, and at σ=2 they differ by under
5 points; the divergence appears only when both the blur is severe *and* the face
is large.
That pattern is physically consistent. At 16 px the resampling has already
removed the high frequencies, so the PSF's shape has nothing left to act on and
all three agree. At 112 px the full spectrum is present and shape decides: a
Gaussian MTF rolls off gently and always leaves *some* energy at every
frequency, so the embedder receives a merely attenuated signal, while a disc MTF
is a jinc that **hits exact zeros** — whole frequency bands annihilated rather
than attenuated, with the bands beyond each zero returning phase-reversed.
Motion sits between them because it ruins one axis and leaves the perpendicular
one untouched.
**The methodological consequence is the important one.** This study originally
swept Gaussian blur alone and concluded blur was a minor effect. On the family
that actually occurs in film, the same nominal severity costs **53% error
instead of 9%** at full resolution. A threshold set from the Gaussian arm would
have been wrong by a factor of five in error rate, and the axis would probably
have been dropped as not worth its cost.
**Defocus is also the case a size gate cannot catch.** Every one of those 112 px
faces is large and confidently detected, and sails through AR-002 untouched.
That, not the Gaussian result, is what justifies a sharpness axis existing at
all.
## Result 4 — variance of Laplacian is anti-predictive at fixed degradation
Pooled across all cells, every candidate scores AUC 0.76–0.80 for predicting
correct identification, with textbook `var_laplacian` top. That number is close
to worthless: it rewards a measure for detecting *how degraded the crop is*,
which all five do. The question a per-observation discount needs is whether, at
a **fixed** degradation, the measure predicts which faces fail:
| measure | Gaussian | Defocus | Motion |
|---|---|---|---|
| `hf_energy_ratio` | **0.530** | **0.521** | **0.557** |
| `norm_var_laplacian` | 0.520 | 0.507 | 0.539 |
| `dir_min_tenengrad` | 0.524 | 0.512 | 0.506 |
| `tenengrad` | 0.433 | 0.437 | 0.457 |
| `var_laplacian` | 0.423 | 0.422 | 0.473 |
Best is 0.557 — barely above chance, and `hf_energy_ratio` wins on all three
families. `var_laplacian` is anti-predictive on all three too, so that finding
does not depend on the blur model.
**The two metrics measure different jobs, and the candidates split along that
line.** On the motion arm `dir_min_tenengrad` has the best *pooled* AUC by a
wide margin — **0.854** against 0.792 for the next — exactly as its synthetic
directional-blur ladder predicted, yet its within-cell AUC there is 0.506. It is
an excellent detector of *how badly smeared a crop is* and no guide at all to
*which face will be recognised*. Pooled AUC is the right metric for a
gross-degradation flag; within-cell AUC is the right one for a per-observation
discount; a measure can be strong at one and useless at the other.
Deciles within the 16 px Gaussian cell, where 1277 failures give the test real
power:
| `var_laplacian` decile | TPI |
|---|---|
| 0.00071–0.00192 (blurriest) | **37.1%** |
| 0.00242–0.00278 | 22.8% |
| 0.00397–0.00447 | 25.7% |
| 0.00625–0.01445 (sharpest) | **14.4%** |
The faces the measure calls sharpest are **2.6x less identifiable** than those
it calls blurriest, monotone across ten bins of 167. Within a cell every crop
received identical degradation, so the residual variance is *native contrast*,
not native detail — and hard shadows, high-contrast lighting, sharpening halos
and JPEG ringing all raise Laplacian variance while making a face harder to
match. The measure reads photographic style and encoding artifacts and calls
them sharpness.
`hf_energy_ratio` is the only candidate with a correctly-signed within-cell
trend (16.2% → 35.3% across the same deciles), being a pure ratio in which the
contrast factor cancels.
**Consequence:** a per-face quality *discount* keyed on variance of Laplacian —
the most widely used blur metric in production vision pipelines — would
systematically down-weight the *more* identifiable faces. It is worse than no
discount.
## Result 5 — as a compute gate, sharpness loses to the size filter
Skipping the embed for crops below a threshold, measured as compute saved
against true identifications lost:
| gate | skipped | true IDs lost | of skipped, doomed anyway |
|---|---|---|---|
| `hf_energy_ratio` < 0.00023 | 10.0% | 7.6% | 37.9% |
| `hf_energy_ratio` < 0.00051 | 20.0% | 15.1% | 38.7% |
| **source size < 24 px** | **16.7%** | **4.7%** | **77.3%** |
At a comparable skip rate the size filter loses **4.7% against sharpness's
15.1%** — three times less damage — and it is free, being a bbox dimension
available before alignment or embedding, where sharpness needs the warped crop
plus a colour convert, three convolutions and a 64×64 DFT.
Restricting to large faces (≥64 px) on the **defocus** arm, where the size
filter is blind, improves the gate's precision 3.5x (37% of skipped crops doomed
versus 10.7% on the Gaussian arm) but not its trade: skip 10%, lose 7.0%.
A hard ceiling explains why. **At 112 px with defocus radius 6 — visually
destroyed — 46.9% of faces still identify correctly and rank-1 is still 94.8%.**
Blur does not determine the outcome, so any gate keyed on apparent blur is
predicting a coin flip. The size filter wins not because size is better
measured, but because *smallness destroys identity more completely than blur
does*: 16 px faces succeed only 23.5% of the time, so discarding them is cheap.
## What this means for the requirements
**Do not gate on sharpness; discount on it.** Heavily defocused faces remain
~47% identifiable, so a gate destroys recoverable evidence. This is the first
hard evidence that AR-028's "**discounts the observation, never deletes the
detection**" is right on the merits rather than merely cautious. Since ranking
survives where confidence does not, the per-track accumulation (AR-025) should
recover much of what a single-frame threshold rejects — which is also the
argument for the discount living in `EvidenceDiscounter` rather than in a filter.
**`var_laplacian` and `tenengrad` are disqualified as discounts** by Result 4,
on all three blur families. They remain usable as coarse *gross-degradation*
detectors, the role in which their pooled AUC is real — the same role the size
filter plays — but they must never weight a per-observation belief.
**`hf_energy_ratio` is the only surviving discount candidate**, best on all
three families, and its within-cell signal (0.52–0.56) is weak enough that
shipping a discount on it needs justification beyond this study.
**`dir_min_tenengrad` earns a different job.** Its pooled 0.854 on the motion arm
makes it the best available detector of gross directional smear — useful as a
per-frame "this shot is unusable" flag, which is a decision about a *frame*, not
a weighting of an *observation*. If AR-029 ships two measures for two roles, this
is the second one, and it must not be confused with the first.
**Model the blur family, not just its amount.** Result 3 makes the choice of
degradation model a first-order design decision rather than a detail: the same
matched severity costs 9% or 53% error depending on the PSF. Any future study
that sweeps blur must state which family it used and why.
**Any discount curve must be flat then steep**, not linear or sigmoid over the
measure. Blur costs nothing until it costs a great deal.
## Limitations
- **Cooperative population.** Gallery mugshots are frontal and well-lit;
within-cell failures are likely dominated by cross-view mismatch, which no
sharpness measure can predict. Read the ~chance within-cell AUCs as "sharpness
does not predict the dominant failure mode *here*", not as "sharpness is
meaningless".
- **Uniform grid, not a natural distribution.** Sizes and blur levels are
sampled evenly, so "skip 16.7%" is exactly the 16 px row. The gate comparisons
are like-for-like on identical records, but the absolute savings are not what
a film would show.
- **TensorRT fp16.** A different realisation of the embedder from the fp32 ONNX
reference — VR-005 measured ~0.85 cosine agreement with separation intact.
Gallery and probes share one session so the study is internally consistent,
but the absolute knee belongs to the fp16 space.
- **Blur is applied in the canonical frame**, after resampling, so its width is
independent of the cell's size. Real optics blur before sampling.
- **The top motion rung is an anchor, not an operating point.** Length 21 is a
per-axis σ of 6.1 — 17% of the inter-ocular distance, a streak rather than a
face — and it is swept to bound the curve, not because a frame like that is
worth reasoning about. Its 3.4% TPI at 112 px should not be quoted as a
headline. The same caution applies less severely to defocus radius 6 (σ = 3).
- **Per-axis σ equates spread, not perceptual damage.** It is the fairest single
scalar for comparing PSFs, but Result 3 is precisely the finding that equal
spread does *not* mean equal harm, so the matched-severity tables compare
like-for-like inputs, not like-for-like severity as a face would experience it.
docs/requirements.md
+383
View File
@@ -0,0 +1,383 @@
# scene-actor-extraction — requirements register
Stable IDs for every requirement in [`SPEC.md`](SPEC.md), which holds the prose.
This file is the **authoritative list**; the CI gate reads its denominators from
here (see [`../../SPEC.md`](../../SPEC.md) §6).
**IDs are permanent.** A withdrawn requirement is marked `Withdrawn` and its
number is never reused — renumbering is what produces orphan TRACES tags. This
register replaces the earlier thematic `A1…E8` scheme, which had already produced
an `A1a` and an out-of-order `E6`.
Tag code with `// TRACES: AR-012 | SR-002`.
| Type | Scope |
|---|---|
| `AR` | Algorithm — the extraction pipeline itself |
| `DP` | Deployment — how it runs |
| `IR` | Integration — contracts with other components |
| `GR` | Gallery — building and maintaining actor references |
| `VR` | Validation — parameter studies and benchmarks |
| `UT` / `IT` | Unit / integration tests |
Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
---
## Algorithm (AR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
| AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far |
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted |
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Done** — decode butterfly joined via `SceneBoundaries`; the sampled branch waits for the detector's watermark. Frames past its last scored window are counted as unverified, never assumed boundary-free |
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | **Done** — `src/track_registry.hpp`; window is `[first_seen, last_seen]` of an owned track |
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | **Done** — `last_seen` optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed |
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | **Done** — swap closes at `last_seen` and opens a successor at the swap frame; counted |
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | **Done** — reverse index detects it on the causing update; counted |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | **Done** — `flush()`, idempotent, closes at last sighting or final tick |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief and observation count |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space; replaces `expand_novelty_sim`. Rejections counted |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | **Done** — all three discontinuity signals clear the buffers; ownership comes from the registry, not a second local tally |
| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned |
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association and accumulation both in probability space; `track_max_embed_dist`, `cut_revive_sim` retired |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | Planned |
| AR-029 | Sharpness measure on the **aligned crop**, consumed as a discount and **never as a gate** | SR-002 | Medium | **In Progress** — five candidates implemented (`src/quality.hpp`) and ranked by VR-012 over three blur families. `var_laplacian` and `tenengrad` are **disqualified as discounts**: within a fixed degradation they are anti-predictive on *all three* families (AUC 0.42–0.47; the decile the measure calls sharpest is 2.6× *less* identifiable), since their residual variance is native contrast, not detail. `hf_energy_ratio` is the only correctly-signed survivor, best on all three, and weak (0.52–0.56). `dir_min_tenengrad` is the best *gross-smear detector* (pooled AUC 0.854 on motion) but ~chance within-cell, so it serves a per-frame flag, not a per-observation weight. The parenthetical this row used to carry — "scale-normalised, so it cannot re-measure size" — was wrong: every candidate responds to source size, and the axes are separable for a different reason (see AR-028) |
| AR-030 | Visibility measure from the AR-001 5-point landmarks — extreme pose or occlusion **discounts the observation, never deletes the detection** | SR-002 | Medium | **In Progress** — measure is the AR-005 alignment residual (`estimate_alignment()`), carried on `DetectedFace`; roll/scale invariance and monotonicity under foreshortening asserted. Nothing consumes it as a discount yet |
## Deployment (DP)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| DP-001 | One analysis core; modes are front-ends and must not fork pipeline logic | PR-004 | High | Done |
| DP-002 | Batch CLI over one title | PR-004 | High | Done |
| DP-003 | On-demand resident service with bounded, observable queue | PR-004 | Medium | Planned |
| DP-004 | Opportunistic/idle mode: external trigger, hard stop, implicit re-queue | PR-004 | Medium | Planned |
| DP-005 | Native installer, no Docker; Fedora + Arch | PR-004 | Medium | Planned |
| DP-006 | Background incremental gallery refresh on a timer | PR-003 | Medium | Planned |
| DP-007 | CI builder image, CPU-only, pinned by tag in the Gitea container registry | PR-004 | High | Planned |
| DP-008 | Builder images + release jobs per backend (cpu / cuda / rocm); ship binaries, not engines | PR-004 | Medium | Planned |
## Integration (IR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done |
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | **Done** — `schema_version: 2`; windows are objects with `belief` + `route`; `extraction.*` carries `extinction_sec` and `gallery_scope`; `anneal_sec` removed |
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | **In Progress** — sink builds windows from registry claims and flushes at EOF; the deferred pass (AR-020) does not exist yet, so output is still final at EOF |
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | **Done** — `src/audio_signature.*`; not yet emitted into the truth file (IR-002) |
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | **Done** — `tests/fixtures/audio/`; v1 parameters now normative in server spec §3 |
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | **Done** |
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | **Done** |
| IR-006 | Jellyfin round-trip: pull pending queue, push complete results only | SR-001 | High | Done |
## Gallery (GR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | **Done** — basename + SHA-256 + `embed_dim`; mismatch fatal with no bypass, unstamped warns unless `--require-gallery-stamp`; `scripts/stamp_gallery.py` migrates in place |
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
| GR-007 | Persist harvested embeddings **flagged and reviewable**, never silently equal to baked | SR-005 | Medium | Planned |
| GR-008 | Flag distributional outliers among an actor's references (poisoning guard) — `EXCEPTION: AR-024` | SR-005 | Medium | Planned |
| GR-009 | Human-confirmed associations persist and improve future extractions | §4 | Medium | TBD |
## Validation (VR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| VR-001 | HDF5 post-inference dump at the embedded-frame boundary | PR-002 | High | Done |
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | **Done** — replay driven from committed fixtures in `tests/test_replay_fixtures.cpp`; determinism asserted |
| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done |
| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | **Done** — knee at 24–32 px; 32 px gives 98.1% TPI, 0.0 FPI at every size. Degrades an already-aligned 112×112 crop, so it isolates the embedder and is an **upper bound**; VR-013 measures the same question end to end and AR-002 takes its number, not this one |
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned |
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned |
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned |
| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | **In Progress** — sharpness half done ([`docs/quality-knee.md`](quality-knee.md)): 1670 actors, joint size×blur grid over three blur families (Gaussian, disc defocus, linear motion), 60120 probe-cell records each. Sharpness is **not a sufficient statistic** (equal measured sharpness spans 15.3–91.0% TPI, ordered by source size); **the blur family matters more than its amount** — at matched per-axis σ=3 on a 112 px face, Gaussian/motion/defocus cost 9/18/**53**% error, so a Gaussian-only sweep understates real lens blur fivefold; blur breaks **confidence, not ranking** (rank-1 80.2% where TPI is 15.3%), so FPI never left 0.1% in any of the 108 cells; a sharpness **gate** loses 3× more true presence than the free size filter at equal saving, because even destroyed faces stay 46.9% identifiable. **Pose half not started** — the AR-030 residual is exposed via `sae_embed.alignment_residual` but no pose arm has been run, so the dedicated-landmark-model question is still open |
| VR-014 | Audio-signature **offset recovery on real content** — a known trim recovered from film audio, not from the synthetic golden tone | PR-002 | Medium | **Done** — 40 random in-cap offsets, every one recovered to the nearest frame: **worst error 46 ms against a 500 ms budget**, and 46 ms is the floor rather than a result, since the offset is quantised to whole 92.88 ms frames. The `runtime/2` anchor confirmed through real head-trimmed files (a `delta` trim moves the window by `delta/2`). The one soft spot is **tier labelling, not accuracy**: the score falls with sub-frame misalignment (0.94–0.99 near a frame boundary, 0.69–0.73 at half a frame), so 27/40 correct alignments were demoted to `loose`. ±1 frame of slack in the *score* fixes it — measured, all 40 back to `audio` (min 0.906), false matches unmoved at 0.12–0.16, costing 81 ms of the budget |
| VR-013 | Cross-source identification probe — gallery from one recording, probes from another, swept over input resolution end to end | PR-002 | Medium | **In Progress** — holding 90% of the plateau needs ~50 px end to end against VR-005's ~22 px, the gap being detection and landmark error; **`min_face_px` 40, since 32 admits faces in the falling region** (AR-002). FPI 0.0% at every scale. Ceiling is cross-view, not resolution |
---
## Verification strategy
**CI runs on an Intel N100 with no discrete GPU.** That is a hard constraint on
how each requirement can be verified, and it shapes the test design rather than
merely limiting it.
Four tiers, in decreasing order of preference:
| Tier | Runs in CI | What it covers |
|---|---|---|
| **T1 — Functor unit** | Yes | A KPN node's `operator()` driven directly with hand-built inputs |
| **T2 — Replay** | Yes | The composed pipeline driven from an HDF5 fixture — no GPU, no video |
| **T3 — CPU inference** | Yes, slowly | ORT CPU provider over a handful of frames; smoke tests only |
| **T4 — GPU** | **No** | Throughput, TRT engines, large-gallery GEMM |
### T1 is the primary tier, and KPN is why
**Node functors are plain callable structs, constructed independently of the
network that wraps them** (`main.cpp:186-207` builds them as stack objects;
`ObjectNode` merely adapts them). So a node is testable by constructing it and
calling `operator()` — no channels, no threads, no network, no fixture.
This is already the established pattern, not a proposal:
`tests/test_face_tracker.cpp` "drives the node's `operator()` with hand-built
`EmbeddedSceneFrame`s and inspects the emitted `track_ids`", and does so
"pure, GPU-free, model-free".
The consequence is that most of the redesign is verifiable **without any
fixture at all**: construct exactly the awkward state — a belief swap, two live
tracks converging on one actor, a film ending mid-track, a gap one frame under
the timeout — rather than hunting for a clip that happens to exhibit it.
Four hazards this removes outright:
- **No fixture-provenance risk** for these tests — the inputs are synthetic and
explicit.
- **No "fixture must be replayed from frame 0"** concern — state is constructed
directly.
- **No cross-test state leakage** (e.g. a tracker's `next_id_` persisting) — each
test constructs a fresh functor.
- **No replay-harness nondeterminism** — no channels, so no EOF-tail heuristics
or silent drops.
It also means **a dead upstream producer does not block testing a downstream
consumer.** `is_scene_boundary` currently has no producer (see AR-010), which
would make a *replay* test of the frame-dependent `track_alpha` pass vacuously —
but a T1 test simply constructs a frame with `is_scene_boundary = true` and
asserts the weighting changes. The producer gap is a pipeline defect to fix, not
a verification blocker.
### T2 covers what T1 cannot
Replay remains necessary for **composition** — that the nodes wired together
behave as the sum of their parts — and for realistic data at scale, which
synthetic inputs cannot honestly imitate. It is the tier that would catch a
wiring error, a channel-capacity problem, or an ordering assumption that only
appears under concurrency.
The HDF5 dump (VR-001) captures state after decode → detect → align → embed, so
replay needs no GPU and no video. That was built for the optimizer; it doubles as
CI, which is a strong argument for keeping the schema honest and for replay
driving the *real* nodes rather than a reimplementation (VR-002).
**Fixtures and studies are generated locally**, on the development machine where
the models, galleries and media already exist. CI consumes them; it never
produces them.
**Small committed fixtures are required.** A few HDF5 dumps covering the awkward
cases — a cut, a belief swap, two live tracks converging, a film ending
mid-track, an unknown track that only resolves after expansion — are worth more
than a large corpus, and they are small enough to commit.
**T4 requirements cannot pass in CI, and the gate must not pretend otherwise.**
For these, CI verifies that a test *exists and is tagged*, not that it passes;
the run happens on a GPU host, nightly or manually, and reports separately. A
requirement whose only evidence is a test that never executes should be visible
as such rather than counted as covered.
| Requirement | Tier | Note |
|---|---|---|
| AR-001, AR-005, AR-006 | T3 | Smoke only — correctness of detection/embedding is a model property, not ours |
| AR-002 | T2 | Size filtering is arithmetic on dumped bboxes |
| AR-003, AR-004 | T1 + T4 | Backpressure logic is unit-testable; saturation behaviour needs real load |
| AR-007 … AR-017 | **T2** | The core of the redesign — fully replayable |
| AR-018 … AR-022 | **T2** | Expansion, deferred pass, clustering: all post-embedding |
| AR-023 … AR-025 | T1 | Calibration fit and log-odds accumulation are pure maths |
| AR-026, AR-027 | T4 | GEMM throughput and scaling — GPU host only |
| DP-* | T1 + manual | Lifecycle logic unit-tested; install paths are manual |
| IR-001 … IR-003 | T1 | Serialisation against a golden truth file |
| IR-004, IR-005 | **T1** | Audio signature is CPU DSP — the golden-vector fixture runs anywhere, which is precisely why it is the right cross-repo check |
| GR-001 … GR-005 | T1 + T3 | Gallery assembly is I/O and bookkeeping; embedding is T3 smoke |
| GR-006 … GR-008 | T1 | Tiering and outlier detection operate on stored embeddings |
| VR-* | Out of CI | Studies are run deliberately and their results committed as documents |
**One consequence worth stating:** AR-027 (arbitrary gallery scale) is
structurally unverifiable on the CI host. It needs a GPU host and a synthetic
large gallery, so it is the requirement most likely to silently regress. Its
benchmark (VR-008) should run on a schedule rather than on demand.
### CI never calls a model
**Not "should not" — cannot.** The N100 has no GPU, and even the ONNX Runtime CPU
provider is impractical: a measured run of the embedder on this hardware sits at
~930 ms per frame, so a 77 s clip at 5 fps would take roughly six minutes of
inference alone. Every model invocation therefore happens **locally, ahead of
time**, and CI consumes the result as data.
This is what makes the T1/T2 split load-bearing rather than a preference: T1 and
T2 are the only tiers that can exist in CI at all.
### Fixture corpus — `bali/`
Five clips of **Road to Bali (1952)**, ~77 s each, 480×360, 30 fps, 42 MB total.
Public domain, and that is the reason to use it rather than a convenience:
**derived fixtures — dumps, crops, golden outputs — can be committed without the
rights question that rules out sharing gallery data (SR-005).** A fixture cut
from a copyrighted title could not live in the repository at all.
Two properties to design around rather than discover:
- **480×360 means small faces.** At this resolution a face is often 40–80 px, so
the AR-002 minimum of 40 px (original resolution) sits at the very bottom of
that range: the filter is close to binding, and anything shot wider is lost.
Fixture generation must set `--min-face-px` explicitly and record it, or the
dumps will be sparse for reasons unrelated to what is being tested.
- **77 s is short.** At 1 fps that is 77 frames — too thin to exercise an
extinction window measured in tens of seconds. Generate at 5 fps (≈385 frames,
~1 MB) and record the rate in provenance, since the behaviour under test
changes with it.
> **AR-004 blocks reproducible fixture generation.** A trial run of one clip
> produced 49 frames of an expected ~385, ending at 51 s of 77 s, with the
> diagnostics reporting 285 frames dropped at `camera_pos` and 51 at
> `face_aligner`. Channels overflow and **drop** rather than blocking, and what
> gets dropped depends on timing — so the same command run twice can produce
> different dumps. Golden fixtures cannot be built on that. AR-004 is therefore
> a prerequisite for VR-001 fixtures, not merely a throughput concern for crowd
> scenes.
### Fixtures — precomputed inference, pulled by CI
The N100 cannot run inference at any useful rate, so **inference output is
precomputed on a GPU host and consumed by CI as data.** This converts most of
what looks like GPU work into pure CPU replay.
| Fixture | Contents | Size | Storage |
|---|---|---|---|
| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~0.1–1 MB each | **Committed in-repo** |
| **Corpus dumps** | Full-length titles from the validation corpus | ~21–38 MB each | **Gitea package registry**, pinned by version + checksum |
| **Synthetic gallery** | Random unit-norm embeddings, fixed seed | small | Generated at test time |
| **Golden truth files** | Expected output for each edge-case dump | KB | Committed |
| **Audio golden vectors** | FLAC + expected signature + parameter contract | ~600 KB | Committed, **shared with the plugin repo** |
Edge-case dumps are small enough to commit, and being in-repo means they version
with the code that reads them.
**Corpus dumps go to the Gitea package registry, not Git LFS.** Both are
available — the models already use LFS — but their fetch semantics differ in a
way that matters here. LFS objects are pulled on clone unless a developer
explicitly skips them, so ~38 MB per title behind LFS taxes everyone who clones,
forever, for data that only CI and the optimizer ever read. Registry artifacts
are fetched on demand by the job that needs them.
Rule of thumb: **LFS for what the build needs; the package registry for what a
particular job needs.** Models are the former; corpus dumps and the CI image
(DP-007) are the latter.
Pin by version and verify by checksum on fetch. A fixture that changes silently
under CI is worse than a missing one, because the failure presents as a code
regression.
**Generation must be reproducible and versioned.** A script, run on a GPU host,
regenerates every fixture from source clips; it is re-run when the VR-001 schema
version bumps. A fixture whose provenance is unknown is worse than no fixture,
because it will be trusted.
> **The limitation that must stay visible:** replay fixtures freeze upstream
> behaviour. A test driven from a dump verifies AR-007 onward *given those
> embeddings* — it cannot detect a regression in detection, alignment or
> embedding, because those produced the fixture. Nothing in CI can. That gap is
> covered only by the T3 smoke test and the scheduled GPU run, and it should not
> be papered over by a high replay-coverage number.
### Per-requirement verification plan
| ID | Tier | Test asserts | Edge cases to cover |
|---|---|---|---|
| AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only |
| AR-002 | T2 | Faces below 40 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame |
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block |
| AR-005 | T1 | Known landmarks → expected 112×112 warp; the fit never mirrors | Landmarks near frame edge; degenerate/collinear points; a mirrored set — SVD returns a reflection unless the determinant guard rejects it |
| AR-006 | T3 | Embeddings are unit-norm | Batch smaller than, equal to, larger than `embed_batch_size` |
| AR-007 | T2 | Association picks the right track | Two faces crossing paths; one leaving frame as another enters |
| AR-008 | T2 | One pool; dormant tracks match on embedding, not IoU | Dormant track whose old bbox overlaps a *different* new face — must not match on position |
| AR-009/010 | T2 | Cut/boundary shifts weighting toward embedding | Cut with same people; cut with all-new people |
| AR-011 | T1 | TransNetV2 receives native-rate frames | Source at 24/25/30 fps — dedup window derived, not assumed |
| AR-012 | **T2** | Window spans full track extent, not first recognition | Actor recognised only at track end — window must still start at `first_seen` |
| AR-013 | **T2** | `last_seen` set/unset; window ends at last sighting | Gap just under vs just over timeout; reappearance after timeout → two windows |
| AR-014 | T2 | Belief swap closes one window, opens another | No blended window; no overlap at the swap frame |
| AR-015 | T2 | Two live tracks on one actor trigger re-association | Counter increments |
| AR-016 | **T2** | Every track closed at EOF | Film ending mid-shot — window ends at final frame, not dropped |
| AR-017 | T1 | Claim carries posterior and route | Deferred and pooled routes distinguishable |
| AR-018 | T1 | Band admits only within bounds | At each bound exactly; store never admits below lower bound |
| AR-019 | T2 | Promotion only when all three signals quiet | Cut mid-track blocks promotion |
| AR-020 | **T2** | Unknown resolved after expansion | Track failing at minute 12, resolved at EOF — the ordering-independence claim |
| AR-021 | T2 | Clustering merges same person, respects cannot-link | **Temporally overlapping tracks never merge**; measure how many merges the constraint rejects |
| AR-022 | T1 | Context crops retained, bounded per track | Track running for minutes |
| AR-023 | T1 | Sigmoid fit on synthetic separable data | Too few positive pairs → `valid=false`, fallback engages |
| AR-024 | **Static check** | No bare cosine outside a tagged `EXCEPTION` | Grep-based; this is the invariant's enforcement |
| AR-025 | T1 | Log-odds accumulate; correlated frames discounted | 30 identical frames must **not** reach the certainty of 30 diverse ones |
| AR-026 | T1 + T4 | GEMM path produces same result as reference loop | Equivalence on small input in CI; throughput on GPU host |
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
| AR-028 | **T2** | No embedding reaches the matcher unscored; the vector survives into the dump | Face failing exactly one axis; all three healthy; a face whose landmarks are degenerate — scored, not silently vanished |
| AR-029 | T1 | Synthetic blur ladder → monotonically falling sharpness | Gaussian vs motion blur; **small sharp face vs large soft one** — size must not leak into this axis |
| AR-030 | T1 | Alignment residual rises monotonically with foreshortening | **In-plane roll, scale and translation must leave it at zero** — the property that makes it a pose measure rather than a pose-and-everything-else measure; face size must not shift it; degenerate landmarks report not-ok rather than a number |
| VR-012 | **T4** | Knee located per axis on held-out films | Report each candidate threshold's cost in **lost true presence**, not only its gain in precision — a gate that improves misID by discarding half the cast has not helped |
| VR-013 | **T4** | Identification holds across two recordings of the same people, and degrades to TBI rather than to a wrong name as input resolution falls | Gallery and probes must come from *different* recordings — a hold-one-out over one recording measures a much easier problem and will not surface the cross-view failure. Ground truth is hand-sorted; labels propagated by embedding similarity would keep only the faces the embedder already gets right |
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos |
| VR-014 | **T2** | A known trim offset is recovered from **real film audio**, to the nearest frame | An offset past the ±600-frame cap and unrelated content must both be *declined*, never given a best-effort alignment. Fixture and signature are both CPU-only, so unlike the other VR rows this one is CI-*executable* — though the repo's only workflow today is the traceability gate, so nothing runs it there yet. The signature comes from the shipped C++ through `sae_audio`; a numpy port would be a third implementation nobody checks against the golden vector |
| IR-006 | T1 + manual | Queue pull and result push against a stubbed Jellyfin API | Partial result never pushed; push only after the deferred pass |
| IR-007 | **T1** | Media < 120 s emits no signature at all | Exactly 120 s; just under; zero-length audio. Must match the plugin's cutoff exactly — a caller-varying window length is what SR-004 forbids |
| IR-008 | T1 | `v1:` prefix emitted and honoured on read | Unknown prefix rejected, not guessed |
| GR-009 | T1 | Human-confirmed associations persist and are tier-tagged | Survives a gallery rebuild; distinguishable from baked and harvested |
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides; **unstamped warns, and errors under `SAE_REQUIRE_GALLERY_STAMP`**; same filename + different SHA-256 must still be a mismatch |
| GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected |
| VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks |
Three of these are worth singling out because they verify claims that would
otherwise be assertions: **AR-012** (window starts at `first_seen` even when
recognition comes late) is the entire point of the redesign; **AR-020** (a track
failing mid-film resolves at EOF) is the claim that ordering stops mattering; and
**AR-025** (30 identical frames ≠ 30 diverse ones) is what stops the Bayesian
accumulation from being decoration.
---
## Withdrawn
| ID | Requirement | Reason |
|---|---|---|
| — | `anneal_sec` window merging | Superseded by AR-012/AR-013: a track survives its own gaps, so there is nothing to anneal |
| — | `extinction_sec` actor keep-alive | Superseded by AR-013: windows end at last sighting, which is what this over-claimed |
Both were deleted rather than retained at zero — a field naming a mechanism the
pipeline no longer has is actively misleading (see `SPEC.md` A6.6).
---
## Notes on coverage
- **VR-*** traces to PR-002 (scene-granularity answers) rather than to a system
requirement: parameter studies are single-repo work serving accuracy, and this
is correct rather than a gap.
- **PR-005** (leak nothing) has no `AR`/`DP` row. It is satisfied *structurally*
by SR-004 and GR-005 — the server holds no binary, the gallery never leaves the
instance — not by any component doing something. It cannot be verified by
pointing at code, and it dies the moment either prohibition is relaxed.
docs/traceability.md
+777
View File
@@ -0,0 +1,777 @@
# Requirements traceability matrix
<!-- GENERATED FILE - do not edit by hand. -->
<!-- Regenerate: scripts/traceability/traceability-gate.sh -->
**Generated:** 2026-07-31T14:44:58+00:00
Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`).
## Summary
| Metric | Value |
|---|---|
| Source files scanned | 111 |
| TRACES tags found | 132 |
| EXCEPTION tags found | 0 |
| Requirements defined | 69 |
| Requirements covered | 38 |
| **Coverage** | **55.1%** (38/69) |
| Coverage of CI-executable scope | 67.9% (38/56) |
| Tagged but unexecuted in CI | 5 |
| Orphan tags | 0 |
### By type
| Type | Covered | Tagged but unexecuted | Defined |
|---|---|---|---|
| AR | 22 | 1 | 30 |
| DP | 2 | 0 | 8 |
| IR | 8 | 0 | 8 |
| GR | 5 | 0 | 9 |
| VR | 1 | 4 | 14 |
- **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104, UT-105, UT-106, UT-107, UT-108
- **IT** tags present (separate taxonomy, not counted in coverage): IT-001
- **PR** tags present (separate taxonomy, not counted in coverage): PR-002, PR-004
- **SR** tags present (separate taxonomy, not counted in coverage): SR-001, SR-002, SR-003, SR-005
## Not executable in CI
These requirements have no verification tier this repo's CI host can run, so a tag on them is evidence of *intent*, not of verification. They are never counted as covered.
| ID | Tiers | Tagged in source | Requirement |
|---|---|---|---|
| AR-027 | T4 | yes | Throughput acceptable for **arbitrary** gallery size |
| VR-001 | out-of-ci | yes | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | out-of-ci | yes | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | out-of-ci | yes | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | out-of-ci | yes | Reproducible validation corpus with ground truth |
| VR-005 | out-of-ci | no | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | out-of-ci | no | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | out-of-ci | no | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | out-of-ci | no | Gallery scaling benchmark — throughput vs gallery size |
| VR-010 | out-of-ci | no | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | out-of-ci | no | Rewrite the replay harness for the post-AR-012 output contract |
| VR-012 | T4, out-of-ci | no | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … |
| VR-013 | T4, out-of-ci | no | Cross-source identification probe — gallery from one recording, probe… |
**Tagged but unexecuted:** AR-027, VR-001, VR-002, VR-003, VR-004 — a test exists and is tagged, but this CI host cannot run it. Report those runs separately.
## Orphan tags
A tag naming an ID `requirements.md` does not define. This is what renumbering produces, and what a typo produces.
_None._
## Requirements tracing up to nothing
A register row whose `Traces to` cell names no parent. Work serving no stated goal is how scope creeps in, and it is invisible unless something looks.
_None._
## Recorded exceptions
Deliberate, documented departures from an invariant (`EXCEPTION: XX-nnn <reason>`). Reported separately and never counted as coverage — an exception is a decision to be reviewed, not evidence a requirement is met.
_None._
## Register
| ID | Status | Tier | Traces to | Trace state | Tagged in | Requirement |
|---|---|---|---|---|---|---|
| AR-001 | Done | T3 | SR-002 | covered | `src/nodes/face_detector_node.hpp` | Detect faces in sampled frames; emit bbox, confidence, 5-point landma… |
| AR-002 | Planned | T2 | SR-002 | untagged | - | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's … |
| AR-003 | **Done** — `max_fac… | T1, T2, T4 | SR-002 | covered | `src/config.hpp`, `src/nodes/face_detector_node.hpp`, `src/nodes/identity_matcher_node.hpp` | No fixed per-frame face cap — crowd scenes must not lose background c… |
| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_replay_fixtures.cpp` | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… |
| AR-005 | **Done** — `umeyama… | T1, T3 | SR-002 | covered | `src/face_utils.hpp`, `tests/test_face_utils.cpp` | Align to 112×112 via ArcFace 5-point similarity transform, fitted by … |
| AR-006 | Done | T3 | SR-002 | covered | `src/nodes/embedder_node.hpp` | 512-d L2-normalised embeddings, batched |
| AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… |
| AR-008 | **Done** — one pool… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | One track pool keyed on `last_seen`; no separate revival path |
| AR-009 | Done | T2 | SR-002 | covered | `src/nodes/camera_position_change_detector_node.hpp` | Camera-cut detection (histogram) as an association hint |
| AR-010 | **Done** — decode b… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/scene_boundary_annotator_node.hpp`, `src/nodes/scene_detector_node.hpp`, `src/scene_boundaries.hpp` | Scene-boundary detection (TransNetV2) as an association hint |
| AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… |
| AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition |
| AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… |
| AR-014 | **Done** — swap clo… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Belief swap A→B terminates the track and starts a new one |
| AR-015 | **Done** — reverse … | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Two live tracks owned by one actor ⇒ treat as a detected cut, re-asso… |
| AR-016 | **Done** — `flush()… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | All tracks closed at EOF — a film ends with faces on screen |
| AR-017 | **Done** — `DeadTra… | T1, T2 | SR-002 | covered | `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Every presence claim carries its belief and identification route |
| AR-018 | **Done** — banded a… | T1, T2 | SR-005 | covered | `src/config.hpp`, `src/gallery/track_gallery.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-subject embedding store with banded admission (novel enough, safe… |
| AR-019 | **Done** — all thre… | T2 | SR-005 | covered | `src/gallery/track_gallery.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-film gallery annex from owned tracks; acquires the non-frontal vi… |
| AR-020 | Planned | T2 | SR-005 | untagged | - | Deferred re-identification of unknown tracks against the final expand… |
| AR-021 | Planned | T2 | SR-005 | untagged | - | Cluster unknown tracks into one entity per person, under temporal can… |
| AR-022 | Planned | T1, T2 | §4 | untagged | - | Capture still-unidentified tracks: embeddings, metadata, **context cr… |
| AR-023 | Done | T1 | SR-002 | covered | `src/gallery/gallery_calibration.hpp`, `src/nodes/identity_matcher_node.hpp` | Fit sigmoid calibration from intra/inter similarity distributions |
| AR-024 | **Done** — associat… | T1, static | SR-002 | covered | `src/config.hpp`, `src/evidence_discount.hpp`, `src/gallery/gallery_calibration.hpp`, `src/gallery/track_gallery.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp`, `src/nodes/identity_matcher_node.hpp` | **Always the calibrated probability, never a raw cosine** — exception… |
| AR-025 | **Done** — log-odds… | T1 | SR-002 | covered | `src/evidence_discount.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-track Bayesian accumulation in log-odds, with correlated-observat… |
| AR-026 | In Progress | T1, T4 | SR-001 | covered | `src/backends/gemm_backend.cpp` | All similarity computed as GEMM, including annex and deferred pass |
| AR-027 | Planned | T4 | SR-001 | tagged, unexecuted | `src/backends/gemm_backend.cpp` | Throughput acceptable for **arbitrary** gallery size |
| AR-028 | Planned | T2 | SR-002 | untagged | - | **Embedding input quality assessed and carried** — every face scored … |
| AR-029 | Planned | T1 | SR-002 | untagged | - | Sharpness measure on the **aligned crop** (scale-normalised, so it ca… |
| AR-030 | **In Progress** — m… | T1 | SR-002 | covered | `src/face_utils.hpp`, `tests/test_face_utils.cpp` | Visibility measure from the AR-001 5-point landmarks — extreme pose o… |
| DP-001 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | One analysis core; modes are front-ends and must not fork pipeline lo… |
| DP-002 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | Batch CLI over one title |
| DP-003 | Planned | T1, manual | PR-004 | untagged | - | On-demand resident service with bounded, observable queue |
| DP-004 | Planned | T1, manual | PR-004 | untagged | - | Opportunistic/idle mode: external trigger, hard stop, implicit re-que… |
| DP-005 | Planned | T1, manual | PR-004 | untagged | - | Native installer, no Docker; Fedora + Arch |
| DP-006 | Planned | T1, manual | PR-003 | untagged | - | Background incremental gallery refresh on a timer |
| DP-007 | Planned | T1, manual | PR-004 | untagged | - | CI builder image, CPU-only, pinned by tag in the Gitea container regi… |
| DP-008 | Planned | T1, manual | PR-004 | untagged | - | Builder images + release jobs per backend (cpu / cuda / rocm); ship b… |
| IR-001 | Done | T1 | SR-003 | covered | `src/nodes/result_sink_node.hpp` | Emit the JRay truth format as sibling `.jray.json` |
| IR-002 | **Done** — `schema_… | T1 | SR-003 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/result_sink_node.hpp` | Windows carry belief + route; `extraction.*` carries `extinction_sec`… |
| IR-003 | **In Progress** — s… | T1 | SR-003 | covered | `src/main.cpp` | Output written **after** the deferred pass, not at EOF |
| IR-004 | **Done** — `src/aud… | T1 | SR-003 | covered | `scripts/validation/test_audio_offset.py`, `src/audio_bindings.cpp`, `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Compute the audio signature exactly per server spec §3 |
| IR-005 | **Done** — `tests/f… | T1 | SR-003 | covered | `src/audio_bindings.cpp`, `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Golden-vector fixture shared with the plugin repo to prove bit-exactn… |
| IR-006 | Done | T1, manual | SR-001 | covered | `scripts/run_from_jellyfin.py` | Jellyfin round-trip: pull pending queue, push complete results only |
| IR-007 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Media < 120 s: emit no signature, apply no sync offset — identical ru… |
| IR-008 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Emit and honour the signature's own `v1:` version prefix |
| GR-001 | Done | T1, T3 | SR-001, SR-005 | covered | `scripts/make_jellyfin_gallery.py` | Build gallery from Jellyfin library cast, TMDB profile fallback |
| GR-002 | Done | T1, T3 | PR-003 | covered | `scripts/make_jellyfin_gallery.py` | Incremental `--merge` refresh without re-embedding known actors |
| GR-003 | Planned | T1, T3 | SR-001 | covered | `src/build_gallery.cpp`, `src/gallery/gallery_calibration.hpp`, `src/gallery/gallery_report.hpp` | Report coverage: zero-image actors, under-referenced actors, dedup, c… |
| GR-004 | **Done** — basename… | T1, T3 | SR-001 | covered | `scripts/filter_gallery.py`, `scripts/make_gallery.py`, `scripts/make_jellyfin_gallery.py`, `scripts/movienet_eval.py`, `scripts/optimizer/fetch_missing_actors.py`, `scripts/optimizer/optimize.py`, `scripts/optimizer/reembed_gallery.py`, `scripts/optimizer/replay.py`, `scripts/sae_embed_loader.py`, `scripts/sae_gallery.py`, `scripts/sae_stamp.py`, `scripts/stamp_gallery.py`, `src/config.hpp`, `src/gallery/embedder_stamp.cpp`, `src/gallery/embedder_stamp.hpp`, `src/gallery/gallery_builder.cpp`, `src/gallery/gallery_store.cpp`, `src/kpn_bindings.cpp`, `src/main.cpp`, `src/nodes/embedding_dump_node.hpp`, `src/scene_preview.cpp`, `src/types.hpp`, `tests/test_gallery_store.cpp` | Stamp embedder identity into the gallery; **hard startup error** on m… |
| GR-005 | Done | T1, T3 | **SR-005** | covered | `src/gallery/gallery_store.hpp` | Gallery data never leaves the instance |
| GR-006 | Planned | T1 | SR-005 | untagged | - | Provenance tiers: baked / harvested / confirmed, distinguishable per … |
| GR-007 | Planned | T1 | SR-005 | untagged | - | Persist harvested embeddings **flagged and reviewable**, never silent… |
| GR-008 | Planned | T1 | SR-005 | untagged | - | Flag distributional outliers among an actor's references (poisoning g… |
| GR-009 | TBD | T1 | §4 | untagged | - | Human-confirmed associations persist and improve future extractions |
| VR-001 | Done | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp`, `tests/test_replay_fixtures.cpp` | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | **Done** — replay d… | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py`, `tests/test_replay_fixtures.cpp` | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/second_score.py` | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/validation/ground_truth.py` | Reproducible validation corpus with ground truth |
| VR-005 | **Done** — knee at … | out-of-ci | PR-002 | untagged | - | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | Planned | out-of-ci | PR-002 | untagged | - | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | Planned | out-of-ci | PR-002 | untagged | - | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | Planned | out-of-ci | PR-002 | untagged | - | Gallery scaling benchmark — throughput vs gallery size |
| VR-009 | Planned | T1, out-of-ci | PR-002 | untagged | - | Verify accumulated posteriors are calibrated against held-out tracks |
| VR-010 | Planned | out-of-ci | PR-002 | untagged | - | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | Planned | out-of-ci | PR-002 | untagged | - | Rewrite the replay harness for the post-AR-012 output contract |
| VR-012 | Planned | T4, out-of-ci | PR-002 | untagged | - | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … |
| VR-013 | **In Progress** — h… | T4, out-of-ci | PR-002 | untagged | - | Cross-source identification probe — gallery from one recording, probe… |
| VR-014 | **Done** — 40 rando… | T2, out-of-ci | PR-002 | covered | `scripts/validation/test_audio_offset.py` | Audio-signature **offset recovery on real content** — a known trim re… |
## Detailed mapping
### AR-001
**Locations:** 1
- [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown`
### AR-003
**Locations:** 3
- [`src/config.hpp:44`](../src/config.hpp#L44) — `Unknown`
- [`src/nodes/face_detector_node.hpp:47`](../src/nodes/face_detector_node.hpp#L47) — `private:`
- [`src/nodes/identity_matcher_node.hpp:166`](../src/nodes/identity_matcher_node.hpp#L166) — `std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);`
### AR-004
**Locations:** 4
- [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;`
- [`src/main.cpp:299`](../src/main.cpp#L299) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:166`](../src/nodes/identity_matcher_node.hpp#L166) — `std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);`
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
### AR-005
**Locations:** 2
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown`
- [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown`
### AR-006
**Locations:** 1
- [`src/nodes/embedder_node.hpp:21`](../src/nodes/embedder_node.hpp#L21) — `struct EmbedderFunc`
### AR-007
**Locations:** 3
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
### AR-008
**Locations:** 3
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
### AR-009
**Locations:** 1
- [`src/nodes/camera_position_change_detector_node.hpp:30`](../src/nodes/camera_position_change_detector_node.hpp#L30) — `struct CameraPositionChangeDetectorFunc`
### AR-010
**Locations:** 9
- [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;`
- [`src/main.cpp:309`](../src/main.cpp#L309) — `Unknown`
- [`src/main.cpp:379`](../src/main.cpp#L379) — `return run_net(std::move(net));`
- [`src/main.cpp:411`](../src/main.cpp#L411) — `Unknown`
- [`src/nodes/scene_boundary_annotator_node.hpp:2`](../src/nodes/scene_boundary_annotator_node.hpp#L2) — `Unknown`
- [`src/nodes/scene_detector_node.hpp:36`](../src/nodes/scene_detector_node.hpp#L36) — `static constexpr std::string_view label() { return "scene_detector"; }`
- [`src/nodes/scene_detector_node.hpp:110`](../src/nodes/scene_detector_node.hpp#L110) — `void flush_remaining()`
- [`src/nodes/scene_detector_node.hpp:144`](../src/nodes/scene_detector_node.hpp#L144) — `void write_output()`
- [`src/scene_boundaries.hpp:2`](../src/scene_boundaries.hpp#L2) — `Unknown`
### AR-012
**Locations:** 9
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/identity_matcher_node.hpp:125`](../src/nodes/identity_matcher_node.hpp#L125) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:272`](../src/nodes/identity_matcher_node.hpp#L272) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-013
**Locations:** 3
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-014
**Locations:** 2
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-015
**Locations:** 2
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-016
**Locations:** 4
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/result_sink_node.hpp:63`](../src/nodes/result_sink_node.hpp#L63) — `void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-017
**Locations:** 3
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-018
**Locations:** 3
- [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown`
- [`src/gallery/track_gallery.hpp:161`](../src/gallery/track_gallery.hpp#L161) — `struct TrackState`
- [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);`
### AR-019
**Locations:** 3
- [`src/gallery/track_gallery.hpp:122`](../src/gallery/track_gallery.hpp#L122) — `void forget(int track_id) { tracks_.erase(track_id); }`
- [`src/nodes/identity_matcher_node.hpp:147`](../src/nodes/identity_matcher_node.hpp#L147) — `MatchedSceneFrame operator()(TrackedSceneFrame tf)`
- [`src/nodes/identity_matcher_node.hpp:285`](../src/nodes/identity_matcher_node.hpp#L285) — `Unknown`
### AR-023
**Locations:** 3
- [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }`
### AR-024
**Locations:** 10
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown`
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/gallery/track_gallery.hpp:132`](../src/gallery/track_gallery.hpp#L132) — `void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }`
- [`src/gallery/track_gallery.hpp:161`](../src/gallery/track_gallery.hpp#L161) — `struct TrackState`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);`
- [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }`
### AR-025
**Locations:** 3
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:125`](../src/nodes/identity_matcher_node.hpp#L125) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:272`](../src/nodes/identity_matcher_node.hpp#L272) — `Unknown`
### AR-026
**Locations:** 1
- [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;`
### AR-027
**Locations:** 1
- [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;`
### AR-030
**Locations:** 2
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown`
- [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown`
### DP-001
**Locations:** 1
- [`src/main.cpp:3`](../src/main.cpp#L3) — `Unknown`
### DP-002
**Locations:** 1
- [`src/main.cpp:3`](../src/main.cpp#L3) — `Unknown`
### GR-001
**Locations:** 1
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### GR-002
**Locations:** 1
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### GR-003
**Locations:** 13
- [`src/build_gallery.cpp:83`](../src/build_gallery.cpp#L83) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:80`](../src/gallery/gallery_calibration.hpp#L80) — `struct GalleryCalibrationStats`
- [`src/gallery/gallery_calibration.hpp:145`](../src/gallery/gallery_calibration.hpp#L145) — `std::vector<bool> actor_eligible(n_actors, false);`
- [`src/gallery/gallery_calibration.hpp:294`](../src/gallery/gallery_calibration.hpp#L294) — `Unknown`
- [`src/gallery/gallery_report.hpp:2`](../src/gallery/gallery_report.hpp#L2) — `Unknown`
- [`src/gallery/gallery_report.hpp:52`](../src/gallery/gallery_report.hpp#L52) — `struct GalleryBuildAudit`
- [`src/gallery/gallery_report.hpp:73`](../src/gallery/gallery_report.hpp#L73) — `struct GalleryReport`
- [`src/gallery/gallery_report.hpp:154`](../src/gallery/gallery_report.hpp#L154) — `inline GalleryReport build_gallery_report(const ActorGallery& gallery,`
- [`src/gallery/gallery_report.hpp:295`](../src/gallery/gallery_report.hpp#L295) — `inline nlohmann::json gallery_report_to_json(const GalleryReport& r)`
- [`src/gallery/gallery_report.hpp:361`](../src/gallery/gallery_report.hpp#L361) — `inline GalleryReport gallery_report_from_json(const nlohmann::json& j)`
- [`src/gallery/gallery_report.hpp:446`](../src/gallery/gallery_report.hpp#L446) — `inline void save_gallery_report(const std::string& path, const GalleryReport& r)`
- [`src/gallery/gallery_report.hpp:454`](../src/gallery/gallery_report.hpp#L454) — `inline GalleryReport load_gallery_report(const std::string& path)`
- [`src/gallery/gallery_report.hpp:464`](../src/gallery/gallery_report.hpp#L464) — `return gallery_report_from_json(j);`
### GR-004
**Locations:** 44
- [`src/config.hpp:54`](../src/config.hpp#L54) — `Unknown`
- [`src/gallery/embedder_stamp.cpp:1`](../src/gallery/embedder_stamp.cpp#L1) — `Unknown`
- [`src/gallery/embedder_stamp.hpp:2`](../src/gallery/embedder_stamp.hpp#L2) — `Unknown`
- [`src/gallery/gallery_builder.cpp:45`](../src/gallery/gallery_builder.cpp#L45) — `ActorGallery build_gallery(const BuildConfig& cfg)`
- [`src/gallery/gallery_store.cpp:82`](../src/gallery/gallery_store.cpp#L82) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/gallery/gallery_store.cpp:167`](../src/gallery/gallery_store.cpp#L167) — `H5::DataSpace scalar(H5S_SCALAR);`
- [`src/gallery/gallery_store.cpp:219`](../src/gallery/gallery_store.cpp#L219) — `Unknown`
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:188`](../src/main.cpp#L188) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:148`](../src/types.hpp#L148) — `struct Actor`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");`
- [`tests/test_gallery_store.cpp:240`](../tests/test_gallery_store.cpp#L240) — `TempFile tf("gallery_json_stamp.json");`
- [`tests/test_gallery_store.cpp:252`](../tests/test_gallery_store.cpp#L252) — `Unknown`
- [`tests/test_gallery_store.cpp:268`](../tests/test_gallery_store.cpp#L268) — `Unknown`
- [`tests/test_gallery_store.cpp:300`](../tests/test_gallery_store.cpp#L300) — `Unknown`
- [`tests/test_gallery_store.cpp:309`](../tests/test_gallery_store.cpp#L309) — `Unknown`
- [`tests/test_gallery_store.cpp:333`](../tests/test_gallery_store.cpp#L333) — `Unknown`
- [`tests/test_gallery_store.cpp:348`](../tests/test_gallery_store.cpp#L348) — `Unknown`
- [`tests/test_gallery_store.cpp:367`](../tests/test_gallery_store.cpp#L367) — `Unknown`
- [`tests/test_gallery_store.cpp:384`](../tests/test_gallery_store.cpp#L384) — `TempFile tf("fake_model.onnx");`
- [`scripts/filter_gallery.py:80`](../scripts/filter_gallery.py#L80) — `if actor_jellyfin_id(a) in cast_ids]`
- [`scripts/make_gallery.py:181`](../scripts/make_gallery.py#L181) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:448`](../scripts/make_jellyfin_gallery.py#L448) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:456`](../scripts/make_jellyfin_gallery.py#L456) — `Unknown`
- [`scripts/movienet_eval.py:65`](../scripts/movienet_eval.py#L65) — `with open(args.gt) as f:`
- [`scripts/optimizer/fetch_missing_actors.py:62`](../scripts/optimizer/fetch_missing_actors.py#L62) — `def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,`
- [`scripts/optimizer/fetch_missing_actors.py:109`](../scripts/optimizer/fetch_missing_actors.py#L109) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:124`](../scripts/optimizer/fetch_missing_actors.py#L124) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/optimize.py:186`](../scripts/optimizer/optimize.py#L186) — `Unknown`
- [`scripts/optimizer/optimize.py:202`](../scripts/optimizer/optimize.py#L202) — `if not Path(f["dump"]).exists():`
- [`scripts/optimizer/reembed_gallery.py:62`](../scripts/optimizer/reembed_gallery.py#L62) — `for i, a in enumerate(ref["actors"], 1):`
- [`scripts/optimizer/replay.py:113`](../scripts/optimizer/replay.py#L113) — `Unknown`
- [`scripts/optimizer/replay.py:253`](../scripts/optimizer/replay.py#L253) — `Unknown`
- [`scripts/sae_embed_loader.py:23`](../scripts/sae_embed_loader.py#L23) — `def resolve_arcface(models_dir: str, arcface: str \| None = None) -> str:`
- [`scripts/sae_gallery.py:171`](../scripts/sae_gallery.py#L171) — `if not _stamp_empty(embedder):`
- [`scripts/sae_gallery.py:200`](../scripts/sae_gallery.py#L200) — `for a in range(len(offset)):`
- [`scripts/sae_stamp.py:3`](../scripts/sae_stamp.py#L3) — `Unknown`
- [`scripts/stamp_gallery.py:4`](../scripts/stamp_gallery.py#L4) — `Unknown`
### GR-005
**Locations:** 1
- [`src/gallery/gallery_store.hpp:15`](../src/gallery/gallery_store.hpp#L15) — `Unknown`
### IR-001
**Locations:** 1
- [`src/nodes/result_sink_node.hpp:2`](../src/nodes/result_sink_node.hpp#L2) — `Unknown`
### IR-002
**Locations:** 5
- [`src/config.hpp:20`](../src/config.hpp#L20) — `struct Config`
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:122`](../src/nodes/result_sink_node.hpp#L122) — `void write_output()`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
### IR-003
**Locations:** 1
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
### IR-004
**Locations:** 18
- [`src/audio_bindings.cpp:3`](../src/audio_bindings.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:265`](../src/audio_signature.cpp#L265) — `std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:320`](../src/audio_signature.cpp#L320) — `std::optional<std::string> signature_from_mono(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:327`](../src/audio_signature.cpp#L327) — `std::optional<std::vector<float>> decode_centre_window(const std::string& path)`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:151`](../tests/test_audio_signature.cpp#L151) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
- [`tests/test_audio_signature.cpp:236`](../tests/test_audio_signature.cpp#L236) — `Unknown`
- [`tests/test_audio_signature.cpp:263`](../tests/test_audio_signature.cpp#L263) — `TempWav w("centred300");`
- [`tests/test_audio_signature.cpp:301`](../tests/test_audio_signature.cpp#L301) — `Unknown`
- [`tests/test_audio_signature.cpp:314`](../tests/test_audio_signature.cpp#L314) — `kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));`
- [`tests/test_audio_signature.cpp:328`](../tests/test_audio_signature.cpp#L328) — `std::vector<float> a(kWindowSamples / 50);`
- [`tests/test_audio_signature.cpp:344`](../tests/test_audio_signature.cpp#L344) — `return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());`
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### IR-005
**Locations:** 6
- [`src/audio_bindings.cpp:3`](../src/audio_bindings.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:138`](../tests/test_audio_signature.cpp#L138) — `Unknown`
### IR-006
**Locations:** 1
- [`scripts/run_from_jellyfin.py:4`](../scripts/run_from_jellyfin.py#L4) — `Unknown`
### IR-007
**Locations:** 8
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:327`](../src/audio_signature.cpp#L327) — `std::optional<std::vector<float>> decode_centre_window(const std::string& path)`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:201`](../tests/test_audio_signature.cpp#L201) — `TempWav w("short30");`
- [`tests/test_audio_signature.cpp:219`](../tests/test_audio_signature.cpp#L219) — `TempWav w("exact120");`
- [`tests/test_audio_signature.cpp:229`](../tests/test_audio_signature.cpp#L229) — `TempWav w("exact120");`
### IR-008
**Locations:** 7
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:320`](../src/audio_signature.cpp#L320) — `std::optional<std::string> signature_from_mono(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
### IT-001
**Locations:** 1
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
### PR-002
**Locations:** 4
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
- [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an`
- [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown`
- [`scripts/validation/ground_truth.py:24`](../scripts/validation/ground_truth.py#L24) — `Unknown`
### PR-004
**Locations:** 1
- [`src/main.cpp:3`](../src/main.cpp#L3) — `Unknown`
### SR-001
**Locations:** 60
- [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;`
- [`src/build_gallery.cpp:83`](../src/build_gallery.cpp#L83) — `Unknown`
- [`src/config.hpp:54`](../src/config.hpp#L54) — `Unknown`
- [`src/gallery/embedder_stamp.cpp:1`](../src/gallery/embedder_stamp.cpp#L1) — `Unknown`
- [`src/gallery/embedder_stamp.hpp:2`](../src/gallery/embedder_stamp.hpp#L2) — `Unknown`
- [`src/gallery/gallery_builder.cpp:45`](../src/gallery/gallery_builder.cpp#L45) — `ActorGallery build_gallery(const BuildConfig& cfg)`
- [`src/gallery/gallery_calibration.hpp:80`](../src/gallery/gallery_calibration.hpp#L80) — `struct GalleryCalibrationStats`
- [`src/gallery/gallery_calibration.hpp:145`](../src/gallery/gallery_calibration.hpp#L145) — `std::vector<bool> actor_eligible(n_actors, false);`
- [`src/gallery/gallery_calibration.hpp:294`](../src/gallery/gallery_calibration.hpp#L294) — `Unknown`
- [`src/gallery/gallery_report.hpp:2`](../src/gallery/gallery_report.hpp#L2) — `Unknown`
- [`src/gallery/gallery_report.hpp:52`](../src/gallery/gallery_report.hpp#L52) — `struct GalleryBuildAudit`
- [`src/gallery/gallery_report.hpp:73`](../src/gallery/gallery_report.hpp#L73) — `struct GalleryReport`
- [`src/gallery/gallery_report.hpp:154`](../src/gallery/gallery_report.hpp#L154) — `inline GalleryReport build_gallery_report(const ActorGallery& gallery,`
- [`src/gallery/gallery_report.hpp:295`](../src/gallery/gallery_report.hpp#L295) — `inline nlohmann::json gallery_report_to_json(const GalleryReport& r)`
- [`src/gallery/gallery_report.hpp:361`](../src/gallery/gallery_report.hpp#L361) — `inline GalleryReport gallery_report_from_json(const nlohmann::json& j)`
- [`src/gallery/gallery_report.hpp:446`](../src/gallery/gallery_report.hpp#L446) — `inline void save_gallery_report(const std::string& path, const GalleryReport& r)`
- [`src/gallery/gallery_report.hpp:454`](../src/gallery/gallery_report.hpp#L454) — `inline GalleryReport load_gallery_report(const std::string& path)`
- [`src/gallery/gallery_report.hpp:464`](../src/gallery/gallery_report.hpp#L464) — `return gallery_report_from_json(j);`
- [`src/gallery/gallery_store.cpp:82`](../src/gallery/gallery_store.cpp#L82) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/gallery/gallery_store.cpp:167`](../src/gallery/gallery_store.cpp#L167) — `H5::DataSpace scalar(H5S_SCALAR);`
- [`src/gallery/gallery_store.cpp:219`](../src/gallery/gallery_store.cpp#L219) — `Unknown`
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:188`](../src/main.cpp#L188) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:148`](../src/types.hpp#L148) — `struct Actor`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");`
- [`tests/test_gallery_store.cpp:240`](../tests/test_gallery_store.cpp#L240) — `TempFile tf("gallery_json_stamp.json");`
- [`tests/test_gallery_store.cpp:252`](../tests/test_gallery_store.cpp#L252) — `Unknown`
- [`tests/test_gallery_store.cpp:268`](../tests/test_gallery_store.cpp#L268) — `Unknown`
- [`tests/test_gallery_store.cpp:300`](../tests/test_gallery_store.cpp#L300) — `Unknown`
- [`tests/test_gallery_store.cpp:309`](../tests/test_gallery_store.cpp#L309) — `Unknown`
- [`tests/test_gallery_store.cpp:333`](../tests/test_gallery_store.cpp#L333) — `Unknown`
- [`tests/test_gallery_store.cpp:348`](../tests/test_gallery_store.cpp#L348) — `Unknown`
- [`tests/test_gallery_store.cpp:367`](../tests/test_gallery_store.cpp#L367) — `Unknown`
- [`tests/test_gallery_store.cpp:384`](../tests/test_gallery_store.cpp#L384) — `TempFile tf("fake_model.onnx");`
- [`scripts/filter_gallery.py:80`](../scripts/filter_gallery.py#L80) — `if actor_jellyfin_id(a) in cast_ids]`
- [`scripts/make_gallery.py:181`](../scripts/make_gallery.py#L181) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:448`](../scripts/make_jellyfin_gallery.py#L448) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:456`](../scripts/make_jellyfin_gallery.py#L456) — `Unknown`
- [`scripts/movienet_eval.py:65`](../scripts/movienet_eval.py#L65) — `with open(args.gt) as f:`
- [`scripts/optimizer/fetch_missing_actors.py:62`](../scripts/optimizer/fetch_missing_actors.py#L62) — `def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,`
- [`scripts/optimizer/fetch_missing_actors.py:109`](../scripts/optimizer/fetch_missing_actors.py#L109) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:124`](../scripts/optimizer/fetch_missing_actors.py#L124) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/optimize.py:186`](../scripts/optimizer/optimize.py#L186) — `Unknown`
- [`scripts/optimizer/optimize.py:202`](../scripts/optimizer/optimize.py#L202) — `if not Path(f["dump"]).exists():`
- [`scripts/optimizer/reembed_gallery.py:62`](../scripts/optimizer/reembed_gallery.py#L62) — `for i, a in enumerate(ref["actors"], 1):`
- [`scripts/optimizer/replay.py:113`](../scripts/optimizer/replay.py#L113) — `Unknown`
- [`scripts/optimizer/replay.py:253`](../scripts/optimizer/replay.py#L253) — `Unknown`
- [`scripts/run_from_jellyfin.py:4`](../scripts/run_from_jellyfin.py#L4) — `Unknown`
- [`scripts/sae_embed_loader.py:23`](../scripts/sae_embed_loader.py#L23) — `def resolve_arcface(models_dir: str, arcface: str \| None = None) -> str:`
- [`scripts/sae_gallery.py:171`](../scripts/sae_gallery.py#L171) — `if not _stamp_empty(embedder):`
- [`scripts/sae_gallery.py:200`](../scripts/sae_gallery.py#L200) — `for a in range(len(offset)):`
- [`scripts/sae_stamp.py:3`](../scripts/sae_stamp.py#L3) — `Unknown`
- [`scripts/stamp_gallery.py:4`](../scripts/stamp_gallery.py#L4) — `Unknown`
### SR-002
**Locations:** 32
- [`src/config.hpp:44`](../src/config.hpp#L44) — `Unknown`
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:299`](../src/main.cpp#L299) — `Unknown`
- [`src/main.cpp:309`](../src/main.cpp#L309) — `Unknown`
- [`src/main.cpp:379`](../src/main.cpp#L379) — `return run_net(std::move(net));`
- [`src/main.cpp:411`](../src/main.cpp#L411) — `Unknown`
- [`src/nodes/camera_position_change_detector_node.hpp:30`](../src/nodes/camera_position_change_detector_node.hpp#L30) — `struct CameraPositionChangeDetectorFunc`
- [`src/nodes/embedder_node.hpp:21`](../src/nodes/embedder_node.hpp#L21) — `struct EmbedderFunc`
- [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown`
- [`src/nodes/face_detector_node.hpp:47`](../src/nodes/face_detector_node.hpp#L47) — `private:`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:125`](../src/nodes/identity_matcher_node.hpp#L125) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:166`](../src/nodes/identity_matcher_node.hpp#L166) — `std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);`
- [`src/nodes/identity_matcher_node.hpp:272`](../src/nodes/identity_matcher_node.hpp#L272) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:63`](../src/nodes/result_sink_node.hpp#L63) — `void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
- [`src/nodes/scene_boundary_annotator_node.hpp:2`](../src/nodes/scene_boundary_annotator_node.hpp#L2) — `Unknown`
- [`src/nodes/scene_detector_node.hpp:36`](../src/nodes/scene_detector_node.hpp#L36) — `static constexpr std::string_view label() { return "scene_detector"; }`
- [`src/nodes/scene_detector_node.hpp:110`](../src/nodes/scene_detector_node.hpp#L110) — `void flush_remaining()`
- [`src/nodes/scene_detector_node.hpp:144`](../src/nodes/scene_detector_node.hpp#L144) — `void write_output()`
- [`src/scene_boundaries.hpp:2`](../src/scene_boundaries.hpp#L2) — `Unknown`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown`
### SR-003
**Locations:** 7
- [`src/audio_bindings.cpp:3`](../src/audio_bindings.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`src/config.hpp:20`](../src/config.hpp#L20) — `struct Config`
- [`src/nodes/result_sink_node.hpp:2`](../src/nodes/result_sink_node.hpp#L2) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:122`](../src/nodes/result_sink_node.hpp#L122) — `void write_output()`
### SR-005
**Locations:** 9
- [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown`
- [`src/gallery/gallery_store.hpp:15`](../src/gallery/gallery_store.hpp#L15) — `Unknown`
- [`src/gallery/track_gallery.hpp:122`](../src/gallery/track_gallery.hpp#L122) — `void forget(int track_id) { tracks_.erase(track_id); }`
- [`src/gallery/track_gallery.hpp:132`](../src/gallery/track_gallery.hpp#L132) — `void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }`
- [`src/gallery/track_gallery.hpp:161`](../src/gallery/track_gallery.hpp#L161) — `struct TrackState`
- [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);`
- [`src/nodes/identity_matcher_node.hpp:147`](../src/nodes/identity_matcher_node.hpp#L147) — `MatchedSceneFrame operator()(TrackedSceneFrame tf)`
- [`src/nodes/identity_matcher_node.hpp:285`](../src/nodes/identity_matcher_node.hpp#L285) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### UT-001
**Locations:** 1
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### UT-101
**Locations:** 5
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:138`](../tests/test_audio_signature.cpp#L138) — `Unknown`
- [`tests/test_audio_signature.cpp:151`](../tests/test_audio_signature.cpp#L151) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
### UT-102
**Locations:** 5
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:201`](../tests/test_audio_signature.cpp#L201) — `TempWav w("short30");`
- [`tests/test_audio_signature.cpp:219`](../tests/test_audio_signature.cpp#L219) — `TempWav w("exact120");`
- [`tests/test_audio_signature.cpp:229`](../tests/test_audio_signature.cpp#L229) — `TempWav w("exact120");`
- [`tests/test_audio_signature.cpp:236`](../tests/test_audio_signature.cpp#L236) — `Unknown`
### UT-103
**Locations:** 2
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:263`](../tests/test_audio_signature.cpp#L263) — `TempWav w("centred300");`
### UT-104
**Locations:** 5
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:301`](../tests/test_audio_signature.cpp#L301) — `Unknown`
- [`tests/test_audio_signature.cpp:314`](../tests/test_audio_signature.cpp#L314) — `kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));`
- [`tests/test_audio_signature.cpp:328`](../tests/test_audio_signature.cpp#L328) — `std::vector<float> a(kWindowSamples / 50);`
- [`tests/test_audio_signature.cpp:344`](../tests/test_audio_signature.cpp#L344) — `return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());`
### UT-105
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### UT-106
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### UT-107
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### UT-108
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### VR-001
**Locations:** 2
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
### VR-002
**Locations:** 2
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
- [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an`
### VR-003
**Locations:** 1
- [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown`
### VR-004
**Locations:** 1
- [`scripts/validation/ground_truth.py:24`](../scripts/validation/ground_truth.py#L24) — `Unknown`
### VR-014
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
experiments/.gitignore
+11
View File
@@ -11,6 +11,17 @@ manifests/
trajectories/
results/
# Cross-source identification study: source clips and the hand-sorted face
# crops. The sorting is human ground truth and expensive to redo, so it goes to
# the artifact registry rather than being regenerated — push it once sorted.
xsource/clips/
xsource/labelling/
xsource/frames/
xsource/cache/
xsource/results_*.json
xsource/failure_analysis.json
xsource/*.jpg
# Raw run logs and scratch scripts (regenerated by every run).
_scratch/
experiments/xsource/README.md
+92
View File
@@ -0,0 +1,92 @@
# xsource — cross-source identification probe (VR-013)
Gallery from **one** recording, probes from **another**, swept over the probe's
input resolution. Complements VR-005, which asked the same question over gallery
mugshots: that one degrades an already-aligned 112×112 crop, holding alignment
perfect, so it isolates the embedder. This one downscales the **whole frame**
before the detector, so detection and landmark regression degrade with it.
Corpus: two Pexels clips of one shoot (4096×2160, 25 fps), four people, all four
present in both.
## Getting the data
Clips, frames and hand-sorted crops are gitignored; they live in the artifact
registry.
scripts/artifacts/pull_artifacts.sh xsource # clips + labelling, frames regenerated
scripts/artifacts/push_artifacts.sh xsource # after correcting labels
Pulling fetches the two clips and the hand-sorted crops, then regenerates the
frames with ffmpeg — ~320 MB of PNG that is deterministic from the clips, so it
is not worth shipping. Extraction settings are pinned in the pull script because
the manifests key on frame filenames *and* on detection order within each frame;
`verify_labels.py` runs at the end and will fail loudly if they drift.
Pull never overwrites an existing `labelling/`. That directory is human ground
truth — somebody looked at all 167 crops and put each one in a folder — and it
is the expensive part of this study, so push it once corrected.
Clips are Pexels-licensed: free to use, no attribution required, but not
CC or MIT. Fine as a frozen CI artifact on private infrastructure; do not
redistribute them as stock content.
## Scripts
| script | does |
|---|---|
| `dump_faces.py` | detect every face, write a context crop per detection + a manifest |
| `redraw_boxes.py` | redraw those crops with the detection boxed, in place |
| `propose_labels.py` | propose labels for one clip from another clip's hand-sorted folders |
| `make_review_site.py` | local `review.html` — current label, crop, better match, correct and export |
| `apply_corrections.py` | apply the exported `corrections.json` |
| `verify_labels.py` | integrity gate: index consistency, duplicates, separation. Exits non-zero on failure |
| `resolution_sweep.py` | the VR-013 measurement |
| `failure_analysis.py` | what explains the misses — pose, size, blur, detector confidence |
| `landmark_voting.py` | average SCRFD's overlapping detections instead of discarding them |
| `pose_label.py` | mesh-estimated head pose, for hand correction (feeds VR-012) |
Everything drives the shipped C++ through `sae_embed`; nothing reimplements
detection, alignment, the embedder or the calibration. Scoring goes through the
production gallery sigmoid — never a raw cosine (AR-024).
LD_PRELOAD=/usr/lib/libcudnn_cnn.so.9 python3 resolution_sweep.py
The preload is needed while ORT's CUDA provider looks for
`cudnnGetConvolutionBackwardDataAlgorithm_v7`, which cuDNN 9 moved into
`libcudnn_cnn.so.9` behind a dispatch stub. Without it everything silently falls
back to CPU.
## What it found
**Resolution is not the binding constraint here.** TPI holds ~41–47% from 4096×2160
down to ~45 px faces, then falls: 23 px → 26%, 18 px → 12%, 14 px → 1.5%. Holding
90% of the plateau needs roughly 50 px end to end, against VR-005's ~22 px — the
gap is detection and landmark error, which VR-005 excludes by construction.
**FPI is 0.0% at every scale.** Resolution loss goes entirely to TBI: the pipeline
stops naming people rather than naming the wrong one.
**The ceiling is cross-view, not resolution.** Every person matches themselves
strongly *within* a recording (sim 0.55–0.85) and collapses *across* the two
(0.14–0.45, threshold 0.335). Only the person with frontal **gallery** references
identified reliably, whatever their probe pose — so the lever is gallery pose
coverage (`docs/pose-expansion.md`), not a better landmark model.
**Landmark voting helps.** SCRFD predicts each face from several anchors and NMS
discards all but one, throwing away a median of 3 landmark estimates per face.
Averaging them, weighted by confidence, lifts cross-clip TPI 41% → 49% for one
forward pass and no extra model. A MediaPipe mesh as landmark source went the
other way (41% → 16%): more stable within a recording, but a ring centroid is not
the annotated landmark ArcFace was trained on, and the embedder punishes the
off-distribution crop.
## Reading these numbers
Four identities, 70 probes, one shoot. The ~47% plateau is pose, not resolution —
half these faces are turned away and never clear threshold at any scale, so the
absolute rates say little and the *shape* is the result. Both clips contain all
four people, so there is no out-of-gallery class and the 10×-weighted out-of-cast
misID is **untested** here; holding one identity out of the gallery would fix
that. And the resolution curve is dominated by the single subject whose gallery
references are frontal.
experiments/xsource/apply_corrections.py
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env python3
"""Apply corrections.json exported from review.html.
python3 apply_corrections.py ~/Downloads/corrections.json [--dry-run]
Moves each crop to the folder you chose. "discard" goes to labelling/<clip>/discard/,
which the sweep ignores — nothing is deleted, so a misclick is recoverable.
Refuses to move a file it cannot find exactly once, rather than guessing: a
half-applied correction set would put a crop in two folders and quietly
duplicate a label.
"""
import sys, json, glob, os, shutil
if len(sys.argv) < 2:
sys.exit(__doc__)
path = sys.argv[1]
DRY = "--dry-run" in sys.argv
corr = json.load(open(path))
if not corr:
sys.exit("no corrections in that file")
moved = skipped = 0
for fname, c in corr.items():
clip, to = c["clip"], c["to"]
hits = glob.glob(f"labelling/{clip}/**/{fname}", recursive=True)
if len(hits) != 1:
print(f"[skip] {fname}: found {len(hits)} copies, expected 1")
skipped += 1
continue
src = hits[0]
dst_dir = f"labelling/{clip}/{to}"
dst = f"{dst_dir}/{fname}"
if os.path.abspath(src) == os.path.abspath(dst):
continue
print(f"{'would move' if DRY else 'move'} {c['from']} -> {to}: {fname}")
if not DRY:
os.makedirs(dst_dir, exist_ok=True)
shutil.move(src, dst)
moved += 1
print(f"\n{moved} moved, {skipped} skipped{' (dry run)' if DRY else ''}")
if not DRY and moved:
print("re-run verify_labels.py to confirm the set is still consistent")
experiments/xsource/dump_faces.py
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env python3
"""Dump face crops from both clips for hand-labelling.
Writes labelling/<clip>/unsorted/<name>.jpg — a context crop around each
detection, big enough to recognise a person by eye. Move them into
labelling/<clip>/person_A/, person_B/, ... and the sweep reads those folders as
ground truth.
Filenames carry a cNN_ cluster-hint prefix so visually similar faces sort next
to each other in a file manager. The hint is only an ordering convenience —
the folder you drop a file into is what counts, and the sweep never reads the
prefix.
Detection and alignment run through the shipped C++ (sae_embed). Every crop
keeps its clip, frame and native-resolution bbox in manifest.json, so probe
detections at reduced scale can be tied back to a labelled face geometrically,
by position, rather than by embedding similarity — which would be circular.
"""
import sys, glob, json, os, shutil
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
M = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/models/"
CLIPS = ["5157339", "5157344"]
MIN_PX = 60
CTX = 256 # context-crop side, for human recognisability
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "arcface_w600k_r50.onnx",
conf=0.5, nms=0.4, max_side=0)
for clip in CLIPS:
out_dir = f"labelling/{clip}/unsorted"
if os.path.isdir(f"labelling/{clip}"):
print(f"[skip] labelling/{clip} exists — not overwriting your sorting",
file=sys.stderr)
continue
os.makedirs(out_dir, exist_ok=True)
entries = []
for p in sorted(glob.glob(f"pex/d{clip}_*.png")):
frame = p.rsplit("_", 1)[-1].split(".")[0]
img = cv2.imread(p)
for i, d in enumerate(eng.detect(img)):
x, y, w, h = d.bbox
if min(w, h) < MIN_PX:
continue
lm = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
emb = np.asarray(eng.embed_crop(crop), dtype=np.float32)
pad = int(0.5 * max(w, h))
x0, y0 = max(0, int(x) - pad), max(0, int(y) - pad)
x1, y1 = min(img.shape[1], int(x + w) + pad), min(img.shape[0], int(y + h) + pad)
ctx = cv2.resize(img[y0:y1, x0:x1], (CTX, CTX))
entries.append({"clip": clip, "frame": frame, "idx": i,
"bbox": [float(x), float(y), float(w), float(h)],
"px": float(min(w, h)), "conf": float(d.confidence),
"emb": emb, "ctx": ctx})
# cluster hint only — greedy, purely to group similar faces in the file list
E = np.stack([e["emb"] for e in entries])
hint = -np.ones(len(entries), int)
k = 0
for i in range(len(entries)):
if hint[i] >= 0:
continue
hint[i] = k
for j in range(i + 1, len(entries)):
if hint[j] < 0 and float(E[i] @ E[j]) > 0.5:
hint[j] = k
k += 1
manifest = []
for e, h in zip(entries, hint):
name = f"c{h:02d}_{e['clip']}_f{e['frame']}_i{e['idx']}_{int(e['px'])}px.jpg"
cv2.imwrite(f"{out_dir}/{name}", e["ctx"])
manifest.append({k: v for k, v in e.items() if k not in ("emb", "ctx")}
| {"file": name, "cluster_hint": int(h)})
json.dump(manifest, open(f"labelling/{clip}/manifest.json", "w"), indent=1)
print(f"[{clip}] {len(manifest)} crops in {out_dir}, {k} cluster hints, "
f"face px {min(m['px'] for m in manifest):.0f}–{max(m['px'] for m in manifest):.0f}",
file=sys.stderr)
experiments/xsource/failure_analysis.py
+174
View File
@@ -0,0 +1,174 @@
#!/usr/bin/env python3
"""What explains the misses? Head pose, face size, blur, detector confidence.
For every hand-labelled probe face, computes the calibrated probability against
its OWN gallery entry — so a low value is a false negative, not a mistake about
who it is — and pairs it with covariates that might explain the failure.
Head pose comes from solvePnP of the 5 landmarks against a canonical 3D face,
giving yaw/pitch/roll in degrees.
CAVEAT, and it matters: the pose estimate is derived from the same 5
landmarks the alignment uses. Where those landmarks are unreliable the pose
estimate is unreliable too, and both degrade for the same reason. So this
can show that failures concentrate at high yaw; it cannot cleanly separate
"the head was turned" from "the landmarks were wrong because the head was
turned". Those are the same physical cause, but not the same fix — the
first argues for gallery pose coverage, the second for a better landmark
source.
A sanity check is printed first: pose is estimated per person, and if it does
not recover what is visible in the review sheets (one subject frontal, another
in profile, another looking down) then the estimate is not worth reading.
Similarities go through the production gallery sigmoid, never compared raw.
"""
import sys, glob, json, os
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
GALLERY_CLIP, PROBE_CLIP = "5157344", "5157339"
PROB_THRESHOLD = 0.754
# Canonical 3D face, ordered as types.hpp:60 —
# [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth.
# The subject's right eye sits to the LEFT in image space, hence the negative X.
FACE_3D = np.array([
(-34.0, 35.0, -28.0),
( 34.0, 35.0, -28.0),
( 0.0, 0.0, 0.0),
(-26.0, -32.0, -25.0),
( 26.0, -32.0, -25.0),
], dtype=np.float64)
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.5, nms=0.4, max_side=0)
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
def head_pose(lm, w, h):
"""yaw, pitch, roll in degrees. Focal length assumed = image width."""
cam = np.array([[w, 0, w / 2], [0, w, h / 2], [0, 0, 1]], dtype=np.float64)
ok, rvec, _ = cv2.solvePnP(FACE_3D, lm.astype(np.float64), cam, None,
flags=cv2.SOLVEPNP_EPNP)
if not ok:
return None
R, _ = cv2.Rodrigues(rvec)
sy = np.sqrt(R[0, 0] ** 2 + R[1, 0] ** 2)
if sy > 1e-6:
pitch = np.degrees(np.arctan2(-R[2, 0], sy))
yaw = np.degrees(np.arctan2(R[1, 0], R[0, 0]))
roll = np.degrees(np.arctan2(R[2, 1], R[2, 2]))
else:
pitch = np.degrees(np.arctan2(-R[2, 0], sy)); yaw = 0.0
roll = np.degrees(np.arctan2(-R[1, 2], R[1, 1]))
# solvePnP's yaw wraps near +/-180 for a face pointing at the camera;
# fold it to a "degrees away from frontal" magnitude.
yaw = ((yaw + 180) % 360) - 180
if abs(yaw) > 90:
yaw = np.sign(yaw) * (180 - abs(yaw))
return yaw, pitch, roll
def collect(clip):
lab = {os.path.basename(p): os.path.basename(os.path.dirname(p))
for p in glob.glob(f"labelling/{clip}/*/*.jpg")
if os.path.basename(os.path.dirname(p)) not in ("discard", "unsorted")}
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
rows = []
for frame in sorted({man[f]["frame"] for f in lab}):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
dets = eng.detect(img)
H, W = img.shape[:2]
for f, person in lab.items():
m = man[f]
if m["frame"] != frame or m["idx"] >= len(dets):
continue
d = dets[m["idx"]]
lm = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
pose = head_pose(lm, W, H)
x, y, w, h = d.bbox
g = cv2.cvtColor(np.asarray(crop), cv2.COLOR_BGR2GRAY)
rows.append({
"person": person, "px": float(min(w, h)), "conf": float(d.confidence),
"yaw": pose[0] if pose else np.nan, "pitch": pose[1] if pose else np.nan,
"roll": pose[2] if pose else np.nan,
"blur": float(cv2.Laplacian(g, cv2.CV_64F).var()),
"emb": np.asarray(eng.embed_crop(crop), dtype=np.float32)})
return rows
gal_rows = collect(GALLERY_CLIP)
prb_rows = collect(PROBE_CLIP)
gal = {}
for r in gal_rows:
gal.setdefault(r["person"], []).append(r["emb"])
gal = {p: np.stack(v) for p, v in gal.items()}
for r in prb_rows:
if r["person"] in gal:
s = float((gal[r["person"]] @ r["emb"]).max()) # best-of-N, own actor
r["p"] = cal.probability(s)
r["sim"] = s
else:
r["p"] = np.nan
rows = [r for r in prb_rows if not np.isnan(r.get("p", np.nan))]
print(f"[data] {len(rows)} labelled probe faces with a gallery entry\n", file=sys.stderr)
# ── sanity check: does the pose estimate recover what the sheets show? ───────
print("pose by person (does this match the review sheets?)")
print(f"{'person':>7}{'n':>5}{'|yaw| med':>11}{'pitch med':>11}{'P med':>8}{'hit rate':>10}")
for p in sorted({r['person'] for r in rows}):
sub = [r for r in rows if r["person"] == p]
print(f"{p:>7}{len(sub):>5}"
f"{np.median([abs(r['yaw']) for r in sub]):>11.1f}"
f"{np.median([r['pitch'] for r in sub]):>11.1f}"
f"{np.median([r['p'] for r in sub]):>8.3f}"
f"{100*np.mean([r['p'] > PROB_THRESHOLD for r in sub]):>9.0f}%")
# ── P binned by each covariate ───────────────────────────────────────────────
def binned(name, key, edges, fmt="{:.0f}"):
print(f"\nP(match) by {name}")
print(f"{'bin':>16}{'n':>5}{'P med':>9}{'hit rate':>10}{'sim med':>9}")
vals = np.array([r[key] for r in rows])
for lo, hi in zip(edges[:-1], edges[1:]):
sub = [r for r, v in zip(rows, vals) if lo <= v < hi]
if not sub:
continue
lbl = f"{fmt.format(lo)}–{fmt.format(hi)}"
print(f"{lbl:>16}{len(sub):>5}"
f"{np.median([r['p'] for r in sub]):>9.3f}"
f"{100*np.mean([r['p'] > PROB_THRESHOLD for r in sub]):>9.0f}%"
f"{np.median([r['sim'] for r in sub]):>9.3f}")
for r in rows:
r["absyaw"] = abs(r["yaw"])
r["abspitch"] = abs(r["pitch"])
binned("|yaw| (deg from frontal)", "absyaw", [0, 10, 20, 30, 45, 60, 91])
binned("|pitch| (deg)", "abspitch", [0, 10, 20, 30, 45, 91])
binned("face size (px)", "px", [0, 130, 150, 175, 200, 400])
binned("blur (laplacian var)", "blur", [0, 50, 150, 400, 1000, 1e9])
binned("detector confidence", "conf", [0.5, 0.6, 0.7, 0.8, 0.9, 1.01], "{:.2f}")
# ── how much does each covariate actually explain? ───────────────────────────
print("\nSpearman rank correlation with P(match):")
def spearman(a, b):
ra = np.argsort(np.argsort(a)); rb = np.argsort(np.argsort(b))
return float(np.corrcoef(ra, rb)[0, 1])
P = np.array([r["p"] for r in rows])
for key, label in [("absyaw", "|yaw|"), ("abspitch", "|pitch|"), ("px", "face px"),
("blur", "blur"), ("conf", "detector conf")]:
v = np.array([r[key] for r in rows])
print(f" {label:>14}: {spearman(v, P):+.3f}")
json.dump([{k: v for k, v in r.items() if k != "emb"} for r in rows],
open("failure_analysis.json", "w"), indent=1, default=float)
experiments/xsource/landmark_voting.py
+165
View File
@@ -0,0 +1,165 @@
#!/usr/bin/env python3
"""Landmark voting: average SCRFD's overlapping detections instead of discarding them.
SCRFD predicts a face from many anchors; NMS keeps the single highest-scoring
box and throws the rest away. Each discarded box carries its own 5-landmark
estimate of the SAME face, so the survivors are one sample from a distribution
we could be averaging over.
baseline conf 0.50, nms 0.40 — the shipped settings, one box per face
voted conf 0.30, nms 0.90 — duplicates survive, then grouped by IoU and
the 5 landmarks averaged, weighted by detection confidence
Why this is worth trying when the mesh failed: the mesh moved the landmarks off
the definition ArcFace was trained on (a lip-ring centroid is not an annotated
mouth corner), and the embedder punished it. A confidence-weighted mean of
SCRFD's OWN landmark predictions is the same kind of point, just with less
variance — it should stay on-distribution while being steadier.
Scored on cross-clip identification through the production sigmoid, which is
the thing that actually broke. Raw similarity shown only to locate the
threshold; it decides nothing.
LD_PRELOAD=/usr/lib/libcudnn_cnn.so.9 python3 landmark_voting.py
"""
import sys, glob, json, os
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed # before cv2 — see alignment_compare.py
import numpy as np
import cv2
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
CLIPS = ["5157344", "5157339"]
PROB_THRESHOLD = 0.754
GROUP_IOU = 0.55 # detections overlapping this much are the same face
MATCH_IOU = 0.35 # tie a detection to the hand-labelled face
base_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.5, nms=0.4, max_side=0)
# Same models, looser suppression: keep the duplicates NMS would have removed.
vote_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.3, nms=0.9, max_side=0)
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
def iou(a, b):
ax, ay, aw, ah = a; bx, by, bw, bh = b
x0, y0 = max(ax, bx), max(ay, by)
x1, y1 = min(ax + aw, bx + bw), min(ay + ah, by + bh)
if x1 <= x0 or y1 <= y0:
return 0.0
i = (x1 - x0) * (y1 - y0)
return i / (aw * ah + bw * bh - i)
def vote(dets):
"""Group overlapping detections, return (bbox, landmarks, conf, n_votes)."""
items = sorted(dets, key=lambda d: -d.confidence)
used, out = [False] * len(items), []
for i, d in enumerate(items):
if used[i]:
continue
grp = [d]
used[i] = True
for j in range(i + 1, len(items)):
if not used[j] and iou(list(d.bbox), list(items[j].bbox)) >= GROUP_IOU:
used[j] = True
grp.append(items[j])
w = np.array([g.confidence for g in grp], dtype=np.float32)
w = w / w.sum()
lms = np.stack([np.array(g.landmarks, dtype=np.float32).reshape(5, 2) for g in grp])
bxs = np.stack([np.array(list(g.bbox), dtype=np.float32) for g in grp])
out.append((( w[:, None] * bxs).sum(0), (w[:, None, None] * lms).sum(0),
float(grp[0].confidence), len(grp)))
return out
def collect(clip):
lab = {os.path.basename(p): os.path.basename(os.path.dirname(p))
for p in glob.glob(f"labelling/{clip}/*/*.jpg")
if os.path.basename(os.path.dirname(p)) not in ("discard", "unsorted")}
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
rows, votes = [], []
for frame in sorted({man[f]["frame"] for f in lab}):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
base = base_eng.detect(img)
voted = vote(vote_eng.detect(img))
for fname, person in lab.items():
m = man[fname]
if m["frame"] != frame or m["idx"] >= len(base):
continue
d = base[m["idx"]]
lm5 = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
c_b = sae_embed.align_face(img, lm5)
# the voted group covering the same face
best, best_v = None, 0.0
for bbox, lms, conf, n in voted:
v = iou(list(bbox), list(d.bbox))
if v > best_v:
best_v, best = v, (lms, n)
c_v = None
if best and best_v >= MATCH_IOU:
c_v = sae_embed.align_face(img, best[0].astype(np.float32))
votes.append(best[1])
rec = {"person": person}
rec["base"] = np.asarray(base_eng.embed_crop(c_b), np.float32) if c_b is not None else None
rec["voted"] = np.asarray(base_eng.embed_crop(c_v), np.float32) if c_v is not None else None
rows.append(rec)
return rows, votes
data, allv = {}, []
for c in CLIPS:
data[c], v = collect(c)
allv += v
print(f"[{c}] {len(data[c])} crops", file=sys.stderr)
print(f"[voting] group size: median {np.median(allv):.0f}, "
f"mean {np.mean(allv):.1f}, max {max(allv)} detections averaged per face",
file=sys.stderr)
GAL, PRB = "5157344", "5157339"
print(f"\ngallery {GAL} -> probe {PRB}, P>{PROB_THRESHOLD}\n")
print(f"{'align':>8}{'person':>8}{'n_gal':>7}{'n_prb':>7}"
f"{'within-clip':>13}{'cross-clip':>12}{'hit rate':>10}")
summary = {}
for key in ("base", "voted"):
gal, prb = {}, {}
for r in data[GAL]:
if r[key] is not None:
gal.setdefault(r["person"], []).append(r[key])
for r in data[PRB]:
if r[key] is not None:
prb.setdefault(r["person"], []).append(r[key])
gal = {p: np.stack(v) for p, v in gal.items()}
prb = {p: np.stack(v) for p, v in prb.items()}
hits = tot = 0
for p in sorted(set(gal) & set(prb)):
pp = prb[p] @ prb[p].T
np.fill_diagonal(pp, -1)
within = float(np.median(pp.max(axis=1))) if len(pp) > 1 else float("nan")
cross = float(np.median((gal[p] @ prb[p].T).max(axis=0)))
h = 0
for e in prb[p]:
bp, bn = 0.0, None
for q in gal:
v = cal.probability(float((gal[q] @ e).max()))
if v > bp:
bp, bn = v, q
if bp > PROB_THRESHOLD and bn == p:
h += 1
hits += h; tot += len(prb[p])
print(f"{key:>8}{p:>8}{len(gal[p]):>7}{len(prb[p]):>7}"
f"{cal.probability(within):>6.3f}/{within:<6.3f}"
f"{cal.probability(cross):>6.3f}/{cross:<5.3f}{100*h/len(prb[p]):>9.0f}%")
summary[key] = (hits, tot)
print(f"{key:>8}{'ALL':>8}{'':>14}{'':>25}{100*hits/max(tot,1):>9.0f}%\n")
hb, tb = summary["base"]; hv, tv = summary["voted"]
print(f"voting vs baseline: {100*hv/max(tv,1) - 100*hb/max(tb,1):+.1f} points "
f"of cross-clip TPI ({hb}/{tb} -> {hv}/{tv})")
experiments/xsource/make_review_site.py
+235
View File
@@ -0,0 +1,235 @@
#!/usr/bin/env python3
"""Build labelling/review.html — a local page for correcting the labels.
One row per crop, ordered most-suspicious first:
left the person it is currently filed under (medoid of that person's
hand-sorted crops, so the reference is one you trust)
centre the crop under review — context with the detection boxed, and
beneath it the 112x112 the embedder actually receives
right the person it matches better, if any, with both probabilities
Pick a destination per row, then Export to download corrections.json and apply
it with apply_corrections.py. Nothing is moved by this script.
Self-contained: images are inlined as data URIs and the page is opened from
disk, so no server runs and no face crop leaves the machine.
Ordering is by P(other) - P(self), both from the global gallery sigmoid, so
rows where the evidence disagrees with the label float to the top and the
agreement cases sink. It is a review order, not a verdict — you are the
arbiter, which is the whole point of labelling by hand.
"""
import sys, glob, json, os, base64
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
EMBEDDER = M + "LVFace-B_Glint360K.onnx"
GALLERY = ROOT + "gallery_lvface.h5"
REF_CLIP = "5157344" # the clip sorted by hand — reference faces come from here
CLIPS = ["5157344", "5157339"]
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=EMBEDDER, conf=0.5, nms=0.4, max_side=0)
cal = sae_embed.gallery_calibration(GALLERY)
def b64(img, size, q=72):
img = cv2.resize(img, (size, size))
ok, buf = cv2.imencode(".jpg", img, [cv2.IMWRITE_JPEG_QUALITY, q])
return "data:image/jpeg;base64," + base64.b64encode(buf).decode() if ok else ""
rows = []
for clip in CLIPS:
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
placed = {os.path.basename(p): (os.path.basename(os.path.dirname(p)), p)
for p in glob.glob(f"labelling/{clip}/*/*.jpg")}
by_frame = {}
for fname, (person, path) in placed.items():
if fname in man and person != "unsorted":
by_frame.setdefault(man[fname]["frame"], []).append((fname, person, path))
for frame, items in sorted(by_frame.items()):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
if img is None:
continue
dets = eng.detect(img)
for fname, person, path in items:
i = man[fname]["idx"]
if i >= len(dets):
continue
lm = np.array(dets[i].landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
rows.append({"clip": clip, "person": person, "file": fname, "path": path,
"px": man[fname]["px"], "aligned": np.asarray(crop),
"emb": np.asarray(eng.embed_crop(crop), dtype=np.float32)})
people = sorted({r["person"] for r in rows})
E = np.stack([r["emb"] for r in rows])
lab = np.array([people.index(r["person"]) for r in rows])
S = E @ E.T
np.fill_diagonal(S, -1.0)
# reference face per person: medoid of their REF_CLIP crops
ref_img = {}
for k, p in enumerate(people):
idx = [i for i in np.where(lab == k)[0] if rows[i]["clip"] == REF_CLIP]
if not idx:
idx = list(np.where(lab == k)[0])
if not idx:
continue
sub = S[np.ix_(idx, idx)].copy()
medoid = idx[int(np.argmax(sub.mean(axis=1)))]
ref_img[p] = b64(rows[medoid]["aligned"], 112)
items = []
for i, r in enumerate(rows):
k = lab[i]
same = [j for j in np.where(lab == k)[0] if j != i]
p_self = cal.probability(float(S[i, same].max())) if same else 0.0
best_other, p_other = None, 0.0
for k2, p2 in enumerate(people):
if k2 == k:
continue
other = np.where(lab == k2)[0]
if not len(other):
continue
pv = cal.probability(float(S[i, other].max()))
if pv > p_other:
p_other, best_other = pv, p2
ctx = cv2.imread(r["path"])
items.append({
"file": r["file"], "clip": r["clip"], "person": r["person"],
"px": int(r["px"]), "p_self": round(p_self, 3), "p_other": round(p_other, 3),
"other": best_other, "delta": round(p_other - p_self, 3),
"ctx": b64(ctx, 150) if ctx is not None else "",
"ali": b64(r["aligned"], 112),
})
items.sort(key=lambda x: -x["delta"])
payload = json.dumps({"people": people, "refs": ref_img, "items": items})
HTML = """<meta charset="utf-8"><title>JRay — label review</title>
<style>
:root{color-scheme:dark;--bg:#14161a;--fg:#e6e8ea;--mut:#8b929c;--line:#262b33;--warn:#e0654a;--ok:#4a9d6a}
body{margin:0;background:var(--bg);color:var(--fg);font:14px/1.5 system-ui,sans-serif}
header{position:sticky;top:0;background:#181b20;border-bottom:1px solid var(--line);
padding:12px 18px;display:flex;gap:18px;align-items:center;flex-wrap:wrap;z-index:5}
h1{font-size:15px;margin:0;font-weight:600}
.stat{color:var(--mut);font-size:13px}
button{background:#232830;color:var(--fg);border:1px solid var(--line);border-radius:6px;
padding:7px 13px;cursor:pointer;font:inherit}
button:hover{background:#2c323c}
button.go{background:#2f5d43;border-color:#3c7555}
.row{display:grid;grid-template-columns:150px 1fr 190px;gap:20px;align-items:center;
padding:14px 18px;border-bottom:1px solid var(--line)}
.row.flag{background:#1e1719}
.row.done{opacity:.4}
.cell{display:flex;gap:10px;align-items:center}
img{border-radius:5px;display:block;background:#000}
.lab{font-weight:600;font-size:15px}
.mut{color:var(--mut);font-size:12px}
.p{font-variant-numeric:tabular-nums}
.hi{color:var(--warn);font-weight:600}
.choices{display:flex;flex-wrap:wrap;gap:6px}
.choices button{padding:5px 10px;font-size:13px}
.choices button.sel{background:#2f5d43;border-color:#3c7555}
.legend{padding:10px 18px;color:var(--mut);font-size:12px;border-bottom:1px solid var(--line)}
</style>
<header>
<h1>Label review</h1>
<span class="stat" id="stat"></span>
<button id="exp" class="go">Export corrections.json</button>
<button id="onlyflag">Show only disagreements</button>
</header>
<div class="legend">Left: the person this crop is filed under. Centre: the crop (context with the
detection boxed, and the 112&times;112 the embedder actually sees). Right: the person it matches
better, if any. Ordered by P(other) &minus; P(self) &mdash; disagreements first.</div>
<div id="list"></div>
<script>
const D = __PAYLOAD__;
const choice = {};
const list = document.getElementById('list');
function render(){
list.innerHTML = '';
const flagOnly = document.body.dataset.flag === '1';
for (const it of D.items){
if (flagOnly && it.delta <= 0) continue;
const row = document.createElement('div');
row.className = 'row' + (it.delta > 0 ? ' flag' : '') + (choice[it.file] ? ' done' : '');
const left = document.createElement('div');
left.className = 'cell';
left.innerHTML = `<img src="${D.refs[it.person]||''}" width="72" height="72">
<div><div class="lab">${it.person}</div>
<div class="mut p">P(self) ${it.p_self.toFixed(3)}</div></div>`;
const mid = document.createElement('div');
mid.className = 'cell';
mid.innerHTML = `<img src="${it.ctx}" width="120" height="120">
<img src="${it.ali}" width="90" height="90">
<div><div class="mut">${it.clip} &middot; ${it.px}px</div>
<div class="mut">${it.file}</div></div>`;
const right = document.createElement('div');
const worse = it.delta > 0;
right.innerHTML = it.other
? `<div class="cell"><img src="${D.refs[it.other]||''}" width="56" height="56">
<div><div class="lab ${worse?'hi':''}">${it.other}</div>
<div class="mut p ${worse?'hi':''}">P ${it.p_other.toFixed(3)}</div></div></div>`
: '<div class="mut">—</div>';
const ch = document.createElement('div');
ch.className = 'choices';
for (const p of D.people.concat(['discard'])){
const b = document.createElement('button');
b.textContent = p === it.person ? p + ' (keep)' : p;
if (choice[it.file] === p || (!choice[it.file] && p === it.person)) b.classList.add('sel');
b.onclick = () => { choice[it.file] = p; render(); };
ch.appendChild(b);
}
right.appendChild(ch);
row.append(left, mid, right);
list.appendChild(row);
}
const changed = Object.entries(choice).filter(([f,p]) =>
p !== (D.items.find(i=>i.file===f)||{}).person).length;
document.getElementById('stat').textContent =
`${D.items.length} crops · ${D.items.filter(i=>i.delta>0).length} disagreements · ${changed} changes staged`;
}
document.getElementById('onlyflag').onclick = () => {
document.body.dataset.flag = document.body.dataset.flag === '1' ? '0' : '1';
render();
};
document.getElementById('exp').onclick = () => {
const out = {};
for (const it of D.items){
const p = choice[it.file] || it.person;
if (p !== it.person) out[it.file] = {from: it.person, to: p, clip: it.clip};
}
const blob = new Blob([JSON.stringify(out, null, 1)], {type:'application/json'});
const a = document.createElement('a');
a.href = URL.createObjectURL(blob); a.download = 'corrections.json'; a.click();
};
render();
</script>
"""
os.makedirs("labelling", exist_ok=True)
out = "labelling/review.html"
with open(out, "w") as f:
f.write(HTML.replace("__PAYLOAD__", payload))
size = os.path.getsize(out) / 1e6
flagged = sum(1 for i in items if i["delta"] > 0)
print(f"{out} {size:.1f} MB {len(items)} crops, {flagged} disagreements", file=sys.stderr)
print(f"open file://{os.path.abspath(out)}", file=sys.stderr)
experiments/xsource/pose_label.py
+200
View File
@@ -0,0 +1,200 @@
#!/usr/bin/env python3
"""Estimate head pose per crop, and build a page to confirm or correct it.
Why not solvePnP on the 5 detector landmarks: those landmarks collapse on
turned faces, so the estimator breaks precisely on the crops whose pose we care
about. Run that way it reported the profile subject as the MOST frontal of the
four, which is how we know not to trust it.
Instead the estimate comes from the MediaPipe face mesh (468 points, run via
OpenCV DNN — the same model rPPG-kahn uses) and a symmetry measure that needs
no 3D model:
yaw_ratio = (dL - dR) / (dL + dR)
over left/right symmetric vertex pairs, where dL and dR are each side's
distance from the face midline. Frontal ~ 0, profile -> +/-1. It degrades
gracefully because it averages many pairs rather than trusting any one point,
and it is scale- and translation-free.
It is still an estimate. So this writes pose_review.html with the estimate
PRE-FILLED as a proposal, ordered by confidence, for you to correct — and the
correlation is only run against your corrected labels. If the estimate turns
out to disagree with you often, that is the finding, and the automatic number
gets dropped rather than reported.
Bins are coarse on purpose: frontal / three-quarter / profile / down-or-hidden.
Finer than that and the labelling is slower and less reliable, and the question
("does pose explain the misses") does not need degrees.
"""
import sys, glob, json, os, base64
# sae_embed MUST be imported before cv2: OpenCV's DNN module loads the system
# libonnxruntime, which then shadows the newer one this module links against and
# the import fails on a missing symbol version. Order matters, so do not tidy
# these into alphabetical order.
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
import numpy as np
import cv2
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
MESH = "/home/dtourolle/Development/rPPG-kahn/models/face_landmark.tflite"
CLIPS = ["5157344", "5157339"]
BINS = ["frontal", "three-quarter", "profile", "down-or-hidden"]
# Symmetric vertex pairs (subject-left, subject-right) on the MediaPipe mesh:
# outer eye corners, inner eye corners, cheeks, mouth corners, jaw.
PAIRS = [(33, 263), (133, 362), (130, 359), (243, 463),
(61, 291), (91, 321), (146, 375), (58, 288), (172, 397), (215, 435)]
MIDLINE = [10, 168, 1, 4, 5, 195, 197, 152] # forehead -> nose -> chin
net = cv2.dnn.readNetFromTFLite(MESH)
NAMES = net.getUnconnectedOutLayersNames()
LMI, PRI = NAMES.index("conv2d_21"), NAMES.index("conv2d_31")
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.5, nms=0.4, max_side=0)
def mesh_pose(img, bbox, expand=1.6):
"""(yaw_ratio, presence) or (nan, 0). yaw_ratio in [-1, 1], 0 = frontal."""
x, y, w, h = bbox
cx, cy, s = x + w / 2, y + h / 2, max(w, h) * expand
crop = cv2.getRectSubPix(img, (int(s), int(s)), (float(cx), float(cy)))
net.setInput(cv2.dnn.blobFromImage(crop, 1 / 255.0, (192, 192), (0, 0, 0), swapRB=True))
o = net.forward(NAMES)
pres = 1 / (1 + np.exp(-float(o[PRI].ravel()[0])))
lm = o[LMI].reshape(468, 3)[:, :2]
mid = lm[MIDLINE]
# least-squares midline direction, then signed distance of each pair member
c = mid.mean(axis=0)
u, _, _ = np.linalg.svd(mid - c)
d = (mid - c)
axis = np.linalg.svd(d.T @ d)[0][:, 0] # principal direction of the midline
normal = np.array([-axis[1], axis[0]])
ratios = []
for a, b in PAIRS:
dl = float(np.dot(lm[a] - c, normal))
dr = float(np.dot(lm[b] - c, normal))
if abs(dl) + abs(dr) < 1e-6:
continue
ratios.append((abs(dl) - abs(dr)) / (abs(dl) + abs(dr)))
return (float(np.median(ratios)) if ratios else np.nan), pres
def b64(img, size, q=72):
ok, buf = cv2.imencode(".jpg", cv2.resize(img, (size, size)),
[cv2.IMWRITE_JPEG_QUALITY, q])
return "data:image/jpeg;base64," + base64.b64encode(buf).decode() if ok else ""
items = []
for clip in CLIPS:
lab = {os.path.basename(p): (os.path.basename(os.path.dirname(p)), p)
for p in glob.glob(f"labelling/{clip}/*/*.jpg")
if os.path.basename(os.path.dirname(p)) not in ("discard", "unsorted")}
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
for frame in sorted({man[f]["frame"] for f in lab}):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
dets = eng.detect(img)
for fname, (person, path) in lab.items():
m = man[fname]
if m["frame"] != frame or m["idx"] >= len(dets):
continue
d = dets[m["idx"]]
lm5 = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm5)
if crop is None:
continue
yaw, pres = mesh_pose(img, d.bbox)
a = abs(yaw) if not np.isnan(yaw) else 1.0
guess = ("frontal" if a < 0.15 else "three-quarter" if a < 0.45
else "profile")
if pres < 0.5:
guess = "down-or-hidden" # mesh could not fit at all
ctx = cv2.imread(path)
items.append({"file": fname, "clip": clip, "person": person,
"px": int(m["px"]), "yaw": None if np.isnan(yaw) else round(yaw, 3),
"pres": round(pres, 3), "guess": guess,
"ctx": b64(ctx, 140) if ctx is not None else "",
"ali": b64(np.asarray(crop), 112)})
# least-confident first: near a bin boundary, or the mesh could not fit
def uncertainty(it):
if it["pres"] < 0.5:
return 0.0
a = abs(it["yaw"]) if it["yaw"] is not None else 1.0
return min(abs(a - 0.15), abs(a - 0.45))
items.sort(key=uncertainty)
payload = json.dumps({"bins": BINS, "items": items})
HTML = """<meta charset="utf-8"><title>JRay — head pose labelling</title>
<style>
:root{color-scheme:dark}
body{margin:0;background:#14161a;color:#e6e8ea;font:14px/1.5 system-ui,sans-serif}
header{position:sticky;top:0;background:#181b20;border-bottom:1px solid #262b33;
padding:12px 18px;display:flex;gap:16px;align-items:center;flex-wrap:wrap;z-index:5}
h1{font-size:15px;margin:0}
button{background:#232830;color:#e6e8ea;border:1px solid #262b33;border-radius:6px;
padding:7px 12px;cursor:pointer;font:inherit}
button:hover{background:#2c323c}
button.go{background:#2f5d43;border-color:#3c7555}
.g{display:grid;grid-template-columns:repeat(auto-fill,minmax(300px,1fr));gap:10px;padding:14px}
.c{border:1px solid #262b33;border-radius:8px;padding:9px;display:flex;gap:9px;align-items:center}
.c.edited{border-color:#3c7555}
img{border-radius:5px;background:#000;display:block}
.m{color:#8b929c;font-size:11px}
.b{display:flex;flex-wrap:wrap;gap:4px;margin-top:5px}
.b button{padding:3px 7px;font-size:11px}
.b button.sel{background:#2f5d43;border-color:#3c7555}
</style>
<header><h1>Head pose</h1><span class="m" id="stat"></span>
<button class="go" id="exp">Export pose_labels.json</button></header>
<div class="g" id="g"></div>
<script>
const D=__PAYLOAD__; const pick={};
function render(){
const g=document.getElementById('g'); g.innerHTML='';
for(const it of D.items){
const cur=pick[it.file]||it.guess;
const c=document.createElement('div');
c.className='c'+(pick[it.file]&&pick[it.file]!==it.guess?' edited':'');
const b=D.bins.map(x=>`<button class="${x===cur?'sel':''}" data-f="${it.file}" data-b="${x}">${x}</button>`).join('');
c.innerHTML=`<img src="${it.ctx}" width="88" height="88"><img src="${it.ali}" width="66" height="66">
<div><div class="m">${it.person} · ${it.clip.slice(-3)} · ${it.px}px</div>
<div class="m">yaw ${it.yaw===null?'—':it.yaw} · presence ${it.pres}</div>
<div class="b">${b}</div></div>`;
g.appendChild(c);
}
g.onclick=e=>{const t=e.target; if(t.dataset&&t.dataset.b){pick[t.dataset.f]=t.dataset.b; render();}};
const ed=Object.entries(pick).filter(([f,v])=>v!==(D.items.find(i=>i.file===f)||{}).guess).length;
document.getElementById('stat').textContent=`${D.items.length} crops · ${ed} corrections`;
}
document.getElementById('exp').onclick=()=>{
const out={}; for(const it of D.items) out[it.file]={pose:pick[it.file]||it.guess,
guess:it.guess, yaw:it.yaw, pres:it.pres, person:it.person, clip:it.clip};
const a=document.createElement('a');
a.href=URL.createObjectURL(new Blob([JSON.stringify(out,null,1)],{type:'application/json'}));
a.download='pose_labels.json'; a.click();
};
render();
</script>
"""
out = "labelling/pose_review.html"
open(out, "w").write(HTML.replace("__PAYLOAD__", payload))
from collections import Counter
print(f"{out} {os.path.getsize(out)/1e6:.1f} MB {len(items)} crops", file=sys.stderr)
print(f"estimate: {dict(Counter(i['guess'] for i in items))}", file=sys.stderr)
print("\nestimated pose per person (does this match what you see?):", file=sys.stderr)
for p in sorted({i["person"] for i in items}):
for clip in CLIPS:
sub = [i for i in items if i["person"] == p and i["clip"] == clip]
if sub:
print(f" {p} {clip[-3:]}: {dict(Counter(i['guess'] for i in sub))}",
file=sys.stderr)
print(f"\nopen file://{os.path.abspath(out)}", file=sys.stderr)
experiments/xsource/propose_labels.py
+242
View File
@@ -0,0 +1,242 @@
#!/usr/bin/env python3
"""Propose person labels for one clip using another clip's hand-sorted labels.
Reads the clip you have already sorted (REF_CLIP) as ground truth, then proposes
a person for every crop in the other clip (TARGET_CLIP) and writes them into
matching folders for you to correct.
python3 propose_labels.py # propose, write folders + sheets
python3 propose_labels.py --dry-run # report only, move nothing
Output:
labelling/<target>/unsorted/A|B|C|D/ proposed, same names as the ref clip
labelling/<target>/unsorted/ left in place when no person is
confident enough to name
labelling/review_<person>.jpg contact sheet spanning BOTH clips:
confirmed crops first, then
proposed ones with their P
Correcting it: open a review sheet. Every face on it should be one person. The
lower block is the proposal — move any intruder to the right folder, or back to
unsorted/. The folder a file sits in is the ground truth; nothing downstream
reads the proposed name or its probability.
The proposal is a labelling aid, never the label. Scoring the sweep against
embedding-derived labels would be circular: it keeps the faces the embedder
already gets right and drops the hard ones the sweep exists to find. Your
correction is what breaks that loop, which is why the proposal is deliberately
conservative and leaves anything doubtful unnamed.
Assignment is on the calibrated probability, per-actor best-of-N, exactly as
identity_matcher_node does — never a bare cosine (AR-024). The calibration is
fitted on your labelled reference crops, which is what calibrate_gallery is for.
"""
import sys, glob, json, os, shutil
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
# The embedder and the gallery whose calibration scores it MUST be the same
# model: a Platt fit is specific to one embedding space, so LVFace probabilities
# read through an ArcFace fit are meaningless.
EMBEDDER = M + "LVFace-B_Glint360K.onnx"
GALLERY = ROOT + "gallery_lvface.h5" # 291 actors, cached fit
REF_CLIP, TARGET_CLIP = "5157344", "5157339"
ASSIGN_P = 0.90 # propose a name only when this confident
SHEET_COLS = 8
THUMB = 150
DRY = "--dry-run" in sys.argv
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=EMBEDDER,
conf=0.5, nms=0.4, max_side=0)
def embed_manifest(clip):
"""Re-derive each dumped crop's embedding from its source frame, cached.
The dumped .jpg is a context thumbnail for human eyes; the embedding must
come from the aligned crop the pipeline would actually produce, so the
frame is re-detected and the manifest's idx picks the same face.
Detecting 24 4K frames per clip costs far more than the rest of this script
put together, and the result only changes when the manifest does — so it is
cached and keyed on the manifest's mtime. Delete cache/ to force a redo.
"""
man_path = f"labelling/{clip}/manifest.json"
cache_path = f"cache/emb_{clip}.npz"
os.makedirs("cache", exist_ok=True)
if os.path.exists(cache_path) and \
os.path.getmtime(cache_path) >= os.path.getmtime(man_path):
z = np.load(cache_path, allow_pickle=True)
print(f"[cache] {clip}: {len(z['meta'])} embeddings reused", file=sys.stderr)
return [{**m, "emb": e} for m, e in zip(z["meta"], z["emb"])]
man = json.load(open(man_path))
by_frame = {}
for m in man:
by_frame.setdefault(m["frame"], []).append(m)
out = []
for frame, ms in sorted(by_frame.items()):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
if img is None:
sys.exit(f"missing frames/d{clip}_{frame}.png — extract with\n"
f" ffmpeg -i clips/{clip}.mp4 -vf fps=2 -frames:v 24 "
f"frames/d{clip}_%03d.png")
dets = eng.detect(img)
for m in ms:
if m["idx"] >= len(dets):
continue
d = dets[m["idx"]]
lm = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
out.append({**m, "emb": np.asarray(eng.embed_crop(crop), dtype=np.float32)})
np.savez(cache_path,
meta=np.array([{k: v for k, v in o.items() if k != "emb"} for o in out],
dtype=object),
emb=np.stack([o["emb"] for o in out]))
print(f"[cache] {clip}: {len(out)} embeddings written to {cache_path}",
file=sys.stderr)
return out
def sorted_dirs(clip):
"""Person folders you created, wherever you put them under labelling/<clip>."""
found = {}
for path in glob.glob(f"labelling/{clip}/**/", recursive=True):
name = os.path.basename(path.rstrip("/"))
if name in ("unsorted", "discard") or name.startswith("5157"):
continue
files = [os.path.basename(f) for f in glob.glob(path + "*.jpg")]
if files:
found[name] = files
return found
# ── reference side: your labels ──────────────────────────────────────────────
ref_rows = embed_manifest(REF_CLIP)
ref_dirs = sorted_dirs(REF_CLIP)
if not ref_dirs:
sys.exit(f"no person folders under labelling/{REF_CLIP} — sort that clip first")
file_to_person = {f: p for p, fs in ref_dirs.items() for f in fs}
ref = [(file_to_person[r["file"]], r["emb"]) for r in ref_rows
if r["file"] in file_to_person]
people = sorted({p for p, _ in ref})
print(f"[ref] {REF_CLIP}: {len(ref)} labelled crops over {len(people)} people "
f"{ {p: sum(1 for q, _ in ref if q == p) for p in people} }", file=sys.stderr)
R = np.stack([e for _, e in ref])
r_actor = [people.index(p) for p, _ in ref]
# The global gallery's sigmoid — NOT a fit over these four people. A Platt fit
# over a handful of identities saturates: it will hand back P=0.99 for faces it
# has no basis to separate, which is exactly how a wrong label acquires a
# convincing probability. The production fit spans the whole actor population,
# so a probability means the same thing here as it does in the matcher.
cal = sae_embed.gallery_calibration(GALLERY)
print(f"[calibration] global: {cal} assign boundary = sim "
f"{cal.boundary_at(ASSIGN_P):.4f}", file=sys.stderr)
# ── target side: propose ─────────────────────────────────────────────────────
tgt_rows = embed_manifest(TARGET_CLIP)
T = np.stack([t["emb"] for t in tgt_rows])
r_actor_arr = np.asarray(r_actor)
# per-actor best-of-N for every target crop at once: (n_people, n_target)
best_sim = np.stack([(R[r_actor_arr == people.index(p)] @ T.T).max(axis=0)
for p in people])
proposals = []
for j, t in enumerate(tgt_rows):
k = int(np.argmax(best_sim[:, j]))
prob = cal.probability(float(best_sim[k, j])) # calibrated, never a bare cosine
proposals.append({**t, "person": people[k] if prob >= ASSIGN_P else None,
"p": prob, "top1": people[k]})
# At the production threshold the global fit stays silent on most of these
# faces, which is the honest answer for profile and downward-gaze shots — but a
# labelling aid wants throughput, not caution. --all proposes the top-1 person
# for every crop and orders the review sheets by descending probability, so the
# proposals degrade visibly down the sheet and you can stop correcting where
# they stop being right. The probability is shown, never hidden.
if "--all" in sys.argv:
for x in proposals:
x["person"] = x["top1"]
named = [x for x in proposals if x["person"]]
print(f"[propose] {TARGET_CLIP}: {len(named)}/{len(proposals)} named at P>={ASSIGN_P}; "
f"{len(proposals) - len(named)} left unsorted", file=sys.stderr)
for p in people:
got = [x for x in named if x["person"] == p]
if got:
ps = [x["p"] for x in got]
print(f" {p}: {len(got):>3} crops P {min(ps):.3f}–{max(ps):.3f}", file=sys.stderr)
if DRY:
sys.exit(0)
# ── write proposed folders, mirroring the ref clip's layout ──────────────────
ref_parent = os.path.dirname(next(iter(glob.glob(f"labelling/{REF_CLIP}/**/{people[0]}/",
recursive=True))).rstrip("/"))
tgt_parent = ref_parent.replace(REF_CLIP, TARGET_CLIP)
for p in people:
d = f"{tgt_parent}/{p}"
if os.path.isdir(d): # never clobber corrections already made
print(f"[skip] {d} exists — leaving your sorting alone", file=sys.stderr)
continue
os.makedirs(d, exist_ok=True)
def find_crop(clip, fname):
"""Locate a crop wherever it currently sits under labelling/<clip>."""
hits = glob.glob(f"labelling/{clip}/**/{fname}", recursive=True)
return hits[0] if hits else None
moved = 0
for x in named:
src = find_crop(TARGET_CLIP, x["file"])
dst = f"{tgt_parent}/{x['person']}/{x['file']}"
if src and os.path.abspath(src) != os.path.abspath(dst):
shutil.move(src, dst)
moved += 1
print(f"[write] moved {moved} crops into proposed folders", file=sys.stderr)
# ── review sheets: confirmed block, then proposed block ─────────────────────
def load(clip, person, fname):
for cand in glob.glob(f"labelling/{clip}/**/{person}/{fname}", recursive=True):
return cv2.imread(cand)
return None
for person in people:
conf = [(REF_CLIP, f, None) for f in ref_dirs.get(person, [])]
prop = sorted([(TARGET_CLIP, x["file"], x["p"]) for x in named
if x["person"] == person],
key=lambda t: -t[2]) # most confident first
items = conf + prop
if not items:
continue
rows_n = (len(items) + SHEET_COLS - 1) // SHEET_COLS
sheet = np.full((rows_n * (THUMB + 26), SHEET_COLS * THUMB, 3), 30, np.uint8)
for n, (clip, fname, p) in enumerate(items):
img = load(clip, person, fname)
if img is None:
continue
rr, cc = divmod(n, SHEET_COLS)
y, x = rr * (THUMB + 26), cc * THUMB
sheet[y:y + THUMB, x:x + THUMB] = cv2.resize(img, (THUMB, THUMB))
if p is None:
tag, col = f"{clip[-3:]} CONFIRMED", (170, 170, 170)
else:
tag, col = f"{clip[-3:]} P={p:.2f}", (140, 255, 140)
cv2.putText(sheet, tag, (x + 3, y + THUMB + 17),
cv2.FONT_HERSHEY_SIMPLEX, 0.42, col, 1)
cv2.imwrite(f"labelling/review_{person}.jpg", sheet)
print(f" review_{person}.jpg: {len(conf)} confirmed + {len(prop)} proposed",
file=sys.stderr)
json.dump({x["file"]: {"person": x["person"], "p": x["p"]} for x in proposals},
open(f"labelling/proposed_{TARGET_CLIP}.json", "w"), indent=1)
experiments/xsource/redraw_boxes.py
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env python3
"""Redraw every dumped crop with its detection box marked.
The original thumbnails padded by 0.5x the face on each side for
recognisability, which in a crowded frame pulls a neighbour into shot — often
more prominently than the subject. A label cannot be corrected from a picture
that does not say which face it refers to.
This rewrites each .jpg IN PLACE, wherever it currently sits, so any sorting
already done is preserved: only the pixels change, never the filename or the
folder. Re-run it after dump_faces.py, and re-check any sorting done before it.
"""
import glob, json, os, sys
import cv2
CLIPS = ["5157339", "5157344"]
OUT = 256
for clip in CLIPS:
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
n = 0
for path in glob.glob(f"labelling/{clip}/**/*.jpg", recursive=True):
fname = os.path.basename(path)
m = man.get(fname)
if m is None:
continue
img = cv2.imread(f"frames/d{clip}_{m['frame']}.png")
if img is None:
sys.exit(f"missing frames/d{clip}_{m['frame']}.png")
x, y, w, h = (int(v) for v in m["bbox"])
pad = int(0.55 * max(w, h))
x0, y0 = max(0, x - pad), max(0, y - pad)
x1, y1 = min(img.shape[1], x + w + pad), min(img.shape[0], y + h + pad)
sub = img[y0:y1, x0:x1].copy()
# Box in the sub-image's coordinates, drawn before the resize so the
# line lands exactly on the face at any output size.
cv2.rectangle(sub, (x - x0, y - y0), (x - x0 + w, y - y0 + h), (0, 0, 255), 3)
# Dim everything outside the box so the subject is unmistakable even
# when a neighbour's face is larger or better lit.
mask = sub.copy()
mask[y - y0:y - y0 + h, x - x0:x - x0 + w] = 0
sub = cv2.addWeighted(sub, 1.0, mask, -0.35, 0)
scale = OUT / max(sub.shape[:2])
sub = cv2.resize(sub, (int(sub.shape[1] * scale), int(sub.shape[0] * scale)))
canvas = cv2.copyMakeBorder(
sub, 0, max(0, OUT - sub.shape[0]), 0, max(0, OUT - sub.shape[1]),
cv2.BORDER_CONSTANT, value=(20, 20, 20))[:OUT, :OUT]
cv2.putText(canvas, f"{int(m['px'])}px", (5, OUT - 8),
cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 255), 1)
cv2.imwrite(path, canvas)
n += 1
print(f"[{clip}] redrew {n} crops in place", file=sys.stderr)
experiments/xsource/resolution_sweep.py
+182
View File
@@ -0,0 +1,182 @@
#!/usr/bin/env python3
"""Impact of input resolution on cross-source identification.
Gallery is built from one clip at NATIVE resolution. Probes come from the other
clip with the WHOLE FRAME downscaled before it reaches the detector, so
detection and landmark regression degrade together with the pixels. That is the
measurement VR-005 structurally could not make: it degraded an already-aligned
112x112 crop, holding alignment perfect, so it isolated the embedder's
resolution sensitivity and excluded everything upstream of it.
python3 resolution_sweep.py [--gallery-clip 5157339] [--detector scrfd_500m_bnkps.onnx]
Ground truth
------------
Hand-sorted person folders. Probe detections at reduced scale are tied back to
a labelled face GEOMETRICALLY — the box is mapped to native coordinates and
matched by IoU. Never by embedding similarity, which would be circular: it
would keep the faces the embedder still gets right and silently drop the ones
this sweep exists to find.
A probe whose label is only in the probe clip is OUT OF GALLERY. Naming it is a
true out-of-cast misID, the error the per-scene scorer weights 10x, so it is
counted separately from naming the wrong gallery member.
Metric
------
The calibrated probability from the PRODUCTION gallery sigmoid, never a raw
cosine (AR-024). Per-actor best-of-N similarity -> probability -> accept above
prob_threshold. This is identification, so the matcher's prior applies;
config.hpp has match_prior 0.5, i.e. log_prior_odds = 0.
Everything runs through the shipped C++ via sae_embed.
"""
import sys, glob, json, os, argparse
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
PROB_THRESHOLD = 0.754 # config.hpp:67
LOG_PRIOR_ODDS = 0.0 # config.hpp:61 match_prior=0.5
IOU_MIN = 0.3 # geometric label carry-down
SCALES = [1.0, 0.8, 0.6, 0.5, 0.4, 0.3, 0.25, 0.2, 0.15, 0.12, 0.09, 0.06]
ap = argparse.ArgumentParser()
ap.add_argument("--gallery-clip", default="5157339")
ap.add_argument("--probe-clip", default="5157344")
ap.add_argument("--detector", default="scrfd_500m_bnkps.onnx")
ap.add_argument("--embedder", default="LVFace-B_Glint360K.onnx")
ap.add_argument("--gallery-calibration", default=ROOT + "gallery_lvface.h5")
ap.add_argument("--out", default="results_resolution_sweep.json")
args = ap.parse_args()
eng = sae_embed.FaceEmbedder(detector_model=M + args.detector,
arcface_model=M + args.embedder,
conf=0.5, nms=0.4, max_side=0)
cal = sae_embed.gallery_calibration(args.gallery_calibration)
print(f"[calibration] global: {cal}", file=sys.stderr)
def labelled(clip):
"""{filename: person} from the hand-sorted folders, ignoring discard."""
out = {}
for path in glob.glob(f"labelling/{clip}/*/*.jpg"):
person = os.path.basename(os.path.dirname(path))
if person in ("discard", "unsorted"):
continue
out[os.path.basename(path)] = person
return out
def manifest(clip):
return {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
def iou(a, b):
ax, ay, aw, ah = a; bx, by, bw, bh = b
x0, y0 = max(ax, bx), max(ay, by)
x1, y1 = min(ax + aw, bx + bw), min(ay + ah, by + bh)
if x1 <= x0 or y1 <= y0:
return 0.0
inter = (x1 - x0) * (y1 - y0)
return inter / (aw * ah + bw * bh - inter)
# ── gallery: native resolution, labelled faces only ──────────────────────────
g_lab, g_man = labelled(args.gallery_clip), manifest(args.gallery_clip)
gal = {}
for frame in sorted({g_man[f]["frame"] for f in g_lab}):
img = cv2.imread(f"frames/d{args.gallery_clip}_{frame}.png")
dets = eng.detect(img)
for fname, person in g_lab.items():
m = g_man[fname]
if m["frame"] != frame or m["idx"] >= len(dets):
continue
lm = np.array(dets[m["idx"]].landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
gal.setdefault(person, []).append(np.asarray(eng.embed_crop(crop), dtype=np.float32))
gal = {p: np.stack(v) for p, v in gal.items() if v}
people = sorted(gal)
print(f"[gallery] {args.gallery_clip} @native: "
f"{ {p: len(v) for p, v in gal.items()} }", file=sys.stderr)
# ── probe ground truth at native resolution ──────────────────────────────────
p_lab, p_man = labelled(args.probe_clip), manifest(args.probe_clip)
truth = {} # frame -> [(bbox_native, person)]
for fname, person in p_lab.items():
m = p_man[fname]
truth.setdefault(m["frame"], []).append((m["bbox"], person))
n_out = sum(1 for p in set(p_lab.values()) if p not in people)
print(f"[probe] {args.probe_clip}: {len(p_lab)} labelled faces, "
f"{len(set(p_lab.values()))} people, {n_out} of them out-of-gallery",
file=sys.stderr)
# ── sweep ────────────────────────────────────────────────────────────────────
print(f"\n{'scale':>6}{'frame':>11}{'face px':>9}{'found':>7}{'matched':>9}"
f"{'TPI':>8}{'FPI-in':>8}{'FPI-out':>9}{'TBI':>8}")
results = []
for s in SCALES:
tpi = fpi_in = fpi_out = tbi = 0
n_found = n_matched = 0
pxs = []
for frame, gts in sorted(truth.items()):
img = cv2.imread(f"frames/d{args.probe_clip}_{frame}.png")
if s != 1.0:
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
dets = eng.detect(img)
n_found += len(dets)
for d in dets:
x, y, w, h = d.bbox
native = (x / s, y / s, w / s, h / s) # geometric carry-down
best, best_iou = None, 0.0
for gt_box, person in gts:
v = iou(native, gt_box)
if v > best_iou:
best_iou, best = v, person
if best_iou < IOU_MIN:
continue # spurious / unlabelled
n_matched += 1
pxs.append(min(w, h))
lm = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
tbi += 1 # degenerate alignment
continue
emb = np.asarray(eng.embed_crop(crop), dtype=np.float32)
best_p, best_name = 0.0, None
for p in people: # per-actor best-of-N
prob = cal.probability(float((gal[p] @ emb).max()), LOG_PRIOR_ODDS)
if prob > best_p:
best_p, best_name = prob, p
if best_p <= PROB_THRESHOLD:
tbi += 1
elif best not in people:
fpi_out += 1 # named someone absent from the gallery
elif best_name == best:
tpi += 1
else:
fpi_in += 1
n = max(1, n_matched)
med_px = float(np.median(pxs)) if pxs else 0.0
print(f"{s:>6.2f}{f'{int(4096*s)}x{int(2160*s)}':>11}{med_px:>9.0f}"
f"{n_found:>7}{n_matched:>9}"
f"{100*tpi/n:>7.1f}%{100*fpi_in/n:>7.1f}%{100*fpi_out/n:>8.1f}%{100*tbi/n:>7.1f}%")
results.append({"scale": s, "median_face_px": med_px, "detections": n_found,
"matched_to_truth": n_matched, "tpi_pct": 100*tpi/n,
"fpi_in_gallery_pct": 100*fpi_in/n, "fpi_out_of_gallery_pct": 100*fpi_out/n,
"tbi_pct": 100*tbi/n})
json.dump({"gallery_clip": args.gallery_clip, "probe_clip": args.probe_clip,
"detector": args.detector, "embedder": args.embedder,
"prob_threshold": PROB_THRESHOLD, "log_prior_odds": LOG_PRIOR_ODDS,
"calibration": {"a": cal.a, "b": cal.b},
"gallery_people": people, "results": results},
open(args.out, "w"), indent=2)
print(f"\nwrote {args.out}", file=sys.stderr)
experiments/xsource/verify_labels.py
+167
View File
@@ -0,0 +1,167 @@
#!/usr/bin/env python3
"""Integrity check on the labelled set, before it is used as ground truth.
Checks, loudest failure first:
1. INDEX INTEGRITY. Each crop's embedding is taken by re-detecting its source
frame and indexing with the manifest's `idx`. If detection order is not
reproducible, the thumbnail you sorted and the embedding that gets scored
are different faces — you would see a correct picture and score the wrong
person, with nothing to signal it. Every crop's re-detected bbox is compared
against the manifest's.
2. NO CROP IN TWO FOLDERS, and every manifest entry accounted for — so a
move that half-completed cannot silently duplicate or drop a label.
3. ALIGNMENT. The 112x112 warp is what the embedder actually sees; the
thumbnail is only context for your eyes. verify_<person>.jpg pairs them:
context-with-box on top, the real aligned crop beneath. A profile face whose
alignment has collapsed is obvious there and nowhere else.
4. SEPARATION. Per person, the calibrated P of their own crops against the
other people's, using the global gallery sigmoid. A label set where someone
matches another person better than themselves is mislabelled.
Nothing here changes a label. It reports.
"""
import sys, glob, json, os
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
EMBEDDER = M + "LVFace-B_Glint360K.onnx"
GALLERY = ROOT + "gallery_lvface.h5"
CLIPS = ["5157344", "5157339"]
THUMB = 130
COLS = 10
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=EMBEDDER, conf=0.5, nms=0.4, max_side=0)
fail = 0
rows = []
for clip in CLIPS:
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
# where each crop currently sits -> its label
placed = {}
for path in glob.glob(f"labelling/{clip}/*/*.jpg"):
person = os.path.basename(os.path.dirname(path))
if person in ("discard", "unsorted"):
continue # not people; scoring them would invent an extra identity
fname = os.path.basename(path)
if fname in placed:
print(f"[FAIL] {fname} appears in both {placed[fname][0]} and {person}")
fail += 1
placed[fname] = (person, path)
missing = set(man) - set(placed)
extra = set(placed) - set(man)
if missing:
print(f"[warn] {clip}: {len(missing)} manifest crops not in any folder")
if extra:
print(f"[FAIL] {clip}: {len(extra)} files with no manifest entry: "
f"{sorted(extra)[:3]}")
fail += 1
# index integrity + alignment, frame by frame
by_frame = {}
for fname, (person, path) in placed.items():
if fname in man:
by_frame.setdefault(man[fname]["frame"], []).append((fname, person, path))
bad_idx = 0
for frame, items in sorted(by_frame.items()):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
if img is None:
print(f"[FAIL] missing frames/d{clip}_{frame}.png")
fail += 1
continue
dets = eng.detect(img)
for fname, person, path in items:
m = man[fname]
i = m["idx"]
if i >= len(dets):
print(f"[FAIL] {fname}: idx {i} >= {len(dets)} detections now")
bad_idx += 1
continue
got = [float(v) for v in dets[i].bbox]
want = m["bbox"]
if max(abs(a - b) for a, b in zip(got, want)) > 1.0:
print(f"[FAIL] {fname}: manifest bbox {[round(v) for v in want]} "
f"!= re-detected {[round(v) for v in got]}")
bad_idx += 1
continue
lm = np.array(dets[i].landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
print(f"[warn] {fname}: alignment degenerate, no crop reaches the embedder")
continue
rows.append({"clip": clip, "person": person, "file": fname, "path": path,
"px": m["px"], "aligned": np.asarray(crop),
"emb": np.asarray(eng.embed_crop(crop), dtype=np.float32)})
fail += bad_idx
print(f"[{clip}] {len(placed)} placed, {len(by_frame)} frames, "
f"index mismatches: {bad_idx}")
if not rows:
sys.exit("nothing to verify")
# ── separation, through the global gallery sigmoid ───────────────────────────
cal = sae_embed.gallery_calibration(GALLERY)
E = np.stack([r["emb"] for r in rows])
people = sorted({r["person"] for r in rows})
lab = np.array([people.index(r["person"]) for r in rows])
S = E @ E.T
np.fill_diagonal(S, -1.0)
print(f"\n{'person':>8}{'crops':>7}{'344':>6}{'339':>6}"
f"{'P(self)':>10}{'P(other)':>10}{'worst':>8}")
for k, p in enumerate(people):
mine = np.where(lab == k)[0]
if len(mine) < 2:
continue
self_sim = S[np.ix_(mine, mine)].max(axis=1)
other_sim = S[np.ix_(mine, np.where(lab != k)[0])].max(axis=1)
p_self = np.array([cal.probability(float(s)) for s in self_sim])
p_other = np.array([cal.probability(float(s)) for s in other_sim])
n344 = sum(1 for i in mine if rows[i]["clip"] == "5157344")
n339 = len(mine) - n344
# a crop that matches someone else better than anyone of its own label
worst = int((other_sim > self_sim).sum())
print(f"{p:>8}{len(mine):>7}{n344:>6}{n339:>6}"
f"{np.median(p_self):>10.3f}{np.median(p_other):>10.3f}{worst:>8}")
if worst:
for i in mine[other_sim > self_sim]:
print(f" suspect: {rows[i]['file']} "
f"P(self)={cal.probability(float(self_sim[list(mine).index(i)])):.3f} "
f"< P(other)={cal.probability(float(other_sim[list(mine).index(i)])):.3f}")
# ── verify sheets: context+box over the actual aligned crop ──────────────────
for p in people:
items = [r for r in rows if r["person"] == p]
items.sort(key=lambda r: (r["clip"], r["file"]))
n = len(items)
sheet_rows = (n + COLS - 1) // COLS
H = THUMB * 2 + 22
sheet = np.full((sheet_rows * H, COLS * THUMB, 3), 25, np.uint8)
for j, r in enumerate(items):
rr, cc = divmod(j, COLS)
y, x = rr * H, cc * THUMB
ctx = cv2.imread(r["path"])
if ctx is not None:
sheet[y:y + THUMB, x:x + THUMB] = cv2.resize(ctx, (THUMB, THUMB))
sheet[y + THUMB:y + 2 * THUMB, x:x + THUMB] = cv2.resize(r["aligned"], (THUMB, THUMB))
cv2.putText(sheet, f"{r['clip'][-3:]} {int(r['px'])}px",
(x + 3, y + 2 * THUMB + 15),
cv2.FONT_HERSHEY_SIMPLEX, 0.38, (150, 220, 150), 1)
cv2.imwrite(f"labelling/verify_{p}.jpg", sheet)
print(f" verify_{p}.jpg: {n} crops (top row context, bottom row what the embedder sees)")
print(f"\n{'PASS' if fail == 0 else f'{fail} FAILURES'}")
sys.exit(1 if fail else 0)
external/KPN
Vendored
+1 -1
Submodule external/KPN updated: 4b6e498ba7...6595e6e925
mkdocs.yml
+1
View File
@@ -39,6 +39,7 @@ nav:
- Best Model: best-model.md
- Gallery Scope (Full vs. Limited): gallery-scope.md
- Pose Expansion: pose-expansion.md
- Quality Knee (Blur and Size): quality-knee.md
- LVFace Deep Dive: lvface-deep-dive.md
- Full Experiment Log: model-bakeoff.md
- Service Conversion (proposal): service-conversion.md
scripts/artifacts/pull_artifacts.sh
+67 -1
View File
@@ -11,6 +11,7 @@
# scripts/artifacts/pull_artifacts.sh montage-frames <film-slug> [version]
# scripts/artifacts/pull_artifacts.sh experiment-data [version]
# scripts/artifacts/pull_artifacts.sh report-highlights <name> [version]
# scripts/artifacts/pull_artifacts.sh xsource [version]
# version defaults to "latest" (newest uploaded version, by created_at).
set -euo pipefail
@@ -83,11 +84,71 @@ pull_report_highlight() {
curl -sf "${DL_BASE}/generic/report-highlights/${version}/${name}" -o "${dest}/${name}"
}
pull_xsource() {
local version="$1"
local dest="${REPO_ROOT}/experiments/xsource"
echo "=== xsource (version ${version}) ==="
mkdir -p "${dest}/clips" "${dest}/frames"
for clip in 5157339 5157344; do
if [ -f "${dest}/clips/${clip}.mp4" ]; then
echo " ${clip}.mp4 already present, skipping"
else
echo " fetching ${clip}.mp4..."
curl -sf "${DL_BASE}/generic/xsource/${version}/${clip}.mp4" \
-o "${dest}/clips/${clip}.mp4" \
|| { echo " [warn] ${clip}.mp4 not found at version ${version}" >&2; continue; }
fi
done
if [ -d "${dest}/labelling" ]; then
echo " labelling/ already present — NOT overwriting (it is hand-sorted"
echo " ground truth; move it aside first if you really want the remote copy)"
else
echo " fetching labelling.zip..."
local tmp; tmp="$(mktemp)"
curl -sf "${DL_BASE}/generic/xsource/${version}/labelling.zip" -o "$tmp"
unzip -qo "$tmp" -d "$dest"
rm "$tmp"
fi
# Frames are regenerated rather than shipped: they are ~320 MB of PNG that
# ffmpeg reproduces exactly from the clips. The manifests key on these
# filenames and on detection order within each frame, so the extraction
# settings must match the ones dump_faces.py ran against — hence fps and
# frame count are pinned here rather than left to the caller.
if ! command -v ffmpeg >/dev/null; then
echo " [warn] ffmpeg not found — frames not regenerated; the study" >&2
echo " scripts will fail until you extract them" >&2
return
fi
for clip in 5157339 5157344; do
[ -f "${dest}/clips/${clip}.mp4" ] || continue
if [ -f "${dest}/frames/d${clip}_001.png" ]; then
echo " frames for ${clip} already present, skipping"
continue
fi
echo " extracting frames for ${clip}..."
ffmpeg -v error -i "${dest}/clips/${clip}.mp4" -vf fps=2 -frames:v 24 \
"${dest}/frames/d${clip}_%03d.png"
done
echo " verifying the labelled set..."
if (cd "$dest" && python3 verify_labels.py >/dev/null 2>&1); then
echo " verify_labels.py passed"
else
echo " [warn] verify_labels.py failed — run it directly to see why." >&2
echo " A frame/manifest mismatch means the extraction settings" >&2
echo " differ from the ones the crops were dumped against." >&2
fi
}
if [ $# -eq 0 ]; then
echo "usage: $0 galleries [version]" >&2
echo " $0 montage-frames <film-slug> [version]" >&2
echo " $0 experiment-data [version]" >&2
echo " $0 report-highlights <name> [version]" >&2
echo " $0 xsource [version]" >&2
exit 1
fi
@@ -115,8 +176,13 @@ case "$TARGET" in
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version report-highlights)"
pull_report_highlight "$VERSION" "$NAME"
;;
xsource)
VERSION="${2:-latest}"
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version xsource)"
pull_xsource "$VERSION"
;;
*)
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, or report-highlights)" >&2
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2
exit 1
;;
esac
scripts/artifacts/push_artifacts.sh
+31 -2
View File
@@ -12,6 +12,7 @@
# scripts/artifacts/push_artifacts.sh montage-frames
# scripts/artifacts/push_artifacts.sh experiment-data
# scripts/artifacts/push_artifacts.sh report-highlights
# scripts/artifacts/push_artifacts.sh xsource
# scripts/artifacts/push_artifacts.sh galleries montage-frames experiment-data report-highlights
#
# Package layout (owner=dtourolle, repo=scene-actor-extraction):
@@ -109,8 +110,35 @@ push_report_highlights() {
upload "report-highlights" "germar_beats_xray.jpg" "$src"
}
push_xsource() {
echo "=== xsource (version ${VERSION}) ==="
local root="${REPO_ROOT}/experiments/xsource"
if [ ! -d "$root/labelling" ]; then
echo " no experiments/xsource/labelling found, skipping" >&2
return
fi
# Source recordings. Already compressed, so uploaded as-is rather than zipped.
shopt -s nullglob
for f in "$root"/clips/*.mp4; do
upload "xsource" "$(basename "$f")" "$f"
done
shopt -u nullglob
# The hand-sorted crops and their manifests. This is human ground truth and
# the expensive part of the study — a person looked at every crop and put it
# in a folder. Frames are deliberately NOT pushed: they are deterministic
# from the clips, and pulling regenerates them.
local tmp; tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' RETURN
local zipfile="${tmp}/labelling.zip"
(cd "$root" && zip -qr "$zipfile" labelling -x 'labelling/*.html' -x 'labelling/review_*.jpg' \
-x 'labelling/verify_*.jpg')
upload "xsource" "labelling.zip" "$zipfile"
}
if [ $# -eq 0 ]; then
echo "usage: $0 <galleries|montage-frames|experiment-data|report-highlights> [...]" >&2
echo "usage: $0 <galleries|montage-frames|experiment-data|report-highlights|xsource> [...]" >&2
exit 1
fi
@@ -120,7 +148,8 @@ for target in "$@"; do
montage-frames) push_montage_frames ;;
experiment-data) push_experiment_data ;;
report-highlights) push_report_highlights ;;
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, or report-highlights)" >&2; exit 1 ;;
xsource) push_xsource ;;
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2; exit 1 ;;
esac
done
scripts/build_trt_engines.sh
+37 -12
View File
@@ -6,7 +6,11 @@
# Profiles must match src/arcface_embedder.hpp and src/scrfd_decoder.hpp:
# ArcFace : min=1x3x112x112 opt=Nx3x112x112 max=Nx3x112x112 (N = embed batch)
# SCRFD : 1x3x640x640 (fixed; we letterbox to this)
# TransNetV2 : 1x100x27x48x3 (fixed; scene detector window), input tensor "input"
# TransNetV2 : 1x100x27x48x3 (fixed; scene detector window)
#
# Input tensor names are read from each ONNX model at runtime rather than
# hardcoded, since they differ between models (LVFace-B: "data", arcface_r18:
# "input", arcface_w600k_{r50,mbf}: "input.1").
#
# These trtexec-built engines are *not* picked up by the ORT TRT EP cache —
# ORT uses its own engine format. The point of this script is:
@@ -27,24 +31,43 @@ SCENE_MODEL="${SCENE_MODEL:-$MODELS/transnetv2.onnx}"
run() { echo "+ $*"; "$@"; }
echo "== ArcFace =="
# The input tensor name is not the same across models — LVFace-B uses "data",
# arcface_r18 uses "input", and arcface_w600k_{r50,mbf} use "input.1". A
# hardcoded name makes trtexec fail with "Cannot find input tensor with name
# ...", so read it from the model instead.
input_name() {
python3 - "$1" <<'PY'
import sys
try:
import onnxruntime as ort
except ImportError:
sys.exit("onnxruntime is required to read the model's input name")
sess = ort.InferenceSession(sys.argv[1], providers=["CPUExecutionProvider"])
print(sess.get_inputs()[0].name)
PY
}
ARCFACE_IN="$(input_name "$ARCFACE_MODEL")"
SCRFD_IN="$(input_name "$SCRFD_MODEL")"
echo "== ArcFace == (input tensor: $ARCFACE_IN)"
run trtexec \
--onnx="$ARCFACE_MODEL" \
--fp16 \
--minShapes=input.1:1x3x112x112 \
--optShapes=input.1:${EMBED_BATCH}x3x112x112 \
--maxShapes=input.1:${EMBED_BATCH}x3x112x112 \
--minShapes="$ARCFACE_IN":1x3x112x112 \
--optShapes="$ARCFACE_IN":${EMBED_BATCH}x3x112x112 \
--maxShapes="$ARCFACE_IN":${EMBED_BATCH}x3x112x112 \
--saveEngine="$OUT/arcface.$(basename "$ARCFACE_MODEL" .onnx).b${EMBED_BATCH}.fp16.engine" \
--useCudaGraph
echo
echo "== SCRFD =="
echo "== SCRFD == (input tensor: $SCRFD_IN)"
run trtexec \
--onnx="$SCRFD_MODEL" \
--fp16 \
--minShapes=input.1:1x3x640x640 \
--optShapes=input.1:1x3x640x640 \
--maxShapes=input.1:1x3x640x640 \
--minShapes="$SCRFD_IN":1x3x640x640 \
--optShapes="$SCRFD_IN":1x3x640x640 \
--maxShapes="$SCRFD_IN":1x3x640x640 \
--saveEngine="$OUT/scrfd.$(basename "$SCRFD_MODEL" .onnx).640.fp16.engine" \
--useCudaGraph
@@ -53,12 +76,14 @@ if [[ -f "$SCENE_MODEL" ]]; then
echo "== TransNetV2 (scene detector) =="
# Fixed 1x100x27x48x3 window. The raw-TRT scene detector backend loads this
# engine directly via --scene-detector-engine; the ORT-TRT EP builds its own.
# No --*Shapes here: TransNetV2's input is fully static (1x100x27x48x3
# with no dynamic dimensions), and TensorRT rejects explicit shape
# profiles for such a model — "Static model does not take explicit shapes
# since the shape of inference tensors will be determined by the model
# itself". The shape comes from the model.
run trtexec \
--onnx="$SCENE_MODEL" \
--fp16 \
--minShapes=input:1x100x27x48x3 \
--optShapes=input:1x100x27x48x3 \
--maxShapes=input:1x100x27x48x3 \
--saveEngine="$OUT/transnetv2.100x27x48.fp16.engine" \
--useCudaGraph
else
scripts/filter_gallery.py
+5 -1
View File
@@ -77,7 +77,11 @@ def main():
if missing > 0:
print(f"[warn] {missing} cast member(s) not present in gallery (not yet embedded)", file=sys.stderr)
save_gallery_hdf5({"actors": actors}, Path(args.output))
# TRACES: GR-004 | SR-001 — a filtered gallery holds the SAME vectors as its
# source, so it inherits the source's binding. Dropping the stamp here would
# silently launder a stamped gallery into an unstamped one.
save_gallery_hdf5({"actors": actors}, Path(args.output),
gallery.get("embedder"))
print(f"Saved {len(actors)} actor(s) to {args.output}", file=sys.stderr)
scripts/make_fixtures.sh
Executable
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env bash
# make_fixtures.sh — regenerate the committed replay fixtures.
#
# TRACES: VR-001 | PR-002
#
# CI never calls a model (see docs/requirements.md, "CI never calls a model"):
# the embedder is impractical on the N100 CI host, so inference happens HERE, on
# a machine with a GPU, and CI consumes the HDF5 dumps as data. Everything
# downstream of embedding — tracking, presence windows, belief accumulation,
# expansion — is cheap CPU maths and replays from these files.
#
# Reproducibility is a requirement, not a nicety. A fixture whose provenance is
# unknown is worse than no fixture, because it will be trusted. Every parameter
# that affects the output is pinned below rather than left to a default, and the
# dumps carry the embedder identity and SHA-256 (GR-004) so a replay cannot be
# silently scored against the wrong gallery.
#
# These are byte-reproducible only because node outputs block rather than drop
# on a full channel (AR-004). Before that fix the same command produced
# different dumps run to run, since what got dropped depended on timing.
#
# Source: bali/ — Road to Bali (1952), public domain. That matters: derived
# fixtures can be committed, where anything cut from a copyrighted title could
# not live in the repository at all.
set -euo pipefail
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CLIPS="${CLIPS:-$REPO/../bali}"
GALLERY="${GALLERY:-$REPO/gallery_lvface.h5}"
BIN="${BIN:-$REPO/build/scene_analyze}"
OUT="$REPO/tests/fixtures/dumps"
# Pinned. Changing either invalidates every committed fixture.
# fps 5 — 1 fps over a 77 s clip is 77 frames, too thin to exercise an
# extinction window measured in tens of seconds.
# min-face — 32 px, the VR-005 measured floor (98.1% TPI). The corpus is
# 480x360, so a stricter value would reject most faces present.
FPS=5
MIN_FACE_PX=32
[[ -x "$BIN" ]] || { echo "no scene_analyze at $BIN (set BIN=)" >&2; exit 1; }
[[ -f "$GALLERY" ]] || { echo "no gallery at $GALLERY (set GALLERY=)" >&2; exit 1; }
[[ -d "$CLIPS" ]] || { echo "no clips at $CLIPS (set CLIPS=)" >&2; exit 1; }
mkdir -p "$OUT"
for clip in "$CLIPS"/Road_To_Bali-*.webm; do
n="$(basename "$clip" .webm)"; n="${n##*-}"
echo "── bali_$n"
"$BIN" --movie "$clip" --gallery "$GALLERY" \
--fps "$FPS" --min-face-px "$MIN_FACE_PX" \
--dump-embeddings "$OUT/bali_$n.h5" \
--output /dev/null 2>&1 | grep -E "wrote|dropped" || true
done
echo
echo "Regenerated in $OUT — verify the diff is empty if nothing upstream changed."
echo "A non-empty diff means detection, alignment or embedding moved. That is"
echo "either a regression or a deliberate change, and either way the golden"
echo "outputs derived from these fixtures need reviewing."
scripts/make_gallery.py
+8 -3
View File
@@ -36,8 +36,9 @@ import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from sae_embed_loader import load_embedder
from sae_gallery import download_images, save_gallery, wikidata_image_urls
from sae_embed_loader import load_embedder, resolve_arcface
from sae_gallery import (download_images, embedder_stamp, save_gallery,
wikidata_image_urls)
from sae_tmdb import TMDB_IMG, tmdb_get, tmdb_id_from_imdb
@@ -177,7 +178,11 @@ def main():
output = Path(args.output)
image_root = Path(args.image_dir) if args.image_dir else output.parent / "images"
# TRACES: GR-004 | SR-001 — stamp with the model actually loaded, resolved
# through the same helper load_embedder uses so the two cannot diverge.
arcface_path = resolve_arcface(args.models_dir, args.arcface)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
stamp = embedder_stamp(arcface_path)
# Resolve movie ID
movie_id = args.movie_id
@@ -203,7 +208,7 @@ def main():
if n_actors == 0:
sys.exit("No actors could be processed — check models and images.")
save_gallery(gallery, missing, output)
save_gallery(gallery, missing, output, embedder=stamp)
if __name__ == "__main__":
scripts/make_jellyfin_gallery.py
+100 -16
View File
@@ -1,6 +1,8 @@
#!/usr/bin/env python3
"""make_jellyfin_gallery.py — build a gallery.h5 spanning an entire Jellyfin library.
TRACES: GR-001, GR-002 | SR-001, SR-005
Queries the Jellyfin API for every Movie/Series, collects the unique cast
across the whole library, downloads each actor's headshot directly from
Jellyfin (no TMDB key needed), embeds them with the sae_embed module (SCRFD +
@@ -51,8 +53,9 @@ import requests
sys.path.insert(0, str(Path(__file__).resolve().parent))
import sae_env # noqa: F401 — loads .env into os.environ on import
from sae_embed_loader import load_embedder
from sae_gallery import (download_image, download_images, load_gallery_hdf5,
from sae_embed_loader import load_embedder, resolve_arcface
from sae_gallery import (download_image, download_images, embedder_stamp,
enforce_embedder_stamp, load_gallery_hdf5,
save_gallery, wikidata_image_urls)
from sae_jellyfin import actor_jellyfin_id, jf_get, normalize_jellyfin_url
from sae_tmdb import (
@@ -147,11 +150,14 @@ def download_person_images(base_url: str, api_key: str, person_id: str,
def fetch_actor_images(base_url: str, api_key: str, pid: str, info: dict,
images_per_actor: int, actor_dir: Path,
fetch_imdb_ids: bool, tmdb_key: str | None
fetch_imdb_ids: bool, tmdb_key: str | None,
fetch_overfetch: float = 1.0
) -> tuple[list[Path], str | None, str | None]:
"""Network-bound: download Jellyfin image(s), then fall back to TMDB if short."""
name = info["name"]
print(f"{name} ({pid}) — in {len(info['appearances'])} title(s)", file=sys.stderr)
# Over-fetch target: see the TMDB block below.
tmdb_budget = int(images_per_actor * fetch_overfetch)
image_paths = download_person_images(base_url, api_key, pid, actor_dir, images_per_actor)
# Need the IMDB id for the TMDB /find lookup, to persist it (--fetch-imdb-ids),
@@ -177,8 +183,13 @@ def fetch_actor_images(base_url: str, api_key: str, pid: str, info: dict,
except requests.RequestException as e:
print(f" [warn] {name}: TMDB lookup failed: {e}", file=sys.stderr)
if len(image_paths) < images_per_actor and tmdb_urls:
needed = images_per_actor - len(image_paths)
# Over-fetch from TMDB: near-duplicate stills (the same photo at different
# crops/resolutions) are dropped after embedding, so downloading exactly
# images_per_actor would leave the actor short of that many *distinct*
# embeddings. Pulling extra candidates lets the dedup filter discard
# duplicates while still reaching the target.
if len(image_paths) < tmdb_budget and tmdb_urls:
needed = tmdb_budget - len(image_paths)
print(f" {name}: Jellyfin image missing/incomplete, falling back to TMDB "
f"({len(tmdb_urls)} image(s) available)…", file=sys.stderr)
image_paths += download_images(tmdb_urls, actor_dir, needed, start_index=len(image_paths))
@@ -186,10 +197,10 @@ def fetch_actor_images(base_url: str, api_key: str, pid: str, info: dict,
# Last resort: a CC-licensed Commons headshot via Wikidata, keyed by the
# actor's IMDB id. Catches actors TMDB has no usable image for (or that the
# name search missed entirely).
if len(image_paths) < images_per_actor and imdb_id:
if len(image_paths) < tmdb_budget and imdb_id:
wiki_urls = wikidata_image_urls(imdb_id)
if wiki_urls:
needed = images_per_actor - len(image_paths)
needed = tmdb_budget - len(image_paths)
print(f" {name}: still short, falling back to Wikidata/Commons "
f"({len(wiki_urls)} image(s) available)…", file=sys.stderr)
image_paths += download_images(wiki_urls, actor_dir, needed,
@@ -198,9 +209,39 @@ def fetch_actor_images(base_url: str, api_key: str, pid: str, info: dict,
return image_paths, imdb_id, tmdb_id
# Default cosine-distance tolerance below which two embeddings of the same
# actor are treated as the same image. Embeddings are L2-normalised by the
# backend, so cosine similarity is a plain dot product and the distance is
# 1 - dot. Expanding an actor's photo set via TMDB frequently returns the same
# still at different crops/resolutions; those embed to nearly identical vectors
# and add gallery size and match cost without adding information.
DEDUP_TOL = 1e-3
def _cosine(a, b) -> float:
"""Cosine similarity of two L2-normalised embeddings."""
return float(sum(x * y for x, y in zip(a, b)))
def _near_duplicate(emb, existing, tol: float) -> int | None:
"""Index of the first embedding within `tol` cosine distance of `emb`.
Returns None when `emb` is sufficiently distinct from everything in
`existing`. tol <= 0 disables the check.
"""
if tol <= 0:
return None
for i, prev in enumerate(existing):
if 1.0 - _cosine(emb, prev) < tol:
return i
return None
def embed_actor(pid: str, info: dict, image_paths: list[Path],
imdb_id: str | None, tmdb_id: str | None, embedder, fetch_imdb_ids: bool,
embed_executor: concurrent.futures.ThreadPoolExecutor) -> tuple[dict | None, str | None]:
embed_executor: concurrent.futures.ThreadPoolExecutor,
dedup_tol: float = DEDUP_TOL,
max_embeddings: int = 0) -> tuple[dict | None, str | None]:
"""GPU-bound: run sae_embed, always on embed_executor's single dedicated thread.
onnxruntime's CUDA EP / cudnn_frontend execution plans are not safe to run
@@ -217,20 +258,35 @@ def embed_actor(pid: str, info: dict, image_paths: list[Path],
embeddings = []
source_images = []
n_dup = 0
print(f" {name}: embedding {len(image_paths)} image(s)…", file=sys.stderr)
for path in image_paths:
res = embed_executor.submit(embedder.embed, str(path)).result()
if not res.ok:
print(f" [skip] {name}/{path.name}: {res.error}", file=sys.stderr)
continue
embeddings.append(res.embedding)
emb = res.embedding
dup = _near_duplicate(emb, embeddings, dedup_tol)
if dup is not None:
n_dup += 1
print(f" [dup] {name}/{path.name}: matches {source_images[dup]} "
f"(cos={_cosine(emb, embeddings[dup]):.6f}), not stored",
file=sys.stderr)
continue
embeddings.append(emb)
source_images.append(path.name)
# Stop once we have the requested number of *distinct* embeddings; the
# extra candidates were only fetched to absorb duplicates.
if max_embeddings and len(embeddings) >= max_embeddings:
break
if not embeddings:
print(f" {name}: no valid embeddings, skipping actor", file=sys.stderr)
return None, "no valid embeddings"
print(f" {name}: → {len(embeddings)} embedding(s) stored", file=sys.stderr)
dup_note = f" ({n_dup} near-duplicate(s) dropped)" if n_dup else ""
print(f" {name}: → {len(embeddings)} embedding(s) stored{dup_note}",
file=sys.stderr)
return {
"imdb_id": imdb_id if (fetch_imdb_ids and imdb_id) else "",
"tmdb_id": tmdb_id or "",
@@ -245,12 +301,16 @@ def embed_actor(pid: str, info: dict, image_paths: list[Path],
def process_actor(pid: str, info: dict, base_url: str, api_key: str,
embedder, images_per_actor: int, image_root: Path,
fetch_imdb_ids: bool, tmdb_key: str | None,
embed_executor: concurrent.futures.ThreadPoolExecutor) -> tuple[dict | None, str | None]:
embed_executor: concurrent.futures.ThreadPoolExecutor,
dedup_tol: float = DEDUP_TOL,
fetch_overfetch: float = 1.0) -> tuple[dict | None, str | None]:
safe_name = info["name"].replace(" ", "_")
actor_dir = image_root / f"{pid}_{safe_name}"
image_paths, imdb_id, tmdb_id = fetch_actor_images(
base_url, api_key, pid, info, images_per_actor, actor_dir, fetch_imdb_ids, tmdb_key)
return embed_actor(pid, info, image_paths, imdb_id, tmdb_id, embedder, fetch_imdb_ids, embed_executor)
base_url, api_key, pid, info, images_per_actor, actor_dir, fetch_imdb_ids,
tmdb_key, fetch_overfetch)
return embed_actor(pid, info, image_paths, imdb_id, tmdb_id, embedder, fetch_imdb_ids,
embed_executor, dedup_tol, images_per_actor)
# ── Gallery assembly ─────────────────────────────────────────────────────────
@@ -258,7 +318,9 @@ def process_actor(pid: str, info: dict, base_url: str, api_key: str,
def build_gallery(base_url: str, api_key: str, embedder, item_types: list[str],
images_per_actor: int, image_root: Path,
fetch_imdb_ids: bool, existing_actors: dict,
tmdb_key: str | None = None, workers: int = 8) -> tuple[dict, list[dict]]:
tmdb_key: str | None = None, workers: int = 8,
dedup_tol: float = DEDUP_TOL,
fetch_overfetch: float = 1.0) -> tuple[dict, list[dict]]:
actors = collect_actors(base_url, api_key, item_types)
gallery_actors = []
@@ -298,7 +360,8 @@ def build_gallery(base_url: str, api_key: str, embedder, item_types: list[str],
futures = {
executor.submit(process_actor, pid, info, base_url, api_key, embedder,
images_per_actor, image_root,
fetch_imdb_ids, tmdb_key, embed_executor): (pid, info["name"])
fetch_imdb_ids, tmdb_key, embed_executor,
dedup_tol, fetch_overfetch): (pid, info["name"])
for pid, info in todo
}
for future in concurrent.futures.as_completed(futures):
@@ -343,6 +406,16 @@ def main():
help="Directory containing ONNX models (default: models/)")
parser.add_argument("--arcface", default=None,
help="Path to ArcFace ONNX model (overrides --models-dir selection)")
parser.add_argument("--dedup-tol", type=float, default=DEDUP_TOL,
help=f"Cosine-distance threshold below which a new embedding is treated "
f"as a duplicate of one already stored for that actor and dropped "
f"(default: {DEDUP_TOL}). TMDB often returns the same still at "
f"different crops. Set 0 to keep every embedding.")
parser.add_argument("--overfetch", type=float, default=2.0,
help="Download this multiple of --images-per-actor as candidates, then "
"keep the first N that survive dedup (default: 2.0). Raise it for "
"actors whose TMDB galleries are mostly duplicates; 1.0 disables "
"over-fetching.")
parser.add_argument("--images-per-actor", type=int, default=10,
help="Images to download per actor (default: 10). Jellyfin usually has "
"only 1, so the rest come from the TMDB/Wikidata fallbacks; more "
@@ -372,11 +445,20 @@ def main():
image_root = Path(args.image_dir) if args.image_dir else output.parent / "images"
item_types = [t.strip() for t in args.item_types.split(",") if t.strip()]
# TRACES: GR-004 | SR-001
arcface_path = resolve_arcface(args.models_dir, args.arcface)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
stamp = embedder_stamp(arcface_path)
existing_actors = {}
if args.merge and output.is_file():
existing = load_gallery_hdf5(output)
# TRACES: GR-004 | SR-001 — --merge keeps the existing actors' vectors and
# embeds the new ones with THIS model. If they disagree, the result is one
# gallery holding two incompatible embedding spaces, which is worse than a
# mismatched gallery: no later check can separate them again.
enforce_embedder_stamp(existing.get("embedder"), stamp, str(output),
arcface_path)
for actor in existing.get("actors", []):
pid = actor_jellyfin_id(actor)
if pid:
@@ -392,6 +474,8 @@ def main():
image_root=image_root,
fetch_imdb_ids=args.fetch_imdb_ids,
existing_actors=existing_actors,
dedup_tol=args.dedup_tol,
fetch_overfetch=args.overfetch,
tmdb_key=args.tmdb_key,
workers=args.workers,
)
@@ -403,7 +487,7 @@ def main():
if n_actors == 0:
sys.exit("No actors could be processed — check Jellyfin URL/API key and models.")
save_gallery(gallery, missing, output)
save_gallery(gallery, missing, output, embedder=stamp)
if __name__ == "__main__":
scripts/movienet_eval.py
+7 -2
View File
@@ -22,8 +22,8 @@ from pathlib import Path
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parent))
from sae_embed_loader import load_embedder
from sae_gallery import load_gallery_hdf5
from sae_embed_loader import load_embedder, resolve_arcface
from sae_gallery import load_gallery_hdf5, verify_gallery_stamp
def load_gallery(path: str) -> dict[str, dict]:
@@ -62,6 +62,11 @@ def main():
args = p.parse_args()
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001 — match() below is a bare dot product against the
# gallery's vectors; if the gallery came from another model those numbers are
# noise wearing a similarity's clothes.
verify_gallery_stamp(args.gallery,
resolve_arcface(args.models_dir, args.arcface))
gallery = load_gallery(args.gallery)
print(f"[eval] gallery: {len(gallery)} actors", file=sys.stderr)
scripts/optimizer/SCHEMA.md
+102 -6
View File
@@ -19,10 +19,33 @@ variable-length HDF5 types and reads straight into numpy.
/ (root)
attrs:
schema_version : int = 1
movie : str (source video path)
sample_fps : float
embed_dim : int = 512
# ── what produced the vectors (GR-004) ──────────────────────────────────
embedder_model : str basename of the embedding model
embedder_sha256: str SHA-256 of that model file
# ── what produced the faces (VR-010) ────────────────────────────────────
detector_model : str basename of the detector .onnx
detector_conf : float score floor a detection had to clear to be dumped
detector_nms : float NMS IoU threshold
min_face_px : float minimum box side, ORIGINAL-resolution px (AR-002)
max_faces : int per-frame cap; 0 = uncapped, the default (AR-003)
# ── what produced the frames (VR-010) ───────────────────────────────────
movie : str source video path
sample_fps : float frames analysed per second of movie
start_sec : float seek point
end_sec : float stop point; -1 = end of file
cut_threshold : float histogram correlation below which is_cut fires
dense_scale : float decoded-frame downscale in dense mode; 1 = off
bbox_upscale : float multiply faces/bbox and faces/landmarks by this to
reach original video pixels; 1 when dense_scale is 1
scene_detect : uint8 0/1 — was TransNetV2 running at all (see below)
# ── downstream setting recorded for comparability (VR-010) ──────────────
track_assoc_min_prob : float the run's tracker admission probability
frames/ group — one row per sampled frame
timestamp_sec : float64 [F]
frame_idx : int64 [F]
@@ -33,18 +56,91 @@ variable-length HDF5 types and reads straight into numpy.
faces/ group — one row per detected face, concatenated
embedding : float32 [N, 512] L2-normalised ArcFace embedding
bbox : float32 [N, 4] x, y, w, h in original video pixels
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order
bbox : float32 [N, 4] x, y, w, h in DECODED-frame pixels
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order,
same space as bbox
confidence : float32 [N] detector confidence
```
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.
## Provenance (VR-010)
The attributes above are not documentation; they are the only thing that makes a
dump interpretable. Two dumps of the same film at `detector_conf` 0.5 and 0.7, or
at `dense_scale` 1.0 and 0.5, or with scene detection on and off, are different
measurements of different things — and they are byte-shaped identically. Without
provenance a consumer that mixes them gets a plausible number from an incoherent
input, and nothing anywhere reports a problem.
**`scene_detect` is the one that cannot be inferred.** `is_scene_boundary` is
all-zero both when TransNetV2 found no boundaries in the clip and when it was
never enabled, and those mean opposite things: the first says *this footage has
no shot changes*, the second says *nobody looked*. A consumer that reads the
array alone must guess. The flag is what removes the guess. (`dump_embeddings`
has no `--scene-detect`, so every dump it writes records `false` — which is
exactly the fact the committed fixtures needed to state.)
**`bbox_upscale` is recorded, not applied.** See the coordinate-space note below.
Reading is by name with a default or an existence check on **both** sides —
`replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in
`src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are
additive and `schema_version` stays 1: a pre-VR-010 dump still loads, and a
post-VR-010 dump still reads on old code.
A missing attribute means **unknown**, never a default value. Substituting
`detector_conf = 0.5` for a dump that does not say so manufactures the provenance
the requirement exists to prevent — per `docs/requirements.md`, *"a fixture whose
provenance is unknown is worse than no fixture, because it will be trusted."*
The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these
attributes; re-dump to bind them, as with GR-004.
## Model binding (GR-004)
`embedder_model` / `embedder_sha256` record which embedder produced every vector
in `faces/embedding`. A replay has no live embedder, so **the dump is the embedder
as far as the gallery is concerned**: `replay.py` checks these two attributes
against the gallery's own `/embedder` stamp and refuses to run on a mismatch,
naming both sides. Cross-model cosines are meaningless but look plausible.
The attributes are additive, not a format break — `schema_version` stays 1. Dumps
written before GR-004 simply lack them, which reports as *unverifiable* (a loud
warning, or a hard error under `SAE_REQUIRE_GALLERY_STAMP=1`) rather than as a
pass. Re-dump to bind an old dump; there is no in-place migration, because unlike
a gallery nobody can assert after the fact which model produced a vector.
## Coordinate space — `bbox`, `landmarks`, `bbox_upscale`
`bbox` and `landmarks` are in **decoded-frame pixels**: exactly the numbers SCRFD
produced, untransformed. To reach original video pixels, multiply by
`bbox_upscale`. With `dense_scale == 1` (the default, and every committed
fixture) `bbox_upscale == 1` and the two spaces coincide.
> Earlier revisions of this document claimed the upscale was applied at dump time.
> It never was. `embedding_dump_node.hpp` writes `f.bbox` raw; the upscale lives
> in `identity_matcher_node.hpp`, which is *downstream* of the dump tap. The
> claim was harmless only because `dense_scale` was 1 in practice.
The fix is to record the factor rather than to apply it, because the dump's whole
contract is to be a **faithful tap** at the `EmbeddedSceneFrame` channel — VR-002
requires replay to drive the real nodes, and a replay is only equivalent to the
live run if the tracker is fed the geometry the live tracker saw. Rescaling at
the tap would break that: the replayed tracker would associate on boxes the live
one never received. Two further reasons:
- The matcher's upscale is applied to `bbox` **only**, not to `landmarks`.
Pre-multiplying at the tap would leave the two arrays in different coordinate
spaces inside one file — a worse trap than the one being fixed.
- Pre-multiplying is lossy in the sense that matters: a dump that had been
upscaled would be indistinguishable from one taken at `dense_scale == 1`, so
you would have to record `bbox_upscale` anyway to know which you were holding.
## Invariants
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
- `bbox` is already mapped to original resolution (bbox_upscale applied at dump time),
matching what the identity matcher would emit.
- `bbox` and `landmarks` share one coordinate space; `bbox_upscale` maps both to
original resolution (see above).
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
- EOF sentinel frames are NOT written.
scripts/optimizer/fetch_missing_actors.py
+13 -2
View File
@@ -35,7 +35,9 @@ REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
import sae_env # noqa: E402 loads .env
from sae_tmdb import tmdb_get, tmdb_person_for_imdb, TMDB_IMG # noqa: E402
from sae_gallery import download_images, wikidata_image_urls # noqa: E402
from sae_embed_loader import resolve_arcface # noqa: E402
from sae_gallery import (download_images, embedder_stamp, # noqa: E402
enforce_embedder_stamp, wikidata_image_urls)
from sae_embed_loader import load_embedder # noqa: E402
@@ -57,6 +59,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
src = "TMDB + Wikidata fallback" if use_wikidata else "TMDB"
print(f"[fetch] {len(missing)} missing actors to resolve via {src}", file=sys.stderr)
embedder = load_embedder(build_dir, models_dir, arcface)
stamp = embedder_stamp(resolve_arcface(models_dir, arcface)) # TRACES: GR-004 | SR-001
img_root = Path(tempfile.mkdtemp(prefix="missing_gallery_"))
actors = []
@@ -103,7 +106,9 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
f"(wiki={n_via_wikidata}) no_tmdb={n_no_tmdb} no_img={n_no_img} "
f"no_face={n_no_face}", file=sys.stderr)
Path(out_path).write_text(json.dumps({"actors": actors}, indent=2))
# TRACES: GR-004 | SR-001 — the legacy JSON gallery carries the same stamp as
# the HDF5 one; src/gallery/gallery_store.cpp reads it from either.
Path(out_path).write_text(json.dumps({"embedder": stamp, "actors": actors}, indent=2))
n_emb = sum(len(a["embeddings"]) for a in actors)
print(f"\n[fetch] recovered {n_resolved}/{len(missing)} actors "
f"({n_via_wikidata} via Wikidata), {n_emb} embeddings → {out_path}",
@@ -115,6 +120,12 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
def merge(base_path, add_path, out_path):
base = json.loads(Path(base_path).read_text())
add = json.loads(Path(add_path).read_text())
# TRACES: GR-004 | SR-001 — merging two galleries from different models makes
# ONE file containing two incompatible embedding spaces. Nothing downstream can
# ever untangle that, so this is the one place the check must run before, not
# after, the write.
enforce_embedder_stamp(base.get("embedder"), add.get("embedder"),
str(base_path), str(add_path))
have = {a.get("imdb_id") for a in base["actors"] if a.get("imdb_id")}
added = [a for a in add["actors"] if a.get("imdb_id") not in have]
base["actors"].extend(added)
scripts/optimizer/optimize.py
+23
View File
@@ -34,6 +34,7 @@ from scipy.optimize import differential_evolution
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts" / "optimizer"))
sys.path.insert(0, str(REPO / "scripts" / "validation"))
sys.path.insert(0, str(REPO / "scripts"))
import json as _json
import os
@@ -56,6 +57,8 @@ DE_WORKERS = int(os.environ.get("DE_WORKERS", "1"))
from second_score import score_seconds # noqa: E402 uniform per-second TPI/FPI scoring
from sample_eval import load_gallery_keys # noqa: E402
from replay import dump_embedder_stamp # noqa: E402
from sae_stamp import EmbedderMismatch, verify_gallery_stamp # noqa: E402
_GAL_KEYS: dict = {} # gallery path → key set (fair-recall FN mask), loaded once
_REPLAY_TIMEOUT = 45 # seconds per film; a wedged replay is killed, not left to hang
@@ -180,7 +183,15 @@ def main():
p.add_argument("--seed", type=int, default=0)
p.add_argument("--trajectory", help="write every evaluation here (JSON lines)")
p.add_argument("--out", help="write best config + metrics")
# TRACES: GR-004 | SR-001
p.add_argument("--require-gallery-stamp", action="store_true",
help="unprovable gallery/dump model binding is a hard error, "
"not a warning (also via SAE_REQUIRE_GALLERY_STAMP=1)")
args = p.parse_args()
if args.require_gallery_stamp:
# Set the env var rather than threading a flag through cfg: replays run as
# subprocesses and inherit it, so strictness cannot be lost in the handoff.
os.environ["SAE_REQUIRE_GALLERY_STAMP"] = "1"
films = json.loads(Path(args.manifest).read_text())
for f in films:
@@ -188,6 +199,18 @@ def main():
if not Path(f["dump"]).exists():
sys.exit(f"[opt] missing dump for {f['name']}: {f['dump']}")
# TRACES: GR-004 | SR-001 — every (dump, gallery) pair is checked ONCE here,
# before the first evaluation. A DE sweep is thousands of replays; discovering
# a cross-model pair at the end (or never) means every number it produced was
# noise. Each replay subprocess re-checks its own pair anyway.
for f in films:
try:
verify_gallery_stamp(f["gallery"], stamp=dump_embedder_stamp(f["dump"]),
embedder_desc=f"embedding dump {Path(f['dump']).name}",
require_stamp=args.require_gallery_stamp)
except EmbedderMismatch as e:
sys.exit(f"[opt] {f['name']}: {e}")
names, bounds = [], []
int_knobs = {"track_max_frames_missing", "cut_inactive_max_frames"}
for spec in args.params:
scripts/optimizer/reembed_gallery.py
+9 -3
View File
@@ -27,8 +27,9 @@ from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
from sae_embed_loader import load_embedder # noqa: E402
from sae_gallery import load_gallery_hdf5, save_gallery_hdf5 # noqa: E402
from sae_embed_loader import load_embedder, resolve_arcface # noqa: E402
from sae_gallery import (embedder_stamp, load_gallery_hdf5, # noqa: E402
save_gallery_hdf5)
def find_dir(images_root: Path, jellyfin_id: str, name: str) -> Path | None:
@@ -58,6 +59,11 @@ def main():
ref = load_gallery_hdf5(Path(args.ref))
images_root = Path(args.images)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001 — this script exists to produce a gallery in a
# DIFFERENT model's space from the reference. The output must therefore never
# inherit the reference's stamp; it carries the stamp of --arcface, which is
# the whole point of the bake-off being safe to run.
stamp = embedder_stamp(resolve_arcface(args.models_dir, args.arcface))
out_actors = []
n_ok = n_nodir = n_noemb = 0
@@ -84,7 +90,7 @@ def main():
print(f" [{i}/{total}] ok={n_ok} no_dir={n_nodir} no_emb={n_noemb}",
file=sys.stderr)
save_gallery_hdf5({"actors": out_actors}, Path(args.out))
save_gallery_hdf5({"actors": out_actors}, Path(args.out), stamp)
n_emb = sum(len(a["embeddings"]) for a in out_actors)
print(f"[reembed] {Path(args.arcface).stem}: {n_ok}/{total} actors, {n_emb} embeddings "
f"→ {args.out}", file=sys.stderr)
scripts/optimizer/replay.py
+35 -1
View File
@@ -2,6 +2,8 @@
"""
replay.py — replay a dumped embedding HDF5 through the real KPN downstream nodes.
TRACES: VR-002 | PR-002
Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an
EmbeddedSceneFrame into a Python-assembled KPN network wiring the *real* C++
face_tracker → identity_matcher → scene_tracker, and returns the same presence-window
@@ -25,6 +27,22 @@ import h5py
import numpy as np
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
from sae_stamp import verify_gallery_stamp # noqa: E402
def dump_embedder_stamp(dump_path: str) -> dict:
"""The GR-004 embedder stamp recorded in an embedding dump.
A replay has no live embedder — the dump IS the embedder as far as the gallery
is concerned, so the dump's stamp is what the gallery must be checked against.
Dumps written before GR-004 have no attributes and yield an empty stamp, which
the check reports as unverifiable rather than silently accepting."""
with h5py.File(dump_path, "r") as f:
name = f.attrs.get("embedder_model", "")
sha = f.attrs.get("embedder_sha256", "")
dec = lambda v: v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
return {"model_name": dec(name), "model_sha256": dec(sha), "embed_dim": 512}
def load_frames(dump_path: str, min_conf: float = 0.0):
@@ -92,6 +110,15 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool =
sys.path.insert(0, build_dir)
import sae_kpn
# TRACES: GR-004 | SR-001 — checked here, before any network is built, so a
# cross-model replay dies with one readable error instead of producing a
# plausible-looking score. add_identity_matcher re-checks it C++-side below;
# that is the backstop for any other caller of the binding.
stamp = dump_embedder_stamp(dump_path)
verify_gallery_stamp(gallery, stamp=stamp,
embedder_desc=f"embedding dump {Path(dump_path).name}",
require_stamp=bool(cfg.get("require_gallery_stamp", False)))
frames, movie, fps = load_frames(dump_path, min_conf=float(cfg.get("detector_conf", 0.0)))
net = sae_kpn.Network()
@@ -122,7 +149,8 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool =
cap = len(frames) * 2 + 64
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], cap)
sae_kpn.add_face_tracker(net, "tracker", cfg, cap)
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap)
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap,
stamp["model_name"], stamp["model_sha256"])
sae_kpn.add_scene_tracker(net, "scene", cfg, cap)
net.connect("replay", 0, "tracker", 0)
net.connect("tracker", 0, "matcher", 0)
@@ -221,11 +249,17 @@ def main():
# per-film gallery expansion: promotes pose-varied views of confidently-identified
# actors into an in-memory annex, recovering ~+4 recall at no precision cost.
p.add_argument("--expand-gallery", action="store_true")
# TRACES: GR-004 | SR-001 — promote an unprovable gallery/dump binding from a
# loud warning to a hard error. Measurement sweeps should set this (or
# SAE_REQUIRE_GALLERY_STAMP=1) so no number comes from an unbound pair.
p.add_argument("--require-gallery-stamp", action="store_true")
args = p.parse_args()
cfg = {k: getattr(args, k) for k in CFG_KEYS if getattr(args, k) is not None}
if args.expand_gallery:
cfg["expand_gallery"] = True
if args.require_gallery_stamp:
cfg["require_gallery_stamp"] = True
# stop=True: PyNode::stop() sets stop_flag_ before joining, so the source
# thread's run_loop actually exits. stop=False skips that, leaving stop_flag_
# false forever — the PyNode destructor's jthread.join() then blocks forever
scripts/optimizer/second_score.py
+2
View File
@@ -2,6 +2,8 @@
"""
second_score.py — uniform per-second agreement with X-Ray.
TRACES: VR-003 | PR-002
Unlike scene_score.py (which unions our detections over a whole X-Ray scene), this
samples EVERY SECOND of the film and asks: at second t, do we name the same actors
X-Ray says are on screen?
scripts/run_from_jellyfin.py
+2
View File
@@ -1,6 +1,8 @@
#!/usr/bin/env python3
"""run_from_jellyfin.py — resolve a Jellyfin title to its media file and run scene_analyze.
TRACES: IR-006 | SR-001
Looks up a Movie/Episode in Jellyfin, reads its on-disk Path (Jellyfin and this
tool must share the same media mount), filters the gallery down to that
title's credited cast (via filter_gallery's logic, fewer look-alike
scripts/sae_embed_loader.py
+27 -2
View File
@@ -3,11 +3,26 @@
sae_embed.FaceEmbedder loads both ONNX sessions once and exposes an
embed(path) -> FaceResult method, avoiding the per-process model reload cost
of spawning the embed_faces CLI binary for every image.
resolve_arcface() exposes the same default-resolution logic load_embedder uses,
so a caller can stamp the gallery it is about to write with the model that
actually produced its embeddings (GR-004) — the resolved path, not the CLI
argument, which is often None.
"""
import sys
from pathlib import Path
DEFAULT_ARCFACE = "arcface_w600k_r50.onnx"
def resolve_arcface(models_dir: str, arcface: str | None = None) -> str:
"""The ArcFace/LVFace ONNX path load_embedder would use for these arguments.
TRACES: GR-004 | SR-001 — single source of truth for "which model is this",
so the stamp written into a gallery can never drift from the model loaded."""
return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)
def load_embedder(build_dir: str, models_dir: str, arcface: str | None = None,
conf: float = 0.5, nms: float = 0.4, max_side: int = 500):
@@ -28,9 +43,19 @@ def load_embedder(build_dir: str, models_dir: str, arcface: str | None = None,
models_path = Path(models_dir)
detector_path = str(models_path / "scrfd_500m_bnkps.onnx")
arcface_path = arcface if arcface else str(models_path / "arcface_w600k_r50.onnx")
arcface_path = resolve_arcface(models_dir, arcface)
for model, name in [(detector_path, "SCRFD"), (arcface_path, "ArcFace")]:
if not Path(model).is_file():
sys.exit(f"{name} model not found: {model}\nRun: bash scripts/download_models.sh")
return sae_embed.FaceEmbedder(detector_path, arcface_path, conf, nms, max_side)
# A TRT-backend build cannot load .onnx; it needs pre-built engines from
# scripts/build_trt_engines.sh. Pass them when present (ignored by ORT).
trt = Path(models_path).parent / "trt_cache"
det_engine = trt / "scrfd.scrfd_500m_bnkps.640.fp16.engine"
arc_engine = trt / f"arcface.{Path(arcface_path).stem}.b4.fp16.engine"
return sae_embed.FaceEmbedder(
detector_path, arcface_path, conf, nms, max_side,
str(det_engine) if det_engine.is_file() else "",
str(arc_engine) if arc_engine.is_file() else "",
)
scripts/sae_gallery.py
+60 -10
View File
@@ -6,9 +6,13 @@ make_jellyfin_gallery.download_urls + download_person_images) and the duplicated
Galleries are written directly as HDF5 — never JSON. Same layout the C++ side
reads/writes (src/gallery/gallery_store.cpp): flat [N,512] embeddings + per-actor
offset/count, parallel imdb_id/tmdb_id/jellyfin_id/name string arrays, and a
per-embedding-row source_images array. calibration is left absent (calib_hash=0);
the C++ identity_matcher fits and writes it back into the file on first use.
offset/count, parallel imdb_id/tmdb_id/jellyfin_id/name string arrays, a
per-embedding-row source_images array, and an /embedder group carrying the
GR-004 model binding. calibration is left absent (calib_hash=0); the C++
identity_matcher fits and writes it back into the file on first use.
The GR-004 embedder stamp written into that /embedder group lives in sae_stamp
and is re-exported below, so existing callers keep importing it from here.
"""
import io
@@ -101,11 +105,36 @@ def download_images(urls: list[str], dest_dir: Path, n: int,
return paths
def save_gallery_hdf5(gallery: dict, output: Path) -> None:
# ── GR-004: gallery ↔ embedder binding ───────────────────────────────────────
# Implemented in sae_stamp (kept dependency-light so the optimizer's replay
# subprocesses can import it without pulling requests/Pillow); re-exported here
# because the gallery writers and every existing caller reach for it via this
# module. See src/gallery/embedder_stamp.hpp for the C++ twin and the rationale.
from sae_stamp import ( # noqa: F401
EmbedderMismatch,
check_embedder_stamp,
describe_stamp,
embedder_stamp,
enforce_embedder_stamp,
read_gallery_stamp,
require_gallery_stamp_from_env,
sha256_file,
verify_gallery_stamp,
_as_str,
_stamp_empty,
)
def save_gallery_hdf5(gallery: dict, output: Path, embedder: dict | None = None) -> None:
"""Write a gallery dict ({"actors": [...]}) directly as HDF5 — same schema
src/gallery/gallery_store.cpp reads/writes. No calibration group; the
C++ identity_matcher computes and writes it back into this file on first
use against an unseen set of embeddings."""
use against an unseen set of embeddings.
`embedder` is the GR-004 stamp (see embedder_stamp()); it may also be carried
on the gallery dict under "embedder", which is how a filtered/derived gallery
keeps its binding without the caller having to re-hash anything."""
embedder = embedder if embedder is not None else gallery.get("embedder")
actors = gallery["actors"]
embs, offsets, counts = [], [], []
imdb, tmdb, jf, name, src_images = [], [], [], [], []
@@ -139,8 +168,17 @@ def save_gallery_hdf5(gallery: dict, output: Path) -> None:
f.create_dataset("jellyfin_id", data=np.asarray(jf, dtype=object), dtype=str_t)
f.create_dataset("name", data=np.asarray(name, dtype=object), dtype=str_t)
f.create_dataset("source_images", data=np.asarray(src_images, dtype=object), dtype=str_t)
print(f"Saved: {output} ({len(actors)} actors, {emb_arr.shape[0]} embeddings)",
file=sys.stderr)
# TRACES: GR-004 | SR-001 — omitted entirely when unknown, so "unstamped"
# round-trips as unstamped rather than as a stamp naming no model.
if not _stamp_empty(embedder):
g = f.create_group("embedder")
g.attrs["model_name"] = embedder.get("model_name", "")
g.attrs["model_sha256"] = embedder.get("model_sha256", "")
g.attrs["embed_dim"] = np.int32(embedder.get("embed_dim", 512))
stamp_note = (f", embedder {embedder['model_name']}" if not _stamp_empty(embedder)
else ", NO EMBEDDER STAMP (GR-004)")
print(f"Saved: {output} ({len(actors)} actors, {emb_arr.shape[0]} embeddings"
f"{stamp_note})", file=sys.stderr)
def load_gallery_hdf5(path: Path) -> dict:
@@ -158,6 +196,14 @@ def load_gallery_hdf5(path: Path) -> dict:
if "source_images" in f:
src_images = [s.decode() if isinstance(s, bytes) else s
for s in f["source_images"][:]]
# TRACES: GR-004 | SR-001 — carried through so a derived gallery (filter,
# merge, cast-restrict) keeps the binding of the gallery it came from.
stamp = None
if "embedder" in f:
a = f["embedder"].attrs
stamp = {"model_name": _as_str(a.get("model_name", "")),
"model_sha256": _as_str(a.get("model_sha256", "")),
"embed_dim": int(a.get("embed_dim", 512))}
actors = []
for a in range(len(offset)):
@@ -167,15 +213,19 @@ def load_gallery_hdf5(path: Path) -> dict:
if src_images is not None:
actor["source_images"] = [src_images[s + i] for i in range(n)]
actors.append(actor)
return {"actors": actors}
out = {"actors": actors}
if stamp is not None:
out["embedder"] = stamp
return out
def save_gallery(gallery: dict, missing: list[dict], output: Path) -> None:
def save_gallery(gallery: dict, missing: list[dict], output: Path,
embedder: dict | None = None) -> None:
"""Write the gallery as HDF5 (forcing a .h5 extension) and, if any actors
lack images, a .missing_images.json sidecar."""
if output.suffix not in (".h5", ".hdf5"):
output = output.with_suffix(".h5")
save_gallery_hdf5(gallery, output)
save_gallery_hdf5(gallery, output, embedder)
if missing:
missing_path = output.with_name(output.stem + ".missing_images.json")
scripts/sae_stamp.py
+217
View File
@@ -0,0 +1,217 @@
"""Gallery ↔ embedder model binding (GR-004).
TRACES: GR-004 | SR-001
Python twin of src/gallery/embedder_stamp.{hpp,cpp}; the two implement the same
comparison rules and must stay in agreement. Kept as its own module — rather than
folded into sae_gallery — because scripts/optimizer/replay.py imports it once per
replay subprocess, thousands of times in a DE sweep, and must not pay for
sae_gallery's requests/Pillow imports to ask "were these made by the same model?".
Dependencies here are hashlib, json and h5py, all of which a replay already loads.
A gallery is only valid for the embedder that built it: cosine similarities across
models are meaningless but look plausible, so the mistake is silent and every
measurement taken afterwards is suspect. Identity = model filename + SHA-256 of
the model file. The hash decides (a model re-exported in place keeps its name but
not its bytes); the name is what makes the error readable. See
src/gallery/embedder_stamp.hpp for the full rationale.
"""
import hashlib
import json
import os
import sys
from pathlib import Path
import h5py
def _as_str(v) -> str:
return v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
_STAMP_CACHE: dict = {}
class EmbedderMismatch(RuntimeError):
"""Gallery was built with a different embedder than the one about to be used."""
def sha256_file(path) -> str:
"""Lowercase hex SHA-256 of a file's bytes; "" if it cannot be read."""
path = Path(path)
try:
st = path.stat()
except OSError:
return ""
key = (str(path), st.st_mtime_ns, st.st_size)
if key in _STAMP_CACHE:
return _STAMP_CACHE[key]
h = hashlib.sha256()
try:
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
except OSError:
return ""
_STAMP_CACHE[key] = h.hexdigest()
return _STAMP_CACHE[key]
def embedder_stamp(model_path, embed_dim: int = 512) -> dict:
"""Identify an embedder model file → {"model_name", "model_sha256", "embed_dim"}.
A model file that is absent (e.g. a TRT deployment running from a prebuilt
.engine) yields a name-only stamp: still comparable, just not provable."""
if not model_path:
return {"model_name": "", "model_sha256": "", "embed_dim": embed_dim}
sha = sha256_file(model_path)
if not sha:
print(f"[gallery] cannot hash embedder model {model_path} — model binding "
f"falls back to filename only (GR-004)", file=sys.stderr)
return {"model_name": Path(model_path).name, "model_sha256": sha,
"embed_dim": embed_dim}
def _stamp_empty(s) -> bool:
return not s or (not s.get("model_name") and not s.get("model_sha256"))
def describe_stamp(s) -> str:
if _stamp_empty(s):
return "UNKNOWN"
name = s.get("model_name") or "<unnamed model>"
sha = s.get("model_sha256") or ""
return f"{name} (sha256 {sha[:12]}…)" if sha else f"{name} (sha256 unavailable)"
def require_gallery_stamp_from_env() -> bool:
"""SAE_REQUIRE_GALLERY_STAMP=1 → an unprovable binding is fatal, not a warning."""
return os.environ.get("SAE_REQUIRE_GALLERY_STAMP", "0") not in ("", "0")
def check_embedder_stamp(built_with: dict | None, loading_with: dict | None,
gallery_desc: str = "gallery",
embedder_desc: str = "embedder") -> tuple[str, str]:
"""Pure comparison. Returns (verdict, message); verdict is one of
match / weak_match / unstamped / unknown_embedder / mismatch.
Same rules as compare_embedder_stamps() in src/gallery/embedder_stamp.cpp."""
if _stamp_empty(built_with):
return "unstamped", (
f"gallery '{gallery_desc}' carries no embedder stamp (GR-004).\n"
f" gallery was built with : UNKNOWN — this file predates model binding\n"
f" embedder now loaded : {describe_stamp(loading_with)} [{embedder_desc}]\n"
f" If these are not the same model every similarity from this run is\n"
f" meaningless but will look plausible. Rebuild or re-stamp the gallery\n"
f" (scripts/stamp_gallery.py), or run with SAE_REQUIRE_GALLERY_STAMP=1 to\n"
f" make this a hard error.")
if _stamp_empty(loading_with):
return "unknown_embedder", (
f"cannot identify the embedder being used against gallery "
f"'{gallery_desc}' (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)}\n"
f" embedder now loaded : UNKNOWN [{embedder_desc}]\n"
f" The binding cannot be checked, so it is not being checked.")
mismatch_tail = (
" Cosine similarities between embeddings from different models are\n"
" meaningless but look plausible. Rebuild the gallery with the loaded\n"
" model, or point the embedder at the model the gallery was built with.")
if int(built_with.get("embed_dim", 512)) != int(loading_with.get("embed_dim", 512)):
return "mismatch", (
f"gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)}, "
f"dim={built_with.get('embed_dim')} [{gallery_desc}]\n"
f" embedder now loaded : {describe_stamp(loading_with)}, "
f"dim={loading_with.get('embed_dim')} [{embedder_desc}]\n"
f" Embedding dimensions differ; these are not the same space.")
a, b = built_with.get("model_sha256", ""), loading_with.get("model_sha256", "")
if a and b:
if a == b:
note = ""
if built_with.get("model_name") != loading_with.get("model_name"):
note = (f" (gallery recorded it as '{built_with.get('model_name')}', "
f"loaded from '{loading_with.get('model_name')}' — "
f"same bytes, renamed file)")
return "match", f"embedder binding verified: {describe_stamp(built_with)}{note}"
return "mismatch", (
f"gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
f" gallery was built with : {built_with.get('model_name')} sha256={a}\n"
f" [{gallery_desc}]\n"
f" embedder now loaded : {loading_with.get('model_name')} sha256={b}\n"
f" [{embedder_desc}]\n" + mismatch_tail)
if built_with.get("model_name") and \
built_with.get("model_name") == loading_with.get("model_name"):
return "weak_match", (
f"embedder binding UNPROVEN for gallery '{gallery_desc}' (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)}\n"
f" embedder now loaded : {describe_stamp(loading_with)} [{embedder_desc}]\n"
f" Filenames agree but at least one SHA-256 is unavailable, so an\n"
f" in-place re-export under the same name would not be detected.")
return "mismatch", (
f"gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)} [{gallery_desc}]\n"
f" embedder now loaded : {describe_stamp(loading_with)} [{embedder_desc}]\n"
+ mismatch_tail)
def enforce_embedder_stamp(built_with, loading_with, gallery_desc, embedder_desc,
require_stamp: bool = False) -> str:
"""Apply check_embedder_stamp: raise EmbedderMismatch when fatal, else warn.
A mismatch is fatal unconditionally — there is no bypass, because a mismatch is
a known-wrong state, not an unknown one. The three "cannot prove it" verdicts
warn loudly and become fatal under require_stamp / SAE_REQUIRE_GALLERY_STAMP."""
strict = require_stamp or require_gallery_stamp_from_env()
verdict, msg = check_embedder_stamp(built_with, loading_with,
gallery_desc, embedder_desc)
if verdict == "mismatch":
raise EmbedderMismatch(msg)
if strict and verdict != "match":
raise EmbedderMismatch(
msg + "\n (fatal because SAE_REQUIRE_GALLERY_STAMP is set)")
if verdict == "match":
print(f"[gallery] {msg}", file=sys.stderr)
else:
print(f"\n[gallery] ***** WARNING (GR-004) *****\n{msg}\n"
f"[gallery] ****************************\n", file=sys.stderr)
return verdict
def read_gallery_stamp(path) -> dict | None:
"""The embedder stamp recorded in a gallery file, or None if unstamped.
Handles both the HDF5 /embedder group and the legacy JSON "embedder" object."""
path = Path(path)
if path.suffix in (".h5", ".hdf5"):
with h5py.File(path, "r") as f:
if "embedder" not in f:
return None
a = f["embedder"].attrs
return {"model_name": _as_str(a.get("model_name", "")),
"model_sha256": _as_str(a.get("model_sha256", "")),
"embed_dim": int(a.get("embed_dim", 512))}
data = json.loads(path.read_text())
return data.get("embedder") or None
def verify_gallery_stamp(gallery_path, model_path=None, *, stamp=None,
embedder_desc: str | None = None,
require_stamp: bool = False) -> str:
"""Load a gallery's stamp and check it against a model file (or an explicit
stamp, e.g. one read off an embedding dump). Raises EmbedderMismatch."""
loading = stamp if stamp is not None else embedder_stamp(model_path)
return enforce_embedder_stamp(read_gallery_stamp(gallery_path), loading,
str(gallery_path),
embedder_desc or str(model_path or "unknown"),
require_stamp)
def _as_str(v) -> str:
return v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
scripts/stamp_gallery.py
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""stamp_gallery.py — bind an existing gallery to the embedder that built it.
TRACES: GR-004 | SR-001
Galleries built before model binding carry no embedder stamp. They still load,
but every consumer warns that it cannot tell whether the gallery and the embedder
belong together — and under SAE_REQUIRE_GALLERY_STAMP=1 they refuse to run.
This is the migration path, and the reason the unstamped case is a warning rather
than a hard failure: re-binding an existing gallery costs one command and no
re-embedding, so nobody has to choose between a bricked setup and a check they
route around.
python scripts/stamp_gallery.py --gallery gallery.h5 \\
--arcface models/LVFace-B_Glint360K.onnx
The stamp is an ASSERTION: you are stating which model produced these vectors.
Nothing can verify it from the vectors themselves, which is exactly why the stamp
has to be written at build time going forward. Stamping the wrong model is worse
than leaving it unstamped, because it converts a loud warning into a false
all-clear — so --show it first if you are not certain.
python scripts/stamp_gallery.py --gallery gallery.h5 --show
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
import h5py
sys.path.insert(0, str(Path(__file__).resolve().parent))
from sae_gallery import (describe_stamp, embedder_stamp, # noqa: E402
read_gallery_stamp)
def main() -> int:
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--gallery", required=True, help="gallery .h5 to stamp in place")
p.add_argument("--arcface", help="the ONNX that built it (hashed into the stamp)")
p.add_argument("--show", action="store_true", help="print the current stamp and exit")
p.add_argument("--force", action="store_true",
help="overwrite an existing stamp (refused otherwise)")
args = p.parse_args()
path = Path(args.gallery)
if path.suffix not in (".h5", ".hdf5"):
return err(f"{path}: only HDF5 galleries can be stamped in place")
current = read_gallery_stamp(path)
print(f"{path}: current stamp = {describe_stamp(current)}", file=sys.stderr)
if args.show:
return 0
if not args.arcface:
return err("--arcface is required (or use --show)")
if current and not args.force:
return err("gallery is already stamped — pass --force to overwrite, but be "
"sure: a wrong stamp turns a warning into a false all-clear")
stamp = embedder_stamp(args.arcface)
if not stamp["model_sha256"]:
return err(f"cannot hash {args.arcface} — refusing to write a name-only "
"stamp, which would claim more certainty than it has")
with h5py.File(path, "r+") as f:
if "embedder" in f:
del f["embedder"]
g = f.create_group("embedder")
g.attrs["model_name"] = stamp["model_name"]
g.attrs["model_sha256"] = stamp["model_sha256"]
g.attrs["embed_dim"] = stamp["embed_dim"]
print(f"{path}: stamped with {describe_stamp(stamp)}", file=sys.stderr)
return 0
def err(msg: str) -> int:
print(f"[stamp_gallery] {msg}", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())
scripts/validation/README.md
+21
View File
@@ -79,9 +79,30 @@ The table caches nulls (tmdb ids TMDB has no IMDb id for) and checkpoints, so a
re-run only resolves new ids. TMDB is authoritative for this crosswalk — there is
no clean free bulk `tmdb_person ↔ nm` file, so we query the API once and cache.
## Minimum face size (VR-005)
`min_face_size.py` is a separate, self-contained study: it needs no video and no
ground truth, only the gallery mugshot cache. It holds out one image per actor,
degrades that probe to each candidate face size and matches it against a gallery
held at **native** resolution, reporting TPI/FPI per size — the measurement that
replaces AR-002's 66×66 px estimate.
```bash
python scripts/validation/min_face_size.py \
--images images --gallery gallery_lvface.h5 \
--arcface models/LVFace-B_Glint360K.onnx \
--actors 100 --out experiments/results/vr005_min_face_size
```
FPI grows with the number of actors competing, so a 100-actor run understates it
against a library of thousands: read FPI as relative across sizes, not as an
absolute rate. Re-run per `--arcface` model to see whether `min_face_px` should be
one constant or scale with the embedder (GR-004).
## Files
- `sample_eval.py` — CLI scorer.
- `ground_truth.py` — `XRayGroundTruth`, `MovieNetGroundTruth` loaders.
- `identity.py` — provider-agnostic match keys.
- `tmdb_imdb_map.py` — build/consult the cached `tmdb→imdb` crosswalk.
- `min_face_size.py` — VR-005 probe-size sweep (see above).
- `test_sample_eval.py` — self-contained tests (`python scripts/validation/test_sample_eval.py`).
scripts/validation/min_face_size.py
+828
View File
@@ -0,0 +1,828 @@
#!/usr/bin/env python3
"""
min_face_size.py — VR-005: at what face size do embeddings stop identifying people?
TRACES: VR-005
`min_face_px` is currently a working estimate (AR-002: 66x66 px in original video
resolution). This script replaces the guess with a measurement, using only gallery
mugshots already on disk — no video, no C++ changes.
Protocol
--------
1. Select ~100 gallery actors that have more than one mugshot.
2. Per actor hold out ONE image as the *probe*; that actor's remaining images stay
in the gallery at native resolution.
3. For each target size S, take the probe's native aligned 112x112 crop, downscale
it to SxS and upscale it back to 112x112, then embed. Detail is genuinely
destroyed and then the same warp the pipeline applies is re-applied on top —
which is what a face detected at SxS in a frame actually suffers.
4. Match each degraded probe against the whole gallery.
5. Record, per size, TPI (identified as the correct actor) and FPI (identified as
someone else). Everything else is an unidentified probe (TBI).
The asymmetry is the point: **the gallery stays at native resolution and only the
probe degrades.** That is the production case — reference mugshots are clean, the
face coming out of the video is small. Degrading both sides would measure
something the pipeline never does.
What this deliberately does NOT measure
---------------------------------------
The cosine between the size-S embedding and the native embedding of the *same*
image. That is embedding *drift*, and it answers the wrong question: an embedding
can drift a long way and stay perfectly separable, or drift a little in a
direction that destroys separation. What matters is the decision the pipeline
makes — probe against a competing gallery — so that is what is recorded.
CAVEAT — FPI IS RELATIVE, NOT ABSOLUTE
--------------------------------------
False positives grow with the number of actors competing for the match. A
~100-actor gallery therefore *understates* the false-positive rate against a
production library of thousands. Read the FPI column as a relative curve across
sizes ("FPI is 4x worse at 32 px than at 64 px"), never as the rate you would see
in production. Re-run with `--actors` at production scale before setting a
threshold from an absolute FPI number.
Decision rule
-------------
Per the repo invariant (CLAUDE.md: "always use the calibrated probability, never a
raw cosine"), identification goes through the same path as `identity_matcher_node`:
per-actor best-of-N cosine -> Platt sigmoid P(match) = sigma(a*sim + b + log-prior)
-> accept if P > `prob_threshold`. The sigmoid is fitted here by the same
histogram/gradient-descent procedure as `src/gallery/gallery_calibration.hpp`,
over the native gallery embeddings only (held-out probes are excluded, so the
calibration cannot see the images it will be scored on).
How this runs
-------------
Through the `sae_embed` bindings, which expose the shipped C++ stages directly:
`detect()`, `align_face()`, `embed_crops()` and `GalleryCalibration`. Nothing
here re-implements detection, the ArcFace warp, the embedder or the Platt fit.
That matters most for the calibration. A second copy of the sigmoid is exactly
where "always the calibrated probability, never a raw cosine" (AR-024) gets
broken without anyone noticing, because the copy keeps returning plausible
numbers after the original has moved. Scoring through the binding makes the rule
structural instead of remembered.
The backend is whichever was compiled in. Under `SAE_INFERENCE_BACKEND=ORT`
that is the reference fp32 path, which loads the .onnx directly. A TensorRT fp16
build is a *different realisation* of the same model and its embeddings are
measurably not the same vectors: on LVFace-B_Glint360K the stored TRT-fp16 gallery
agrees with an fp32 recompute of the same mugshot at only ~0.85 cosine, while
same-actor/different-actor separation is essentially unchanged (d' 5.3 vs 5.7).
Nothing here is invalidated by that — gallery and probes go through one session,
so the comparison is internally consistent — but the two embedding spaces are not
interchangeable, and `--verify-against <gallery.h5>` will show ~0.85, not ~1.0,
against a TRT-built gallery. It reports the separation of both sets alongside the
agreement so the two causes are distinguishable: a broken port collapses
separation, a different backend does not.
Secondary output (VR-005): running the sweep per `--arcface` model shows whether
`min_face_px` should be one constant at all, or should scale with the embedder —
which matters because the model is a build-time choice (GR-004).
Usage
-----
python scripts/validation/min_face_size.py \
--images images \
--gallery gallery_lvface.h5 \
--arcface models/LVFace-B_Glint360K.onnx \
--actors 100 --seed 0 \
--out experiments/results/vr005_min_face_size
Writes <out>.csv, <out>.json and <out>.png (plus <out>.per_probe.csv with
--per-probe).
"""
from __future__ import annotations
import argparse
import csv
import json
import random
import re
import sys
import time
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
def _find_sae_embed() -> Path | None:
"""Locate the built sae_embed module.
A git worktree has no build tree of its own, so fall back to the main
checkout via the shared git dir — otherwise running this study from a
feature worktree cannot find the bindings it now depends on.
"""
roots = [REPO]
try:
import subprocess
common = subprocess.run(["git", "-C", str(REPO), "rev-parse",
"--path-format=absolute", "--git-common-dir"],
capture_output=True, text=True, check=True).stdout.strip()
if common:
roots.append(Path(common).parent)
except Exception:
pass
for root in roots:
for b in ("build-ort", "build"):
if list((root / b).glob("sae_embed*.so")):
return root / b
return None
_SAE_BUILD = _find_sae_embed()
if _SAE_BUILD is None:
sys.exit("cannot find the built sae_embed module — build it with\n"
" cmake --build build-ort --target sae_embed")
sys.path.insert(0, str(_SAE_BUILD))
# Before cv2: OpenCV's DNN module loads the system libonnxruntime, which then
# shadows the one sae_embed links against and the import fails on a missing
# symbol version. Order matters here.
import sae_embed
import cv2
import numpy as np
IMAGE_EXTS = (".jpg", ".jpeg", ".png", ".webp")
JELLYFIN_ID_RE = re.compile(r"^[0-9a-f]{32}$")
# Interpolation used for the two halves of the degradation. Downscaling uses
# INTER_AREA (correct low-pass for shrinking, i.e. detail that a small detection
# genuinely never had); upscaling uses INTER_LINEAR, which is what warpAffine in
# align_face() uses when it blows a small detection up to 112x112.
INTERP = {
"area": cv2.INTER_AREA,
"linear": cv2.INTER_LINEAR,
"cubic": cv2.INTER_CUBIC,
"nearest": cv2.INTER_NEAREST,
"lanczos": cv2.INTER_LANCZOS4,
}
# House chart palette, shared with scripts/docs/experiment_charts.py so figures
# across the report read as one set.
INK, MUTED, GRID, SURFACE = "#0b0b0b", "#898781", "#e1e0d9", "#fcfcfb"
BLUE, GREEN, RED, AMBER = "#2a78d6", "#008300", "#e34948", "#eda100"
# ── Production stages, via the sae_embed bindings ─────────────────────────────
# detect / align_face / embed_crops / calibrate_gallery all call the shipped C++.
# There is deliberately no Python re-implementation of any of them: a second copy
# drifts from what ships, and the calibration is the one that must not — AR-024
# requires every similarity to pass through the same sigmoid the matcher uses.
# These two mirror constants in gallery_calibration.hpp. They are NOT a second
# copy of the fit — that is the binding's job — but the script reproduces the
# same dedup and eligibility filtering so the actor counts it reports describe
# the population the C++ actually fitted on. Keep them in step with the header.
MIN_EMB_FOR_POSITIVE = 5
DEDUP_SIM = 1.0 - 1e-7
class Stages:
"""Thin holder so the rest of the script has one object to call."""
def __init__(self, detector: str, arcface: str, conf: float, nms: float,
detector_engine: str = "", arcface_engine: str = ""):
# The engine paths are only consulted by a TRT-backend build, where they
# are mandatory — that backend loads a pre-built .engine and will not
# fall back to reading the .onnx. An ORT build ignores them, so passing
# them unconditionally is safe and keeps one constructor for both.
self.engine = sae_embed.FaceEmbedder(
detector_model=detector, arcface_model=arcface,
conf=conf, nms=nms, max_side=0,
detector_engine=detector_engine, arcface_engine=arcface_engine)
def detect(self, img):
return self.engine.detect(img)
def align(self, img, landmarks):
return sae_embed.align_face(img, np.asarray(landmarks, dtype=np.float32).reshape(5, 2))
def enhance(self, img):
return sae_embed.enhance_for_retry(img)
def embed(self, crops):
"""(N,112,112,3) uint8 BGR -> (N,512) float32.
Chunked at the backend's max_batch: the engine does not split an
oversized request, so handing it a whole gallery at once asks CUDA for
a multi-gigabyte activation buffer and the allocator refuses.
"""
if not len(crops):
return np.zeros((0, 512), dtype=np.float32)
n = max(1, int(self.engine.max_batch))
arr = np.ascontiguousarray(np.stack(crops), dtype=np.uint8)
out = [np.asarray(self.engine.embed_crops(np.ascontiguousarray(arr[i:i + n])))
for i in range(0, len(arr), n)]
return np.concatenate(out, axis=0)
def calibrate_gallery(emb: np.ndarray, actor: np.ndarray) -> dict:
"""The production Platt fit (gallery_calibration.hpp), via the binding."""
cal = sae_embed.calibrate_gallery(
np.ascontiguousarray(emb, dtype=np.float32), [int(a) for a in actor])
print(f"[calibration] a={cal.a:.4f} b={cal.b:.4f} valid={cal.valid} "
f"boundary(P=0.5)=sim{cal.boundary_at(0.5):.4f}", file=sys.stderr)
# Held module-side rather than returned: the returned dict lands in the run
# metadata, and a native object there breaks the JSON dump.
_CAL["cal"] = cal
return {"a": float(cal.a), "b": float(cal.b), "valid": bool(cal.valid)}
def probability(sim, a: float, b: float, log_prior_odds: float = 0.0):
"""P(match) through GalleryCalibration — the C++ sigmoid, not a copy of it."""
cal = _CAL.get("cal")
if cal is None:
raise RuntimeError("probability() called before calibrate_gallery()")
sim = np.asarray(sim, dtype=np.float64)
flat = np.atleast_1d(sim).ravel()
out = np.array([cal.probability(float(v), log_prior_odds) for v in flat])
return out.reshape(sim.shape) if sim.shape else float(out[0])
_CAL: dict = {}
# ── Runtime / actor discovery ─────────────────────────────────────────────────
def normalise_name(name: str) -> str:
return re.sub(r"[^a-z0-9]+", "", name.lower())
def discover_actors(images_root: Path) -> list[dict]:
"""Enumerate the gallery-build image cache: <root>/<jellyfin_id>_<Name>/NN.jpg
(the layout make_jellyfin_gallery.py / reembed_gallery.py use)."""
actors = []
for d in sorted(p for p in images_root.iterdir() if p.is_dir()):
imgs = sorted(p for p in d.iterdir()
if p.is_file() and p.suffix.lower() in IMAGE_EXTS)
if not imgs:
continue
head, _, tail = d.name.partition("_")
if JELLYFIN_ID_RE.match(head) and tail:
jellyfin_id, name = head, tail.replace("_", " ")
else:
jellyfin_id, name = "", d.name.replace("_", " ")
actors.append({"dir": d, "jellyfin_id": jellyfin_id, "name": name,
"images": imgs})
return actors
def gallery_keys(gallery_path: Path) -> tuple[set[str], set[str]]:
"""(jellyfin ids, normalised names) of the actors an existing gallery holds."""
from sae_gallery import load_gallery_hdf5
g = load_gallery_hdf5(gallery_path)
ids = {a.get("jellyfin_id", "") for a in g["actors"] if a.get("jellyfin_id")}
names = {normalise_name(a.get("name", "")) for a in g["actors"] if a.get("name")}
return ids, names
# ── Degradation ───────────────────────────────────────────────────────────────
def degrade(crop: np.ndarray, size: int, down: int, up: int) -> np.ndarray:
"""Throw away everything a face detected at size x size never had, then warp
it back up to the 112x112 the embedder is fed."""
if size == 112:
return crop
small = cv2.resize(crop, (size, size), interpolation=down)
return cv2.resize(small, (112, 112), interpolation=up)
# ── Reporting ─────────────────────────────────────────────────────────────────
CAVEAT = (
"CAVEAT: FPI grows with gallery size. This ran against {n_actors} actors, so it "
"UNDERSTATES the false-positive rate of a production library of thousands. Read "
"FPI as relative across sizes, not as an absolute rate."
)
def write_plot(rows: list[dict], out_png: Path, meta: dict) -> None:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
plt.rcParams.update({
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE,
"savefig.facecolor": SURFACE, "text.color": INK,
"axes.edgecolor": MUTED, "axes.labelcolor": INK,
"xtick.color": MUTED, "ytick.color": MUTED,
"axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.8,
"axes.spines.top": False, "axes.spines.right": False,
})
sizes = [r["size_px"] for r in rows]
fig, ax = plt.subplots(figsize=(9, 5.6))
ax.plot(sizes, [100 * r["tpi_rate"] for r in rows], "-o", color=GREEN,
lw=2, label="TPI — identified, correct actor")
ax.plot(sizes, [100 * r["fpi_rate"] for r in rows], "-s", color=RED,
lw=2, label="FPI — identified, wrong actor")
ax.plot(sizes, [100 * r["unidentified_rate"] for r in rows], color=MUTED,
marker="^", lw=1.4, ls="--", label="unidentified (below P threshold)")
ax.plot(sizes, [100 * r["rank1_rate"] for r in rows], ":", color=BLUE,
lw=1.6, label="rank-1 correct (ignoring threshold)")
op = meta.get("operating_point")
if op:
ax.axvline(op, color=AMBER, lw=1.6, ls="-.", zorder=1)
ax.annotate(f"operating point {op} px", xy=(op, 50),
xytext=(4, 0), textcoords="offset points",
color=AMBER, fontsize=9, rotation=90, va="center")
ax.set_xlabel("probe face size before upscaling (px)")
ax.set_ylabel("% of probes")
ax.set_ylim(-2, 102)
ax.set_xticks(sizes)
ax.set_title(f"VR-005 — identification vs. probe face size\n"
f"{meta['model']}, {meta['n_actors']} actors, "
f"{meta['n_probes']} probes/size, gallery at native resolution",
fontsize=11, loc="left")
ax.legend(frameon=False, fontsize=9, loc="center left")
fig.text(0.01, 0.005, CAVEAT.format(n_actors=meta["n_actors"]),
fontsize=7.5, color=MUTED, wrap=True)
fig.tight_layout(rect=(0, 0.05, 1, 1))
out_png.parent.mkdir(parents=True, exist_ok=True)
fig.savefig(out_png, dpi=150)
plt.close(fig)
def pick_operating_point(rows: list[dict], retention: float, fpi_slack: float) -> int | None:
"""Smallest size that keeps `retention` of the undegraded (112 px control)
TPI rate and does not add more than `fpi_slack` absolute FPI over it.
A stated rule, not a magic number — change the rule, not the answer."""
control = next((r for r in rows if r["size_px"] == 112), None)
if control is None or control["n_probes"] == 0:
return None
tpi_floor = retention * control["tpi_rate"]
fpi_ceil = control["fpi_rate"] + fpi_slack
ok = [r["size_px"] for r in rows
if r["tpi_rate"] >= tpi_floor and r["fpi_rate"] <= fpi_ceil]
return min(ok) if ok else None
# ── Main ──────────────────────────────────────────────────────────────────────
def main() -> int:
p = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--images", required=True,
help="gallery image cache root (<jellyfin_id>_<Name>/NN.jpg)")
p.add_argument("--gallery", default=None,
help="gallery .h5 — restricts the actor pool to its members")
p.add_argument("--out", default=str(REPO / "experiments/results/vr005_min_face_size"),
help="output path prefix (.csv/.json/.png are appended)")
p.add_argument("--per-probe", action="store_true",
help="also write <out>.per_probe.csv, one row per probe per size")
p.add_argument("--actors", type=int, default=100, help="actors to sample (default 100)")
p.add_argument("--min-images", type=int, default=2,
help="minimum mugshots for an actor to be eligible (default 2)")
p.add_argument("--probes-per-actor", type=int, default=1,
help="images held out per actor; 1 is the VR-005 protocol")
p.add_argument("--seed", type=int, default=0, help="actor/probe selection seed")
p.add_argument("--keep-duplicates", action="store_true",
help="keep mugshots that are the same photograph twice; by "
"default they are dropped, since a probe identical to a "
"gallery reference is identified for free at every size")
p.add_argument("--sizes", default="12,16,20,24,32,40,48,56,64,72,80,96,112",
help="comma-separated probe sizes; 112 is the undegraded control")
p.add_argument("--models-dir", default=str(REPO / "models"))
p.add_argument("--arcface", default=None,
help="embedder ONNX (default <models-dir>/LVFace-B_Glint360K.onnx)")
p.add_argument("--detector", default=None,
help="SCRFD ONNX (default <models-dir>/scrfd_500m_bnkps.onnx)")
p.add_argument("--conf", type=float, default=0.5, help="detector confidence")
p.add_argument("--nms", type=float, default=0.4, help="detector NMS IoU")
p.add_argument("--max-side", type=int, default=500,
help="downscale mugshots to this longest side before detection, "
"matching the gallery builders' embedder settings")
p.add_argument("--prob-threshold", type=float, default=0.754,
help="accept if P(match) exceeds this (Config::prob_threshold)")
p.add_argument("--match-prior", type=float, default=0.5,
help="base-rate prior (Config::match_prior)")
p.add_argument("--calib-a", type=float, default=None,
help="override the fitted sigmoid scale instead of fitting")
p.add_argument("--calib-b", type=float, default=None,
help="override the fitted sigmoid bias instead of fitting")
p.add_argument("--down-interp", default="area", choices=sorted(INTERP),
help="interpolation for the 112 -> S downscale (default area)")
p.add_argument("--up-interp", default="linear", choices=sorted(INTERP),
help="interpolation for the S -> 112 upscale (default linear, "
"as warpAffine uses in align_face)")
p.add_argument("--tpi-retention", type=float, default=0.95,
help="operating point keeps this fraction of the control TPI rate")
p.add_argument("--fpi-slack", type=float, default=0.01,
help="operating point may add at most this absolute FPI over control")
p.add_argument("--verify-against", default=None,
help="gallery .h5 built from --images with the same model: report "
"agreement and separation of recomputed vs stored embeddings "
"(a TensorRT-built gallery will not agree; see the docstring)")
args = p.parse_args()
if (args.calib_a is None) != (args.calib_b is None):
return err("--calib-a and --calib-b must be given together")
models_dir = Path(args.models_dir)
arcface = Path(args.arcface) if args.arcface else models_dir / "LVFace-B_Glint360K.onnx"
detector = Path(args.detector) if args.detector else models_dir / "scrfd_500m_bnkps.onnx"
for path, what in ((arcface, "embedder"), (detector, "detector")):
if not path.is_file():
return err(f"{what} model not found: {path}\n"
f"Run: bash scripts/download_models.sh")
images_root = Path(args.images)
if not images_root.is_dir():
return err(f"image cache not found: {images_root}")
sizes = sorted({int(s) for s in args.sizes.split(",") if s.strip()})
if not sizes:
return err("--sizes is empty")
if args.probes_per_actor < 1:
return err("--probes-per-actor must be >= 1")
cv2.setRNGSeed(args.seed) # estimateAffinePartial2D's RANSAC draws from this
# ── actor pool ────────────────────────────────────────────────────────────
pool = discover_actors(images_root)
print(f"[select] {len(pool)} actor dirs with images under {images_root}",
file=sys.stderr)
if args.gallery:
ids, names = gallery_keys(Path(args.gallery))
pool = [a for a in pool
if (a["jellyfin_id"] and a["jellyfin_id"] in ids)
or normalise_name(a["name"]) in names]
print(f"[select] {len(pool)} of them are in {args.gallery}", file=sys.stderr)
need = max(args.min_images, args.probes_per_actor + 1)
eligible = [a for a in pool if len(a["images"]) >= need]
print(f"[select] {len(eligible)} have >= {need} mugshots", file=sys.stderr)
if len(eligible) < 2:
return err(f"need at least 2 actors with >= {need} mugshots; found "
f"{len(eligible)}. Build the image cache first "
f"(scripts/make_jellyfin_gallery.py) or lower --min-images.")
rng = random.Random(args.seed)
selected = sorted(rng.sample(eligible, min(args.actors, len(eligible))),
key=lambda a: a["dir"].name)
if len(selected) < args.actors:
print(f"[select] WARNING: only {len(selected)} eligible actors, "
f"--actors {args.actors} requested. FPI is gallery-size dependent — "
f"see the caveat.", file=sys.stderr)
# ── detect + align every mugshot of the selected actors, once ─────────────
stages = Stages(str(detector), str(arcface), args.conf, args.nms)
print(f"[models] detector={detector.name} embedder={arcface.name} "
f"batch={stages.engine.max_batch} (provider chosen by the C++ backend: "
f"CUDA, then ROCm, then CPU)", file=sys.stderr)
t0 = time.time()
crops: list[np.ndarray] = []
rows: list[dict] = [] # parallel to crops: {actor, actor_idx, image}
actors: list[dict] = []
n_nodetect = 0
for a in selected:
actor_crops, actor_paths = [], []
for img_path in a["images"]:
img = cv2.imread(str(img_path))
if img is None:
n_nodetect += 1
continue
if args.max_side > 0 and max(img.shape[:2]) > args.max_side:
s = args.max_side / max(img.shape[:2])
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
faces = stages.detect(img)
if not faces:
enhanced = stages.enhance(img)
faces = stages.detect(enhanced)
if faces:
img = enhanced
if not faces:
n_nodetect += 1
continue
best = max(faces, key=lambda f: f.confidence)
crop = stages.align(img, best.landmarks)
if crop is None:
n_nodetect += 1
continue
actor_crops.append(crop)
actor_paths.append(img_path)
if len(actor_crops) < args.probes_per_actor + 1:
continue
ai = len(actors)
actors.append({"name": a["name"], "jellyfin_id": a["jellyfin_id"],
"dir": a["dir"].name, "n_images": len(actor_crops)})
for crop, img_path in zip(actor_crops, actor_paths):
rows.append({"actor_idx": ai, "image": str(img_path)})
crops.append(crop)
if len(actors) % 20 == 0:
print(f" [align] {len(actors)}/{len(selected)} actors, "
f"{len(crops)} crops", file=sys.stderr)
if len(actors) < 2:
return err(f"only {len(actors)} actors survived detection/alignment — "
f"nothing to match against")
print(f"[align] {len(actors)} actors, {len(crops)} aligned crops, "
f"{n_nodetect} images skipped (no face / unreadable) in "
f"{time.time() - t0:.1f}s", file=sys.stderr)
actor_of = np.array([r["actor_idx"] for r in rows], dtype=int)
# ── embed everything at native resolution ────────────────────────────────
t0 = time.time()
native = stages.embed(crops)
print(f"[embed] {len(crops)} native crops in {time.time() - t0:.1f}s",
file=sys.stderr)
if args.verify_against:
verify_embeddings(Path(args.verify_against), rows, native, actor_of)
# ── drop duplicate mugshots ───────────────────────────────────────────────
# The cache holds the same photograph twice for some actors (two provider
# URLs, one picture). A probe that is identical to a gallery reference is
# identified for free at every size, which flatters the whole curve, so
# remove duplicates the same way calibrate_gallery does.
n_dup = 0
if not args.keep_duplicates:
keep = np.ones(len(rows), bool)
for ai in range(len(actors)):
kept: list[int] = []
for i in np.nonzero(actor_of == ai)[0]:
if any(float(native[i] @ native[k]) > DEDUP_SIM for k in kept):
keep[i] = False
else:
kept.append(int(i))
n_dup = int((~keep).sum())
# An actor left with too few distinct mugshots to hold one out drops out.
counts = np.bincount(actor_of[keep], minlength=len(actors))
drop_actor = counts < args.probes_per_actor + 1
keep &= ~drop_actor[actor_of]
remap = np.full(len(actors), -1, dtype=int)
remap[~drop_actor] = np.arange(int((~drop_actor).sum()))
actors = [a for a, d in zip(actors, drop_actor) if not d]
rows = [r for r, k in zip(rows, keep) if k]
crops = [c for c, k in zip(crops, keep) if k]
native = native[keep]
actor_of = remap[actor_of[keep]]
for r, ai in zip(rows, actor_of):
r["actor_idx"] = int(ai)
for ai, a in enumerate(actors):
a["n_images"] = int(np.sum(actor_of == ai))
print(f"[dedup] dropped {n_dup} duplicate mugshots and "
f"{int(drop_actor.sum())} actors left with too few; "
f"{len(actors)} actors, {len(rows)} images remain", file=sys.stderr)
if len(actors) < 2:
return err("fewer than 2 actors survive de-duplication — the image "
"cache holds too few distinct mugshots")
# ── hold out the probes ───────────────────────────────────────────────────
is_probe = np.zeros(len(rows), bool)
for ai in range(len(actors)):
idx = np.nonzero(actor_of == ai)[0]
# Seeded per actor so the choice does not depend on iteration order.
r = random.Random(f"{args.seed}:{actors[ai]['dir']}")
for pick in r.sample(list(idx), args.probes_per_actor):
is_probe[pick] = True
probe_rows = np.nonzero(is_probe)[0]
gal_rows = np.nonzero(~is_probe)[0]
print(f"[holdout] {len(probe_rows)} probes held out, "
f"{len(gal_rows)} gallery embeddings remain", file=sys.stderr)
gal_emb = native[gal_rows]
gal_actor = actor_of[gal_rows]
probe_actor = actor_of[probe_rows]
# Per-actor column masks for the best-of-N scan (identity_matcher_node).
actor_cols = [np.nonzero(gal_actor == ai)[0] for ai in range(len(actors))]
have_refs = np.array([len(c) > 0 for c in actor_cols])
if not have_refs.all():
return err("an actor ended up with no gallery references left; "
"raise --min-images")
# ── calibration ───────────────────────────────────────────────────────────
if args.calib_a is not None:
cal = {"a": args.calib_a, "b": args.calib_b, "valid": True}
print(f"[calibration] using supplied a={cal['a']} b={cal['b']}", file=sys.stderr)
else:
cal = calibrate_gallery(gal_emb, gal_actor)
if not cal["valid"]:
return err(
"calibration could not be fitted, and this study will not fall back to a "
"raw cosine threshold (CLAUDE.md invariant). Use more actors with >= "
f"{MIN_EMB_FOR_POSITIVE} mugshots, or pass --calib-a/--calib-b from a "
"production gallery.")
log_prior_odds = float(np.log(args.match_prior / (1.0 - args.match_prior)))
# ── sweep ─────────────────────────────────────────────────────────────────
down, up = INTERP[args.down_interp], INTERP[args.up_interp]
probe_crops = [crops[i] for i in probe_rows]
results, per_probe = [], []
for size in sizes:
t0 = time.time()
degraded = [degrade(c, size, down, up) for c in probe_crops]
q = stages.embed(degraded)
sims = q @ gal_emb.T # [n_probe, n_gal]
best_per_actor = np.stack([sims[:, cols].max(axis=1) for cols in actor_cols],
axis=1) # [n_probe, n_actor]
best_actor = best_per_actor.argmax(axis=1)
best_sim = best_per_actor.max(axis=1)
p_match = np.asarray(probability(best_sim, cal["a"], cal["b"], log_prior_odds))
accept = p_match > args.prob_threshold
correct = best_actor == probe_actor
tpi = int(np.sum(accept & correct))
fpi = int(np.sum(accept & ~correct))
unid = int(np.sum(~accept))
n = len(probe_rows)
results.append({
"size_px": size,
"n_probes": n,
"tpi": tpi, "fpi": fpi, "unidentified": unid,
"tpi_rate": tpi / n, "fpi_rate": fpi / n, "unidentified_rate": unid / n,
"rank1_rate": float(np.mean(correct)),
"mean_best_sim": float(np.mean(best_sim)),
"mean_p_match": float(np.mean(p_match)),
"mean_sim_true_actor": float(np.mean(
best_per_actor[np.arange(n), probe_actor])),
})
if args.per_probe:
for j in range(n):
per_probe.append({
"size_px": size,
"probe_image": rows[probe_rows[j]]["image"],
"true_actor": actors[probe_actor[j]]["name"],
"matched_actor": actors[best_actor[j]]["name"],
"best_sim": float(best_sim[j]),
"p_match": float(p_match[j]),
"outcome": ("TPI" if accept[j] and correct[j]
else "FPI" if accept[j] else "unidentified"),
})
print(f"[sweep] {size:3d}px TPI {tpi:4d} ({100 * tpi / n:5.1f}%) "
f"FPI {fpi:4d} ({100 * fpi / n:5.1f}%) "
f"unid {unid:4d} ({100 * unid / n:5.1f}%) "
f"rank1 {100 * np.mean(correct):5.1f}% "
f"[{time.time() - t0:.1f}s]", file=sys.stderr)
op = pick_operating_point(results, args.tpi_retention, args.fpi_slack)
# ── outputs ───────────────────────────────────────────────────────────────
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
# Append rather than with_suffix() so a prefix containing a dot keeps its name.
csv_path = out.with_name(out.name + ".csv")
json_path = out.with_name(out.name + ".json")
png_path = out.with_name(out.name + ".png")
fields = list(results[0].keys())
with open(csv_path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=fields)
w.writeheader()
w.writerows(results)
meta = {
"requirement": "VR-005",
"caveat": CAVEAT.format(n_actors=len(actors)),
"model": arcface.stem,
"detector": detector.stem,
"backend": "sae_embed / the compiled-in inference backend (fp32 ONNX under "
"SAE_INFERENCE_BACKEND=ORT; a TensorRT fp16 build is a different "
"embedding space)",
"n_actors": len(actors),
"n_probes": len(probe_rows),
"n_gallery_embeddings": len(gal_rows),
"probes_per_actor": args.probes_per_actor,
"seed": args.seed,
"sizes": sizes,
"prob_threshold": args.prob_threshold,
"match_prior": args.match_prior,
"calibration": cal,
"calibration_source": "supplied" if args.calib_a is not None else "fitted",
"sim_boundary_at_threshold": float(
(np.log(args.prob_threshold / (1 - args.prob_threshold))
- cal["b"] - log_prior_odds) / cal["a"]),
"down_interp": args.down_interp,
"up_interp": args.up_interp,
"max_side": args.max_side,
"duplicate_mugshots_dropped": n_dup,
"operating_point_rule": (
f"smallest size retaining >= {args.tpi_retention:.0%} of the 112 px "
f"control TPI rate with <= +{args.fpi_slack:.1%} absolute FPI"),
"operating_point": op,
"images_skipped_no_face": n_nodetect,
"curve": results,
"actors": actors,
}
json_path.write_text(json.dumps(meta, indent=2) + "\n")
if per_probe:
pp = out.with_name(out.name + ".per_probe.csv")
with open(pp, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(per_probe[0].keys()))
w.writeheader()
w.writerows(per_probe)
print(f"[out] {pp}", file=sys.stderr)
write_plot(results, png_path, meta)
# ── stdout report ─────────────────────────────────────────────────────────
print(f"\nVR-005 — minimum face size, {arcface.stem}")
print(f"{len(actors)} actors, {len(probe_rows)} probes/size, "
f"{len(gal_rows)} gallery embeddings at native resolution")
print(f"identify when P>{args.prob_threshold}, i.e. cosine above "
f"{meta['sim_boundary_at_threshold']:.4f} under the calibration fitted "
f"on this gallery\n")
print(f"{'size':>5} {'TPI':>8} {'FPI':>8} {'unid':>8} {'rank1':>8} {'mean sim':>9}")
for r in results:
print(f"{r['size_px']:>5} {100 * r['tpi_rate']:>7.1f}% "
f"{100 * r['fpi_rate']:>7.1f}% {100 * r['unidentified_rate']:>7.1f}% "
f"{100 * r['rank1_rate']:>7.1f}% {r['mean_best_sim']:>9.4f}")
print(f"\noperating point: {op if op else 'none of the swept sizes qualifies'}"
f" ({meta['operating_point_rule']})")
print(f"\n{meta['caveat']}")
print(f"\n[out] {csv_path}\n[out] {json_path}\n[out] {png_path}")
return 0
def _separation(emb: np.ndarray, actor: np.ndarray) -> tuple[float, float, float]:
"""(mean same-actor sim, mean different-actor sim, d') — the property that has
to survive for an embedding space to be usable, whatever its coordinates."""
iu, ju = np.triu_indices(len(actor), k=1)
sims = (emb @ emb.T)[iu, ju]
same = actor[iu] == actor[ju]
pos = sims[same & (sims < 0.9999)] # drop duplicate source images
neg = sims[~same]
if pos.size < 2 or neg.size < 2:
return float("nan"), float("nan"), float("nan")
d = (pos.mean() - neg.mean()) / np.sqrt(0.5 * (pos.var() + neg.var()))
return float(pos.mean()), float(neg.mean()), float(d)
def verify_embeddings(gallery_path: Path, rows: list[dict], native: np.ndarray,
actor_of: np.ndarray) -> None:
"""Cross-check this script's ONNX port against a gallery built by the C++
pipeline from the same mugshots.
Agreement is ~1.0 only if that gallery was built with the same backend. A
TensorRT fp16 build lands around 0.85 on LVFace-B while separating just as
well, so the separation figures — not the agreement — are what says whether
the port is sound."""
from sae_gallery import load_gallery_hdf5
g = load_gallery_hdf5(gallery_path)
stored: dict[tuple[str, str], np.ndarray] = {}
for a in g["actors"]:
key = a.get("jellyfin_id") or normalise_name(a.get("name", ""))
for e, src in zip(a.get("embeddings", []), a.get("source_images", [])):
if src:
stored[(key, src)] = np.asarray(e, np.float32)
sims, paired_mine, paired_ref, paired_actor = [], [], [], []
for i, r in enumerate(rows):
path = Path(r["image"])
head, _, _ = path.parent.name.partition("_")
key = head if JELLYFIN_ID_RE.match(head) else normalise_name(
path.parent.name.replace("_", " "))
ref = stored.get((key, path.name))
if ref is None or ref.shape != native[i].shape:
continue
ref = ref / max(float(np.linalg.norm(ref)), 1e-6)
sims.append(float(native[i] @ ref))
paired_mine.append(native[i])
paired_ref.append(ref)
paired_actor.append(actor_of[i])
if not sims:
print(f"[verify] no overlap with {gallery_path} — nothing checked",
file=sys.stderr)
return
sims_arr = np.asarray(sims)
print(f"[verify] {len(sims)} embeddings vs {gallery_path.name}: "
f"mean cos={sims_arr.mean():.4f} min={sims_arr.min():.4f}",
file=sys.stderr)
act = np.asarray(paired_actor)
for label, mat in (("this script", np.asarray(paired_mine)),
("stored gallery", np.asarray(paired_ref))):
pos, neg, d = _separation(mat, act)
print(f"[verify] {label:>14s}: same-actor {pos:.3f} "
f"different-actor {neg:.3f} d'={d:.2f}", file=sys.stderr)
if sims_arr.mean() < 0.99:
print("[verify] embeddings differ from the stored gallery. If d' is "
"comparable this is a backend difference (e.g. a TensorRT fp16 "
"build), not a broken port; the study is self-consistent either "
"way. If d' collapsed, the port is wrong.", file=sys.stderr)
def err(msg: str) -> int:
print(f"error: {msg}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())
scripts/validation/quality_knee.py
+667
View File
@@ -0,0 +1,667 @@
#!/usr/bin/env python3
"""
quality_knee.py — VR-012: what does a blurred or small face cost in identification,
and which sharpness measure predicts it?
TRACES: VR-012, AR-028, AR-029
VR-005 located the size floor by degrading held-out gallery mugshots and watching
TPI/FPI fall. This does the same over a **joint size x blur grid**, and adds the
part that makes the result usable at inference.
Why a joint grid and not two sweeps
-----------------------------------
A 16 px face upscaled to 112 has already lost its high frequencies, so additional
blur costs it far less than it costs a 112 px one. Sweeping the axes separately
measures each in the presence of an implicit "other axis at its best" and misses
that interaction entirely — and the interaction is the whole question, because
AR-002 already gates on size and AR-029 proposes to discount on sharpness. If
identity loss turns out to be a function of the sharpness measure alone, then one
axis carries the information and discounting on both double-counts. If a
small-but-sharp and a large-but-blurred probe at equal measure lose different
amounts, the axes are genuinely separate and both belong.
Why sigma is not the answer
---------------------------
Sigma is a lab variable. At inference nothing knows how blurred a face is, so a
knee expressed in sigma cannot be acted on. What AR-028/AR-030 can consume is
measure value -> expected identity reliability
so the controlled degradation exists to *select and calibrate the measure*, and
the measure is what ships. Every candidate is therefore scored on every degraded
crop, and the candidates are ranked by how well each predicts the identification
outcome (AUC over probe-cell records), not by how smooth its ladder looks.
Protocol (VR-005's, extended)
-----------------------------
1. Every gallery actor with at least `--min-images` mugshots. At the default 3,
holding one out still leaves two references per actor.
2. Hold out ONE image per actor as the probe; the rest stay in the gallery at
native resolution. Only the probe degrades — reference mugshots are clean and
the face coming out of the video is not, which is the production case.
3. For each (size, sigma) cell: downscale the probe crop to size x size and back
to 112 (the sampling loss), then Gaussian blur at sigma canonical px (the
optical/motion loss). Resolution first, then blur, so sigma always means the
same thing in the frame AR-029 measures in, whatever the cell's size.
4. Score all five AR-029 candidates on the degraded crop, through the C++
binding.
5. Embed, match against the whole gallery, record TPI/FPI/unidentified.
Decision rule is the pipeline's: per-actor best-of-N cosine -> Platt sigmoid ->
accept if P > prob_threshold. Never a raw cosine (CLAUDE.md invariant, AR-024).
Everything runs through `sae_embed` — detection, the ArcFace warp, the embedder,
the sharpness measures and the calibration are all the shipped C++. Nothing here
re-implements a pipeline stage in numpy; the analysis on top of the recorded
numbers (AUC, knee location) is analysis and is numpy's job.
CAVEAT — FPI IS RELATIVE, NOT ABSOLUTE
--------------------------------------
False positives grow with the number of actors competing. Read FPI as a curve
across cells, not as a production rate. This runs the whole eligible gallery
rather than VR-005's 100-actor sample, so the understatement is much smaller,
but a production library is larger still.
Usage
-----
python scripts/validation/quality_knee.py \
--images images --gallery gallery_lvface.h5 \
--arcface models/LVFace-B_Glint360K.onnx \
--min-images 3 --out experiments/results/vr012_quality_knee
"""
from __future__ import annotations
import argparse
import csv
import json
import random
import sys
import time
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
sys.path.insert(0, str(Path(__file__).resolve().parent))
# min_face_size owns the shared scaffolding — actor discovery, the sae_embed
# locator, the Stages wrapper, the calibration-through-the-binding and the house
# plot palette. Importing it keeps one copy of each; a second copy of the
# calibration path in particular is what AR-024 exists to prevent.
import min_face_size as vr005 # noqa: E402
from min_face_size import ( # noqa: E402
DEDUP_SIM, INTERP, Stages, calibrate_gallery, discover_actors,
gallery_keys, normalise_name, probability, err,
INK, MUTED, GRID, SURFACE, BLUE, GREEN, RED, AMBER,
)
import cv2 # noqa: E402
import numpy as np # noqa: E402
import sae_embed # noqa: E402
# The five AR-029 candidates, in quality.hpp's order. Names match the binding's
# attributes so the CSV columns and the C++ fields cannot drift apart.
MEASURES = ["var_laplacian", "norm_var_laplacian", "tenengrad",
"hf_energy_ratio", "dir_min_tenengrad"]
# ── Degradation ───────────────────────────────────────────────────────────────
def disc_kernel(radius: float) -> np.ndarray:
"""The circle-of-confusion PSF of a defocused lens.
Optical defocus is **not** Gaussian, and the difference is not cosmetic. A
lens out of focus spreads a point into a uniform disc, whose transfer
function is a jinc — `2·J1(x)/x` — which crosses zero and goes negative.
Defocus therefore reverses contrast at particular spatial frequencies and
can leave *more* energy in some high bands than a Gaussian of the same
nominal width. A Gaussian MTF is strictly positive and monotonically
decreasing and does neither.
That matters here beyond realism: defocus is how a face ends up **large and
useless**. A focus pull, a shallow depth of field, an actor stepping off the
focal plane — all leave a big, confidently-detected face carrying no usable
detail, and all sail straight through a size gate. Gaussian blur was the one
family that mostly co-occurs with small faces, which is precisely why
sharpness looked redundant against AR-002 on the first grid.
The disc is supersampled 8x before downsampling so its edge is
anti-aliased; a hard-edged binary disc at small radii is a poor circle and
its spectrum carries the staircase, not the optics.
"""
ss = 8
n = int(np.ceil(radius)) * 2 + 1
hi = np.zeros((n * ss, n * ss), np.float32)
c = (n * ss - 1) / 2.0
y, x = np.ogrid[:n * ss, :n * ss]
hi[((x - c) ** 2 + (y - c) ** 2) <= (radius * ss) ** 2] = 1.0
k = hi.reshape(n, ss, n, ss).mean(axis=(1, 3))
s = k.sum()
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
def motion_kernel(length: int, angle_deg: float) -> np.ndarray:
"""Linear motion blur — a camera pan or a moving subject.
Directional by construction: it destroys detail along one axis and leaves
the perpendicular axis untouched. That is the property that separates the
AR-029 candidates, since a measure normalising by total energy divides out
the loss and reads a heavy smear as mild (see tests/test_quality.cpp).
"""
k = np.zeros((length, length), np.float32)
k[length // 2, :] = 1.0
m = cv2.getRotationMatrix2D(((length - 1) / 2.0, (length - 1) / 2.0),
angle_deg, 1.0)
k = cv2.warpAffine(k, m, (length, length))
s = k.sum()
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
def degrade(crop: np.ndarray, size: int, level: float, kind: str,
down: int, up: int, angle: float = 0.0) -> np.ndarray:
"""Resolution loss, then blur of the requested family.
Order matters and this one is deliberate. Sampling happens in the source
frame, so the downscale/upscale pair models a face that was `size` px when
detected. The blur is then applied in the canonical frame, so `level` means
the same number of canonical pixels in every cell of the grid — which is what
lets the two axes be read independently. Blurring first would make the
effective width depend on the cell's size, and the grid would no longer be
factorial.
`level` is the family's natural parameter: Gaussian sigma, disc radius, or
motion length in canonical px. They are NOT equivalent at equal numbers —
matching families by parameter would compare different amounts of damage, so
the analysis matches them on measured effect instead.
"""
out = crop
if size != 112:
small = cv2.resize(out, (size, size), interpolation=down)
out = cv2.resize(small, (112, 112), interpolation=up)
if level > 0:
if kind == "gaussian":
out = cv2.GaussianBlur(out, (0, 0), level, level)
elif kind == "disc":
out = cv2.filter2D(out, -1, disc_kernel(level))
elif kind == "motion":
out = cv2.filter2D(out, -1, motion_kernel(int(round(level)), angle))
else:
raise ValueError(f"unknown blur kind: {kind}")
return out
def score_sharpness(crop: np.ndarray) -> dict:
"""All five candidates, from the shipped C++ (quality.hpp)."""
s = sae_embed.assess_sharpness(np.ascontiguousarray(crop))
d = {m: float(getattr(s, m)) for m in MEASURES}
d["ok"] = bool(s.ok)
return d
# ── Analysis ──────────────────────────────────────────────────────────────────
def auc(scores: np.ndarray, positive: np.ndarray) -> float:
"""Area under the ROC for `scores` predicting `positive`, by the rank
(Mann-Whitney U) identity. 0.5 is chance; 1.0 is a measure that orders every
correctly-identified probe above every failure.
This is the ranking criterion for AR-029. A measure earns the job by
predicting *the decision the pipeline makes*, not by having a tidy response
to synthetic blur — a candidate can be beautifully monotone in sigma and
still be a poor guide to whether this particular face will be recognised.
"""
pos = scores[positive]
neg = scores[~positive]
if pos.size == 0 or neg.size == 0:
return float("nan")
order = np.argsort(np.concatenate([pos, neg]), kind="mergesort")
ranks = np.empty(order.size, dtype=np.float64)
ranks[order] = np.arange(1, order.size + 1)
# Average ranks over ties, or a measure with many equal values is scored
# arbitrarily by input order.
vals = np.concatenate([pos, neg])
sv = vals[order]
i = 0
while i < sv.size:
j = i
while j + 1 < sv.size and sv[j + 1] == sv[i]:
j += 1
if j > i:
ranks[order[i:j + 1]] = ranks[order[i:j + 1]].mean()
i = j + 1
r_pos = ranks[:pos.size].sum()
return float((r_pos - pos.size * (pos.size + 1) / 2) / (pos.size * neg.size))
def knee_from_measure(records: list[dict], measure: str, retention: float,
n_bins: int = 20) -> dict:
"""Where on `measure`'s own scale does identification start to fall apart?
Bins the probe-cell records by measure value and reports the TPI rate in
each. The threshold is the lowest bin edge whose bin and every bin above it
retain `retention` of the undegraded control's TPI rate — a stated rule, so
changing the answer means changing the rule rather than picking a number.
"""
vals = np.array([r[measure] for r in records], dtype=np.float64)
tpi = np.array([r["outcome"] == "TPI" for r in records])
control = np.array([r["size_px"] == 112 and r["sigma"] == 0.0 for r in records])
if control.sum() == 0:
return {}
floor = retention * float(tpi[control].mean())
# Quantile edges: the measures have wildly different scales and heavy tails,
# so equal-width bins would put almost everything in one bucket.
edges = np.unique(np.quantile(vals, np.linspace(0, 1, n_bins + 1)))
if edges.size < 3:
return {}
idx = np.clip(np.digitize(vals, edges[1:-1]), 0, edges.size - 2)
bins = []
for b in range(edges.size - 1):
m = idx == b
if m.sum() == 0:
continue
bins.append({"lo": float(edges[b]), "hi": float(edges[b + 1]),
"n": int(m.sum()), "tpi_rate": float(tpi[m].mean()),
"fpi_rate": float(np.mean([r["outcome"] == "FPI"
for r, k in zip(records, m) if k]))})
# Walk down from the top; the threshold is where retention first breaks.
thr = None
for b in reversed(bins):
if b["tpi_rate"] < floor:
thr = b["hi"]
break
return {"measure": measure, "control_tpi": float(tpi[control].mean()),
"tpi_floor": floor, "threshold": thr, "bins": bins}
# ── Plot ──────────────────────────────────────────────────────────────────────
def write_plots(cells: list[dict], records: list[dict], ranking: list[dict],
out_png: Path, meta: dict) -> None:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
plt.rcParams.update({
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE,
"savefig.facecolor": SURFACE, "text.color": INK,
"axes.edgecolor": MUTED, "axes.labelcolor": INK,
"xtick.color": MUTED, "ytick.color": MUTED,
"axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.8,
"axes.spines.top": False, "axes.spines.right": False,
})
sizes = sorted({c["size_px"] for c in cells})
sigmas = sorted({c["sigma"] for c in cells})
fig, axes = plt.subplots(1, 3, figsize=(17, 5.4))
# (a) the joint grid as TPI heat map
grid = np.full((len(sigmas), len(sizes)), np.nan)
for c in cells:
grid[sigmas.index(c["sigma"]), sizes.index(c["size_px"])] = 100 * c["tpi_rate"]
im = axes[0].imshow(grid, origin="lower", aspect="auto", cmap="viridis",
vmin=0, vmax=100)
axes[0].set_xticks(range(len(sizes)), [str(s) for s in sizes])
axes[0].set_yticks(range(len(sigmas)), [f"{s:g}" for s in sigmas])
axes[0].set_xlabel("probe size before upscaling (px)")
axes[0].set_ylabel("Gaussian sigma (canonical px)")
axes[0].set_title("TPI % over the joint grid", fontsize=11, loc="left")
axes[0].grid(False)
fig.colorbar(im, ax=axes[0], fraction=0.046)
# (b) TPI against the winning measure — the curve a discount is built from
best = ranking[0]["measure"]
vals = np.array([r[best] for r in records])
tpi = np.array([r["outcome"] == "TPI" for r in records])
edges = np.unique(np.quantile(vals, np.linspace(0, 1, 21)))
centres, rates = [], []
for i in range(edges.size - 1):
m = (vals >= edges[i]) & (vals <= edges[i + 1])
if m.sum() > 20:
centres.append(0.5 * (edges[i] + edges[i + 1]))
rates.append(100 * tpi[m].mean())
axes[1].plot(centres, rates, "-o", color=GREEN, lw=2)
axes[1].set_xscale("log")
axes[1].set_xlabel(f"{best} (log scale)")
axes[1].set_ylabel("TPI %")
axes[1].set_title(f"identification vs the measure\nbest predictor: {best} "
f"(AUC {ranking[0]['auc']:.3f})", fontsize=11, loc="left")
# (c) how well each candidate predicts the decision
names = [r["measure"] for r in ranking]
aucs = [r["auc"] for r in ranking]
axes[2].barh(range(len(names)), aucs, color=BLUE)
axes[2].axvline(0.5, color=RED, lw=1.4, ls="--")
axes[2].set_yticks(range(len(names)), names, fontsize=9)
axes[2].set_xlim(0.4, 1.0)
axes[2].set_xlabel("AUC — predicts correct identification")
axes[2].set_title("AR-029 candidate ranking", fontsize=11, loc="left")
axes[2].invert_yaxis()
fig.suptitle(f"VR-012 — quality knee, {meta['model']}, {meta['n_actors']} actors, "
f"{meta['n_probes']} probes x {len(cells)} cells",
fontsize=12, x=0.01, ha="left")
fig.tight_layout(rect=(0, 0.02, 1, 0.97))
out_png.parent.mkdir(parents=True, exist_ok=True)
fig.savefig(out_png, dpi=150)
plt.close(fig)
# ── Main ──────────────────────────────────────────────────────────────────────
def main() -> int:
p = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--images", default=str(REPO / "images"))
p.add_argument("--gallery", default=str(REPO / "gallery_lvface.h5"))
p.add_argument("--out", default=str(REPO / "experiments/results/vr012_quality_knee"))
p.add_argument("--actors", type=int, default=0,
help="cap the actor pool (0 = every eligible actor, the default: "
"FPI is gallery-size dependent and the whole gallery is the "
"least understated estimate available)")
p.add_argument("--min-images", type=int, default=3,
help="minimum mugshots to be eligible (default 3, so holding one "
"out still leaves two references)")
p.add_argument("--seed", type=int, default=0)
p.add_argument("--sizes", default="16,24,32,48,64,112",
help="probe sizes before upscaling; 112 is undegraded")
p.add_argument("--sigmas", default="0,0.5,1,1.5,2,3",
help="blur level in canonical px; 0 is unblurred. Meaning "
"depends on --blur-kind: Gaussian sigma, disc radius, "
"or motion length")
p.add_argument("--blur-kind", default="gaussian",
choices=["gaussian", "disc", "motion"],
help="blur family. gaussian is a soft-focus stand-in; disc "
"is the circle-of-confusion PSF of real optical "
"defocus (non-Gaussian, jinc MTF with zero crossings); "
"motion is a linear smear. The last two are how a face "
"ends up large and useless, which a size gate cannot "
"catch")
p.add_argument("--motion-angle", type=float, default=0.0,
help="motion blur direction in degrees (--blur-kind motion)")
p.add_argument("--keep-duplicates", action="store_true")
p.add_argument("--models-dir", default=str(REPO / "models"))
p.add_argument("--arcface", default=None)
p.add_argument("--detector", default=None)
p.add_argument("--conf", type=float, default=0.5)
p.add_argument("--nms", type=float, default=0.4)
p.add_argument("--max-side", type=int, default=500)
# Required by a TRT-backend build, ignored by an ORT one. A TensorRT fp16
# run is a different realisation of the embedder — VR-005 measured ~0.85
# cosine agreement with the fp32 ONNX path on LVFace-B, with separation
# essentially intact — so a knee located here belongs to the fp16 space.
# The study stays internally consistent because gallery and probes are both
# embedded in this one session.
p.add_argument("--detector-engine", default="",
help="pre-built SCRFD .engine (TRT builds only)")
p.add_argument("--arcface-engine", default="",
help="pre-built ArcFace .engine (TRT builds only)")
p.add_argument("--prob-threshold", type=float, default=0.754)
p.add_argument("--match-prior", type=float, default=0.5)
p.add_argument("--tpi-retention", type=float, default=0.95)
p.add_argument("--down-interp", default="area", choices=sorted(INTERP))
p.add_argument("--up-interp", default="linear", choices=sorted(INTERP))
args = p.parse_args()
models_dir = Path(args.models_dir)
arcface = Path(args.arcface) if args.arcface else models_dir / "LVFace-B_Glint360K.onnx"
detector = Path(args.detector) if args.detector else models_dir / "scrfd_500m_bnkps.onnx"
for path, what in ((arcface, "embedder"), (detector, "detector")):
if not path.is_file():
return err(f"{what} model not found: {path}")
images_root = Path(args.images)
if not images_root.is_dir():
return err(f"image cache not found: {images_root}")
sizes = sorted({int(s) for s in args.sizes.split(",") if s.strip()})
sigmas = sorted({float(s) for s in args.sigmas.split(",") if s.strip()})
cv2.setRNGSeed(args.seed)
# ── actor pool ────────────────────────────────────────────────────────────
pool = discover_actors(images_root)
print(f"[select] {len(pool)} actor dirs under {images_root}", file=sys.stderr)
if args.gallery and Path(args.gallery).is_file():
ids, names = gallery_keys(Path(args.gallery))
pool = [a for a in pool
if (a["jellyfin_id"] and a["jellyfin_id"] in ids)
or normalise_name(a["name"]) in names]
print(f"[select] {len(pool)} are in {args.gallery}", file=sys.stderr)
eligible = [a for a in pool if len(a["images"]) >= args.min_images]
print(f"[select] {len(eligible)} have >= {args.min_images} mugshots",
file=sys.stderr)
if len(eligible) < 2:
return err(f"need at least 2 eligible actors; found {len(eligible)}")
rng = random.Random(args.seed)
selected = (sorted(rng.sample(eligible, min(args.actors, len(eligible))),
key=lambda a: a["dir"].name)
if args.actors else eligible)
# ── detect + align every mugshot once ─────────────────────────────────────
stages = Stages(str(detector), str(arcface), args.conf, args.nms,
args.detector_engine, args.arcface_engine)
print(f"[models] detector={detector.name} embedder={arcface.name} "
f"batch={stages.engine.max_batch}", file=sys.stderr)
t0 = time.time()
crops, rows, actors = [], [], []
n_nodetect = 0
for a in selected:
actor_crops, actor_paths = [], []
for img_path in a["images"]:
img = cv2.imread(str(img_path))
if img is None:
n_nodetect += 1
continue
if args.max_side > 0 and max(img.shape[:2]) > args.max_side:
s = args.max_side / max(img.shape[:2])
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
faces = stages.detect(img)
if not faces:
enhanced = stages.enhance(img)
faces = stages.detect(enhanced)
if faces:
img = enhanced
if not faces:
n_nodetect += 1
continue
best = max(faces, key=lambda f: f.confidence)
crop = stages.align(img, best.landmarks)
if crop is None:
n_nodetect += 1
continue
actor_crops.append(crop)
actor_paths.append(img_path)
if len(actor_crops) < 2:
continue
ai = len(actors)
actors.append({"name": a["name"], "jellyfin_id": a["jellyfin_id"],
"dir": a["dir"].name, "n_images": len(actor_crops)})
for crop, img_path in zip(actor_crops, actor_paths):
rows.append({"actor_idx": ai, "image": str(img_path)})
crops.append(crop)
if len(actors) % 200 == 0:
print(f" [align] {len(actors)}/{len(selected)} actors, "
f"{len(crops)} crops", file=sys.stderr)
if len(actors) < 2:
return err(f"only {len(actors)} actors survived detection/alignment")
print(f"[align] {len(actors)} actors, {len(crops)} crops, {n_nodetect} skipped "
f"in {time.time() - t0:.1f}s", file=sys.stderr)
actor_of = np.array([r["actor_idx"] for r in rows], dtype=int)
t0 = time.time()
native = stages.embed(crops)
print(f"[embed] {len(crops)} native crops in {time.time() - t0:.1f}s",
file=sys.stderr)
# ── drop duplicate mugshots ───────────────────────────────────────────────
if not args.keep_duplicates:
keep = np.ones(len(rows), bool)
for ai in range(len(actors)):
kept: list[int] = []
for i in np.nonzero(actor_of == ai)[0]:
if any(float(native[i] @ native[k]) > DEDUP_SIM for k in kept):
keep[i] = False
else:
kept.append(int(i))
n_dup = int((~keep).sum())
counts = np.bincount(actor_of[keep], minlength=len(actors))
drop_actor = counts < 2
keep &= ~drop_actor[actor_of]
remap = np.full(len(actors), -1, dtype=int)
remap[~drop_actor] = np.arange(int((~drop_actor).sum()))
actors = [a for a, d in zip(actors, drop_actor) if not d]
rows = [r for r, k in zip(rows, keep) if k]
crops = [c for c, k in zip(crops, keep) if k]
native = native[keep]
actor_of = remap[actor_of[keep]]
print(f"[dedup] dropped {n_dup} duplicates and {int(drop_actor.sum())} "
f"actors; {len(actors)} actors, {len(rows)} images remain",
file=sys.stderr)
# ── hold out one probe per actor ──────────────────────────────────────────
is_probe = np.zeros(len(rows), bool)
for ai in range(len(actors)):
idx = np.nonzero(actor_of == ai)[0]
r = random.Random(f"{args.seed}:{actors[ai]['dir']}")
is_probe[r.choice(list(idx))] = True
probe_rows = np.nonzero(is_probe)[0]
gal_rows = np.nonzero(~is_probe)[0]
print(f"[holdout] {len(probe_rows)} probes, {len(gal_rows)} gallery embeddings",
file=sys.stderr)
gal_emb = native[gal_rows]
gal_actor = actor_of[gal_rows]
probe_actor = actor_of[probe_rows]
actor_cols = [np.nonzero(gal_actor == ai)[0] for ai in range(len(actors))]
if not all(len(c) for c in actor_cols):
return err("an actor has no gallery references left; raise --min-images")
cal = calibrate_gallery(gal_emb, gal_actor)
if not cal["valid"]:
return err("calibration could not be fitted; this study will not fall back "
"to a raw cosine threshold (CLAUDE.md invariant)")
log_prior_odds = float(np.log(args.match_prior / (1.0 - args.match_prior)))
# ── the grid ──────────────────────────────────────────────────────────────
down, up = INTERP[args.down_interp], INTERP[args.up_interp]
probe_crops = [crops[i] for i in probe_rows]
cells, records = [], []
n = len(probe_rows)
for size in sizes:
for sigma in sigmas:
t0 = time.time()
degraded = [degrade(c, size, sigma, args.blur_kind, down, up,
args.motion_angle) for c in probe_crops]
sharp = [score_sharpness(d) for d in degraded]
q = stages.embed(degraded)
sims = q @ gal_emb.T
best_per_actor = np.stack([sims[:, c].max(axis=1) for c in actor_cols],
axis=1)
best_actor = best_per_actor.argmax(axis=1)
best_sim = best_per_actor.max(axis=1)
p_match = np.asarray(probability(best_sim, cal["a"], cal["b"],
log_prior_odds))
accept = p_match > args.prob_threshold
correct = best_actor == probe_actor
tpi = int(np.sum(accept & correct))
fpi = int(np.sum(accept & ~correct))
unid = int(np.sum(~accept))
cell = {"size_px": size, "sigma": sigma, "blur_kind": args.blur_kind,
"n_probes": n,
"tpi": tpi, "fpi": fpi, "unidentified": unid,
"tpi_rate": tpi / n, "fpi_rate": fpi / n,
"unidentified_rate": unid / n,
"rank1_rate": float(np.mean(correct)),
"mean_p_match": float(np.mean(p_match))}
for m in MEASURES:
cell[f"mean_{m}"] = float(np.mean([s[m] for s in sharp]))
cells.append(cell)
for j in range(n):
rec = {"size_px": size, "sigma": sigma,
"blur_kind": args.blur_kind,
"probe_image": rows[probe_rows[j]]["image"],
"p_match": float(p_match[j]),
"outcome": ("TPI" if accept[j] and correct[j]
else "FPI" if accept[j] else "unidentified")}
rec.update({m: sharp[j][m] for m in MEASURES})
records.append(rec)
print(f"[grid] {size:3d}px {args.blur_kind[:4]} {sigma:<4g} TPI {100*tpi/n:5.1f}% "
f"FPI {100*fpi/n:5.1f}% unid {100*unid/n:5.1f}% "
f"rank1 {100*np.mean(correct):5.1f}% [{time.time()-t0:.1f}s]",
file=sys.stderr)
# ── rank the candidates, then locate the knee on the winner ───────────────
is_tpi = np.array([r["outcome"] == "TPI" for r in records])
ranking = sorted(
({"measure": m,
"auc": auc(np.array([r[m] for r in records], dtype=np.float64), is_tpi)}
for m in MEASURES),
key=lambda d: -d["auc"])
knees = [knee_from_measure(records, r["measure"], args.tpi_retention)
for r in ranking]
# ── outputs ───────────────────────────────────────────────────────────────
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
csv_path = out.with_name(out.name + ".csv")
with open(csv_path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(cells[0].keys()))
w.writeheader()
w.writerows(cells)
rec_path = out.with_name(out.name + ".records.csv")
with open(rec_path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(records[0].keys()))
w.writeheader()
w.writerows(records)
meta = {
"requirement": "VR-012",
"model": arcface.stem, "detector": detector.stem,
"n_actors": len(actors), "n_probes": len(probe_rows),
"n_gallery_embeddings": len(gal_rows),
"min_images": args.min_images, "seed": args.seed,
"sizes": sizes, "sigmas": sigmas, "blur_kind": args.blur_kind,
"motion_angle": args.motion_angle,
"prob_threshold": args.prob_threshold, "match_prior": args.match_prior,
"calibration": cal,
"measure_ranking": ranking,
"knees": knees,
"sharpness_window": list(sae_embed.sharpness_window()),
"caveat": (f"FPI grows with gallery size; this ran against {len(actors)} "
f"actors and still understates a production library."),
"grid": cells,
}
json_path = out.with_name(out.name + ".json")
json_path.write_text(json.dumps(meta, indent=2) + "\n")
png_path = out.with_name(out.name + ".png")
write_plots(cells, records, ranking, png_path, meta)
# ── stdout report ─────────────────────────────────────────────────────────
print(f"\nVR-012 — quality knee, {arcface.stem}")
print(f"{len(actors)} actors, {len(probe_rows)} probes x {len(cells)} cells\n")
print(f"{'size':>5} {'sigma':>6} {'TPI':>8} {'FPI':>8} {'unid':>8} {'rank1':>8}")
for c in cells:
print(f"{c['size_px']:>5} {c['sigma']:>6g} {100*c['tpi_rate']:>7.1f}% "
f"{100*c['fpi_rate']:>7.1f}% {100*c['unidentified_rate']:>7.1f}% "
f"{100*c['rank1_rate']:>7.1f}%")
print("\nAR-029 candidate ranking — AUC for predicting correct identification:")
for r in ranking:
print(f" {r['measure']:>20} {r['auc']:.4f}")
print(f"\n[out] {csv_path}\n[out] {rec_path}\n[out] {json_path}\n[out] {png_path}")
print(f"\n{meta['caveat']}")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/validation/test_audio_offset.py
+370
View File
@@ -0,0 +1,370 @@
#!/usr/bin/env python3
"""
VR-014 — the v1 audio signature recovers a known trim offset on real audio.
TRACES: UT-105, UT-106, UT-107, UT-108 | VR-014 | IR-004
python scripts/validation/test_audio_offset.py [build_dir]
The golden vector (IR-005) proves the *arithmetic* is identical in both
producers. It cannot prove the thing the signature exists for: that when the
same cut arrives trimmed differently, sliding one signature against the other
finds the true alignment and only the true alignment. Its fixture is a synthetic
tone sweep, which is pathologically easy to align; film dialogue and score are
not, and that is what this measures.
The signature is computed by the **shipped C++**, through the `sae_audio`
nanobind module — never a numpy port. A third implementation of a fingerprint
whose whole value rests on three implementations agreeing byte for byte would be
the one nobody checks against the golden vector.
The slide *is* written here in numpy, deliberately: matching is the consumer's
algorithm (server SPEC.md section 3), owned by the server and the jRay plugin,
not by this repo. Writing it out is what makes this a test of the signature
rather than a test of somebody's matcher.
Two independent offset mechanisms are checked, because they can fail
separately:
* a **window offset** (UT-105) — two 120 s excerpts taken from different
points, which is the alignment search itself; and
* a **head trim** (UT-106) — a real file with delta seconds removed from the
front, which additionally exercises the runtime/2 anchor: the window follows
the midpoint, so cutting delta from the head moves it by delta/2, not delta.
That factor of two is the easiest thing in the whole feature to get wrong
and nothing else checks it.
"""
import base64
import random
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
import numpy as np
REPO = Path(__file__).resolve().parent.parent.parent
BUILD = Path(sys.argv[1]) if len(sys.argv) > 1 else REPO / "build"
sys.path.insert(0, str(BUILD))
import sae_audio # noqa: E402
FIXTURE = REPO / "tests" / "fixtures" / "audio" / "bali_offset_200s.flac"
TONE = REPO / "tests" / "fixtures" / "audio" / "jray_audio_v1_tone.flac"
# Server SPEC.md section 3, "Matching and offset recovery". The cap is the
# spec's, not a convenience: +/-600 frames is ~56 s, which covers realistic trim
# differences, and an offset outside it must be declined rather than guessed at.
SEARCH_CAP_FRAMES = 600
AUDIO_TIER = 0.85
LOOSE_TIER = 0.60
TRIALS = 40
SEED = 20250731
HOP_SEC = sae_audio.hop_size / sae_audio.sample_rate
# What the offset is actually *for*: shifting scene windows, which are seconds
# long. Half a second of error is invisible against them, and that budget is
# what makes the numbers below readable — an offset is quantised to whole
# frames, so no correct answer can be worse than half a frame (46 ms) and the
# feature has an order of magnitude in hand before anything is at stake.
OFFSET_BUDGET_SEC = 0.5
def peak_bins(signature):
"""The per-frame peak band index, which is what the slide compares.
The `v1:` prefix is checked against the constant the C++ exports rather
than a literal, so a producer bump cannot be silently parsed as v1 here
(IR-008).
"""
prefix = sae_audio.version_prefix
if not signature.startswith(prefix):
raise AssertionError(f"signature is not {prefix!r}: {signature[:8]!r}")
packed = np.frombuffer(base64.b64decode(signature[len(prefix):]), dtype=np.uint8)
if np.any(packed & 0x80):
raise AssertionError("reserved bit set — not a structurally valid signature")
return packed >> 2
def best_match(reference, query, cap=SEARCH_CAP_FRAMES, slack=0):
"""Slide `query` against `reference`; return (score, offset_frames).
`offset` is how many frames later the query's window begins, so
``query[i]`` lines up with ``reference[i + offset]``. Score is the fraction
of overlapping frames whose peak bin agrees, exactly as the spec defines it.
`slack` widens what counts as agreement to a frame within +/-slack, which is
not the spec's rule — it is the candidate remedy UT-108 measures. It changes
the *score* only; the offset it reports is still a whole-frame alignment.
"""
best_score, best_offset = -1.0, 0
for offset in range(-cap, cap + 1):
if offset >= 0:
a, b = reference[offset:], query[: len(query) - offset]
else:
a, b = reference[: len(reference) + offset], query[-offset:]
n = min(len(a), len(b))
if n < 100: # too little overlap to mean anything
continue
a, b = a[:n], b[:n]
if slack == 0:
agree = a == b
else:
agree = np.zeros(n, dtype=bool)
for shift in range(-slack, slack + 1):
shifted = np.roll(a, shift)
# 255 is not a band index, so the wrapped end can never agree.
if shift > 0:
shifted[:shift] = 255
elif shift < 0:
shifted[shift:] = 255
agree |= shifted == b
score = float(np.mean(agree))
if score > best_score:
best_score, best_offset = score, offset
return best_score, best_offset
def decode_mono(path):
"""The whole fixture as float32 mono at 11025 Hz — the signature's own rate."""
raw = subprocess.run(
["ffmpeg", "-nostdin", "-v", "error", "-i", str(path),
"-ac", "1", "-ar", str(sae_audio.sample_rate), "-f", "f32le", "-"],
capture_output=True, check=True).stdout
return np.frombuffer(raw, dtype="<f4")
def trim_head(source, seconds, out):
"""`source` with `seconds` removed from the front — a differently trimmed release."""
subprocess.run(
["ffmpeg", "-nostdin", "-v", "error", "-y", "-ss", f"{seconds:.3f}", "-i", str(source),
"-ac", "1", "-ar", str(sae_audio.sample_rate), "-sample_fmt", "s16",
"-c:a", "flac", str(out)], check=True)
return out
def tier(score):
if score >= AUDIO_TIER:
return "audio"
return "loose" if score >= LOOSE_TIER else "none"
# ── The random trial set, signed once and reused ─────────────────────────────
def random_trials(pcm):
"""(reference bins, [(expected_frames, query bins)]) for TRIALS excerpts.
Signing 40 windows is the expensive part of this file, so UT-105 and UT-108
share one set — they ask different questions of the same measurements.
"""
window = sae_audio.window_samples
reference = peak_bins(sae_audio.signature_from_mono(pcm[:window]))
rng = random.Random(SEED)
queries = []
for _ in range(TRIALS):
# Within the search cap: past it, no offset is recoverable by
# construction, which UT-107 checks separately.
start = rng.randrange(0, SEARCH_CAP_FRAMES * sae_audio.hop_size)
signature = sae_audio.signature_from_mono(pcm[start:start + window])
assert signature is not None, "a full window must always sign"
queries.append((start / sae_audio.hop_size, peak_bins(signature)))
return reference, queries
# ── UT-105 — window offsets from random excerpt starts ───────────────────────
def test_random_window_offsets(reference, queries):
"""Every in-cap offset is recovered to the nearest frame, on real audio."""
rows = []
for want, query in queries:
score, offset = best_match(reference, query)
rows.append((want, offset, score, abs(want - round(want))))
expected = np.array([r[0] for r in rows])
offset = np.array([r[1] for r in rows])
score = np.array([r[2] for r in rows])
subframe = np.array([r[3] for r in rows])
error = np.abs(offset - expected)
# The offset is quantised to whole frames, so the best any correct answer
# can do is half a frame — 46 ms. What matters is the budget that half-frame
# is measured against, and it is an order of magnitude away from it.
assert error.max() <= 1.0, f"offset missed by {error.max():.2f} frames"
assert error.max() * HOP_SEC <= OFFSET_BUDGET_SEC, (
f"offset error {error.max() * HOP_SEC:.3f}s exceeds the {OFFSET_BUDGET_SEC}s budget")
# Never mistaken for different content. This is the floor that matters: the
# audio genuinely is the same cut, so a "no match" would be a false negative
# on the case the feature exists for.
assert score.min() >= LOOSE_TIER, f"same content scored {score.min():.3f}"
# An offset that lands near a frame boundary has no excuse: it should reach
# the top tier, and does.
aligned = subframe <= 0.1
assert aligned.any(), "seed no longer produces a near-aligned trial"
assert score[aligned].min() >= AUDIO_TIER, (
f"near-frame-aligned offset scored only {score[aligned].min():.3f}")
print(f"UT-105 {TRIALS} random window offsets, all within the +/-600 frame cap")
print(f" offset error : max {error.max():.2f} frames"
f" = {error.max() * HOP_SEC * 1000:.0f} ms, against a"
f" {OFFSET_BUDGET_SEC * 1000:.0f} ms budget")
print(f" score : min {score.min():.3f} median {np.median(score):.3f}"
f" max {score.max():.3f}")
print(" score by sub-frame misalignment — the offset is exact in every row:")
for lo, hi in ((0.0, 0.1), (0.1, 0.2), (0.2, 0.3), (0.3, 0.4), (0.4, 0.5)):
m = (subframe >= lo) & (subframe < hi)
if m.any():
print(f" {lo:.1f}-{hi:.1f} frame n={m.sum():2d}"
f" score {score[m].min():.3f}-{score[m].max():.3f}"
f" tier {tier(np.median(score[m]))}")
counts = {t: int(sum(1 for s in score if tier(s) == t)) for t in ("audio", "loose", "none")}
print(f" tiers : {counts}")
return counts
# ── UT-106 — head trims through real files, including the runtime/2 anchor ───
def test_head_trims():
"""A release with delta seconds of head removed aligns at delta/2 frames."""
reference = peak_bins(sae_audio.compute_signature(str(FIXTURE)))
results = []
with tempfile.TemporaryDirectory() as tmp:
for delta in (7.0, 23.5, 41.25, 60.0):
trimmed = trim_head(FIXTURE, delta, Path(tmp) / f"trim_{delta}.flac")
signature = sae_audio.compute_signature(str(trimmed))
assert signature is not None, f"trim of {delta}s should still sign"
score, offset = best_match(reference, peak_bins(signature))
# The window follows the midpoint, so removing delta from the head
# moves it by delta/2 — not by delta.
expected = (delta / 2.0) / HOP_SEC
assert abs(offset - expected) <= 1.0, (
f"head trim {delta}s: expected ~{expected:.1f} frames, got {offset}")
assert score >= LOOSE_TIER, f"head trim {delta}s scored {score:.3f}"
results.append((delta, expected, offset, score))
print("UT-106 head trims through the real decode path (compute_signature on a file)")
for delta, expected, offset, score in results:
print(f" -{delta:6.2f}s head expected {expected:7.2f} fr"
f" recovered {offset:5d} score {score:.3f} ({tier(score)})")
# ── UT-107 — what must NOT match ─────────────────────────────────────────────
def test_declines(pcm, reference):
"""Out-of-cap offsets and unrelated content are declined, not guessed at."""
window = sae_audio.window_samples
beyond = int(75.0 * sae_audio.sample_rate) # ~807 frames, past the cap
assert beyond + window <= len(pcm), "fixture too short for the out-of-cap case"
far = peak_bins(sae_audio.signature_from_mono(pcm[beyond:beyond + window]))
score_beyond, offset_beyond = best_match(reference, far)
assert score_beyond < LOOSE_TIER, (
f"an offset past the cap scored {score_beyond:.3f} at {offset_beyond} — "
"the search invented an alignment rather than declining")
tone = peak_bins(sae_audio.compute_signature(str(TONE)))
score_tone, offset_tone = best_match(reference, tone)
assert score_tone < LOOSE_TIER, f"unrelated content scored {score_tone:.3f}"
print("UT-107 declines rather than guesses")
print(f" offset past the +/-600 frame cap : best {score_beyond:.3f}"
f" at {offset_beyond} ({tier(score_beyond)})")
print(f" unrelated content (tone fixture) : best {score_tone:.3f}"
f" at {offset_tone} ({tier(score_tone)})")
return far, tone
# ── UT-108 — the sub-frame demotion, and what one frame of slack costs ───────
def test_scoring_slack(reference, queries, far, tone):
"""Measured: +/-1 frame of slack in the *score* restores the `audio` tier.
UT-105 leaves a real question open. Every offset is right, but two thirds of
them score below the server's 0.85 `audio` threshold purely because the two
windows' frame grids do not coincide — so a correctly aligned release is
demoted to `loose`, which is the tier meaning "possibly the same cut,
degraded audio". The obvious remedy is to stop demanding that frames line up
exactly, and the question is what that costs in discrimination.
Nothing is asserted about the spec's own rule here; this measures a
candidate change to it, which is the server's to make (SPEC.md section 3).
"""
print("UT-108 cost of relaxing the score's frame alignment")
print(f" {'slack':>5} {'audio':>6} {'loose':>6} {'none':>5}"
f" {'min true':>9} {'worst err':>10} {'unrelated':>10} {'out-of-cap':>11}")
measured = {}
for slack in (0, 1, 2):
score, error = [], []
for want, query in queries:
s, offset = best_match(reference, query, slack=slack)
score.append(s)
error.append(abs(offset - want))
score, error = np.array(score), np.array(error)
false_tone, _ = best_match(reference, tone, slack=slack)
false_far, _ = best_match(reference, far, slack=slack)
counts = {t: int(sum(1 for s in score if tier(s) == t)) for t in ("audio", "loose", "none")}
measured[slack] = (score, error, max(false_tone, false_far))
print(f" {slack:>5} {counts['audio']:>6} {counts['loose']:>6} {counts['none']:>5}"
f" {score.min():>9.3f} {error.max() * HOP_SEC * 1000:>7.0f} ms"
f" {false_tone:>10.3f} {false_far:>11.3f}")
score, error, worst_false = measured[1]
# One frame of slack lifts every correct alignment to the top tier...
assert score.min() >= AUDIO_TIER, (
f"one frame of slack still leaves a true match at {score.min():.3f}")
# ...without narrowing the gap that makes the threshold mean anything...
assert worst_false < LOOSE_TIER, (
f"slack lifted a false match to {worst_false:.3f}")
# ...and the offset it costs is still far inside the budget: the score's
# peak flattens slightly, so the argmax can pick an adjacent frame.
assert error.max() * HOP_SEC <= OFFSET_BUDGET_SEC, (
f"slack cost {error.max() * HOP_SEC:.3f}s of offset accuracy")
print(f" +/-1 frame: every true match reaches `audio` (min {score.min():.3f}),"
f" worst false stays at {worst_false:.3f},")
print(f" and the offset costs {error.max() * HOP_SEC * 1000:.0f} ms of a"
f" {OFFSET_BUDGET_SEC * 1000:.0f} ms budget. +/-2 buys nothing more.")
def main():
if not FIXTURE.exists():
print(f"missing fixture {FIXTURE} — regenerate with make_offset_fixture.sh", file=sys.stderr)
return 2
if shutil.which("ffmpeg") is None:
print("this validation needs the ffmpeg CLI to trim the fixture", file=sys.stderr)
return 2
print(f"VR-014 audio-signature offset recovery on {FIXTURE.name}")
print(f" {sae_audio.expected_frames} frames per signature,"
f" {HOP_SEC * 1000:.2f} ms per frame, cap +/-{SEARCH_CAP_FRAMES} frames")
pcm = decode_mono(FIXTURE)
reference, queries = random_trials(pcm)
counts = test_random_window_offsets(reference, queries)
test_head_trims()
far, tone = test_declines(pcm, reference)
test_scoring_slack(reference, queries, far, tone)
print()
print(f"PASS — every in-cap offset recovered to the nearest frame, worst"
f" {1000 * HOP_SEC / 2:.0f} ms against a {OFFSET_BUDGET_SEC * 1000:.0f} ms budget.")
if counts["audio"] < TRIALS:
# Stated rather than asserted against the spec's rule: the offset is
# right in every case, so this is the 0.85 threshold meeting a sub-frame
# shift, not a defect in the signature. The threshold was calibrated on
# a re-encode at zero offset, where the score is 1.00. UT-108 measures
# the remedy; adopting it is the server spec's call, not this repo's.
print(f"NOTE — under the spec's exact-frame score only {counts['audio']}/{TRIALS}"
f" reach `audio`; {counts['loose']} are demoted to `loose` by sub-frame"
" shift alone. See UT-108.")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/vendor/jray-project
Vendored Submodule
+1
Submodule scripts/vendor/jray-project added at 17106f3370
src/audio_bindings.cpp
+125
View File
@@ -0,0 +1,125 @@
// sae_audio — Python module wrapping the v1 audio signature (audio_signature.*).
//
/// TRACES: IR-004, IR-005 | SR-003
//
// Exists so a study or a test can drive the **shipped** signature code from
// Python instead of porting the DSP to numpy. A numpy port would be a third
// implementation of a fingerprint that only works if every implementation
// agrees byte for byte, and it would be the one nobody checks against the
// golden vector — so the offset-recovery validation (VR-014) calls this.
//
// Bound with nanobind, as `sae_embed` and `sae_kpn` are. Not pybind11: a second
// binding framework in one build is a second set of ABI and lifetime rules to
// get right, for a module that needs nothing nanobind lacks.
//
// The module deliberately stops at the producer's edge. Matching — sliding one
// signature against another and scoring the overlap — is the *consumer's*
// algorithm (server SPEC §3, and the jRay plugin implements it), so it is not
// bound here and a caller writing a slide in numpy is not re-implementing
// anything this repo owns.
#include "audio_signature.hpp"
#include <nanobind/nanobind.h>
#include <nanobind/ndarray.h>
#include <nanobind/stl/optional.h>
#include <nanobind/stl/pair.h>
#include <nanobind/stl/string.h>
#include <nanobind/stl/vector.h>
#include <cstdint>
#include <optional>
#include <string>
#include <utility>
#include <vector>
namespace nb = nanobind;
using namespace nb::literals;
using namespace sae::audio;
namespace {
using MonoArray = nb::ndarray<const float, nb::ndim<1>, nb::c_contig, nb::device::cpu>;
// Hand the vector's buffer to Python without copying 1.3 M samples, and let a
// capsule own it: the array outlives this call, so the storage has to as well.
nb::object own_as_ndarray(std::vector<float>&& samples) {
auto* held = new std::vector<float>(std::move(samples));
nb::capsule owner(held, [](void* p) noexcept {
delete static_cast<std::vector<float>*>(p);
});
const std::size_t n = held->size();
return nb::cast(nb::ndarray<nb::numpy, float, nb::ndim<1>>(held->data(), {n}, owner));
}
std::vector<float> to_vector(const MonoArray& a) {
return std::vector<float>(a.data(), a.data() + a.shape(0));
}
} // namespace
NB_MODULE(sae_audio, m) {
m.doc() =
"JRay v1 audio signature (JRay-public-server SPEC.md section 3), as the "
"extraction pipeline computes it. The constants below are the contract: "
"changing any of them is a v1 -> v2 change.";
m.attr("sample_rate") = kSampleRate;
m.attr("frame_size") = kFrameSize;
m.attr("hop_size") = kHopSize;
m.attr("num_bands") = kNumBands;
m.attr("band_lo_hz") = kBandLoHz;
m.attr("band_hi_hz") = kBandHiHz;
m.attr("window_sec") = kWindowSec;
m.attr("window_samples") = kWindowSamples;
m.attr("expected_frames") = kExpectedFrames;
m.attr("version_prefix") = std::string(kVersionPrefix);
m.def(
"compute_signature",
[](const std::string& path) { return compute_signature(path); },
"path"_a,
"Signature of the 120 s window centred on the media's midpoint, or None "
"for media shorter than the window (IR-007), media with no audio "
"stream, and any decode failure — degradation, never an exception.");
m.def(
"decode_centre_window",
[](const std::string& path) -> nb::object {
std::optional<std::vector<float>> mono = decode_centre_window(path);
if (!mono) {
return nb::none();
}
return own_as_ndarray(std::move(*mono));
},
"path"_a,
"The decoded centre window as float32 mono at 11025 Hz, or None. Exposed "
"so a caller can slice or perturb real audio and re-sign it without "
"going back through a container.");
m.def(
"signature_from_mono",
[](const MonoArray& mono) { return signature_from_mono(to_vector(mono)); },
"mono"_a,
"Signature of mono float32 samples already at 11025 Hz, in [-1, 1). None "
"when fewer than one whole frame is given.");
m.def(
"pack_frames",
[](const MonoArray& mono) {
std::vector<std::uint8_t> packed = pack_frames(to_vector(mono));
return nb::bytes(reinterpret_cast<const char*>(packed.data()), packed.size());
},
"mono"_a,
"One packed byte per whole STFT frame: (band << 2) | energy_class. This "
"is the payload the signature base64-encodes.");
m.def(
"band_fft_bins",
[] {
const auto& table = band_fft_bins();
return std::vector<std::pair<int, int>>(table.begin(), table.end());
},
"The half-open FFT bin range owned by each of the 32 log-spaced bands.");
}
src/audio_signature.cpp
+428
View File
@@ -0,0 +1,428 @@
// ── JRay audio signature, v1 — implementation ────────────────────────────────
//
/// TRACES: IR-004, IR-007, IR-008 | SR-003
//
// The contract this implements is documented in full in audio_signature.hpp;
// read that before changing anything here. Every constant is load-bearing: the
// JRay Jellyfin plugin computes the same bytes in C#, and a signature that
// differs in any parameter simply does not match.
//
// Audio decode is a *second stream from an existing dependency* — the pipeline
// already links libavformat/libavcodec/libavutil for video (ffmpeg_decoder.hpp);
// this adds libswresample for the downmix+resample, no new project dependency.
// The FFT is written out here rather than pulled from a library for the same
// reason the plugin vendors one: it is a fixed, fully specified transform, and
// a dependency whose version could change the numerics is a liability when the
// output has to be bit-identical across two languages.
#include "audio_signature.hpp"
extern "C" {
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libavutil/avutil.h>
#include <libavutil/channel_layout.h>
#include <libavutil/opt.h>
#include <libavutil/samplefmt.h>
#include <libswresample/swresample.h>
}
#include <algorithm>
#include <cmath>
#include <cstring>
#include <vector>
namespace sae::audio {
namespace {
constexpr double kPi = 3.14159265358979323846;
// ── Band table ───────────────────────────────────────────────────────────────
// edge[b] = 300 * 10^(b/32); band b owns FFT bins [k_lo[b], k_lo[b+1]).
// ceil() of the edge in bins, so membership is decided once by integers rather
// than by a float comparison per bin per frame. The bands tile [112, 1115)
// contiguously with no gap and no overlap, which is what lets the frame energy
// below be accumulated from the per-band sums.
std::array<std::pair<int, int>, kNumBands> build_band_table() {
const double hz_per_bin = static_cast<double>(kSampleRate) / kFrameSize;
std::array<int, kNumBands + 1> k{};
for (int b = 0; b <= kNumBands; ++b) {
const double edge = kBandLoHz * std::pow(kBandHiHz / kBandLoHz,
static_cast<double>(b) / kNumBands);
k[b] = static_cast<int>(std::ceil(edge / hz_per_bin));
}
std::array<std::pair<int, int>, kNumBands> tbl{};
for (int b = 0; b < kNumBands; ++b) tbl[b] = {k[b], k[b + 1]};
return tbl;
}
// Hann, periodic: w[n] = 0.5 * (1 - cos(2*pi*n/N)). Not the symmetric (N-1)
// variant — the two differ, and the difference is observable.
const std::vector<double>& hann_window() {
static const std::vector<double> w = [] {
std::vector<double> v(kFrameSize);
for (int n = 0; n < kFrameSize; ++n)
v[n] = 0.5 * (1.0 - std::cos(2.0 * kPi * n / kFrameSize));
return v;
}();
return w;
}
// ── Radix-2 decimation-in-time complex FFT, in place, no normalisation ──────
// Twiddles are precomputed per stage from cos/sin of -2*pi*j/len so the angle
// is an exactly reproducible double in any language and only the libm rounding
// of cos/sin (≤1 ulp) can differ — orders of magnitude below the decision
// margins in the golden fixture.
struct FftTables {
std::vector<int> rev; // bit-reversal permutation
std::vector<std::vector<double>> wr, wi; // per stage
};
const FftTables& fft_tables() {
static const FftTables t = [] {
FftTables f;
f.rev.resize(kFrameSize);
int bits = 0;
while ((1 << bits) < kFrameSize) ++bits;
for (int i = 0; i < kFrameSize; ++i) {
int r = 0;
for (int b = 0; b < bits; ++b)
if (i & (1 << b)) r |= 1 << (bits - 1 - b);
f.rev[i] = r;
}
for (int len = 2; len <= kFrameSize; len <<= 1) {
const int half = len / 2;
std::vector<double> cr(half), ci(half);
for (int j = 0; j < half; ++j) {
const double ang = -2.0 * kPi * j / len;
cr[j] = std::cos(ang);
ci[j] = std::sin(ang);
}
f.wr.push_back(std::move(cr));
f.wi.push_back(std::move(ci));
}
return f;
}();
return t;
}
void fft_4096(std::vector<double>& re, std::vector<double>& im) {
const FftTables& t = fft_tables();
for (int i = 0; i < kFrameSize; ++i) {
const int j = t.rev[i];
if (i < j) { std::swap(re[i], re[j]); std::swap(im[i], im[j]); }
}
int stage = 0;
for (int len = 2; len <= kFrameSize; len <<= 1, ++stage) {
const int half = len / 2;
const std::vector<double>& wr = t.wr[stage];
const std::vector<double>& wi = t.wi[stage];
for (int base = 0; base < kFrameSize; base += len) {
for (int j = 0; j < half; ++j) {
const int a = base + j;
const int b = a + half;
const double tr = re[b] * wr[j] - im[b] * wi[j];
const double ti = re[b] * wi[j] + im[b] * wr[j];
re[b] = re[a] - tr; im[b] = im[a] - ti;
re[a] = re[a] + tr; im[a] = im[a] + ti;
}
}
}
}
int energy_class(double r) {
if (r < kEnergyClassEdges[0]) return 0;
if (r < kEnergyClassEdges[1]) return 1;
if (r < kEnergyClassEdges[2]) return 2;
return 3;
}
// ── FFmpeg RAII ─────────────────────────────────────────────────────────────
struct DecodeCtx {
AVFormatContext* fmt = nullptr;
AVCodecContext* dec = nullptr;
SwrContext* swr = nullptr;
AVFrame* frm = nullptr;
AVPacket* pkt = nullptr;
~DecodeCtx() {
if (swr) swr_free(&swr);
if (frm) av_frame_free(&frm);
if (pkt) av_packet_free(&pkt);
if (dec) avcodec_free_context(&dec);
if (fmt) avformat_close_input(&fmt);
}
};
bool open_resampler(DecodeCtx& c, const AVFrame* f) {
#if LIBAVUTIL_VERSION_INT >= AV_VERSION_INT(57, 24, 100)
AVChannelLayout out_layout;
av_channel_layout_default(&out_layout, 1); // mono
AVChannelLayout in_layout;
if (av_channel_layout_copy(&in_layout, &f->ch_layout) < 0) return false;
if (in_layout.nb_channels <= 0) {
av_channel_layout_uninit(&in_layout);
av_channel_layout_default(&in_layout, 1);
}
const int rc = swr_alloc_set_opts2(
&c.swr,
&out_layout, AV_SAMPLE_FMT_FLT, kSampleRate,
&in_layout, static_cast<AVSampleFormat>(f->format),
f->sample_rate ? f->sample_rate : kSampleRate,
0, nullptr);
av_channel_layout_uninit(&in_layout);
av_channel_layout_uninit(&out_layout);
if (rc < 0 || !c.swr) return false;
#else
const int64_t in_layout = f->channel_layout
? static_cast<int64_t>(f->channel_layout)
: av_get_default_channel_layout(f->channels ? f->channels : 1);
c.swr = swr_alloc_set_opts(
nullptr,
AV_CH_LAYOUT_MONO, AV_SAMPLE_FMT_FLT, kSampleRate,
in_layout, static_cast<AVSampleFormat>(f->format),
f->sample_rate ? f->sample_rate : kSampleRate,
0, nullptr);
if (!c.swr) return false;
#endif
return swr_init(c.swr) >= 0;
}
// Push one decoded frame (or a flush) through the resampler, dropping the
// leading `to_skip` output samples, and append to `out`.
void drain(SwrContext* swr, const AVFrame* f, int in_rate,
std::size_t& to_skip, std::vector<float>& out) {
const int64_t delay = swr_get_delay(swr, in_rate ? in_rate : kSampleRate);
const int in_n = f ? f->nb_samples : 0;
const int max_out = static_cast<int>(av_rescale_rnd(
delay + in_n, kSampleRate, in_rate ? in_rate : kSampleRate, AV_ROUND_UP)) + 32;
if (max_out <= 0) return;
std::vector<float> buf(static_cast<std::size_t>(max_out));
uint8_t* dst = reinterpret_cast<uint8_t*>(buf.data());
const int n = swr_convert(swr, &dst, max_out,
f ? const_cast<const uint8_t**>(f->extended_data) : nullptr,
in_n);
if (n <= 0) return;
std::size_t produced = static_cast<std::size_t>(n);
std::size_t off = 0;
if (to_skip) {
const std::size_t drop = std::min(to_skip, produced);
to_skip -= drop;
off = drop;
produced -= drop;
}
if (produced)
out.insert(out.end(), buf.begin() + off, buf.begin() + off + produced);
}
} // namespace
// ── Public surface ──────────────────────────────────────────────────────────
const std::array<std::pair<int, int>, kNumBands>& band_fft_bins() {
static const std::array<std::pair<int, int>, kNumBands> tbl = build_band_table();
return tbl;
}
std::string base64_encode(const std::uint8_t* data, std::size_t n) {
static constexpr char kAlphabet[] =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
std::string out;
out.reserve(((n + 2) / 3) * 4);
std::size_t i = 0;
for (; i + 3 <= n; i += 3) {
const std::uint32_t v = (std::uint32_t(data[i]) << 16) |
(std::uint32_t(data[i + 1]) << 8) |
std::uint32_t(data[i + 2]);
out += kAlphabet[(v >> 18) & 0x3F];
out += kAlphabet[(v >> 12) & 0x3F];
out += kAlphabet[(v >> 6) & 0x3F];
out += kAlphabet[v & 0x3F];
}
if (i < n) {
const bool two = (n - i) == 2;
const std::uint32_t v = (std::uint32_t(data[i]) << 16) |
(two ? (std::uint32_t(data[i + 1]) << 8) : 0u);
out += kAlphabet[(v >> 18) & 0x3F];
out += kAlphabet[(v >> 12) & 0x3F];
out += two ? kAlphabet[(v >> 6) & 0x3F] : '=';
out += '=';
}
return out;
}
std::uint64_t fnv1a64(const void* data, std::size_t n) {
const auto* p = static_cast<const std::uint8_t*>(data);
std::uint64_t h = 0xcbf29ce484222325ULL;
for (std::size_t i = 0; i < n; ++i) {
h ^= p[i];
h *= 0x100000001b3ULL;
}
return h;
}
/// TRACES: IR-004
std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono) {
if (mono.size() < static_cast<std::size_t>(kFrameSize)) return {};
const std::size_t nframes = 1 + (mono.size() - kFrameSize) / kHopSize;
const auto& bands = band_fft_bins();
const auto& win = hann_window();
const int k_lo = bands.front().first;
const int k_hi = bands.back().second; // exclusive
const double bin_count = static_cast<double>(k_hi - k_lo);
std::vector<double> re(kFrameSize), im(kFrameSize);
std::vector<std::uint8_t> peak(nframes);
std::vector<double> energy(nframes);
for (std::size_t f = 0; f < nframes; ++f) {
const float* src = mono.data() + f * kHopSize;
for (int n = 0; n < kFrameSize; ++n) {
re[n] = static_cast<double>(src[n]) * win[n];
im[n] = 0.0;
}
fft_4096(re, im);
// Per-band mean magnitude; the bands tile the 300–3000 Hz range with no
// gaps, so the frame's band-limited energy is the sum of the band sums.
double best = -1.0, total = 0.0;
int best_b = 0;
for (int b = 0; b < kNumBands; ++b) {
double sum = 0.0;
for (int k = bands[b].first; k < bands[b].second; ++k)
sum += std::sqrt(re[k] * re[k] + im[k] * im[k]);
total += sum;
const double mean = sum / (bands[b].second - bands[b].first);
if (mean > best) { best = mean; best_b = b; } // ties → lowest index
}
peak[f] = static_cast<std::uint8_t>(best_b);
energy[f] = total / bin_count;
}
// Reference is the upper median of the frame energies: an actually observed
// value (no averaging of the two middle samples), so it is bit-reproducible,
// gain-invariant and barely moves when the window is trimmed.
std::vector<double> sorted = energy;
std::sort(sorted.begin(), sorted.end());
const double ref = sorted[sorted.size() / 2];
std::vector<std::uint8_t> out(nframes);
for (std::size_t f = 0; f < nframes; ++f) {
const double r = std::log10((energy[f] + kEnergyEps) / (ref + kEnergyEps));
out[f] = static_cast<std::uint8_t>(((peak[f] & 0x1F) << 2) |
(energy_class(r) & 0x03));
}
return out;
}
/// TRACES: IR-004, IR-008
std::optional<std::string> signature_from_mono(const std::vector<float>& mono) {
const std::vector<std::uint8_t> packed = pack_frames(mono);
if (packed.empty()) return std::nullopt;
return std::string(kVersionPrefix) + base64_encode(packed.data(), packed.size());
}
/// TRACES: IR-004, IR-007
std::optional<std::vector<float>> decode_centre_window(const std::string& path) {
av_log_set_level(AV_LOG_ERROR);
DecodeCtx c;
if (avformat_open_input(&c.fmt, path.c_str(), nullptr, nullptr) < 0)
return std::nullopt;
if (avformat_find_stream_info(c.fmt, nullptr) < 0) return std::nullopt;
if (c.fmt->duration == AV_NOPTS_VALUE) return std::nullopt;
const double duration = static_cast<double>(c.fmt->duration) / AV_TIME_BASE;
// IR-007 — the window underflows, so there is no signature and no sync
// offset downstream. The plugin applies the identical rule.
if (duration < kWindowSec) return std::nullopt;
const int idx = av_find_best_stream(c.fmt, AVMEDIA_TYPE_AUDIO, -1, -1, nullptr, 0);
if (idx < 0) return std::nullopt; // no audio → no signature
AVStream* st = c.fmt->streams[idx];
const AVCodec* codec = avcodec_find_decoder(st->codecpar->codec_id);
if (!codec) return std::nullopt;
c.dec = avcodec_alloc_context3(codec);
if (!c.dec) return std::nullopt;
if (avcodec_parameters_to_context(c.dec, st->codecpar) < 0) return std::nullopt;
c.dec->thread_count = 0;
if (avcodec_open2(c.dec, codec, nullptr) < 0) return std::nullopt;
const double start_sec = duration / 2.0 - kWindowSec / 2.0;
// Seek to a packet at or before the window start; the exact start is then
// reached by discarding the leading output samples, which is what
// `ffmpeg -ss <t> -i <file>` does and therefore what the plugin sees.
if (start_sec > 0.0) {
const int64_t tgt = av_rescale_q(
static_cast<int64_t>(start_sec * AV_TIME_BASE), AV_TIME_BASE_Q, st->time_base);
if (av_seek_frame(c.fmt, idx, tgt, AVSEEK_FLAG_BACKWARD) >= 0)
avcodec_flush_buffers(c.dec);
}
c.frm = av_frame_alloc();
c.pkt = av_packet_alloc();
if (!c.frm || !c.pkt) return std::nullopt;
std::vector<float> mono;
mono.reserve(kWindowSamples + kSampleRate);
std::size_t to_skip = 0;
bool have_swr = false;
int in_rate = kSampleRate;
bool eof = false;
while (mono.size() < kWindowSamples && !eof) {
const int rr = av_read_frame(c.fmt, c.pkt);
if (rr < 0) {
eof = true;
avcodec_send_packet(c.dec, nullptr); // flush the decoder
} else if (c.pkt->stream_index != idx) {
av_packet_unref(c.pkt);
continue;
} else {
avcodec_send_packet(c.dec, c.pkt);
av_packet_unref(c.pkt);
}
while (avcodec_receive_frame(c.dec, c.frm) == 0) {
if (!have_swr) {
if (!open_resampler(c, c.frm)) return std::nullopt;
have_swr = true;
in_rate = c.frm->sample_rate ? c.frm->sample_rate : kSampleRate;
int64_t pts = c.frm->best_effort_timestamp;
if (pts == AV_NOPTS_VALUE) pts = c.frm->pts;
const double t0 = (pts == AV_NOPTS_VALUE)
? start_sec : av_q2d(st->time_base) * static_cast<double>(pts);
const double lead = start_sec - t0;
to_skip = lead > 0.0
? static_cast<std::size_t>(std::llround(lead * kSampleRate)) : 0;
}
drain(c.swr, c.frm, in_rate, to_skip, mono);
av_frame_unref(c.frm);
if (mono.size() >= kWindowSamples) break;
}
}
if (have_swr && mono.size() < kWindowSamples)
drain(c.swr, nullptr, in_rate, to_skip, mono); // flush the resampler
if (mono.empty()) return std::nullopt;
// Truncate to exactly 120.000 s so the frame count is 1288 for every input
// and does not wobble with seek granularity or the resampler tail.
if (mono.size() > kWindowSamples) mono.resize(kWindowSamples);
return mono;
}
/// TRACES: IR-004, IR-005, IR-007, IR-008
std::optional<std::string> compute_signature(const std::string& path) {
const std::optional<std::vector<float>> mono = decode_centre_window(path);
if (!mono) return std::nullopt;
return signature_from_mono(*mono);
}
} // namespace sae::audio
src/audio_signature.hpp
+158
View File
@@ -0,0 +1,158 @@
#pragma once
// ── JRay audio signature, v1 ─────────────────────────────────────────────────
//
/// TRACES: IR-004, IR-005, IR-007, IR-008 | SR-003
//
// A content-derived spectral-peak signature taken from the *centre* of the
// media, so a truth file is self-identifying: a consumer can tell whether a
// local file is the same cut as the one a manifest describes, and recover the
// frame offset when it is the same cut trimmed differently.
//
// The construction is owned by `JRay-public-server/SPEC.md` §3 and is
// reproduced by the JRay Jellyfin plugin in C#. **The two implementations must
// agree byte for byte** — a signature that differs in any parameter simply does
// not match, which defeats the entire point. Every deviation is therefore a
// breaking change and must go through the `v1:` prefix (see kVersionPrefix).
//
// Server spec §3, restated:
//
// 1. Decode a 120 s window centred on the midpoint (runtime/2 ± 60 s).
// 2. Downmix to mono, resample to 11025 Hz.
// 3. STFT: 4096-sample frame, 1024-sample hop, Hann window (~1290 frames).
// 4. Per frame, log-magnitude spectrum over 300–3000 Hz.
// 5. 32 logarithmically spaced bins; peak bin index + coarse 2-bit energy
// class.
// 6. Pack one byte per frame; base64-encode.
// 7. Prefix `v1:`.
//
// ── Details the server spec leaves open, pinned here for v1 ──────────────────
//
// The prose above is not sufficient to reproduce a byte stream, so the choices
// below are the contract. They are mirrored in
// `tests/fixtures/audio/jray_audio_v1_golden.json`, which is the artefact
// shared with the plugin repo (IR-005).
//
// Arithmetic All DSP in IEEE-754 **double**. float32 is not sufficient:
// the golden fixture has frames whose two strongest bands are
// within 1.3% of each other, which double resolves identically
// everywhere and float32 does not.
// Sample scale FFmpeg's native s16→flt conversion, x * (1/32768), then
// widened to double. Values in [-1, 1).
// Framing Only whole frames: n_frames = 1 + (n_samples - 4096) / 1024,
// integer division, 0 when n_samples < 4096. A 120.000 s
// window is 1 323 000 samples → **1288 frames**.
// ("~1290" in the spec; the server accepts a tolerance.)
// Window Hann, **periodic**: w[n] = 0.5 * (1 - cos(2*pi*n/4096)).
// Not the symmetric (N-1) variant.
// Transform Plain radix-2 decimation-in-time complex FFT over 4096 real
// samples (imag = 0), no normalisation. Magnitude is
// sqrt(re² + im²). Twiddles from cos/sin of
// -2*pi*k/len computed in double.
// Band edges edge[b] = 300 * (3000/300)^(b/32), b = 0..32. Band b spans
// FFT bins [k_lo[b], k_lo[b+1]) with
// k_lo[b] = ceil(edge[b] * 4096 / 11025) — i.e. bins 112..1114
// inclusive, 8 bins in the narrowest band. Precomputed as an
// integer table so no float comparison decides membership.
// Band value **Mean** of the linear magnitudes in the band. Mean, not
// sum, so a wide high band is not favoured over a narrow low
// one; magnitude, not power, because it is an energy proxy and
// more codec-robust than a single bin's peak.
// Peak bin argmax over the 32 band values; ties resolve to the **lowest
// index**. The log of step 4 is a monotone squash and so
// cannot change an argmax — it is applied only where it is
// observable, in the energy class below.
// Energy class The spec says "coarse 2-bit energy class" and no more. v1
// defines it as the frame's band-limited energy relative to
// the window, which is invariant to gain (loudness
// normalisation must not change a signature) and robust to
// trimming (the median barely moves):
// E_f = mean magnitude over *all* FFT bins 112..1114
// Eref = median over frames of E_f, taken as the upper
// median sorted[n/2] — no averaging of the two middle
// values, so the reference is always an actual
// observed value and is bit-reproducible
// r = log10((E_f + 1e-12) / (Eref + 1e-12))
// class = 0 if r < -0.6, 1 if r < -0.2, 2 if r < 0.2, else 3
// The thresholds deliberately straddle r = 0 rather than sit
// on it, so the median frame itself is not on a boundary.
// Byte layout bit 7 = 0 (reserved), bits 6..2 = 5-bit band index,
// bits 1..0 = 2-bit energy class:
// byte = (band << 2) | class → always 0..127
// This is the structural constraint the server validates on
// upload (§3 "Validation and abuse").
// Base64 Standard alphabet A–Za–z0–9+/ with '=' padding.
//
// ── Short media (IR-007) ─────────────────────────────────────────────────────
//
// `runtime/2 ± 60 s` underflows below 120 s, so **no signature is emitted** and
// no sync offset is applied downstream. Both producers apply the identical
// rule; diverging here would break exactly the short items most likely to be
// misidentified. `compute_signature` returns `std::nullopt`.
//
// The same nullopt is returned for a file with no audio stream, an unopenable
// file, or an unknown duration. UR-9 is an enhancement and must never be able
// to break a fetch — degradation, not failure.
#include <array>
#include <cstddef>
#include <cstdint>
#include <optional>
#include <string>
#include <utility>
#include <vector>
namespace sae::audio {
// ── Contract constants — changing any of these is a `v1:` bump ───────────────
inline constexpr int kSampleRate = 11025;
inline constexpr int kFrameSize = 4096;
inline constexpr int kHopSize = 1024;
inline constexpr int kNumBands = 32;
inline constexpr double kBandLoHz = 300.0;
inline constexpr double kBandHiHz = 3000.0;
inline constexpr double kWindowSec = 120.0;
inline constexpr double kEnergyEps = 1e-12;
// Class thresholds on log10(E_frame / E_median); see the header comment.
inline constexpr double kEnergyClassEdges[3] = {-0.6, -0.2, 0.2};
// 120.000 s at 11025 Hz. The decoded window is truncated to exactly this so the
// frame count does not wobble with seek granularity or resampler tail.
inline constexpr std::size_t kWindowSamples =
static_cast<std::size_t>(kWindowSec * kSampleRate); // 1 323 000
inline constexpr std::size_t kExpectedFrames =
1 + (kWindowSamples - kFrameSize) / kHopSize; // 1288
static_assert(kWindowSamples == 1323000, "120 s at 11025 Hz");
static_assert(kExpectedFrames == 1288, "server spec's ~1290 frames");
/// The version prefix is the signature's own, separate from `schema_version`:
/// a future change to the DSP chain must be *detectable* rather than silently
/// producing non-matching signatures (IR-008).
inline constexpr const char* kVersionPrefix = "v1:";
/// FFT bin range [first, last) for each of the 32 log-spaced bands.
/// Computed once from the constants above; exposed so the golden fixture can
/// assert the table itself, not merely the signature it produces.
const std::array<std::pair<int, int>, kNumBands>& band_fft_bins();
/// Decode the centre window of `path` as mono float PCM at 11025 Hz.
/// nullopt when the media is shorter than 120 s (IR-007), has no audio stream,
/// or cannot be opened. Never throws.
std::optional<std::vector<float>> decode_centre_window(const std::string& path);
/// One packed byte per whole STFT frame. Empty when `mono` is shorter than one
/// frame. This is the payload that gets base64-encoded.
std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono);
/// `v1:` + base64(pack_frames(mono)). nullopt when no whole frame fits.
std::optional<std::string> signature_from_mono(const std::vector<float>& mono);
/// Decode + sign. The one call the pipeline makes. nullopt per IR-007 and on
/// any decode failure — degradation, not failure.
std::optional<std::string> compute_signature(const std::string& path);
// ── Small utilities, exposed for the golden-fixture test ────────────────────
std::string base64_encode(const std::uint8_t* data, std::size_t n);
/// FNV-1a 64. Used only to pin the *decoded PCM* in the golden fixture, so a
/// codec-level difference is distinguishable from a DSP-level one.
std::uint64_t fnv1a64(const void* data, std::size_t n);
} // namespace sae::audio
src/backends/gemm_backend.cpp
+36 -4
View File
@@ -34,9 +34,23 @@ constexpr int kDim = 512;
#if defined(SAE_GEMM_CPU)
#if defined(SAE_GEMM_CBLAS)
#include <cblas.h>
#endif
// ── CPU reference engine ──────────────────────────────────────────────────────
// Portable, dependency-free path used for CI and as the correctness oracle for
// the GPU backends. The gallery is L2-normalised (as are the queries), so each
// Used for CI and as the correctness oracle for the GPU backends.
//
// TRACES: AR-026, AR-027 | SR-001
// Backed by CBLAS (OpenBLAS) when available, falling back to a scalar loop when
// not. The fallback is portable but scales badly: scoring one face against a
// 5000-embedding gallery is 2.6 MFLOP, and a crowded frame multiplies that by
// the face count. Since AR-003 removed the per-frame face cap and CI has no GPU,
// the CPU path is now the one that has to hold up under a library-scale gallery
// (AR-027) rather than merely be correct.
//
// The fallback is kept rather than made mandatory so the build has no hard new
// dependency, and so the two can be diffed when a similarity looks wrong. The gallery is L2-normalised (as are the queries), so each
// similarity is a plain dot product. S is stored column-major to match the GPU
// backends: the gallery similarities for face fi start at result + fi*n_gallery.
class SimilarityEngine final : public ISimilarityEngine {
@@ -47,7 +61,13 @@ public:
gallery_row_major + static_cast<size_t>(n_gallery) * kDim)
{
host_sims_.resize(static_cast<size_t>(max_faces_) * n_gallery_);
std::cerr << "[similarity] CPU reference engine: gallery resident in host RAM ("
std::cerr << "[similarity] CPU engine ("
#if defined(SAE_GEMM_CBLAS)
<< "CBLAS"
#else
<< "scalar fallback — no CBLAS; expect poor scaling on a large gallery"
#endif
<< "): gallery resident in host RAM ("
<< (gallery_.size() * sizeof(float)) / (1024 * 1024) << " MiB)\n";
}
@@ -58,7 +78,18 @@ public:
if (n_faces > max_faces_)
throw std::runtime_error("SimilarityEngine: n_faces exceeds max_faces");
// S(g, f) col-major = dot(gallery[g], query[f]).
// S(g, f) col-major = dot(gallery[g], query[f]). Viewed as row-major
// [n_faces x n_gallery] that is exactly query * gallery^T, so it is one
// GEMM rather than a loop nest.
#if defined(SAE_GEMM_CBLAS)
cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans,
/*M=*/n_faces, /*N=*/n_gallery_, /*K=*/kDim,
/*alpha=*/1.0f,
query_row_major, /*lda=*/kDim,
gallery_.data(), /*ldb=*/kDim,
/*beta=*/0.0f,
host_sims_.data(), /*ldc=*/n_gallery_);
#else
for (int f = 0; f < n_faces; ++f) {
const float* q = query_row_major + static_cast<size_t>(f) * kDim;
float* out = host_sims_.data() + static_cast<size_t>(f) * n_gallery_;
@@ -69,6 +100,7 @@ public:
out[g] = acc;
}
}
#endif
return host_sims_.data();
}
src/backends/ort_provider.hpp
+12 -1
View File
@@ -23,12 +23,23 @@
enum class OrtProvider { CPU, CUDA, ROCm, TensorRT };
inline OrtProvider detect_ort_provider() {
// ORT returns these in its own preference order (TensorRT, CUDA, ..., CPU
// last), so the first recognised entry is the best available and the loop
// returns on it.
auto available = Ort::GetAvailableProviders();
for (const auto& p : available) {
// Only the TensorRT *EP* is a build-time opt-in — it needs the headers
// and the profile plumbing below. CUDA is not: it is a plain ORT
// provider, and gating its detection on the TRT flag (as this did) made
// the CUDA branch unreachable in every build that did not also ask for
// TensorRT. The symptom is silent rather than loud — inference simply
// runs on the CPU and everything still returns correct answers — which
// is why it survived: a 300-actor VR-012 grid cell took 76 s on the CPU
// with the GPU idle at 212 MiB.
#ifdef SAE_ORT_WITH_TRT_EP
if (p == "TensorrtExecutionProvider") return OrtProvider::TensorRT;
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
#endif
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
if (p == "ROCMExecutionProvider") return OrtProvider::ROCm;
}
return OrtProvider::CPU;
src/backends/trt_backend.cpp
+11 -2
View File
@@ -511,8 +511,17 @@ public:
}
std::lock_guard<std::mutex> lk(mu_);
context_->setInputShape(input_name_.c_str(),
nvinfer1::Dims5{1, kWindow, kFrameH, kFrameW, 3});
// TensorRT 10 removed the fixed-rank Dims5 helper (Dims2..Dims4 remain
// in NvInferLegacyDims.h). Build the rank-5 shape via the generic Dims,
// which works on both 8.x and 10.x.
nvinfer1::Dims shape{};
shape.nbDims = 5;
shape.d[0] = 1;
shape.d[1] = kWindow;
shape.d[2] = kFrameH;
shape.d[3] = kFrameW;
shape.d[4] = 3;
context_->setInputShape(input_name_.c_str(), shape);
check_cuda(cudaMemcpyAsync(d_input_, buf.data(), buf.size() * 4,
cudaMemcpyHostToDevice, stream_), "H2D input");
src/build_gallery.cpp
+26
View File
@@ -18,6 +18,8 @@
// --nms <f> NMS IoU threshold (default: 0.4)
#include "gallery/gallery_builder.hpp"
#include "gallery/gallery_report.hpp"
#include "gallery/gallery_calibration.hpp"
#include "gallery/gallery_store.hpp"
#include "config.hpp"
@@ -77,6 +79,30 @@ int main(int argc, char** argv) {
}
save_gallery(output_path, gallery);
std::cerr << "Gallery saved to: " << output_path << "\n";
/// TRACES: GR-003 | SR-001
// Fit the calibration here and persist what it learned. The matcher
// fits the same sigmoid at analysis time, but that is the wrong place
// to audit a gallery from: by then the answer is per-run and nobody is
// looking. Build time is when the gallery's quality is decided, and a
// gallery can be quietly bad — heavily overlapping intra/inter
// distributions, actors with no usable image — while looking fine.
std::vector<Embedding> flat;
std::vector<int> flat_actor;
for (int ai = 0; ai < static_cast<int>(gallery.actors.size()); ++ai)
for (const auto& e : gallery.actors[ai].embeddings) {
flat.push_back(e);
flat_actor.push_back(ai);
}
GalleryCalibrationStats stats;
GalleryCalibration cal = calibrate_gallery(flat, flat_actor, &stats);
const GalleryReport report =
build_gallery_report(gallery, cal, stats, nullptr, output_path);
const std::string report_path = gallery_report_path(output_path);
save_gallery_report(report_path, report);
std::cerr << "Gallery report saved to: " << report_path << "\n";
} catch (const std::exception& e) {
std::cerr << "Fatal: " << e.what() << "\n";
return 1;
src/config.hpp
+48 -16
View File
@@ -16,7 +16,13 @@ enum class Verbosity {
struct Config {
// ── Input ─────────────────────────────────────────────────────────────────
std::string movie_path;
std::string gallery_path; // gallery.json produced by build_gallery
std::string gallery_path;
// TRACES: IR-002 | SR-003
// "global" (matched against the whole library) or "limited" (this title's
// credited cast only). The strongest single quality signal when two
// manifests compete for the same cut: identical gallery_size can mean very
// different recall depending on which was used.
std::string gallery_scope{"global"}; // gallery.json produced by build_gallery
// ── Output ───────────────────────────────────────────────────────────────
std::string output_path; // annotations.json
@@ -35,11 +41,24 @@ struct Config {
// ── Detection (SCRFD-500MF via cv::dnn::Net) ──────────────────────────────
std::string detector_model;
std::string detector_engine; // optional path to pre-built TRT engine; bypasses ORT
int max_faces{10}; // pipeline cap: keep only the N largest faces
// TRACES: AR-003 | SR-002
// 0 = no cap, the default. A fixed cap discards the SMALLEST faces first,
// which are exactly the background cast X-Ray still credits with scene
// membership. Per-frame cost is contained by backpressure (AR-004) rather
// than by throwing work away. Set >0 only to bound a pathological source.
int max_faces{0};
float min_face_px{40.f}; // discard detections narrower or shorter than this
float detector_conf{0.5f};
float detector_nms{0.4f};
/// TRACES: GR-004 | SR-001
// Gallery ↔ embedder binding. A gallery built with a different model than the
// one loaded here is a hard error, always. This flag additionally promotes
// "cannot prove they match" (unstamped legacy gallery, or a name-only match
// because the ONNX could not be hashed) from a loud warning to a hard error.
// Also settable via SAE_REQUIRE_GALLERY_STAMP=1. Measurement runs want it on.
bool require_gallery_stamp{false}; // --require-gallery-stamp
// ── Recognition (ArcFace ONNX) ────────────────────────────────────────────
std::string arcface_model;
std::string arcface_engine; // optional path to a pre-built TRT engine; bypasses ORT
@@ -86,21 +105,25 @@ struct Config {
float dense_scale{1.0f}; // dense-mode frame downscale (1 = off)
// ── Face tracking (frame-to-frame) ───────────────────────────────────────
float track_alpha{0.4f}; // cost weight: 0=embedding only, 1=spatial only
/// TRACES: AR-007, AR-008, AR-024 | SR-002
// track_alpha is the *base* weight, used on ordinary frames. It is
// frame-dependent (AR-007): on is_cut / is_scene_boundary, and for any track
// that is no longer on screen, it drops to 0 (embedding only), because
// position carries no information across a viewpoint change or a gap.
float track_alpha{0.4f}; // base cost weight: 0=embedding only, 1=spatial only
float track_min_iou{0.1f}; // IoU below which spatial link alone is rejected
float track_max_embed_dist{0.7f}; // cosine dist above which embedding link alone is rejected
int track_max_frames_missing{5}; // expire track after N consecutive missed frames
// ── Cross-cut track re-association ────────────────────────────────────────
// A camera-angle change (Frame::is_cut) breaks spatial (IoU) continuity but
// not identity: the same people are usually still on screen from a new angle.
// Instead of destroying tracks on a cut, the tracker parks them in an
// inactive pool. A post-cut detection whose raw cosine similarity to a parked
// track's last-frame embedding is ≥ cut_revive_sim revives that track_id
// (identity continuity survives the cut); otherwise it starts a fresh track.
// Parked tracks that go unrevived for cut_inactive_max_frames are dropped.
float cut_revive_sim{0.50f}; // min raw cosine sim (last-frame emb) to revive across a cut
int cut_inactive_max_frames{5}; // drop a parked track after N frames without revival
// Minimum P(same person) for an association to be admissible on appearance
// alone. This replaces track_max_embed_dist (a raw cosine distance, AR-024).
// 0.5 is not a tuned constant: it is the decision boundary. Below it the pair
// is more likely two people than one, and no amount of IoU makes that a link
// worth asserting on identity grounds.
float track_assoc_min_prob{0.5f};
// How long a track that has gone off screen stays available for association
// before the registry reaps it and emits its presence claim (AR-013).
// Replaces track_max_frames_missing: a frame count silently changed meaning
// with sample_fps, and the same number had to be guessed twice (once for an
// ordinary miss, once for a cut). Seconds mean one thing at any sample rate.
double track_extinction_sec{5.0};
// ── Scene tracking ────────────────────────────────────────────────────────
// extinction_sec re-tuned by DE against X-Ray per-second presence, 4-film rep4
@@ -126,6 +149,15 @@ struct Config {
// opposite of the earlier assumption that it only helps restricted galleries.
bool expand_gallery{true}; // master switch
int expand_buffer_size{20}; // per-track diversity buffer capacity
// TRACES: AR-018, AR-024 | SR-005
// Banded admission for the per-subject store, in PROBABILITY space. An
// embedding joins only if P(same person) against something already stored
// lands inside [lo, hi]: above hi it is redundant, below lo it is evidence
// the track is not one person. Replaces expand_novelty_sim, a raw cosine.
// Working values pending VR-007; sweep both bounds, they fail in opposite
// directions.
float expand_band_lo{0.90f};
float expand_band_hi{0.95f};
float expand_novelty_sim{0.55f}; // promote only embeddings whose best sim to the
// actor's refs is below this (gallery-far / novel)
float expand_track_spread_max{0.60f}; // reject promotion if the retained buffer's
src/embed_faces.cpp
+154 -17
View File
@@ -25,11 +25,25 @@
// expected to contain exactly one subject). A warning is printed to stderr
// when more than one face is found.
//
// --all-faces emits every detection instead, which is what a caller analysing
// a frame rather than a gallery portrait needs:
// [ { "image": "frame.png",
// "faces": [ {"bbox": [...], "landmarks": [[x,y] x5],
// "confidence": 0.89, "embedding": [...]} , ... ] } ]
//
// --calibration <gallery> additionally emits the gallery's fitted Platt
// sigmoid, so a non-Python client can turn a similarity into P(match) with the
// same parameters the C++ matcher uses. Output becomes
// {"calibration": {...}, "images": [...]}. Clients must score through it:
// AR-024 requires the calibrated probability, never a bare cosine — a raw
// threshold means something different for every model, gallery and face size.
//
// This binary is intentionally a thin wrapper around the same ONNX models
// used by scene_analyze, so embeddings are guaranteed compatible.
#include "config.hpp"
#include "face_utils.hpp"
#include "gallery/gallery_store.hpp"
#include "inference/face_detector.hpp"
#include "inference/face_embedder.hpp"
@@ -100,6 +114,77 @@ static void save_debug(const std::string& dir,
cv::imwrite(dir + "/" + stem + "_aligned.jpg", aligned);
}
// ── Process one image, keeping every detection ────────────────────────────────
// The --all-faces path. Same detect → align → embed chain as process() below,
// but without the highest-confidence reduction: a frame legitimately contains
// several people, and dropping all but one is a gallery-portrait assumption.
// Faces that fail alignment are reported with a null embedding rather than
// silently dropped, so a caller can count what the detector found against what
// survived the ArcFace warp.
struct MultiFaceResult {
std::string image_path;
std::string error; // set only when the image itself failed
std::vector<FaceResult> faces;
};
static MultiFaceResult process_all(
const std::string& path,
const std::function<std::vector<DetectedFace>(const cv::Mat&)>& detect,
const std::function<Embedding(const cv::Mat&)>& embed_one,
int max_side,
const std::string& debug_dir = "") {
MultiFaceResult out;
out.image_path = path;
cv::Mat img = cv::imread(path);
if (img.empty()) {
out.error = "cannot read image";
return out;
}
if (max_side > 0) {
const int big = std::max(img.cols, img.rows);
if (big > max_side) {
const double s = static_cast<double>(max_side) / big;
cv::resize(img, img, {}, s, s, cv::INTER_AREA);
}
}
std::vector<DetectedFace> faces = detect(img);
if (faces.empty()) {
cv::Mat enhanced = enhance_for_retry(img);
faces = detect(enhanced);
if (!faces.empty())
img = enhanced;
}
if (faces.empty()) {
out.error = "no face detected";
return out;
}
for (const auto& face : faces) {
FaceResult r;
r.image_path = path;
r.confidence = face.confidence;
r.bbox[0] = face.bbox.x; r.bbox[1] = face.bbox.y;
r.bbox[2] = face.bbox.width; r.bbox[3] = face.bbox.height;
r.landmarks = face.landmarks;
cv::Mat crop = align_face(img, face.landmarks);
if (crop.empty()) {
r.error = "alignment failed";
} else {
r.ok = true;
r.embedding = embed_one(crop);
if (!debug_dir.empty())
save_debug(debug_dir, path, img, face, crop);
}
out.faces.push_back(std::move(r));
}
return out;
}
// ── Process one image ─────────────────────────────────────────────────────────
static FaceResult process(const std::string& path,
@@ -177,6 +262,8 @@ int main(int argc, char** argv) {
std::string arcface_model = kDefaultArcfaceModel;
std::string arcface_engine;
std::string debug_dir;
std::string calibration_gallery;
bool all_faces = false;
float conf = 0.5f, nms = 0.4f;
int max_side = 500;
std::vector<std::string> images;
@@ -190,13 +277,16 @@ int main(int argc, char** argv) {
else if (std::strcmp(argv[i], "--nms") == 0 && i+1 < argc) { nms = std::stof(argv[++i]); }
else if (std::strcmp(argv[i], "--save-debug") == 0 && i+1 < argc) { debug_dir = argv[++i]; }
else if (std::strcmp(argv[i], "--max-side") == 0 && i+1 < argc) { max_side = std::stoi(argv[++i]); }
else if (std::strcmp(argv[i], "--calibration")== 0 && i+1 < argc) { calibration_gallery = argv[++i]; }
else if (std::strcmp(argv[i], "--all-faces") == 0) { all_faces = true; }
else if (argv[i][0] != '-') { images.push_back(argv[i]); }
else { std::cerr << "[warn] unknown flag: " << argv[i] << "\n"; }
}
if (images.empty()) {
std::cerr << "Usage: embed_faces [--detector <path>] [--arcface <path>] "
"[--save-debug <dir>] [--max-side <N>] image1.jpg ...\n";
"[--save-debug <dir>] [--max-side <N>] [--all-faces] "
"[--calibration <gallery>] image1.jpg ...\n";
return 1;
}
@@ -217,31 +307,78 @@ int main(int argc, char** argv) {
std::function<Embedding(const cv::Mat&)> embed_one =
[&](const cv::Mat& c) { return embedder->embed_one(c); };
// One face's fields, shared by both output shapes.
auto face_json = [](const FaceResult& r) {
json f;
f["confidence"] = r.confidence;
f["bbox"] = {r.bbox[0], r.bbox[1], r.bbox[2], r.bbox[3]};
json lms = json::array();
for (const auto& pt : r.landmarks) lms.push_back({pt.x, pt.y});
f["landmarks"] = std::move(lms);
if (r.ok) f["embedding"] = std::vector<float>(r.embedding.begin(),
r.embedding.end());
else { f["embedding"] = nullptr; f["error"] = r.error; }
return f;
};
// Process images and build JSON output
json output = json::array();
json images_out = json::array();
for (const auto& path : images) {
std::cerr << "[embed_faces] " << path << "\n";
FaceResult res = process(path, detect, embed_one, max_side, debug_dir);
json entry;
entry["image"] = res.image_path;
if (res.ok) {
entry["embedding"] = std::vector<float>(res.embedding.begin(),
res.embedding.end());
entry["confidence"] = res.confidence;
entry["bbox"] = {res.bbox[0], res.bbox[1], res.bbox[2], res.bbox[3]};
json lms = json::array();
for (const auto& pt : res.landmarks) lms.push_back({pt.x, pt.y});
entry["landmarks"] = std::move(lms);
entry["image"] = path;
if (all_faces) {
MultiFaceResult res = process_all(path, detect, embed_one, max_side, debug_dir);
if (!res.error.empty()) {
entry["faces"] = json::array();
entry["error"] = res.error;
std::cerr << " [skip] " << res.error << "\n";
} else {
json faces = json::array();
for (const auto& f : res.faces) faces.push_back(face_json(f));
entry["faces"] = std::move(faces);
}
} else {
entry["embedding"] = nullptr;
entry["error"] = res.error;
std::cerr << " [skip] " << res.error << "\n";
FaceResult res = process(path, detect, embed_one, max_side, debug_dir);
if (res.ok) {
entry.merge_patch(face_json(res));
} else {
entry["embedding"] = nullptr;
entry["error"] = res.error;
std::cerr << " [skip] " << res.error << "\n";
}
}
output.push_back(std::move(entry));
images_out.push_back(std::move(entry));
}
std::cout << output.dump() << "\n";
// Without --calibration the output stays a bare array, unchanged, so
// existing callers (build_gallery, fetch_missing_actors) are unaffected.
if (calibration_gallery.empty()) {
std::cout << images_out.dump() << "\n";
return 0;
}
ActorGallery gallery = load_gallery(calibration_gallery);
if (!gallery.calib_valid)
std::cerr << "[warn] " << calibration_gallery
<< " carries no valid calibration; a client cannot convert a "
"similarity to a probability from it (AR-024)\n";
json out;
out["calibration"] = {
{"a", gallery.calib_a},
{"b", gallery.calib_b},
{"valid", gallery.calib_valid},
{"form", "P(match) = 1/(1+exp(-(a*similarity + b + log_prior_odds)))"},
{"note", "Score through this. AR-024: a bare cosine threshold means "
"something different for every model, gallery and face size. "
"log_prior_odds = log(p0/(1-p0)) for a base-rate prior p0; use "
"0 for association (are these two faces one person)."},
};
out["images"] = std::move(images_out);
std::cout << out.dump() << "\n";
return 0;
}
src/evidence_discount.hpp
+131
View File
@@ -0,0 +1,131 @@
#pragma once
/// TRACES: AR-024, AR-025 | SR-002
///
/// EvidenceDiscounter — how much a single observation is allowed to move a
/// track's belief.
///
/// **The independence problem.** Per-frame identity evidence is accumulated as
/// log-odds along a track (AR-025), which is only valid for *independent*
/// observations. Consecutive frames of one track are nothing of the kind: near
/// identical pose, lighting and expression. Treating them as independent drives
/// the posterior to certainty on what is effectively one measurement — thirty
/// frames of the same face at the same angle is not thirty pieces of evidence.
///
/// The mitigation is to weight each observation by how much it *adds*: a view
/// the track has already contributed is discounted toward zero, a genuinely new
/// pose counts in full. This reuses the same judgement the diversity buffer
/// makes for gallery expansion (AR-019) — which embeddings on a track are
/// mutually distinct — rather than inventing a second notion of novelty.
///
/// Owned by TrackRegistry rather than left to callers. A caller that forgot to
/// discount, or applied it twice, would silently produce confident wrong
/// answers, and the registry is the one place where all evidence converges.
///
/// **Similarity enters as a calibrated probability, never a raw cosine**
/// (AR-024): "is this the same view" is a decision, and a bare cosine threshold
/// means something different for every model and every face size.
#include "types.hpp"
#include <algorithm>
#include <cmath>
#include <functional>
#include <vector>
class EvidenceDiscounter {
public:
/// cosine similarity → P(same view). Supplied by the caller so the
/// calibration fitted for the active embedder is used (AR-023/AR-024).
using Calibrate = std::function<float(float)>;
struct Config {
int max_views{8}; ///< distinct views remembered per track
float admit_below{0.6f}; ///< P(same view) under this ⇒ a new view
/// Ceiling on the correlation between two observations of one track.
///
/// This is what bounds the accumulation. `n_eff = n / (1 + (n-1)·rho)`
/// tends to `1/rho` as `n` grows, so `rho_max` sets how much a single
/// repeated view can ever be worth: 0.5 caps it at two observations,
/// no matter how long the shot runs.
///
/// 0.5 caps a repeated view at two independent observations' worth,
/// which is what lets a track the matcher accepts on frame after frame
/// actually become owned. Higher values starve ownership; the sweep
/// (VR-007) decides where it belongs.
///
/// It is capped below 1 deliberately. P(same view) near 1 says the two
/// crops look alike; it does not say the second carries no information.
/// A fresh frame is a fresh detection, a fresh alignment and a fresh
/// noise realisation, so a little independent evidence survives even a
/// perfectly held pose. Setting this to 1 recovers the original bug —
/// belief frozen after the first frame.
float rho_max{0.5f};
};
// Two constructors rather than a defaulted argument: `Config{}` as a default
// argument would reference Config's own member initializers before the
// enclosing class is complete, which is ill-formed.
explicit EvidenceDiscounter(Calibrate cal)
: cal_(std::move(cal)), cfg_() {}
EvidenceDiscounter(Calibrate cal, Config cfg)
: cal_(std::move(cal)), cfg_(cfg) {}
/// The marginal evidence one observation adds, in units of independent
/// observations.
///
/// Each frame is a Bayesian update, so confidence must keep growing — but
/// correlated observations must grow it less, and must not grow it without
/// bound. The standard treatment is **effective sample size**:
///
/// n_eff(n) = n / (1 + (n-1)·rho)
///
/// and this returns `n_eff(n) - n_eff(n-1)`, the gain from *this* frame.
/// The shape is right at both ends: with rho = 0 every frame counts fully
/// and the belief accumulates linearly, while as rho rises the series
/// converges on `1/rho` and a held pose stops adding no matter how long it
/// is held.
///
/// The two failure modes it sits between are both real and both were hit:
/// a weight of 0 for repeats froze the belief after one frame, so a track
/// recognised on 318 frames was owned on none; a constant floor grew it
/// linearly forever, so a long shot could out-argue genuinely varied
/// evidence purely by lasting longer.
///
/// `rho` is estimated from P(same view) against the closest stored view,
/// capped by `rho_max`. The first observation has nothing to be redundant
/// with and counts in full.
/// `n_seen` is the count of observations already folded into THIS track.
/// It is a parameter rather than discounter state because one discounter
/// serves every track: holding the count internally would pool unrelated
/// tracks into one effective sample, so a busy film would silently discount
/// each track by how many others happened to be on screen.
float weight(std::vector<Embedding>& views, int n_seen, const Embedding& e) const {
if (views.empty()) {
views.push_back(e);
return 1.0f;
}
float p_same = 0.0f;
for (const auto& v : views)
p_same = std::max(p_same, cal_(cosine_similarity(v, e)));
const float rho = std::min(cfg_.rho_max, std::max(0.0f, p_same));
const float n_prev = static_cast<float>(std::max(1, n_seen));
const float n_now = n_prev + 1.0f;
auto n_eff = [rho](float n) { return n / (1.0f + (n - 1.0f) * rho); };
const float w = std::max(0.0f, n_eff(n_now) - n_eff(n_prev));
if (p_same < cfg_.admit_below &&
static_cast<int>(views.size()) < cfg_.max_views) {
views.push_back(e);
}
return w;
}
private:
Calibrate cal_;
Config cfg_;
};
src/face_embedder_engine.hpp
+31 -5
View File
@@ -32,16 +32,23 @@ struct FaceEmbedResult {
class FaceEmbedderEngine {
public:
// detector_engine/arcface_engine are optional paths to pre-built TensorRT
// engines. They are required when built with SAE_INFERENCE_BACKEND=TRT
// (which cannot load .onnx directly) and ignored by the ORT backend.
FaceEmbedderEngine(const std::string& detector_model,
const std::string& arcface_model,
float conf = 0.5f, float nms = 0.4f, int max_side = 500)
float conf = 0.5f, float nms = 0.4f, int max_side = 500,
const std::string& detector_engine = "",
const std::string& arcface_engine = "")
: max_side_(max_side)
{
Config cfg;
cfg.detector_model = detector_model;
cfg.arcface_model = arcface_model;
cfg.detector_conf = conf;
cfg.detector_nms = nms;
cfg.detector_model = detector_model;
cfg.arcface_model = arcface_model;
cfg.detector_engine = detector_engine;
cfg.arcface_engine = arcface_engine;
cfg.detector_conf = conf;
cfg.detector_nms = nms;
detector_ = make_face_detector(cfg);
embedder_ = make_face_embedder(cfg);
}
@@ -105,6 +112,25 @@ public:
return res;
}
// ── Stage accessors ──────────────────────────────────────────────────────
// embed_mat() above is the whole detect→align→embed chain, which is the
// right entry point for embedding a gallery image. Studies that need to
// intervene between the stages — swapping the landmark source, degrading a
// crop before it reaches the embedder — drive these instead, so they still
// exercise the shipped detector, alignment and embedder rather than a
// re-implementation of them.
std::vector<DetectedFace> detect(const cv::Mat& img) { return detector_->detect(img); }
Embedding embed_crop(const cv::Mat& crop) { return embedder_->embed_one(crop); }
// Batched form. A study embedding thousands of crops one at a time pays the
// per-call overhead thousands of times over; the backend already batches.
std::vector<Embedding> embed_crops(const std::vector<cv::Mat>& crops) {
return embedder_->embed(crops);
}
int max_batch() const { return embedder_->max_batch(); }
private:
std::unique_ptr<IFaceDetector> detector_;
std::unique_ptr<IFaceEmbedder> embedder_;
src/face_utils.hpp
+130 -11
View File
@@ -1,25 +1,144 @@
#pragma once
/// TRACES: AR-005, AR-030 | SR-002
#include "types.hpp"
#include <opencv2/calib3d.hpp>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <cmath>
// ── align_face ────────────────────────────────────────────────────────────────
// Produces a 112×112 BGR crop using the ArcFace 5-point similarity transform.
// Returns an empty Mat if the affine fit fails (degenerate detection).
inline cv::Mat align_face(const cv::Mat& img,
const std::array<cv::Point2f, 5>& landmarks) {
std::vector<cv::Point2f> src(landmarks.begin(), landmarks.end());
std::vector<cv::Point2f> dst(5);
// ── umeyama_similarity ────────────────────────────────────────────────────────
// Closed-form least-squares similarity transform (rotation + uniform scale +
// translation, 4 DoF) mapping `src` onto `dst`, by Umeyama's solution.
//
// This is the estimator InsightFace aligns with — skimage's SimilarityTransform
// is `_umeyama(..., estimate_scale=True)` — and therefore the one that produced
// the crops ArcFace and LVFace were *trained* on. The canonical warp is part of
// the input distribution, not a free implementation choice (AR-011).
//
// Deliberately **not** `cv::estimateAffinePartial2D(..., cv::RANSAC)`:
//
// - A robust estimator earns a small residual by discarding the points that
// disagree with the model. On a turned face those are precisely the
// foreshortened landmarks — the pose signal AR-030 exists to measure. RANSAC
// would suppress exactly the quantity we want to read.
// - With five points and a two-point minimal sample there is almost no
// redundancy, so it cannot distinguish a mis-detected landmark from honest
// out-of-plane rotation. The robustness is nominal.
// - It is RNG-driven (`cv::theRNG()` is thread-local); this is exact, so
// replay determinism stops depending on thread scheduling.
//
// Returns an empty Mat when the source points are degenerate (all coincident).
inline cv::Mat umeyama_similarity(const std::array<cv::Point2f, 5>& src,
const std::array<cv::Point2f, 5>& dst) {
constexpr int N = 5;
double mu_sx = 0, mu_sy = 0, mu_dx = 0, mu_dy = 0;
for (int i = 0; i < N; ++i) {
mu_sx += src[i].x; mu_sy += src[i].y;
mu_dx += dst[i].x; mu_dy += dst[i].y;
}
mu_sx /= N; mu_sy /= N; mu_dx /= N; mu_dy /= N;
// var_src and the cross-covariance Σ = (1/N) Σ (d - μ_d)(s - μ_s)ᵀ
double var_s = 0;
cv::Matx22d sigma = cv::Matx22d::zeros();
for (int i = 0; i < N; ++i) {
const double sx = src[i].x - mu_sx, sy = src[i].y - mu_sy;
const double dx = dst[i].x - mu_dx, dy = dst[i].y - mu_dy;
var_s += sx * sx + sy * sy;
sigma(0, 0) += dx * sx; sigma(0, 1) += dx * sy;
sigma(1, 0) += dy * sx; sigma(1, 1) += dy * sy;
}
var_s /= N;
sigma *= 1.0 / N;
if (var_s < 1e-12) return {}; // every source point coincides — no scale
cv::Mat w, u, vt;
cv::SVD::compute(cv::Mat(sigma), w, u, vt, cv::SVD::FULL_UV);
const cv::Matx22d U (u.at<double>(0,0), u.at<double>(0,1),
u.at<double>(1,0), u.at<double>(1,1));
const cv::Matx22d Vt(vt.at<double>(0,0), vt.at<double>(0,1),
vt.at<double>(1,0), vt.at<double>(1,1));
// A similarity may rotate but never mirror: if the fit came out
// orientation-reversing, flip the least-significant singular direction.
cv::Matx22d S = cv::Matx22d::eye();
if (cv::determinant(U) * cv::determinant(Vt) < 0) S(1, 1) = -1;
const cv::Matx22d R = U * S * Vt;
const double c = (w.at<double>(0) * S(0,0) + w.at<double>(1) * S(1,1)) / var_s;
cv::Mat M(2, 3, CV_64F);
M.at<double>(0,0) = c * R(0,0); M.at<double>(0,1) = c * R(0,1);
M.at<double>(1,0) = c * R(1,0); M.at<double>(1,1) = c * R(1,1);
M.at<double>(0,2) = mu_dx - c * (R(0,0) * mu_sx + R(0,1) * mu_sy);
M.at<double>(1,2) = mu_dy - c * (R(1,0) * mu_sx + R(1,1) * mu_sy);
return M;
}
// ── Alignment ─────────────────────────────────────────────────────────────────
// The 5-point fit, plus what it could not explain.
//
// `residual` is the RMS landmark error in **canonical 112×112 pixels** after the
// best similarity fit. Two properties make it the AR-030 visibility measure:
//
// - The similarity transform absorbs rotation, uniform scale and translation
// exactly, so the residual is by construction the part of the deformation a
// similarity *cannot* explain — out-of-plane rotation and foreshortening,
// plus landmark noise. In-plane roll contributes nothing. The "roll must not
// read as yaw" failure is excluded structurally rather than by tuning.
// - The destination frame is fixed, so a 40 px face and a 400 px face are both
// measured in the same canonical space. The measure cannot silently
// re-express face size (already AR-002's job) the way a raw-pixel one would.
//
// It also responds to occlusion and to plainly broken landmark sets, which a
// yaw-angle estimator by construction does not.
struct Alignment {
cv::Mat M; ///< 2×3 CV_64F: source pixels → canonical 112×112
float residual{0.f}; ///< RMS canonical-pixel error; 0 ⇒ a perfect fit
bool ok{false}; ///< false ⇒ degenerate landmarks, no transform
};
/// Fit the canonical ArcFace template to `landmarks` and report the misfit.
inline Alignment estimate_alignment(const std::array<cv::Point2f, 5>& landmarks) {
std::array<cv::Point2f, 5> dst;
for (int i = 0; i < 5; ++i) dst[i] = {kArcFaceRef[i][0], kArcFaceRef[i][1]};
cv::Mat M = cv::estimateAffinePartial2D(src, dst, cv::noArray(), cv::RANSAC, 3.0);
if (M.empty()) return {};
Alignment a;
a.M = umeyama_similarity(landmarks, dst);
if (a.M.empty()) return a;
double sq = 0;
for (int i = 0; i < 5; ++i) {
const double x = a.M.at<double>(0,0) * landmarks[i].x
+ a.M.at<double>(0,1) * landmarks[i].y + a.M.at<double>(0,2);
const double y = a.M.at<double>(1,0) * landmarks[i].x
+ a.M.at<double>(1,1) * landmarks[i].y + a.M.at<double>(1,2);
const double ex = x - dst[i].x, ey = y - dst[i].y;
sq += ex * ex + ey * ey;
}
a.residual = static_cast<float>(std::sqrt(sq / 5.0));
a.ok = true;
return a;
}
// ── align_face ────────────────────────────────────────────────────────────────
// Produces a 112×112 BGR crop using the ArcFace 5-point similarity transform.
// Returns an empty Mat if the fit fails (degenerate detection). When
// `residual_out` is non-null it receives the AR-030 misfit for the same fit —
// free, since the transform has already been computed.
inline cv::Mat align_face(const cv::Mat& img,
const std::array<cv::Point2f, 5>& landmarks,
float* residual_out = nullptr) {
const Alignment a = estimate_alignment(landmarks);
if (!a.ok) return {};
if (residual_out) *residual_out = a.residual;
cv::Mat crop;
cv::warpAffine(img, crop, M, {112, 112},
cv::warpAffine(img, crop, a.M, {112, 112},
cv::INTER_LINEAR, cv::BORDER_CONSTANT, {0, 0, 0});
return crop;
}
src/gallery/embedder_stamp.cpp
+326
View File
@@ -0,0 +1,326 @@
/// TRACES: GR-004 | SR-001
#include "embedder_stamp.hpp"
#include "types.hpp"
#include <algorithm>
#include <cstdlib>
#include <cstring>
#include <filesystem>
#include <fstream>
#include <iostream>
#include <map>
#include <mutex>
#include <sstream>
#include <stdexcept>
#include <vector>
namespace fs = std::filesystem;
// ── SHA-256 (FIPS 180-4) ──────────────────────────────────────────────────────
// Self-contained rather than pulled from OpenSSL: the gallery library already
// links OpenCV, HDF5, FFmpeg and a GPU backend, and the unit tests deliberately
// link none of those crypto stacks. ~80 lines of table-driven code is cheaper
// than another find_package that CI has to satisfy on an Intel N100.
namespace {
struct Sha256 {
uint32_t h[8] = {0x6a09e667u, 0xbb67ae85u, 0x3c6ef372u, 0xa54ff53au,
0x510e527fu, 0x9b05688cu, 0x1f83d9abu, 0x5be0cd19u};
uint64_t len = 0;
uint8_t buf[64]{};
size_t buf_n = 0;
static uint32_t ror(uint32_t x, int n) { return (x >> n) | (x << (32 - n)); }
void block(const uint8_t* p) {
static const uint32_t k[64] = {
0x428a2f98u,0x71374491u,0xb5c0fbcfu,0xe9b5dba5u,0x3956c25bu,0x59f111f1u,
0x923f82a4u,0xab1c5ed5u,0xd807aa98u,0x12835b01u,0x243185beu,0x550c7dc3u,
0x72be5d74u,0x80deb1feu,0x9bdc06a7u,0xc19bf174u,0xe49b69c1u,0xefbe4786u,
0x0fc19dc6u,0x240ca1ccu,0x2de92c6fu,0x4a7484aau,0x5cb0a9dcu,0x76f988dau,
0x983e5152u,0xa831c66du,0xb00327c8u,0xbf597fc7u,0xc6e00bf3u,0xd5a79147u,
0x06ca6351u,0x14292967u,0x27b70a85u,0x2e1b2138u,0x4d2c6dfcu,0x53380d13u,
0x650a7354u,0x766a0abbu,0x81c2c92eu,0x92722c85u,0xa2bfe8a1u,0xa81a664bu,
0xc24b8b70u,0xc76c51a3u,0xd192e819u,0xd6990624u,0xf40e3585u,0x106aa070u,
0x19a4c116u,0x1e376c08u,0x2748774cu,0x34b0bcb5u,0x391c0cb3u,0x4ed8aa4au,
0x5b9cca4fu,0x682e6ff3u,0x748f82eeu,0x78a5636fu,0x84c87814u,0x8cc70208u,
0x90befffau,0xa4506cebu,0xbef9a3f7u,0xc67178f2u};
uint32_t w[64];
for (int i = 0; i < 16; ++i)
w[i] = (uint32_t(p[i * 4]) << 24) | (uint32_t(p[i * 4 + 1]) << 16) |
(uint32_t(p[i * 4 + 2]) << 8) | uint32_t(p[i * 4 + 3]);
for (int i = 16; i < 64; ++i) {
uint32_t s0 = ror(w[i - 15], 7) ^ ror(w[i - 15], 18) ^ (w[i - 15] >> 3);
uint32_t s1 = ror(w[i - 2], 17) ^ ror(w[i - 2], 19) ^ (w[i - 2] >> 10);
w[i] = w[i - 16] + s0 + w[i - 7] + s1;
}
uint32_t a = h[0], b = h[1], c = h[2], d = h[3];
uint32_t e = h[4], f = h[5], g = h[6], hh = h[7];
for (int i = 0; i < 64; ++i) {
uint32_t S1 = ror(e, 6) ^ ror(e, 11) ^ ror(e, 25);
uint32_t ch = (e & f) ^ (~e & g);
uint32_t t1 = hh + S1 + ch + k[i] + w[i];
uint32_t S0 = ror(a, 2) ^ ror(a, 13) ^ ror(a, 22);
uint32_t mj = (a & b) ^ (a & c) ^ (b & c);
uint32_t t2 = S0 + mj;
hh = g; g = f; f = e; e = d + t1;
d = c; c = b; b = a; a = t1 + t2;
}
h[0] += a; h[1] += b; h[2] += c; h[3] += d;
h[4] += e; h[5] += f; h[6] += g; h[7] += hh;
}
void update(const uint8_t* p, size_t n) {
len += n;
while (n) {
size_t take = std::min(n, size_t(64) - buf_n);
std::memcpy(buf + buf_n, p, take);
buf_n += take; p += take; n -= take;
if (buf_n == 64) { block(buf); buf_n = 0; }
}
}
std::string hex() {
uint64_t bits = len * 8;
uint8_t pad = 0x80;
update(&pad, 1);
uint8_t zero = 0;
while (buf_n != 56) update(&zero, 1);
uint8_t tail[8];
for (int i = 0; i < 8; ++i) tail[i] = uint8_t(bits >> (56 - i * 8));
// update() would re-count these into len, but len is already frozen in bits.
std::memcpy(buf + buf_n, tail, 8);
block(buf);
buf_n = 0;
static const char* d = "0123456789abcdef";
std::string out;
out.reserve(64);
for (int i = 0; i < 8; ++i)
for (int s = 28; s >= 0; s -= 4)
out += d[(h[i] >> s) & 0xF];
return out;
}
};
// (path, mtime, size) → digest. Hashing a 250 MB ONNX is cheap but not free, and
// the optimizer constructs many networks in one process against the same model.
std::mutex g_hash_mu;
std::map<std::string, std::string> g_hash_cache;
std::string short_hash(const std::string& hex) {
return hex.size() > 12 ? hex.substr(0, 12) + "…" : hex;
}
} // namespace
std::string sha256_hex(const std::string& bytes) {
Sha256 s;
s.update(reinterpret_cast<const uint8_t*>(bytes.data()), bytes.size());
return s.hex();
}
std::string sha256_file_hex(const std::string& path) {
if (path.empty()) return "";
std::error_code ec;
auto size = fs::file_size(path, ec);
if (ec) return "";
auto mtime = fs::last_write_time(path, ec);
if (ec) return "";
std::ostringstream key;
key << path << '|' << size << '|'
<< mtime.time_since_epoch().count();
{
std::lock_guard<std::mutex> lk(g_hash_mu);
auto it = g_hash_cache.find(key.str());
if (it != g_hash_cache.end()) return it->second;
}
std::ifstream f(path, std::ios::binary);
if (!f) return "";
Sha256 s;
std::vector<char> chunk(1 << 20);
while (f) {
f.read(chunk.data(), static_cast<std::streamsize>(chunk.size()));
std::streamsize got = f.gcount();
if (got > 0) s.update(reinterpret_cast<const uint8_t*>(chunk.data()),
static_cast<size_t>(got));
}
std::string hex = s.hex();
std::lock_guard<std::mutex> lk(g_hash_mu);
g_hash_cache[key.str()] = hex;
return hex;
}
// ── EmbedderStamp ─────────────────────────────────────────────────────────────
std::string EmbedderStamp::describe() const {
std::string name = model_name.empty() ? "<unnamed model>" : model_name;
if (model_sha256.empty())
return name + " (sha256 unavailable)";
return name + " (sha256 " + short_hash(model_sha256) + ")";
}
EmbedderStamp make_embedder_stamp(const std::string& model_path) {
EmbedderStamp s;
if (model_path.empty()) return s;
s.model_name = fs::path(model_path).filename().string();
s.model_sha256 = sha256_file_hex(model_path);
if (s.model_sha256.empty())
std::cerr << "[gallery] cannot hash embedder model " << model_path
<< " — model binding falls back to filename only (GR-004)\n";
return s;
}
bool require_gallery_stamp_from_env() {
const char* v = std::getenv("SAE_REQUIRE_GALLERY_STAMP");
return v && *v && std::strcmp(v, "0") != 0;
}
// ── Comparison ────────────────────────────────────────────────────────────────
StampCheck compare_embedder_stamps(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc,
const std::string& embedder_desc) {
StampCheck out;
std::ostringstream m;
// The gallery predates GR-004 (or was written by a tool that does not stamp).
if (built_with.empty()) {
out.verdict = StampVerdict::unstamped;
m << "gallery '" << gallery_desc << "' carries no embedder stamp (GR-004).\n"
<< " gallery was built with : UNKNOWN — this file predates model binding\n"
<< " embedder now loaded : " << loading_with.describe()
<< " [" << embedder_desc << "]\n"
<< " If these are not the same model every similarity from this run is\n"
<< " meaningless but will look plausible. Rebuild or re-stamp the gallery\n"
<< " (scripts/stamp_gallery.py), or run with SAE_REQUIRE_GALLERY_STAMP=1 to\n"
<< " make this a hard error.";
out.message = m.str();
return out;
}
// Gallery is stamped but we cannot say what is about to embed.
if (loading_with.empty()) {
out.verdict = StampVerdict::unknown_embedder;
m << "cannot identify the embedder being used against gallery '"
<< gallery_desc << "' (GR-004).\n"
<< " gallery was built with : " << built_with.describe() << "\n"
<< " embedder now loaded : UNKNOWN [" << embedder_desc << "]\n"
<< " The binding cannot be checked, so it is not being checked.";
out.message = m.str();
return out;
}
const bool have_both_hashes =
!built_with.model_sha256.empty() && !loading_with.model_sha256.empty();
// Embedding width disagreeing is a mismatch on its own terms — different
// spaces entirely, and it will not even be caught by a cosine that "looks fine".
if (built_with.embed_dim != loading_with.embed_dim) {
out.verdict = StampVerdict::mismatch;
m << "gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
<< " gallery was built with : " << built_with.describe()
<< ", dim=" << built_with.embed_dim << " [" << gallery_desc << "]\n"
<< " embedder now loaded : " << loading_with.describe()
<< ", dim=" << loading_with.embed_dim << " [" << embedder_desc << "]\n"
<< " Embedding dimensions differ; these are not the same space.";
out.message = m.str();
return out;
}
if (have_both_hashes) {
if (built_with.model_sha256 == loading_with.model_sha256) {
out.verdict = StampVerdict::match;
m << "embedder binding verified: " << built_with.describe();
if (built_with.model_name != loading_with.model_name)
m << " (gallery recorded it as '" << built_with.model_name
<< "', loaded from '" << loading_with.model_name
<< "' — same bytes, renamed file)";
out.message = m.str();
return out;
}
out.verdict = StampVerdict::mismatch;
m << "gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
<< " gallery was built with : " << built_with.model_name
<< " sha256=" << built_with.model_sha256 << "\n"
<< " [" << gallery_desc << "]\n"
<< " embedder now loaded : " << loading_with.model_name
<< " sha256=" << loading_with.model_sha256 << "\n"
<< " [" << embedder_desc << "]\n"
<< " Cosine similarities between embeddings from different models are\n"
<< " meaningless but look plausible. Rebuild the gallery with the loaded\n"
<< " model, or point the embedder at the model the gallery was built with.";
out.message = m.str();
return out;
}
// One side has no hash (e.g. a TRT deployment with the .onnx absent). Names
// are all we have; agreeing on them is evidence, not proof.
if (!built_with.model_name.empty() &&
built_with.model_name == loading_with.model_name) {
out.verdict = StampVerdict::weak_match;
m << "embedder binding UNPROVEN for gallery '" << gallery_desc << "' (GR-004).\n"
<< " gallery was built with : " << built_with.describe() << "\n"
<< " embedder now loaded : " << loading_with.describe()
<< " [" << embedder_desc << "]\n"
<< " Filenames agree but at least one SHA-256 is unavailable, so an\n"
<< " in-place re-export under the same name would not be detected.";
out.message = m.str();
return out;
}
out.verdict = StampVerdict::mismatch;
m << "gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
<< " gallery was built with : " << built_with.describe()
<< " [" << gallery_desc << "]\n"
<< " embedder now loaded : " << loading_with.describe()
<< " [" << embedder_desc << "]\n"
<< " Cosine similarities between embeddings from different models are\n"
<< " meaningless but look plausible. Rebuild the gallery with the loaded\n"
<< " model, or point the embedder at the model the gallery was built with.";
out.message = m.str();
return out;
}
void enforce_embedder_stamp(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc,
const std::string& embedder_desc,
bool require_stamp) {
const bool strict = require_stamp || require_gallery_stamp_from_env();
StampCheck chk = compare_embedder_stamps(built_with, loading_with,
gallery_desc, embedder_desc);
if (chk.fatal(strict)) {
if (chk.verdict != StampVerdict::mismatch)
throw std::runtime_error(chk.message +
"\n (fatal because SAE_REQUIRE_GALLERY_STAMP / --require-gallery-stamp is set)");
throw std::runtime_error(chk.message);
}
if (chk.verdict == StampVerdict::match) {
std::cerr << "[gallery] " << chk.message << "\n";
} else {
std::cerr << "\n[gallery] ***** WARNING (GR-004) *****\n"
<< chk.message << "\n"
<< "[gallery] ****************************\n\n";
}
}
void verify_gallery_embedder(const ActorGallery& gallery,
const std::string& gallery_path,
const std::string& arcface_model_path,
bool require_stamp) {
enforce_embedder_stamp(gallery.embedder,
make_embedder_stamp(arcface_model_path),
gallery_path,
arcface_model_path.empty() ? "no --arcface given"
: arcface_model_path,
require_stamp);
}
src/gallery/embedder_stamp.hpp
+115
View File
@@ -0,0 +1,115 @@
#pragma once
/// TRACES: GR-004 | SR-001
//
// Gallery ↔ embedder binding.
//
// A gallery is only valid for the embedder that built it. Cosine similarities
// between embeddings from two different models are meaningless but *look*
// plausible — nothing crashes, nothing is obviously wrong, and every number
// measured downstream is quietly garbage. So the embedder's identity is stamped
// into the gallery at build time and checked by every consumer at load time.
//
// ── What identifies an embedder ───────────────────────────────────────────────
// Two fields, carried together:
//
// model_name basename of the model file, e.g. "LVFace-B_Glint360K.onnx"
// model_sha256 hex SHA-256 of that file's bytes
//
// The hash is what *decides*; the name is what a human *reads*. Neither alone is
// enough:
//
// • A name alone is a promise, not a fact. Models get re-exported, re-quantised
// and overwritten in place under an unchanged filename — which is precisely
// the case where the weights differ and nothing else does. A name-only stamp
// is blind to exactly the failure it exists to catch.
// • A hash alone is correct but unreadable: "expected 3f2a… got 9c1b…" tells an
// operator nothing about what to do next.
//
// SHA-256 over the file bytes is derived from the artefact rather than asserted
// about it, is stable across machines and filesystems, and needs no registry to
// be kept up to date. Cost is ~0.1 s for a 250 MB ONNX, paid once per process
// (results are memoised on path+mtime+size), which is noise next to model load.
//
// ── Degraded and legacy cases ─────────────────────────────────────────────────
// A TRT-backend deployment may run from a prebuilt .engine with the source .onnx
// absent, so the hash cannot be computed. Then the name is compared alone and the
// result is reported as a *weak* match — believed, not proven.
//
// Galleries built before GR-004 carry no stamp at all. They warn loudly rather
// than fail, because the state is unknown rather than known-bad, and because
// hard-failing every pre-existing gallery would make the check something people
// route around rather than trust. Set require_stamp (or SAE_REQUIRE_GALLERY_STAMP=1)
// to promote "unknown" to a hard error — that is the mode measurement work runs in.
//
// A *mismatch* is always fatal, in every mode, with no bypass.
#include <cstdint>
#include <string>
struct EmbedderStamp {
std::string model_name; // basename of the model file
std::string model_sha256; // lowercase hex SHA-256 of the file's bytes ("" = unavailable)
int32_t embed_dim{512};
bool empty() const { return model_name.empty() && model_sha256.empty(); }
// "LVFace-B_Glint360K.onnx (sha256 3f2a1c4d…)" — for error messages.
std::string describe() const;
};
// Identify the model at `model_path`. Missing/unreadable file → name filled from
// the path, hash left empty (the weak-match path). Empty path → empty stamp.
EmbedderStamp make_embedder_stamp(const std::string& model_path);
enum class StampVerdict {
match, // hashes agree — binding proven
weak_match, // names agree, no hash on one side — believed, unproven
unstamped, // gallery predates GR-004 / was written without a stamp
unknown_embedder, // gallery is stamped but the loaded embedder can't be identified
mismatch, // proven different models — always fatal
};
struct StampCheck {
StampVerdict verdict{StampVerdict::match};
std::string message; // human-readable, names BOTH sides
// A mismatch is fatal unconditionally. The three "cannot prove it" verdicts
// are fatal only in strict mode.
bool fatal(bool require_stamp) const {
return verdict == StampVerdict::mismatch ||
(require_stamp && verdict != StampVerdict::match);
}
};
// Pure comparison — no file I/O, no model loading. This is the unit under test.
// `gallery_desc`/`embedder_desc` are only used to make the message locatable
// (a gallery path, a dump path, "the embedder being loaded", …).
StampCheck compare_embedder_stamps(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc = "gallery",
const std::string& embedder_desc = "embedder");
// Apply the comparison: throw std::runtime_error on a fatal verdict, otherwise
// log to stderr. `require_stamp` is OR-ed with SAE_REQUIRE_GALLERY_STAMP.
void enforce_embedder_stamp(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc,
const std::string& embedder_desc,
bool require_stamp);
// Convenience for the common consumer shape: "I loaded this gallery and I am
// about to embed with this model file." Hashes the model, then enforces.
struct ActorGallery;
void verify_gallery_embedder(const ActorGallery& gallery,
const std::string& gallery_path,
const std::string& arcface_model_path,
bool require_stamp);
// SAE_REQUIRE_GALLERY_STAMP=1 → treat an unprovable binding as fatal.
bool require_gallery_stamp_from_env();
// Lowercase hex SHA-256. Exposed so a test can pin the digest against the
// published vectors, which is what guarantees the C++ and Python (hashlib)
// stamps of the same file agree.
std::string sha256_hex(const std::string& bytes);
std::string sha256_file_hex(const std::string& path); // "" if unreadable
src/gallery/gallery_builder.cpp
+7
View File
@@ -1,5 +1,6 @@
#include "gallery_builder.hpp"
#include "config.hpp"
#include "embedder_stamp.hpp"
#include "face_utils.hpp"
#include "inference/face_detector.hpp"
#include "inference/face_embedder.hpp"
@@ -41,6 +42,12 @@ ActorGallery build_gallery(const BuildConfig& cfg) {
ActorGallery gallery;
/// TRACES: GR-004 | SR-001
// Stamp before the first embedding exists, so there is no window in which a
// gallery holds vectors without recording what produced them.
gallery.embedder = make_embedder_stamp(cfg.arcface_model);
std::cerr << "[build_gallery] embedder: " << gallery.embedder.describe() << "\n";
for (const auto& actor_dir : fs::directory_iterator(cfg.gallery_root)) {
if (!actor_dir.is_directory()) continue;
src/gallery/gallery_builder.hpp
+1
View File
@@ -1,4 +1,5 @@
#pragma once
#include "gallery/gallery_report.hpp"
#include "types.hpp"
#include <string>
src/gallery/gallery_calibration.hpp
+105 -1
View File
@@ -1,4 +1,5 @@
#pragma once
/// TRACES: AR-023 | SR-002
#include "types.hpp"
#include <algorithm>
@@ -8,6 +9,7 @@
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <functional>
#include <iostream>
#include <stdexcept>
#include <string>
@@ -48,14 +50,75 @@ struct GalleryCalibration {
}
};
/// TRACES: AR-023, AR-024 | SR-002
///
/// cosine → P(same person). The one probability space the pipeline reasons in.
///
/// Handed to every stage that has to decide whether two embeddings are the same
/// person — track association (AR-007), evidence discounting (AR-025), identity
/// matching — so a threshold of 0.5 means the same thing in all of them. A stage
/// that thresholded a raw cosine instead would be using a number that means
/// something different for every model, gallery and face size (AR-024).
///
/// **No prior term.** `log_prior_odds` adjusts for the gallery's base rate, which
/// is a question about *which of N actors*; association asks whether two faces
/// are one person, where the balanced fit is the right answer. Passing the
/// matcher's prior here would silently bias tracking by the size of the cast.
inline std::function<float(float)> same_person_probability(const GalleryCalibration& cal) {
if (!cal.valid) {
// Loud, because the failure mode is invisible: an untuned sigmoid still
// returns plausible probabilities, and every threshold downstream of it
// is then a guess wearing a calibrated number's clothes.
std::cerr << "[calibration] WARNING: no fitted calibration — association and "
"evidence weighting fall back to the untuned default sigmoid "
"(a=" << cal.a << ", b=" << cal.b << "). Probabilities are "
"not meaningful for this embedder.\n";
}
return [cal](float similarity) { return cal.probability(similarity); };
}
/// TRACES: GR-003 | SR-001
///
/// Everything the fit learns about the gallery on its way to two numbers.
///
/// The fit computes per-actor dedup counts, which actors can supply positive
/// pairs at all, and the two similarity distributions the sigmoid is derived
/// from — and then returns only (a, b, valid). GR-003 exists because that is the
/// evidence for whether the calibration, and so every threshold expressed in its
/// probability space (AR-024), rests on anything. Filling this struct costs
/// nothing: the values already exist at the point they are copied out.
///
/// Per-actor vectors are indexed by the actor index used in `flat_actor`.
struct GalleryCalibrationStats {
int n_actors = 0;
int min_embeddings_for_positive = 0;
float dedup_sim_threshold = 0.f;
std::vector<int> distinct_per_actor; // after near-duplicate removal
std::vector<int> duplicates_removed_per_actor;
std::vector<char> eligible; // 1 = supplies positive pairs
int hist_bins = 0; // over sim ∈ [-1, 1]
std::vector<double> intra_hist;
std::vector<double> inter_hist;
double n_intra_pairs = 0.0;
double n_inter_pairs = 0.0;
double train_accuracy_pct = 0.0;
};
// Fit a logistic sigmoid to gallery pair similarities.
// Positive pairs: same actor, different reference images.
// Negative pairs: different actors (all cross-actor embedding pairs).
// Class weights balance the (typically skewed) pos/neg ratio.
// Requires ≥2 positive pairs and ≥1 negative pair.
//
// `stats` is optional (GR-003): pass one to receive the dedup, eligibility and
// distribution detail the fit would otherwise discard.
inline GalleryCalibration calibrate_gallery(
const std::vector<Embedding>& flat_emb,
const std::vector<int>& flat_actor)
const std::vector<int>& flat_actor,
GalleryCalibrationStats* stats = nullptr)
{
constexpr int kMinEmbeddingsForPositive = 5;
constexpr float kDedupSimThreshold = 1.f - 1e-7f; // sim above this = duplicate
@@ -79,6 +142,21 @@ inline GalleryCalibration calibrate_gallery(
std::vector<bool> actor_eligible(n_actors, false);
int n_eligible = 0;
/// TRACES: GR-003 | SR-001
// Record what the filter did, per actor, while the counts still exist.
if (stats) {
*stats = GalleryCalibrationStats{};
stats->n_actors = n_actors;
stats->min_embeddings_for_positive = kMinEmbeddingsForPositive;
stats->dedup_sim_threshold = kDedupSimThreshold;
stats->distinct_per_actor.assign(n_actors, 0);
stats->duplicates_removed_per_actor.assign(n_actors, 0);
stats->eligible.assign(n_actors, 0);
stats->hist_bins = kHistBins;
stats->intra_hist.assign(kHistBins, 0.0);
stats->inter_hist.assign(kHistBins, 0.0);
}
for (int ai = 0; ai < n_actors; ++ai) {
std::vector<Embedding> kept;
for (const auto& e : by_actor[ai]) {
@@ -92,6 +170,12 @@ inline GalleryCalibration calibrate_gallery(
actor_eligible[ai] = true;
++n_eligible;
}
if (stats) {
stats->distinct_per_actor[ai] = static_cast<int>(kept.size());
stats->duplicates_removed_per_actor[ai] =
static_cast<int>(by_actor[ai].size() - kept.size());
stats->eligible[ai] = actor_eligible[ai] ? 1 : 0;
}
for (auto& e : kept) {
flat_emb_dedup.push_back(e);
flat_actor_dedup.push_back(ai);
@@ -99,6 +183,14 @@ inline GalleryCalibration calibrate_gallery(
}
const int n = static_cast<int>(flat_emb_dedup.size());
// Nothing to fit and nothing to multiply. Returning here keeps the report
// buildable for a degenerate gallery instead of handing cv::gemm an empty
// matrix; the per-actor stats above are already filled and still useful.
if (n == 0) {
std::cerr << "[calibration] no embeddings — calibration skipped\n";
return {};
}
std::cerr << "[calibration] dedup: " << flat_emb.size() << " -> " << n
<< " embeddings (" << n_eligible << "/" << n_actors
<< " actors have >= " << kMinEmbeddingsForPositive
@@ -199,6 +291,17 @@ inline GalleryCalibration calibrate_gallery(
double n_pos = 0.0, n_neg = 0.0;
for (int b = 0; b < kHistBins; ++b) { n_pos += pos_hist[b]; n_neg += neg_hist[b]; }
/// TRACES: GR-003 | SR-001
// The two distributions the sigmoid is about to be fitted from. Emitted
// whether or not the fit succeeds — a failed fit is exactly the case where
// someone needs to see why.
if (stats) {
stats->intra_hist = pos_hist;
stats->inter_hist = neg_hist;
stats->n_intra_pairs = n_pos;
stats->n_inter_pairs = n_neg;
}
if (n_pos < 2 || n_neg < 1) {
std::cerr << "[calibration] insufficient pairs (+" << n_pos
<< "/-" << n_neg << ") — calibration skipped\n";
@@ -253,6 +356,7 @@ inline GalleryCalibration calibrate_gallery(
correct += (sig > 0.5f) ? pos_hist[b] : neg_hist[b];
}
double acc = 100.0 * correct / total;
if (stats) stats->train_accuracy_pct = acc;
GalleryCalibration cal{a, bias, true};
std::cerr << "[calibration] sigmoid fitted:"
src/gallery/gallery_report.hpp
+495
View File
@@ -0,0 +1,495 @@
#pragma once
/// TRACES: GR-003 | SR-001
///
/// The gallery build report — what the gallery *is*, written next to it.
///
/// A gallery is a silent artefact: it loads, it scores, it never complains. The
/// two ways it fails are both invisible from the outside.
///
/// 1. **An actor with zero usable images can never be recognised.** They are
/// dropped at build time (`gallery_builder.cpp` skips a directory whose
/// images all fail detection or alignment), so afterwards nothing in the
/// file records that they were ever meant to be there. Every scene they
/// appear in is a guaranteed miss, and recall is capped at a number nobody
/// computed. This is the single most useful line in the report.
/// 2. **A gallery can be quietly bad and look fine.** The Platt sigmoid
/// (AR-023) is fitted from two distributions — intra-class (same actor,
/// different reference) and inter-class (different actors) similarity — and
/// *every* threshold in the pipeline is expressed in the probability space
/// that fit defines (AR-024): identity acceptance, track association,
/// expansion admission, cluster merging. If those two distributions overlap
/// heavily the fit is weak, and every downstream decision silently inherits
/// that weakness while still reporting confident-looking probabilities. The
/// fit already computes the distributions and throws them away; emitting
/// them is what makes the quality of the whole probability space auditable
/// instead of assumed.
///
/// The report is therefore a build artefact, not a debug aid: it is the only
/// place the recall ceiling and the calibration's conditioning are written down.
///
/// **On the histograms being in cosine space.** They bin raw similarity, and
/// that is not an AR-024 violation: no decision is taken here. These two
/// distributions are the *input* the calibration is fitted from — they cannot be
/// expressed in the probability space the calibration defines, because that
/// space is their output. GR-003 asks for exactly this ("the intra/inter
/// distributions behind it"), for the same reason GR-008 characterises an
/// actor's reference spread in the metric space: shape is a property of the
/// metric, decisions are a property of the probability.
#include "gallery/gallery_calibration.hpp"
#include "gallery/embedder_stamp.hpp"
#include "types.hpp"
#include <nlohmann/json.hpp>
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <fstream>
#include <stdexcept>
#include <string>
#include <vector>
/// TRACES: GR-003 | SR-001
///
/// Per-actor image accounting from the build pass, including the actors that
/// produced nothing and were therefore dropped from the gallery.
///
/// Filled by `build_gallery()`. It has to be collected there and cannot be
/// recovered later: by the time a gallery exists, an actor with no usable image
/// is indistinguishable from an actor who was never requested.
struct GalleryBuildAudit {
struct ActorImages {
std::string imdb_id;
std::string name;
int images_seen = 0; // candidate image files in the actor's directory
int images_used = 0; // ...that yielded an embedding
int unreadable = 0; // cv::imread failed
int no_face = 0; // detector found nothing
int align_failed = 0; // 5-point warp failed
};
std::vector<ActorImages> actors; // every directory seen, in build order
};
/// TRACES: GR-003 | SR-001
struct GalleryReport {
// One row per actor the build considered. Actors with references == 0 are
// the zero-usable-image case: present in the source tree, absent from the
// gallery, unrecognisable for the life of the file.
struct Actor {
std::string imdb_id;
std::string name;
int images_seen = -1; // -1 = unknown (report built without a build audit)
int references = 0; // embeddings stored in the gallery
int distinct_references = 0; // ...after near-duplicate removal
int duplicates_removed = 0;
bool eligible_for_positive_pairs = false;
};
// The two distributions the sigmoid is fitted from, as the fit itself saw
// them: counts per similarity bin over [sim_min, sim_max].
struct Distributions {
int bins = 0;
float sim_min = -1.f;
float sim_max = 1.f;
std::vector<double> intra; // same actor, different reference image
std::vector<double> inter; // different actors
double intra_pairs = 0.0;
double inter_pairs = 0.0;
double intra_mean = 0.0;
double inter_mean = 0.0;
// Normalised histogram intersection, Σ_b min(p_intra[b], p_inter[b]).
// 0 = perfectly separated, 1 = indistinguishable. This is the number
// that says whether the calibration — and so every threshold expressed
// in its probability space — rests on anything.
double overlap = 0.0;
};
// GR-003 / AR-023 open question, reported but NOT applied. The spec asks for
// a gallery-derived prior of intra/(intra+inter); the shipped default is
// 0.5. Persisting the distributions makes the real value computable, so the
// decision can be taken on evidence rather than left implicit. Behaviour is
// unchanged: `applied` is always false here.
struct Prior {
double derived = 0.0; // intra_pairs / (intra_pairs + inter_pairs)
double derived_log_odds = 0.0; // log(p/(1-p)), the term AR-023 would add
float configured_default = 0.5f;
bool applied = false;
std::string note;
};
std::string schema{"sae.gallery_report/1"};
std::string gallery_path;
EmbedderStamp embedder;
// ── Summary ──────────────────────────────────────────────────────────────
int actors_total = 0; // considered (gallery + zero-usable)
int actors_in_gallery = 0;
int actors_zero_usable = 0;
int actors_below_positive_threshold = 0;
int64_t embeddings_total = 0;
int64_t distinct_embeddings_total = 0;
int64_t duplicates_removed_total = 0;
double mean_embeddings_per_actor = 0.0; // over actors in the gallery
int min_embeddings_for_positive_pairs = 0;
float dedup_similarity_threshold = 0.f;
// ── Calibration ──────────────────────────────────────────────────────────
float calib_a = 10.f;
float calib_b = -5.f;
bool calib_valid = false;
uint64_t calib_hash = 0;
double calib_train_accuracy_pct = 0.0;
float calib_boundary_p50 = 0.f; // similarity at which P(match) = 0.5
Distributions distributions;
Prior prior;
std::vector<Actor> actors;
// Names duplicated out of `actors` so the two failure modes are greppable
// without a JSON query. These are the lines a human reads first.
std::vector<std::string> zero_usable;
std::vector<std::string> below_positive_threshold;
};
/// TRACES: GR-003 | SR-001
///
/// Assembles the report from the three things that know a piece of the answer:
/// the gallery itself (who is in it, with how many references), the calibration
/// stats (dedup, eligibility, the two distributions), and the build audit (who
/// was considered and produced nothing). The audit is optional — a report built
/// from a stored gallery simply cannot know about the actors that never made it.
///
/// `stats` is indexed by actor index, so the flat arrays handed to
/// `calibrate_gallery()` must have used the gallery's own actor ordering.
inline GalleryReport build_gallery_report(const ActorGallery& gallery,
const GalleryCalibration& cal,
const GalleryCalibrationStats& stats,
const GalleryBuildAudit* audit = nullptr,
const std::string& gallery_path = "",
float configured_prior = 0.5f)
{
GalleryReport r;
r.gallery_path = gallery_path;
r.embedder = gallery.embedder;
r.calib_a = cal.a;
r.calib_b = cal.b;
r.calib_valid = cal.valid;
r.calib_hash = gallery.calib_hash;
r.calib_train_accuracy_pct = stats.train_accuracy_pct;
r.calib_boundary_p50 = cal.boundary_at(0.5f);
r.min_embeddings_for_positive_pairs = stats.min_embeddings_for_positive;
r.dedup_similarity_threshold = stats.dedup_sim_threshold;
auto audit_for = [&](const ActorGallery::Actor& a) -> const GalleryBuildAudit::ActorImages* {
if (!audit) return nullptr;
for (const auto& e : audit->actors) {
if (!a.imdb_id.empty() && e.imdb_id == a.imdb_id) return &e;
if (a.imdb_id.empty() && e.name == a.name) return &e;
}
return nullptr;
};
for (size_t i = 0; i < gallery.actors.size(); ++i) {
const auto& ga = gallery.actors[i];
GalleryReport::Actor row;
row.imdb_id = ga.imdb_id;
row.name = ga.name;
row.references = static_cast<int>(ga.embeddings.size());
if (const auto* au = audit_for(ga)) row.images_seen = au->images_seen;
if (i < stats.distinct_per_actor.size()) {
row.distinct_references = stats.distinct_per_actor[i];
row.duplicates_removed = stats.duplicates_removed_per_actor[i];
row.eligible_for_positive_pairs = stats.eligible[i] != 0;
} else {
// No calibration stats for this actor (the fit never saw them).
// Report the raw count rather than a fabricated distinct count.
row.distinct_references = row.references;
}
r.embeddings_total += row.references;
r.distinct_embeddings_total += row.distinct_references;
r.duplicates_removed_total += row.duplicates_removed;
if (!row.eligible_for_positive_pairs) {
++r.actors_below_positive_threshold;
r.below_positive_threshold.push_back(row.name);
}
r.actors.push_back(std::move(row));
}
r.actors_in_gallery = static_cast<int>(gallery.actors.size());
// Actors the build considered and could not use at all. They are not in the
// gallery, so this is the only record that they exist.
if (audit) {
for (const auto& e : audit->actors) {
if (e.images_used > 0) continue;
GalleryReport::Actor row;
row.imdb_id = e.imdb_id;
row.name = e.name;
row.images_seen = e.images_seen;
row.references = 0;
r.zero_usable.push_back(e.name);
r.actors.push_back(std::move(row));
}
}
r.actors_zero_usable = static_cast<int>(r.zero_usable.size());
r.actors_total = r.actors_in_gallery + r.actors_zero_usable;
r.mean_embeddings_per_actor =
r.actors_in_gallery > 0
? static_cast<double>(r.embeddings_total) / r.actors_in_gallery
: 0.0;
// ── The two distributions, straight out of the fit ───────────────────────
auto& d = r.distributions;
d.bins = stats.hist_bins;
d.sim_min = -1.f;
d.sim_max = 1.f;
d.intra = stats.intra_hist;
d.inter = stats.inter_hist;
d.intra_pairs = stats.n_intra_pairs;
d.inter_pairs = stats.n_inter_pairs;
if (d.bins > 0) {
const double bin_w = (d.sim_max - d.sim_min) / d.bins;
double si = 0.0, se = 0.0;
for (int b = 0; b < d.bins; ++b) {
const double centre = d.sim_min + (b + 0.5) * bin_w;
si += d.intra[b] * centre;
se += d.inter[b] * centre;
}
if (d.intra_pairs > 0.0) d.intra_mean = si / d.intra_pairs;
if (d.inter_pairs > 0.0) d.inter_mean = se / d.inter_pairs;
if (d.intra_pairs > 0.0 && d.inter_pairs > 0.0) {
double ov = 0.0;
for (int b = 0; b < d.bins; ++b)
ov += std::min(d.intra[b] / d.intra_pairs, d.inter[b] / d.inter_pairs);
d.overlap = ov;
}
}
// ── The prior AR-023 leaves open — computed, reported, not applied ────────
r.prior.configured_default = configured_prior;
r.prior.applied = false;
const double pair_total = d.intra_pairs + d.inter_pairs;
if (pair_total > 0.0) {
r.prior.derived = d.intra_pairs / pair_total;
const double p = std::clamp(r.prior.derived, 1e-12, 1.0 - 1e-12);
r.prior.derived_log_odds = std::log(p / (1.0 - p));
}
r.prior.note =
"AR-023 specifies a gallery-derived prior of intra/(intra+inter); the shipped "
"match_prior default is 0.5 (calibrated sigmoid used directly). The derived value "
"is the base rate of same-actor pairs among ALL enumerated gallery pairs, so it "
"falls as the cast grows (roughly (k-1)/((k-1)+(A-1)k) for A actors with k "
"references each) — it is a property of gallery size as much as of the embedder. "
"Reported here as evidence; NOT applied. Behaviour is unchanged until the choice "
"is recorded in the spec.";
return r;
}
// ── JSON ─────────────────────────────────────────────────────────────────────
/// TRACES: GR-003 | SR-001
inline nlohmann::json gallery_report_to_json(const GalleryReport& r) {
nlohmann::json j;
j["schema"] = r.schema;
j["gallery_path"] = r.gallery_path;
j["embedder"] = {{"model_name", r.embedder.model_name},
{"model_sha256", r.embedder.model_sha256},
{"embed_dim", r.embedder.embed_dim}};
j["summary"] = {
{"actors_total", r.actors_total},
{"actors_in_gallery", r.actors_in_gallery},
{"actors_zero_usable", r.actors_zero_usable},
{"actors_below_positive_threshold", r.actors_below_positive_threshold},
{"embeddings_total", r.embeddings_total},
{"distinct_embeddings_total", r.distinct_embeddings_total},
{"duplicates_removed_total", r.duplicates_removed_total},
{"mean_embeddings_per_actor", r.mean_embeddings_per_actor},
{"min_embeddings_for_positive_pairs", r.min_embeddings_for_positive_pairs},
{"dedup_similarity_threshold", r.dedup_similarity_threshold}};
j["calibration"] = {
{"a", r.calib_a},
{"b", r.calib_b},
{"valid", r.calib_valid},
{"hash", r.calib_hash},
{"train_accuracy_pct", r.calib_train_accuracy_pct},
{"boundary_p50", r.calib_boundary_p50}};
const auto& d = r.distributions;
j["distributions"] = {
{"bins", d.bins},
{"sim_min", d.sim_min},
{"sim_max", d.sim_max},
{"intra", d.intra},
{"inter", d.inter},
{"intra_pairs", d.intra_pairs},
{"inter_pairs", d.inter_pairs},
{"intra_mean", d.intra_mean},
{"inter_mean", d.inter_mean},
{"overlap", d.overlap}};
j["prior"] = {
{"derived", r.prior.derived},
{"derived_log_odds", r.prior.derived_log_odds},
{"configured_default", r.prior.configured_default},
{"applied", r.prior.applied},
{"note", r.prior.note}};
j["zero_usable"] = r.zero_usable;
j["below_positive_threshold"] = r.below_positive_threshold;
j["actors"] = nlohmann::json::array();
for (const auto& a : r.actors) {
j["actors"].push_back({
{"imdb_id", a.imdb_id},
{"name", a.name},
{"images_seen", a.images_seen},
{"references", a.references},
{"distinct_references", a.distinct_references},
{"duplicates_removed", a.duplicates_removed},
{"eligible_for_positive_pairs", a.eligible_for_positive_pairs}});
}
return j;
}
/// TRACES: GR-003 | SR-001
inline GalleryReport gallery_report_from_json(const nlohmann::json& j) {
GalleryReport r;
r.schema = j.value("schema", std::string{});
r.gallery_path = j.value("gallery_path", std::string{});
if (j.contains("embedder")) {
const auto& je = j.at("embedder");
r.embedder.model_name = je.value("model_name", "");
r.embedder.model_sha256 = je.value("model_sha256", "");
r.embedder.embed_dim = je.value("embed_dim", 512);
}
if (j.contains("summary")) {
const auto& s = j.at("summary");
r.actors_total = s.value("actors_total", 0);
r.actors_in_gallery = s.value("actors_in_gallery", 0);
r.actors_zero_usable = s.value("actors_zero_usable", 0);
r.actors_below_positive_threshold = s.value("actors_below_positive_threshold", 0);
r.embeddings_total = s.value("embeddings_total", int64_t{0});
r.distinct_embeddings_total = s.value("distinct_embeddings_total", int64_t{0});
r.duplicates_removed_total = s.value("duplicates_removed_total", int64_t{0});
r.mean_embeddings_per_actor = s.value("mean_embeddings_per_actor", 0.0);
r.min_embeddings_for_positive_pairs = s.value("min_embeddings_for_positive_pairs", 0);
r.dedup_similarity_threshold = s.value("dedup_similarity_threshold", 0.f);
}
if (j.contains("calibration")) {
const auto& c = j.at("calibration");
r.calib_a = c.value("a", 10.f);
r.calib_b = c.value("b", -5.f);
r.calib_valid = c.value("valid", false);
r.calib_hash = c.value("hash", uint64_t{0});
r.calib_train_accuracy_pct = c.value("train_accuracy_pct", 0.0);
r.calib_boundary_p50 = c.value("boundary_p50", 0.f);
}
if (j.contains("distributions")) {
const auto& d = j.at("distributions");
r.distributions.bins = d.value("bins", 0);
r.distributions.sim_min = d.value("sim_min", -1.f);
r.distributions.sim_max = d.value("sim_max", 1.f);
r.distributions.intra = d.value("intra", std::vector<double>{});
r.distributions.inter = d.value("inter", std::vector<double>{});
r.distributions.intra_pairs = d.value("intra_pairs", 0.0);
r.distributions.inter_pairs = d.value("inter_pairs", 0.0);
r.distributions.intra_mean = d.value("intra_mean", 0.0);
r.distributions.inter_mean = d.value("inter_mean", 0.0);
r.distributions.overlap = d.value("overlap", 0.0);
}
if (j.contains("prior")) {
const auto& p = j.at("prior");
r.prior.derived = p.value("derived", 0.0);
r.prior.derived_log_odds = p.value("derived_log_odds", 0.0);
r.prior.configured_default = p.value("configured_default", 0.5f);
r.prior.applied = p.value("applied", false);
r.prior.note = p.value("note", "");
}
r.zero_usable = j.value("zero_usable", std::vector<std::string>{});
r.below_positive_threshold = j.value("below_positive_threshold", std::vector<std::string>{});
if (j.contains("actors")) {
for (const auto& ja : j.at("actors")) {
GalleryReport::Actor a;
a.imdb_id = ja.value("imdb_id", "");
a.name = ja.value("name", "");
a.images_seen = ja.value("images_seen", -1);
a.references = ja.value("references", 0);
a.distinct_references = ja.value("distinct_references", 0);
a.duplicates_removed = ja.value("duplicates_removed", 0);
a.eligible_for_positive_pairs = ja.value("eligible_for_positive_pairs", false);
r.actors.push_back(std::move(a));
}
}
return r;
}
// "<dir>/cast.h5" → "<dir>/cast.report.json". A known gallery extension is
// replaced rather than appended to, so the report sits beside the gallery under
// the same stem.
inline std::string gallery_report_path(const std::string& gallery_path) {
auto slash = gallery_path.find_last_of("/\\");
auto dot = gallery_path.find_last_of('.');
std::string stem =
(dot != std::string::npos && (slash == std::string::npos || dot > slash))
? gallery_path.substr(0, dot)
: gallery_path;
return stem + ".report.json";
}
/// TRACES: GR-003 | SR-001
inline void save_gallery_report(const std::string& path, const GalleryReport& r) {
std::ofstream out(path);
if (!out.is_open())
throw std::runtime_error("save_gallery_report: cannot write " + path);
out << gallery_report_to_json(r).dump(2) << "\n";
}
/// TRACES: GR-003 | SR-001
inline GalleryReport load_gallery_report(const std::string& path) {
std::ifstream in(path);
if (!in.is_open())
throw std::runtime_error("load_gallery_report: cannot open " + path);
nlohmann::json j;
in >> j;
return gallery_report_from_json(j);
}
/// TRACES: GR-003 | SR-001
///
/// The report's headline, on stderr, at build time. The file is the audit trail;
/// this is what stops a bad gallery from being shipped without anyone noticing.
inline void log_gallery_report(const GalleryReport& r) {
std::cerr << "[gallery-report] " << r.actors_in_gallery << " actors / "
<< r.embeddings_total << " embeddings"
<< " (mean " << r.mean_embeddings_per_actor << " per actor)\n";
if (r.actors_zero_usable > 0) {
std::cerr << "[gallery-report] WARNING: " << r.actors_zero_usable
<< " actor(s) have NO usable image — they can never be recognised:\n";
for (const auto& n : r.zero_usable) std::cerr << " - " << n << "\n";
}
if (r.actors_below_positive_threshold > 0) {
std::cerr << "[gallery-report] " << r.actors_below_positive_threshold
<< " actor(s) below " << r.min_embeddings_for_positive_pairs
<< " distinct references — they contribute no positive pairs and "
"weaken the calibration\n";
}
if (r.duplicates_removed_total > 0)
std::cerr << "[gallery-report] " << r.duplicates_removed_total
<< " near-duplicate reference(s) removed\n";
std::cerr << "[gallery-report] calibration valid=" << r.calib_valid
<< " a=" << r.calib_a << " b=" << r.calib_b
<< " intra/inter overlap=" << r.distributions.overlap
<< " (intra mean=" << r.distributions.intra_mean
<< ", inter mean=" << r.distributions.inter_mean << ")\n";
std::cerr << "[gallery-report] gallery-derived prior would be "
<< r.prior.derived << " (log-odds " << r.prior.derived_log_odds
<< "); shipped default " << r.prior.configured_default
<< " is in force — reported, not applied\n";
}
src/gallery/gallery_store.cpp
+40
View File
@@ -79,6 +79,21 @@ static ActorGallery load_gallery_hdf5(const std::string& path) {
gallery.actors.push_back(std::move(actor));
}
/// TRACES: GR-004 | SR-001
// Absent /embedder group == a gallery written before model binding existed.
// It stays readable; verify_gallery_embedder() decides what that means.
if (file.nameExists("embedder")) {
H5::Group eg = file.openGroup("embedder");
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
if (eg.attrExists("model_name"))
eg.openAttribute("model_name").read(str, gallery.embedder.model_name);
if (eg.attrExists("model_sha256"))
eg.openAttribute("model_sha256").read(str, gallery.embedder.model_sha256);
if (eg.attrExists("embed_dim"))
eg.openAttribute("embed_dim").read(H5::PredType::NATIVE_INT32,
&gallery.embedder.embed_dim);
}
if (file.nameExists("calibration")) {
H5::Group cal = file.openGroup("calibration");
cal.openAttribute("a").read(H5::PredType::NATIVE_FLOAT, &gallery.calib_a);
@@ -149,6 +164,20 @@ static void save_gallery_hdf5(const std::string& path, const ActorGallery& galle
write_str_dataset(file, "name", name);
write_str_dataset(file, "source_images", src_images);
/// TRACES: GR-004 | SR-001
// Bind the file to the embedder that produced its vectors. Written only when
// known — an empty stamp must round-trip as "unstamped", not as a stamp
// claiming an unnamed model.
if (!gallery.embedder.empty()) {
H5::Group eg = file.createGroup("embedder");
H5::DataSpace scalar(H5S_SCALAR);
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
eg.createAttribute("model_name", str, scalar).write(str, gallery.embedder.model_name);
eg.createAttribute("model_sha256", str, scalar).write(str, gallery.embedder.model_sha256);
eg.createAttribute("embed_dim", H5::PredType::NATIVE_INT32, scalar)
.write(H5::PredType::NATIVE_INT32, &gallery.embedder.embed_dim);
}
if (gallery.calib_hash != 0) {
H5::Group cal = file.createGroup("calibration");
H5::DataSpace scalar(H5S_SCALAR);
@@ -186,6 +215,17 @@ ActorGallery load_gallery(const std::string& path) {
<< std::chrono::duration<double>(t1 - t0).count() << "s\n";
ActorGallery gallery;
/// TRACES: GR-004 | SR-001
// Optional top-level "embedder" object, matching the HDF5 /embedder group.
// Written by the JSON-era helper scripts; absent in anything older.
if (j.contains("embedder") && j.at("embedder").is_object()) {
const auto& je = j.at("embedder");
gallery.embedder.model_name = je.value("model_name", "");
gallery.embedder.model_sha256 = je.value("model_sha256", "");
gallery.embedder.embed_dim = je.value("embed_dim", 512);
}
for (const auto& ja : j.at("actors")) {
ActorGallery::Actor actor;
actor.imdb_id = ja.value("imdb_id", "");
src/gallery/gallery_store.hpp
+14
View File
@@ -6,12 +6,25 @@
// legacy gallery.json files are still readable for backward compatibility but
// save_gallery always writes HDF5 regardless of the requested extension.
//
// GR-005 is preserved here by absence: this is the only path that serialises a
// gallery, and it reads and writes the local filesystem only. There is no
// upload, no client, and no encoder that could put an embedding on a wire — the
// public server refuses to carry one (SR-004/UR-012), and the prohibition holds
// on this side by there being nothing that would try.
//
/// TRACES: GR-005 | SR-005
//
// HDF5 layout:
// /embeddings float32 [N, 512] all actors' refs concatenated, row-major
// /offset int64 [A] first row of actor a in /embeddings
// /count int32 [A] number of refs for actor a
// /imdb_id /tmdb_id /jellyfin_id /name : variable-length string [A]
// /source_images : variable-length string [N], parallel to /embeddings rows
// /embedder/model_name : scalar var-len string attr — embedder file basename
// /embedder/model_sha256 : scalar var-len string attr — SHA-256 of that file
// /embedder/embed_dim : scalar int32 attr
// The GR-004 model binding. Absent group == unstamped
// (pre-GR-004 file); see gallery/embedder_stamp.hpp.
// /calibration/a, /b : scalar float32 attrs — Platt-sigmoid P(match|sim) fit
// /calibration/valid : scalar int8 attr (0/1)
// /calibration/hash : scalar uint64 attr — hash of the embeddings the fit
@@ -19,6 +32,7 @@
//
// Legacy JSON format (read-only):
// {
// "embedder": {"model_name": "...", "model_sha256": "...", "embed_dim": 512},
// "actors": [
// {
// "imdb_id": "nm0000093", // optional, "" if unknown
src/gallery/track_gallery.hpp
+70 -1
View File
@@ -2,6 +2,8 @@
#include "types.hpp"
#include "config.hpp"
#include <functional>
#include <cmath>
#include <cstdio>
#include <iostream>
@@ -117,6 +119,27 @@ struct TrackGallery {
// matcher when it observes a cut or track disappearance.
void forget(int track_id) { tracks_.erase(track_id); }
/// TRACES: AR-019 | SR-005
/// The registry's verdict on who this track is. Authoritative: it comes from
/// the Bayesian accumulation (AR-025), where the local tally counted raw
/// accepted frames and so weighted thirty near-identical looks the same as
/// thirty distinct ones.
void set_owner(int track_id, int actor_idx) {
if (track_id < 0 || actor_idx < 0) return;
tracks_[track_id].registry_owner = actor_idx;
}
/// TRACES: AR-024 | SR-005
/// Supply the calibration belonging to the active embedder. Without it the
/// band falls back to treating cosine as probability, which is wrong but
/// bounded — and the default is loud in the header rather than silent.
void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }
void set_band(float lo, float hi) { band_lo_ = lo; band_hi_ = hi; }
/// Embeddings the band refused. A store that admits nothing is as wrong as
/// one that admits everything, and neither is visible without this.
std::size_t band_rejected() const { return rejected_; }
// Drop every track buffer (scene cut / EOF). Mirrors face_tracker's clear.
void clear_tracks() { tracks_.clear(); }
@@ -132,11 +155,43 @@ private:
std::map<int, int> actor_votes; // actor_idx → accepted-frame count
int accepted_frames{0};
bool promoted{false};
int registry_owner{-1}; ///< AR-019: authoritative
};
/// TRACES: AR-018, AR-024 | SR-005
/// Banded admission: an embedding joins the store only if its similarity to
/// something already there falls **inside a band**.
///
/// above the upper bound → redundant. It is another look at a pose the
/// store already covers, and adding it teaches the annex nothing while
/// costing a slot that a novel view could have used.
/// below the lower bound → suspect. Within one track every face is the
/// same person by construction, so an embedding unlike everything else
/// on the track is evidence the construction failed — a track-ID
/// collision or a bad detection. Admitting it is how an actor's annex
/// gets poisoned with someone else's face.
///
/// Both bounds are calibrated probabilities, never raw cosines (AR-024): a
/// bare similarity threshold means something different for every model and
/// every face size, and this gate has to hold across both.
///
/// The first embedding is always admitted — there is nothing for it to be
/// redundant with, and nothing to contradict it.
bool admit(const TrackState& ts, const Embedding& emb) const {
if (ts.buf.empty()) return true;
float p_max = 0.f;
for (const auto& b : ts.buf)
p_max = std::max(p_max, calibrate_(cosine_similarity(b.emb, emb)));
return p_max >= band_lo_ && p_max <= band_hi_;
}
void insert_into_buffer(TrackState& ts, const Embedding& emb,
float gal_sim, const cv::Mat& crop)
{
if (!admit(ts, emb)) { ++rejected_; return; }
BufEntry e;
e.emb = emb;
e.gal_sim = gal_sim;
@@ -166,7 +221,7 @@ private:
void promote(int track_id, TrackState& ts) {
ts.promoted = true; // idempotent: never promote a track twice
int actor = plurality_actor(ts);
int actor = owning_actor(ts);
if (actor < 0) return;
// ── Safety gate: internal spread ─────────────────────────────────────
@@ -202,6 +257,13 @@ private:
<< annex_.size() << "\n";
}
/// Prefer the registry's verdict; fall back to the local tally only when no
/// registry is attached (unit tests, replay harness).
static int owning_actor(const TrackState& ts) {
if (ts.registry_owner >= 0) return ts.registry_owner;
return plurality_actor(ts);
}
static int plurality_actor(const TrackState& ts) {
int best = -1, best_votes = 0;
for (const auto& [ai, v] : ts.actor_votes) {
@@ -231,6 +293,13 @@ private:
#endif
}
/// cosine → P(same person). The one probability space the pipeline reasons
/// in; see gallery_calibration.hpp's same_person_probability.
std::function<float(float)> calibrate_{[](float c) { return std::max(0.f, c); }};
float band_lo_{0.90f};
float band_hi_{0.95f};
std::size_t rejected_{0}; ///< admissions refused by the band
bool enabled_;
int buffer_size_;
float novelty_sim_;
src/kpn_bindings.cpp
+28 -2
View File
@@ -15,6 +15,7 @@
#include "types.hpp"
#include "config.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/face_tracker_node.hpp"
#include "nodes/identity_matcher_node.hpp"
@@ -163,6 +164,9 @@ static Config config_from_dict(nb::dict d) {
getd("anneal_sec", cfg.anneal_sec);
// gallery expansion (usually off for sweeps; expose so it can be toggled)
if (d.contains("expand_gallery")) cfg.expand_gallery = nb::cast<bool>(d["expand_gallery"]);
/// TRACES: GR-004 | SR-001
if (d.contains("require_gallery_stamp"))
cfg.require_gallery_stamp = nb::cast<bool>(d["require_gallery_stamp"]);
return cfg;
}
@@ -210,8 +214,17 @@ NB_MODULE(sae_kpn, m) {
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "config"_a, "capacity"_a = 16);
/// TRACES: GR-004 | SR-001
// embedder_model / embedder_sha256 identify whatever produced the embeddings
// that will be fed in. In a replay those come from the dump's own stamp (see
// scripts/optimizer/SCHEMA.md), because there is no live embedder in the
// network — the dump *is* the embedder as far as this gallery is concerned.
// Passing neither leaves the binding unverifiable, which warns loudly and is
// fatal under SAE_REQUIRE_GALLERY_STAMP.
m.def("add_identity_matcher", [](Net& net, std::string name, std::string gallery_path,
nb::dict cfg_dict, std::size_t cap) {
nb::dict cfg_dict, std::size_t cap,
std::string embedder_model,
std::string embedder_sha256) {
Config cfg = config_from_dict(cfg_dict);
cfg.gallery_path = gallery_path; // needed to persist refreshed calibration back
// Cache loaded galleries by path so a threshold sweep (many networks, same
@@ -222,11 +235,24 @@ NB_MODULE(sae_kpn, m) {
if (it == cache.end())
it = cache.emplace(gallery_path,
std::make_shared<ActorGallery>(load_gallery(gallery_path))).first;
// Checked on every construction, not only on the cache miss: the same
// process may replay several dumps against one cached gallery.
EmbedderStamp feeding;
feeding.model_name = std::move(embedder_model);
feeding.model_sha256 = std::move(embedder_sha256);
enforce_embedder_stamp(it->second->embedder, feeding, gallery_path,
feeding.model_name.empty()
? "embeddings fed into this network"
: feeding.model_name,
cfg.require_gallery_stamp);
auto node = std::make_shared<kpn::ObjectVariantNodeWrapper<
IdentityMatcherFunc, SaeVariant, kpn::in<"tracked">, kpn::out<"matched">>>(
cap, *it->second, cfg);
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "gallery"_a, "config"_a, "capacity"_a = 16);
}, "net"_a, "name"_a, "gallery"_a, "config"_a, "capacity"_a = 16,
"embedder_model"_a = "", "embedder_sha256"_a = "");
m.def("add_scene_tracker", [](Net& net, std::string name, nb::dict cfg_dict, std::size_t cap) {
Config cfg = config_from_dict(cfg_dict);
src/main.cpp
+124 -9
View File
@@ -1,5 +1,9 @@
// scene_analyze — identify actors in a movie using a KPN pipeline
//
// TRACES: DP-001, DP-002 | PR-004
// One analysis core; the CLI is a front-end over it and must not fork pipeline
// logic. Other deployment modes (DP-003, DP-004) wrap this same core.
//
// KPN topology (release build):
//
// [frame_source] ──Frame──► [face_detector] ──SceneFrame──► [face_aligner]
@@ -45,6 +49,7 @@
#include "config.hpp"
#include "types.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/frame_source_node.hpp"
#include "nodes/camera_position_change_detector_node.hpp"
@@ -55,6 +60,8 @@
#include "nodes/identity_matcher_node.hpp"
#include "nodes/scene_tracker_node.hpp"
#include "nodes/scene_detector_node.hpp"
#include "scene_boundaries.hpp"
#include "nodes/scene_boundary_annotator_node.hpp"
#include "nodes/result_sink_node.hpp"
#include "nodes/embedding_dump_node.hpp"
#ifdef SAE_DEBUG
@@ -76,6 +83,18 @@
// ── CLI parsing ───────────────────────────────────────────────────────────────
/// TRACES: AR-010, AR-004 | SR-002
/// How deeply the sampled branch is buffered behind the dense one. TransNetV2
/// needs kWindow (100) dense frames before it can score any of them, so the face
/// branch must lag by at least that much or it asks about frames nobody has
/// looked at yet. Backpressure turns depth into lag: the fanout blocks on the
/// slower branch rather than dropping, so the detector simply runs ahead.
static constexpr std::size_t kSceneJoinDepth = 256;
/// Set when the scene branch is built, so shutdown can report whether the join
/// actually worked.
static std::shared_ptr<SceneBoundaries> scene_stats;
static Config parse_args(int argc, char** argv) {
Config cfg;
cfg.detector_model = kDefaultDetectorModel;
@@ -114,6 +133,7 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--detector")) cfg.detector_model = next();
else if (arg("--detector-engine")) cfg.detector_engine = next();
else if (arg("--arcface")) cfg.arcface_model = next();
else if (arg("--require-gallery-stamp")) cfg.require_gallery_stamp = true;
else if (arg("--arcface-engine")) cfg.arcface_engine = next();
else if (arg("--conf")) cfg.detector_conf = std::stof(next());
else if (arg("--max-faces")) cfg.max_faces = std::stoi(next());
@@ -122,10 +142,8 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--ratio-ceil")) cfg.match_ratio_ceil = std::stof(next());
else if (arg("--track-alpha")) cfg.track_alpha = std::stof(next());
else if (arg("--track-min-iou")) cfg.track_min_iou = std::stof(next());
else if (arg("--track-max-embed")) cfg.track_max_embed_dist = std::stof(next());
else if (arg("--track-max-missing")) cfg.track_max_frames_missing = std::stoi(next());
else if (arg("--cut-revive-sim")) cfg.cut_revive_sim = std::stof(next());
else if (arg("--cut-inactive-max")) cfg.cut_inactive_max_frames = std::stoi(next());
else if (arg("--track-min-prob")) cfg.track_assoc_min_prob = std::stof(next());
else if (arg("--track-extinction")) cfg.track_extinction_sec = std::stod(next());
else if (arg("--anneal")) cfg.anneal_sec = std::stod(next());
else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
@@ -167,6 +185,11 @@ int main(int argc, char** argv) {
ActorGallery gallery;
try {
gallery = load_gallery(cfg.gallery_path);
/// TRACES: GR-004 | SR-001
// Hard startup error before a single frame is decoded: a gallery built
// with another embedder yields plausible-looking, meaningless matches.
verify_gallery_embedder(gallery, cfg.gallery_path, cfg.arcface_model,
cfg.require_gallery_stamp);
} catch (const std::exception& e) {
std::cerr << "Gallery error: " << e.what() << "\n";
return 1;
@@ -183,10 +206,36 @@ int main(int argc, char** argv) {
FaceDetectorFunc detector_fn{cfg};
FaceAlignerFunc aligner_fn;
EmbedderFunc embedder_fn{cfg};
FaceTrackerFunc ftracker_fn{cfg};
// Constructed before the tracker: it fits (or loads) the calibration, and
// the tracker must decide in that same probability space (AR-024).
IdentityMatcherFunc matcher_fn {gallery, cfg};
/// TRACES: AR-007, AR-008, AR-012, AR-024 | SR-002
// The registry is created here and shared, not owned by a node: track state
// is not a stage in the stream, it is state several stages read and write,
// and its final answer is only known when a track dies.
auto same_person = same_person_probability(matcher_fn.calibration());
TrackRegistry::Config reg_cfg;
reg_cfg.extinction_sec = cfg.track_extinction_sec;
auto registry = std::make_shared<TrackRegistry>(
reg_cfg, EvidenceDiscounter(same_person));
matcher_fn.set_registry(registry);
FaceTrackerFunc ftracker_fn{cfg, registry, same_person};
SceneTrackerFunc tracker_fn {cfg};
ResultSinkFunc sink_fn {cfg, done};
/// TRACES: AR-012, AR-016 | IR-002, IR-003 | SR-002
// A reaped track goes straight to the aggregator, so the registry holds only
// live tracks and its size is bounded by concurrent on-screen faces rather
// than growing with the film.
registry->on_track_dead([&sink_fn](const DeadTrack& d) { sink_fn.add_claim(d); });
// AR-016: a film ends with faces on screen and those tracks have not timed
// out. Without this flush the closing scene's cast is silently never
// emitted — a loss that reads as a recognition miss, not a bookkeeping bug.
sink_fn.set_pre_write_hook([registry](double last_ts) { registry->flush(last_ts); });
#ifdef SAE_DEBUG
DebugRendererFunc debug_fn {cfg};
#endif
@@ -247,15 +296,45 @@ int main(int argc, char** argv) {
net.stop();
net.print_diagnostics();
/// TRACES: AR-004 | SR-002
// A dropped frame does not degrade a result, it silently changes one —
// the output is a claim about footage that was never analysed, and
// nothing in the file says so. Since AR-004 made data pushes block, a
// drop can no longer happen on the data path, so any drop here means
// either that fix regressed (it lives in the KPN submodule, one line,
// easy to lose in an update) or a channel was disabled mid-run.
//
// Reporting it in a footer and exiting 0 made both invisible: the run
// "succeeded" and the truth file looked complete. Fail instead.
/// TRACES: AR-010 | SR-002
if (scene_stats) {
std::cerr << "[scene_annotate] boundaries=" << scene_stats->count()
<< " scored_through=" << scene_stats->scored_through() << "s";
// The tail is expected: frames after the detector's last full
// window are never covered, and no amount of buffering changes
// that. They are counted rather than silently treated as
// boundary-free, which is the distinction that matters.
if (scene_stats->outran() > 0)
std::cerr << " unscored=" << scene_stats->outran()
<< " frame(s) past the detector's last window — treated as"
" boundary-free, which is unverified rather than known";
std::cerr << "\n";
}
bool dropped = false;
{
std::lock_guard<std::mutex> lk(event_mtx);
if (!overflow_counts.empty()) {
std::cerr << "[main] dropped frames (channel overflow):\n";
dropped = true;
std::cerr << "[main] ERROR: frames were dropped (channel overflow):\n";
for (const auto& [name, count] : overflow_counts)
std::cerr << " " << name << ": " << count << "\n";
std::cerr << "[main] The output would describe footage that was never "
"analysed. Refusing to report success.\n";
}
}
return node_crashed.load(std::memory_order_acquire) ? 1 : 0;
if (node_crashed.load(std::memory_order_acquire)) return 1;
return dropped ? 2 : 0;
};
// ── Build static network and run ──────────────────────────────────────────
@@ -296,6 +375,21 @@ int main(int argc, char** argv) {
if (cfg.scene_detect) {
scene_done.store(false, std::memory_order_release); // now a real terminal branch
SceneDetectorFunc scene_fn{cfg, scene_done};
/// TRACES: AR-010 | SR-002
// The join of the decode butterfly. source fans out to the dense
// TransNetV2 branch and the sampled face branch; boundaries found on the
// first have to reach the second, and cannot ride the frames because the
// branches run in parallel.
//
// TransNetV2 buffers kWindow frames before it can score any of them, so
// the face branch must lag by at least that much or it will ask about
// frames nobody has looked at yet. Channel depth is what creates the lag:
// with backpressure (AR-004) the fanout blocks on the slower branch, so
// a deep face-branch channel lets the detector run ahead by its window
// rather than dropping anything.
auto boundaries = std::make_shared<SceneBoundaries>();
scene_fn.set_boundaries(boundaries);
kpn::ObjectNode<SceneDetectorFunc, kpn::in<"dense">, kpn::out<>, "scene_detector", 0>
scene_node(scene_fn, 128);
@@ -312,13 +406,34 @@ int main(int argc, char** argv) {
return true;
}
return false;
}, 32);
}, kSceneJoinDepth);
/// TRACES: AR-010 | SR-002
// Stamp is_scene_boundary from the detector's published verdict. tol is
// half a sample interval: the two branches sample at different rates, so
// a boundary found on a dense frame rarely lands exactly on a sampled
// one, and half an interval attributes it to the nearest sampled frame
// and no further.
//
// outran() counts frames that arrived before the detector had scored
// them. Nonzero means the join depth is too shallow for the window, and
// those frames were annotated from an incomplete verdict — which would
// otherwise look exactly like "no boundary here".
SceneBoundaryAnnotatorFunc annotate_fn{boundaries, 0.5 / cfg.sample_fps};
kpn::ObjectNode<SceneBoundaryAnnotatorFunc, kpn::in<"frame">, kpn::out<"frame">,
"scene_annotate", 0> annotate(annotate_fn, kSceneJoinDepth);
// Reported at shutdown: without this the join is unverifiable, and an
// annotator that never fired looks identical to footage with no
// boundaries.
scene_stats = boundaries;
auto net = kpn::make_network(
kpn::edge(source.output<"raw">(), campos.input<"raw">()),
kpn::edge(source.output<"raw">(), scene_node.input<"dense">()),
kpn::edge(campos.output<"frame">(), decimate.input<0>()),
kpn::edge(decimate.output<0>(), detector.input<"frame">()),
kpn::edge(decimate.output<0>(), annotate.input<"frame">()),
kpn::edge(annotate.output<"frame">(), detector.input<"frame">()),
kpn::edge(detector.output<"scene">(), aligner.input<"scene">()),
kpn::edge(aligner.output<"aligned">(), embedder.input<"aligned">()),
kpn::edge(embedder.output<"embedded">(), ftracker.input<"embedded">()),
src/nodes/camera_position_change_detector_node.hpp
+2 -1
View File
@@ -26,7 +26,8 @@
// The node is a pure pass-through: it forwards the Frame unchanged except for
// is_cut, so it slots between frame_source and face_detector without altering the
// downstream contract. eof frames are forwarded immediately without processing.
//
/// TRACES: AR-009 | SR-002
struct CameraPositionChangeDetectorFunc {
static constexpr std::string_view label() { return "camera_position_change_detector"; }
src/nodes/embedder_node.hpp
+2 -1
View File
@@ -17,7 +17,8 @@
// All crops in one frame are batched into a single forward pass (capped at
// embed_batch_size). The backend serialises itself; we only call it from the
// single embedder thread.
//
/// TRACES: AR-006 | SR-002
struct EmbedderFunc {
static constexpr std::string_view label() { return "embedder"; }
src/nodes/embedding_dump_node.hpp
+183 -12
View File
@@ -1,14 +1,113 @@
#pragma once
/// TRACES: VR-001, VR-010 | PR-002
#include "types.hpp"
#include "config.hpp"
#include "gallery/embedder_stamp.hpp"
#include <H5Cpp.h>
#include <atomic>
#include <cstdint>
#include <iostream>
#include <optional>
#include <string>
#include <type_traits>
#include <vector>
// ── DumpProvenance ────────────────────────────────────────────────────────────
/// TRACES: VR-010 | PR-002
// Everything that determined a dump's *content*, read back tolerantly.
//
// Two dumps of the same film with different detector thresholds, a different
// `dense_scale`, or scene detection on versus off are different measurements of
// different things — but they are byte-shaped identically, so a consumer that
// mixes them gets a plausible number from an incoherent input. GR-004 closed the
// worst case (a cross-model replay, where every cosine is meaningless); this
// closes the rest.
//
// Every field is optional because dumps written before VR-010 lack the
// attributes. A missing field reads as *unknown*, never as a default — a
// silently-defaulted `detector_conf` is exactly the fabricated provenance the
// requirement exists to prevent ("a fixture whose provenance is unknown is worse
// than no fixture, because it will be trusted").
struct DumpProvenance {
// Model identity
std::optional<std::string> embedder_model; // GR-004
std::optional<std::string> embedder_sha256; // GR-004
std::optional<std::string> detector_model;
// Sampling
std::optional<std::string> movie;
std::optional<float> sample_fps;
std::optional<double> start_sec;
std::optional<double> end_sec; // -1 = to end of file
// Detection — what the run admitted into the dump
std::optional<float> detector_conf;
std::optional<float> detector_nms;
std::optional<float> min_face_px;
std::optional<int> max_faces; // 0 = uncapped (AR-003)
// Frame geometry
std::optional<float> dense_scale;
std::optional<float> bbox_upscale; // faces/bbox × this = original-resolution px
std::optional<float> cut_threshold;
// Scene detection. The reason this flag exists: `is_scene_boundary` is
// all-zero both when TransNetV2 found no boundaries and when it never ran,
// and no amount of staring at the array distinguishes them.
std::optional<bool> scene_detect;
// Downstream knob that shaped nothing in the dump but everything a replay is
// compared against — recorded so a sweep can be told apart from the baseline.
std::optional<float> track_assoc_min_prob;
};
// Read whatever provenance a dump carries. Never throws on a missing attribute;
// an old dump simply yields a DumpProvenance full of empty optionals.
inline DumpProvenance read_dump_provenance(const H5::H5File& f) {
DumpProvenance p;
auto str = [&](const char* n, std::optional<std::string>& out) {
if (!f.attrExists(n)) return;
// Written as a variable-length string, so the read must name the same
// type explicitly — the default would truncate to a fixed length.
H5::StrType vlen(H5::PredType::C_S1, H5T_VARIABLE);
std::string v;
f.openAttribute(n).read(vlen, v);
out = v;
};
auto num = [&](const char* n, const H5::PredType& dt, auto& out) {
if (!f.attrExists(n)) return;
typename std::decay_t<decltype(out)>::value_type v{};
f.openAttribute(n).read(dt, &v);
out = v;
};
str("embedder_model", p.embedder_model);
str("embedder_sha256", p.embedder_sha256);
str("detector_model", p.detector_model);
str("movie", p.movie);
num("sample_fps", H5::PredType::NATIVE_FLOAT, p.sample_fps);
num("start_sec", H5::PredType::NATIVE_DOUBLE, p.start_sec);
num("end_sec", H5::PredType::NATIVE_DOUBLE, p.end_sec);
num("detector_conf", H5::PredType::NATIVE_FLOAT, p.detector_conf);
num("detector_nms", H5::PredType::NATIVE_FLOAT, p.detector_nms);
num("min_face_px", H5::PredType::NATIVE_FLOAT, p.min_face_px);
num("max_faces", H5::PredType::NATIVE_INT, p.max_faces);
num("dense_scale", H5::PredType::NATIVE_FLOAT, p.dense_scale);
num("bbox_upscale", H5::PredType::NATIVE_FLOAT, p.bbox_upscale);
num("cut_threshold", H5::PredType::NATIVE_FLOAT, p.cut_threshold);
num("track_assoc_min_prob", H5::PredType::NATIVE_FLOAT, p.track_assoc_min_prob);
if (f.attrExists("scene_detect")) {
uint8_t v = 0;
f.openAttribute("scene_detect").read(H5::PredType::NATIVE_UINT8, &v);
p.scene_detect = (v != 0);
}
return p;
}
// ── EmbeddingDumpFunc ─────────────────────────────────────────────────────────
// KPN sink that taps the EmbeddedSceneFrame channel and writes the per-frame face
// metadata + embeddings to one HDF5 file (schema: scripts/optimizer/SCHEMA.md).
@@ -25,12 +124,44 @@ struct EmbeddingDumpFunc {
: path_(cfg.dump_embeddings_path), movie_(cfg.movie_path),
sample_fps_(cfg.sample_fps), done_(done)
{
std::cerr << "[embedding_dump] writing " << path_ << "\n";
/// TRACES: GR-004 | SR-001
// A dump is a bag of embeddings with no model attached, replayed against a
// gallery hours or weeks later — the same silent cross-model hazard as the
// gallery itself, so it carries the same stamp.
stamp_ = make_embedder_stamp(cfg.arcface_model);
/// TRACES: VR-010 | PR-002
// The rest of what determined this file's content. Captured from the live
// Config at construction, so it describes the run that is being written
// rather than whatever config happens to be lying around at read time.
prov_.detector_model = basename_of(cfg.detector_model);
prov_.detector_conf = cfg.detector_conf;
prov_.detector_nms = cfg.detector_nms;
prov_.min_face_px = cfg.min_face_px;
prov_.max_faces = cfg.max_faces;
prov_.cut_threshold = cfg.cut_threshold;
prov_.dense_scale = cfg.dense_scale;
prov_.start_sec = cfg.start_sec;
prov_.end_sec = cfg.end_sec;
prov_.scene_detect = cfg.scene_detect;
prov_.track_assoc_min_prob = cfg.track_assoc_min_prob;
std::cerr << "[embedding_dump] writing " << path_
<< " embedder: " << stamp_.describe()
<< " detector: " << *prov_.detector_model
<< " @conf " << cfg.detector_conf
<< " scene_detect=" << (cfg.scene_detect ? "on" : "off") << "\n";
}
void operator()(EmbeddedSceneFrame ef) {
if (ef.source.eof) { flush(); return; }
/// TRACES: VR-010 | PR-002
// Taken from the frames themselves, not recomputed from dense_scale — the
// factor the source actually stamped on them is the one that maps
// faces/bbox back to original resolution, whatever rule produced it.
if (!prov_.bbox_upscale) prov_.bbox_upscale = ef.source.bbox_upscale;
const int32_t n = static_cast<int32_t>(ef.faces.size());
ts_.push_back(ef.source.timestamp_sec);
fidx_.push_back(ef.source.frame_idx);
@@ -63,9 +194,18 @@ struct EmbeddingDumpFunc {
}
private:
// Root attributes are additive: schema_version stays 1 across VR-010, because
// every reader takes attributes by name with a default (replay.py) or an
// existence check (read_dump_provenance), so an old dump loses nothing and a
// new dump breaks nothing. A bump is for a change to the *datasets*.
static constexpr int kSchemaVersion = 1;
static constexpr int kEmbedDim = 512;
static std::string basename_of(const std::string& path) {
const auto slash = path.find_last_of("/\\");
return slash == std::string::npos ? path : path.substr(slash + 1);
}
template<typename T>
void write_vec(H5::Group& g, const char* name, const std::vector<T>& v,
const H5::PredType& dtype, hsize_t cols = 0) {
@@ -77,20 +217,49 @@ private:
if (!v.empty()) ds.write(v.data(), dtype);
}
static void attr_str(H5::H5File& f, const char* name, const std::string& v) {
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
f.createAttribute(name, str, H5::DataSpace(H5S_SCALAR)).write(str, v);
}
template<typename T>
static void attr_num(H5::H5File& f, const char* name, const H5::PredType& dt, T v) {
f.createAttribute(name, dt, H5::DataSpace(H5S_SCALAR)).write(dt, &v);
}
void write_hdf5() {
H5::H5File file(path_, H5F_ACC_TRUNC);
// root attrs
auto scalar = H5::DataSpace(H5S_SCALAR);
auto ver = file.createAttribute("schema_version", H5::PredType::NATIVE_INT, scalar);
int sv = kSchemaVersion; ver.write(H5::PredType::NATIVE_INT, &sv);
auto ed = file.createAttribute("embed_dim", H5::PredType::NATIVE_INT, scalar);
int dim = kEmbedDim; ed.write(H5::PredType::NATIVE_INT, &dim);
auto fps = file.createAttribute("sample_fps", H5::PredType::NATIVE_FLOAT, scalar);
fps.write(H5::PredType::NATIVE_FLOAT, &sample_fps_);
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
auto mv = file.createAttribute("movie", str, scalar);
mv.write(str, movie_);
attr_num(file, "schema_version", H5::PredType::NATIVE_INT, kSchemaVersion);
attr_num(file, "embed_dim", H5::PredType::NATIVE_INT, kEmbedDim);
attr_num(file, "sample_fps", H5::PredType::NATIVE_FLOAT, sample_fps_);
attr_str(file, "movie", movie_);
/// TRACES: GR-004 | SR-001
attr_str(file, "embedder_model", stamp_.model_name);
attr_str(file, "embedder_sha256", stamp_.model_sha256);
/// TRACES: VR-010 | PR-002
attr_str(file, "detector_model", prov_.detector_model.value_or(""));
attr_num(file, "detector_conf", H5::PredType::NATIVE_FLOAT, *prov_.detector_conf);
attr_num(file, "detector_nms", H5::PredType::NATIVE_FLOAT, *prov_.detector_nms);
attr_num(file, "min_face_px", H5::PredType::NATIVE_FLOAT, *prov_.min_face_px);
attr_num(file, "max_faces", H5::PredType::NATIVE_INT, *prov_.max_faces);
attr_num(file, "cut_threshold", H5::PredType::NATIVE_FLOAT, *prov_.cut_threshold);
attr_num(file, "dense_scale", H5::PredType::NATIVE_FLOAT, *prov_.dense_scale);
// Recorded, NOT applied — faces/bbox stays in the detector's own frame
// space so a replay feeds the tracker exactly what the live run fed it.
attr_num(file, "bbox_upscale", H5::PredType::NATIVE_FLOAT,
prov_.bbox_upscale.value_or(1.f));
attr_num(file, "start_sec", H5::PredType::NATIVE_DOUBLE, *prov_.start_sec);
attr_num(file, "end_sec", H5::PredType::NATIVE_DOUBLE, *prov_.end_sec);
attr_num(file, "track_assoc_min_prob", H5::PredType::NATIVE_FLOAT,
*prov_.track_assoc_min_prob);
// 0/1, matching the uint8 booleans in frames/. Tells "TransNetV2 found no
// boundaries" apart from "TransNetV2 never ran", which is/was the same
// all-zero is_scene_boundary array either way.
attr_num(file, "scene_detect", H5::PredType::NATIVE_UINT8,
static_cast<uint8_t>(*prov_.scene_detect ? 1 : 0));
H5::Group frames = file.createGroup("frames");
write_vec(frames, "timestamp_sec", ts_, H5::PredType::NATIVE_DOUBLE);
@@ -110,7 +279,9 @@ private:
<< conf_.size() << " faces → " << path_ << "\n";
}
std::string path_, movie_;
std::string path_, movie_;
EmbedderStamp stamp_;
DumpProvenance prov_;
float sample_fps_;
std::atomic<bool>& done_;
std::atomic<bool> written_{false};
src/nodes/face_aligner_node.hpp
+5 -1
View File
@@ -24,11 +24,15 @@ struct FaceAlignerFunc {
crops.reserve(sf.faces.size());
for (auto& face : sf.faces) {
cv::Mat crop = align_face(sf.source.image, face.landmarks);
// The AR-030 misfit comes from the transform the warp already needs,
// so visibility costs no extra fit.
float residual = -1.f;
cv::Mat crop = align_face(sf.source.image, face.landmarks, &residual);
if (crop.empty()) {
std::cerr << "[face_aligner] degenerate detection skipped\n";
continue;
}
face.alignment_residual = residual;
good_faces.push_back(face);
crops.push_back(std::move(crop));
}
src/nodes/face_detector_node.hpp
+6 -1
View File
@@ -1,4 +1,5 @@
#pragma once
/// TRACES: AR-001 | SR-002
#include "config.hpp"
#include "inference/face_detector.hpp"
@@ -43,7 +44,11 @@ struct FaceDetectorFunc {
[](const DetectedFace& a, const DetectedFace& b) {
return a.bbox.area() > b.bbox.area();
});
if (static_cast<int>(faces.size()) > max_faces_)
// TRACES: AR-003 | SR-002
// Largest-first ordering is kept regardless: it is load-bearing for
// deterministic association, since the Hungarian solver tie-breaks on
// index order (see the replay determinism test).
if (max_faces_ > 0 && static_cast<int>(faces.size()) > max_faces_)
faces.resize(max_faces_);
return {std::move(f), std::move(faces)};
src/nodes/face_tracker_node.hpp
+169 -162
View File
@@ -1,102 +1,124 @@
#pragma once
/// TRACES: AR-007, AR-008, AR-024 | SR-002
///
/// FaceTrackerFunc — KPN node that links face detections into tracks.
///
/// **The registry is the tracker's state.** The node owns no track map of its
/// own: it drives `TrackRegistry` through a `FrameScope` and reads the same
/// `Track` objects everything else reads. Two parallel copies could disagree,
/// and every divergence would surface as a wrong presence window rather than as
/// a crash — silently, and only in the output.
///
/// **One candidate pool** (AR-008). `last_seen` alone distinguishes a track that
/// is on screen from one that is dormant, and it only affects whether IoU means
/// anything. There is no parked pool and no revival branch: re-associating a
/// track whose face was lost — across a cut or not — is ordinary inter-frame
/// association, and it falls out of the embedding comparison already being done.
///
/// Assignment cost (track i, detection j):
///
/// p = P(same person | cosine(track mean, detection)) ← calibrated
/// alpha = base weight, or 0 when position carries no information
/// cost = alpha·(1 − IoU) + (1 − alpha)·(1 − p)
///
/// gated to INF unless the pair is admissible on position *or* on identity.
///
/// **alpha is frame- and track-dependent** (AR-007). It falls to 0 —
/// embedding only — when either:
/// - the frame is flagged `is_cut` / `is_scene_boundary`: the viewpoint
/// changed, so the same person is at a new position; or
/// - the track is dormant (`last_seen` set): time has passed since its box was
/// last observed, so that box is stale regardless of cuts.
/// Both are the same statement — spatial continuity is broken — arrived at from
/// two directions, which is why they collapse into one rule rather than two
/// branches.
///
/// **Everything is thresholded in probability space** (AR-024). The cosine goes
/// through the calibration before it is compared to anything; the raw-cosine
/// constants `track_max_embed_dist` and `cut_revive_sim` are retired.
#include "types.hpp"
#include "config.hpp"
#include "track_registry.hpp"
#include <algorithm>
#include <cmath>
#include <functional>
#include <iostream>
#include <limits>
#include <map>
#include <memory>
#include <stdexcept>
#include <string_view>
#include <vector>
// ── FaceTrackerFunc ───────────────────────────────────────────────────────────
// KPN node: links face detections across consecutive frames using the Hungarian
// algorithm on a combined spatial (IoU) + embedding (cosine distance) cost.
//
// Each track accumulates a running directional mean of its ArcFace embeddings
// (averaged then re-normalised to the unit sphere), used as the embedding side
// of the assignment cost below for more stable track continuity.
//
// Assignment cost (track i, detection j):
// cost = alpha * (1 - IoU) + (1-alpha) * min(cosine_dist/2, 1)
// Gated to INF when IoU < min_iou AND cosine_dist > max_embed_dist.
//
// Unmatched tracks have their frames_missing counter incremented; they are
// expired once frames_missing > max_frames_missing.
//
// Cross-cut re-association. A camera-angle change (Frame::is_cut, set by
// camera_position_change_detector) destroys spatial (IoU) continuity — the same
// person reappears at a new position — but not identity. On a cut the tracker
// does NOT discard its tracks; it parks them in an inactive pool keyed by their
// last-frame raw embedding. A post-cut detection whose raw cosine similarity to
// a parked track's last-frame embedding is ≥ cut_revive_sim revives that track:
// the original track_id, mean embedding and n_frames are restored (only the bbox
// jumps to the new detection), so identity continuity survives the cut. Parked
// tracks left unrevived for cut_inactive_max_frames are finally dropped.
struct FaceTrackerFunc {
static constexpr std::string_view label() { return "face_tracker"; }
struct TrackState {
cv::Rect2f bbox;
Embedding mean_emb{};
Embedding last_emb{}; // raw embedding of the most recent matched frame
int n_frames{0};
int frames_missing{0};
};
/// cosine similarity → P(same person). Supplied by the caller so the fit
/// belonging to the active embedder is used (AR-023/AR-024) — the same
/// pattern, and normally the same function object, as
/// `EvidenceDiscounter::Calibrate`.
using Calibrate = std::function<float(float)>;
explicit FaceTrackerFunc(const Config& cfg)
: alpha_(cfg.track_alpha)
/// The registry is a constructor argument, not an option: a tracker without
/// one would have to keep its own tracks, which is the defect this replaces.
FaceTrackerFunc(const Config& cfg,
std::shared_ptr<TrackRegistry> registry,
Calibrate calibrate)
: registry_(std::move(registry))
, calibrate_(std::move(calibrate))
, alpha_base_(cfg.track_alpha)
, min_iou_(cfg.track_min_iou)
, max_embed_dist_(cfg.track_max_embed_dist)
, max_missing_(cfg.track_max_frames_missing)
, revive_sim_(cfg.cut_revive_sim)
, inactive_max_(cfg.cut_inactive_max_frames)
, min_assoc_prob_(cfg.track_assoc_min_prob)
{
std::cerr << "[face_tracker] alpha=" << alpha_
if (!registry_)
throw std::invalid_argument("face_tracker: registry must not be null");
if (!calibrate_)
throw std::invalid_argument("face_tracker: a calibration is required — "
"association is decided in probability space");
std::cerr << "[face_tracker] alpha_base=" << alpha_base_
<< " min_iou=" << min_iou_
<< " max_embed_dist=" << max_embed_dist_
<< " max_missing=" << max_missing_
<< " cut_revive_sim=" << revive_sim_
<< " cut_inactive_max=" << inactive_max_ << "\n";
<< " min_assoc_prob=" << min_assoc_prob_ << "\n";
}
TrackedSceneFrame operator()(EmbeddedSceneFrame ef) {
if (ef.source.eof) {
tracks_.clear();
inactive_.clear();
// Deliberately does *not* flush the registry. The identity matcher
// runs downstream and its votes for the final frames are still in
// flight; reaping here would drop them (they would land on ids that
// no longer exist and show up as dropped_votes). AR-016's flush
// belongs at the pipeline's termination point, after the last vote.
boxes_.clear();
TrackedSceneFrame out;
out.source = std::move(ef.source);
return out;
}
const int n_det = static_cast<int>(ef.embeddings.size());
const double t = ef.source.timestamp_sec;
const int n_det = static_cast<int>(ef.embeddings.size());
// Camera-angle change: park active tracks instead of destroying them so
// they can be revived by identity (raw last-frame embedding cosine) once
// the same people reappear from the new angle.
if (ef.source.is_cut && !tracks_.empty()) {
std::cerr << "[face_tracker] cut — parking " << tracks_.size()
<< " track(s) into inactive pool\n";
for (auto& [tid, ts] : tracks_) {
ts.frames_missing = 0; // repurpose as time-since-parked counter
inactive_[tid] = std::move(ts);
}
tracks_.clear();
}
// Unconditional: the clock must advance on frames with no detections
// too, or a track only dies when some unrelated face happens to appear
// and a film that ends mid-track never closes it (AR-013).
auto scope = registry_->begin_frame(t);
// Age the inactive pool every frame and drop tracks parked too long.
for (auto it = inactive_.begin(); it != inactive_.end(); ) {
it->second.frames_missing++;
it = (it->second.frames_missing > inactive_max_)
? inactive_.erase(it) : std::next(it);
}
// One pool (AR-008) — on-screen and dormant tracks compete together.
std::vector<Track*> cands = scope.candidates();
const int n_trk = static_cast<int>(cands.size());
// Snapshot active track IDs so the map can be modified safely below
std::vector<int> tids;
tids.reserve(tracks_.size());
for (auto& [tid, _] : tracks_) tids.push_back(tid);
const int n_trk = static_cast<int>(tids.size());
prune_boxes(cands);
std::vector<Spatial*> sp(n_trk);
for (int ti = 0; ti < n_trk; ++ti)
sp[ti] = &boxes_.try_emplace(cands[ti]->id, Spatial{{}, t, false})
.first->second;
// AR-007 — the frame half of the frame-dependent weighting. Both flags
// say the same thing to the tracker: whatever was at that position is
// not there any more.
const bool viewpoint_change =
ef.source.is_cut || ef.source.is_scene_boundary;
// ── Cost matrix [n_trk × n_det] ──────────────────────────────────────
constexpr float INF_COST = 1e6f;
@@ -104,109 +126,108 @@ struct FaceTrackerFunc {
std::vector<float>(n_det, INF_COST));
for (int ti = 0; ti < n_trk; ++ti) {
const TrackState& ts = tracks_[tids[ti]];
// Spatial continuity holds only for a track that was on screen, whose
// box we have actually observed, on a frame that did not change the
// viewpoint. Otherwise the box is stale and IoU is noise.
const bool spatial_meaningful =
sp[ti]->observed && cands[ti]->on_screen() && !viewpoint_change;
const float alpha = spatial_meaningful ? alpha_base_ : 0.f;
for (int di = 0; di < n_det; ++di) {
float iou_v = iou(ts.bbox, ef.faces[di].bbox);
float emb_d = (ts.n_frames > 0)
? 1.f - cosine_similarity(ts.mean_emb, ef.embeddings[di])
: 1.f;
if (iou_v < min_iou_ && emb_d > max_embed_dist_) continue;
float s = 1.f - iou_v;
float e = std::min(emb_d * 0.5f, 1.f);
cost[ti][di] = alpha_ * s + (1.f - alpha_) * e;
// AR-024 — the cosine is converted before it is used for
// anything, including the gate below.
const float p = calibrate_(
cosine_similarity(cands[ti]->mean, ef.embeddings[di]));
const float iou_v = spatial_meaningful
? iou(sp[ti]->bbox, ef.faces[di].bbox) : 0.f;
// Either signal on its own can admit a link: a face that moved a
// little but whose embedding degraded (blur, profile turn) is
// still linkable on position, and a face that jumped across the
// frame is still linkable on identity. Neither ⇒ no link.
const bool spatial_ok = spatial_meaningful && iou_v >= min_iou_;
const bool identity_ok = p >= min_assoc_prob_;
if (!spatial_ok && !identity_ok) continue;
cost[ti][di] = alpha * (1.f - iou_v) + (1.f - alpha) * (1.f - p);
}
}
// ── Hungarian assignment ──────────────────────────────────────────────
// ── Hungarian assignment ─────────────────────────────────────────────
std::vector<int> assign(n_trk, -1);
if (n_trk > 0 && n_det > 0)
assign = hungarian(cost, n_trk, n_det);
// ── Build output frame ────────────────────────────────────────────────
// ── Build output frame ───────────────────────────────────────────────
TrackedSceneFrame out;
out.source = ef.source;
out.faces = ef.faces;
out.crops = ef.crops;
out.embeddings = ef.embeddings;
out.source = ef.source;
out.faces = ef.faces;
out.crops = ef.crops;
out.embeddings = ef.embeddings;
out.track_ids.assign(n_det, -1);
std::vector<bool> det_matched(n_det, false);
// Update matched tracks
for (int ti = 0; ti < n_trk; ++ti) {
int di = assign[ti];
bool valid = (di >= 0 && di < n_det && cost[ti][di] < INF_COST * 0.5f);
TrackState& ts = tracks_[tids[ti]];
const int di = assign[ti];
const int id = cands[ti]->id;
const bool valid =
(di >= 0 && di < n_det && cost[ti][di] < INF_COST * 0.5f);
if (!valid) {
ts.frames_missing++;
// Only a track that *was* on screen can become lost, and it
// becomes lost as of its last sighting, never as of now — the
// gap after the final sighting is never claimed (AR-013). A
// track already dormant is left alone so its extinction clock
// keeps running from the right instant.
if (cands[ti]->on_screen()) scope.mark_lost(id, sp[ti]->last_ts);
continue;
}
update_mean(ts.mean_emb, ts.n_frames, ef.embeddings[di]);
ts.last_emb = ef.embeddings[di];
ts.bbox = ef.faces[di].bbox;
ts.n_frames++;
ts.frames_missing = 0;
det_matched[di] = true;
out.track_ids[di] = tids[ti];
scope.mark_seen(id, t, ef.embeddings[di]);
sp[ti]->bbox = ef.faces[di].bbox;
sp[ti]->last_ts = t;
sp[ti]->observed = true;
det_matched[di] = true;
out.track_ids[di] = id;
}
// Handle unmatched detections: first try to revive a parked track by
// identity (raw last-frame embedding cosine), else start a fresh track.
for (int di = 0; di < n_det; ++di) {
if (det_matched[di]) continue;
int tid = revive_from_inactive(ef.embeddings[di]);
if (tid >= 0) {
// Restore the parked track: keep its identity statistics
// (mean_emb, n_frames), jump the bbox to the new detection.
TrackState ts = std::move(inactive_[tid]);
inactive_.erase(tid);
update_mean(ts.mean_emb, ts.n_frames, ef.embeddings[di]);
ts.last_emb = ef.embeddings[di];
ts.bbox = ef.faces[di].bbox;
ts.n_frames++;
ts.frames_missing = 0;
tracks_[tid] = std::move(ts);
out.track_ids[di] = tid;
std::cerr << "[face_tracker] revived track " << tid
<< " across cut\n";
continue;
}
tid = next_id_++;
TrackState ts;
ts.bbox = ef.faces[di].bbox;
ts.mean_emb = ef.embeddings[di];
ts.last_emb = ef.embeddings[di];
ts.n_frames = 1;
tracks_[tid] = ts;
out.track_ids[di] = tid;
}
// Expire stale tracks
for (auto it = tracks_.begin(); it != tracks_.end(); ) {
it = (it->second.frames_missing > max_missing_)
? tracks_.erase(it) : std::next(it);
const int id = scope.create(t, ef.embeddings[di]);
boxes_[id] = Spatial{ef.faces[di].bbox, t, true};
out.track_ids[di] = id;
}
// No reaping here: begin_frame's tick owns the extinction sweep, so
// there is exactly one place a track can die.
return out;
}
private:
// Pick the parked track whose last-frame embedding is most similar to emb,
// returning its id if that raw cosine similarity clears revive_sim_, else -1.
// The caller removes the returned track from the pool, so a later detection in
// the same frame cannot claim it again.
int revive_from_inactive(const Embedding& emb) const {
int best_tid = -1;
float best_sim = revive_sim_; // threshold is the bar to beat (inclusive)
for (const auto& [tid, ts] : inactive_) {
float sim = cosine_similarity(ts.last_emb, emb);
if (sim >= best_sim) { best_sim = sim; best_tid = tid; }
// subsequent ties keep the later id; harmless, all clear the threshold
// ── Spatial annotation ───────────────────────────────────────────────────
// The one piece of per-track state the registry does not hold, because it is
// not about presence: where the face was, and when it was last seen there.
// Keyed by registry track id and pruned against `candidates()` every frame,
// so it cannot outlive or contradict the registry — it annotates the pool
// rather than duplicating it.
struct Spatial {
cv::Rect2f bbox{};
double last_ts{0.0}; ///< timestamp of the last frame this track matched
bool observed{false}; ///< false until a detection has been assigned
};
// Drop boxes for ids the registry no longer has. `candidates()` is the
// authority on what exists; anything else is a leak (and, for a reused id,
// would be a stale box attached to a different person).
void prune_boxes(const std::vector<Track*>& cands) {
if (boxes_.size() == cands.size()) return; // common case: nothing died
std::map<int, Spatial> kept;
for (const Track* t : cands) {
auto it = boxes_.find(t->id);
if (it != boxes_.end()) kept.emplace(t->id, it->second);
}
return best_tid;
boxes_.swap(kept);
}
// IoU of two axis-aligned bounding boxes
@@ -220,17 +241,6 @@ private:
return inter / (a.width * a.height + b.width * b.height - inter);
}
// Online directional mean: average then re-normalise to unit sphere
static void update_mean(Embedding& mean, int n_prev, const Embedding& emb) {
float norm_sq = 0.f;
for (int k = 0; k < 512; ++k) {
mean[k] = (mean[k] * n_prev + emb[k]) / (n_prev + 1);
norm_sq += mean[k] * mean[k];
}
float inv = 1.f / std::sqrt(norm_sq);
for (int k = 0; k < 512; ++k) mean[k] *= inv;
}
// O(n³) potential-based Hungarian algorithm (Jonker-Volgenant / Kuhn-Munkres).
// Returns assign[row] = col (0-indexed), or -1 when row is matched to a
// padded virtual column (i.e., unmatched). Rectangular matrices are padded
@@ -292,13 +302,10 @@ private:
return ans;
}
std::map<int, TrackState> tracks_;
std::map<int, TrackState> inactive_; // parked across a cut, keyed by track id
int next_id_{0};
float alpha_;
std::shared_ptr<TrackRegistry> registry_;
Calibrate calibrate_;
std::map<int, Spatial> boxes_; ///< track id → where it was, when
float alpha_base_;
float min_iou_;
float max_embed_dist_;
int max_missing_;
float revive_sim_;
int inactive_max_;
float min_assoc_prob_;
};
src/nodes/identity_matcher_node.hpp
+81 -11
View File
@@ -5,6 +5,7 @@
#include "gallery/gallery_store.hpp"
#include "gallery/gallery_calibration.hpp"
#include "gallery/track_gallery.hpp"
#include "track_registry.hpp"
#include <cstdint>
#include <cstring>
@@ -105,8 +106,28 @@ struct IdentityMatcherFunc {
flat_emb_[i].data(), 512 * sizeof(float));
sim_engine_ = make_similarity_engine(host_gallery.data(), n_gallery_, kMaxFaces);
/// TRACES: AR-018, AR-024 | SR-005
// The expansion store thresholds in the same probability space as
// association and evidence weighting, so a "0.9" means one thing
// pipeline-wide rather than three.
track_gallery_.set_calibration(same_person_probability(cal_));
}
/// TRACES: AR-023, AR-024 | SR-002
/// The fitted sigmoid. Exposed because the matcher is where it gets fitted
/// (and cached back to the gallery), but it is not the matcher's private
/// property: track association and evidence weighting must threshold in the
/// *same* probability space, or a "0.5" in one stage and a "0.5" in another
/// mean different things. See `same_person_probability`.
const GalleryCalibration& calibration() const { return cal_; }
/// TRACES: AR-012, AR-025 | SR-002
/// Where per-frame identity evidence reaches the registry. Optional: with no
/// registry attached the matcher behaves exactly as before, which keeps the
/// replay harness and the unit tests working unchanged.
void set_registry(std::shared_ptr<TrackRegistry> r) { registry_ = std::move(r); }
// Runtime setter — lets a persistent pipeline be reused across a threshold sweep
// without rebuilding the (expensive, gallery-resident) matcher. The gallery,
// calibration and GPU sim-engine stay put; only the accept threshold changes.
@@ -123,27 +144,51 @@ struct IdentityMatcherFunc {
// mix embeddings from two viewpoints under one buffer, so we still drop
// every diversity buffer here — a revived track simply re-accumulates its
// buffer from post-cut frames. Stale cross-cut embeddings are never promoted.
if (tf.source.is_cut) track_gallery_.clear_tracks();
/// TRACES: AR-019 | SR-005
// Promotion may only borrow same-identity evidence from a span where
// identity is certain, so ALL THREE discontinuity signals clear the
// buffers, not just the histogram cut:
// is_cut — camera-angle change
// is_scene_boundary — different scene (AR-010; previously never set,
// so this half of the gate was dead)
// The third, an identity contradiction (AR-015), is enforced by the
// registry: a track whose belief swapped is closed outright, so it can
// no longer promote anything.
if (tf.source.is_cut || tf.source.is_scene_boundary)
track_gallery_.clear_tracks();
const int n_faces = static_cast<int>(tf.embeddings.size());
std::vector<IdentifiedActor> actors;
actors.reserve(n_faces);
if (n_faces == 0) return {std::move(tf.source), {}};
if (n_faces > kMaxFaces)
throw std::runtime_error("identity_matcher: n_faces exceeds kMaxFaces");
std::vector<float> host_query(static_cast<size_t>(n_faces) * 512);
for (int fi = 0; fi < n_faces; ++fi) {
std::memcpy(host_query.data() + static_cast<size_t>(fi) * 512,
tf.embeddings[fi].data(), 512 * sizeof(float));
/// TRACES: AR-003, AR-004 | SR-002
// kMaxFaces sizes the similarity engine's preallocated buffer, so it
// bounds MEMORY, not how many faces a frame may contain. It used to
// throw above the bound, which made it a hard cap on crowd scenes by
// accident; now the frame is scored in batches of that size.
//
// Faces per frame are unbounded (AR-003) because X-Ray credits scene
// membership to background cast too, and a fixed cap discards exactly
// those — the smallest faces are dropped first. Cost is contained by
// backpressure (AR-004), which slows the producer, rather than by
// silently throwing work away.
std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);
for (int base = 0; base < n_faces; base += kMaxFaces) {
const int chunk = std::min(kMaxFaces, n_faces - base);
for (int k = 0; k < chunk; ++k) {
std::memcpy(host_query.data() + static_cast<size_t>(k) * 512,
tf.embeddings[base + k].data(), 512 * sizeof(float));
}
// S (N_gallery × n_faces) col-major: face fi's gallery sims at sims + fi*n_gallery.
const float* host_sims = sim_engine_->compute(host_query.data(), n_faces);
// S (N_gallery × chunk) col-major: face k's gallery sims at sims + k*n_gallery.
const float* host_sims = sim_engine_->compute(host_query.data(), chunk);
for (int fi = 0; fi < n_faces; ++fi) {
const float* sims = host_sims + static_cast<size_t>(fi) * n_gallery_;
for (int ci = 0; ci < chunk; ++ci) {
const int fi = base + ci;
const float* sims = host_sims + static_cast<size_t>(ci) * n_gallery_;
std::vector<float> best_sim(gallery_.actors.size(),
-std::numeric_limits<float>::max());
@@ -224,11 +269,35 @@ struct IdentityMatcherFunc {
// face (annex already folded in above); the track's diversity buffer
// keeps the gallery-far views and promotes them once the track is
// confirmed. No-op unless --expand-gallery is set.
// TRACES: AR-012, AR-025 | SR-002
// Every scored face is evidence, not only the accepted ones: a run of
// near-misses for one actor is itself informative, and discarding it
// would make ownership depend on a per-frame threshold the redesign
// exists to stop relying on. The registry discounts for correlation
// and decides ownership from the accumulated posterior (AR-025).
if (registry_ && best_actor >= 0 && tf.track_ids[fi] >= 0) {
const float p = cal_.valid
? cal_.probability(best_s, log_prior_odds_)
: std::max(0.f, best_s);
registry_->observe(tf.track_ids[fi], best_actor, p, tf.embeddings[fi]);
}
// TRACES: AR-019 | SR-005
// Ownership is the registry's, computed once. TrackGallery used to
// tally its own plurality vote over accepted frames, which meant two
// different answers to "who is this track" could coexist — and the
// expansion one ignored the Bayesian accumulation entirely.
if (registry_ && tf.track_ids[fi] >= 0) {
if (auto owner = registry_->owner(tf.track_ids[fi]))
track_gallery_.set_owner(tf.track_ids[fi], *owner);
}
track_gallery_.observe(tf.track_ids[fi], tf.embeddings[fi],
best_actor, best_s, accept, tf.crops[fi]);
actors.push_back(std::move(ia));
}
} // chunk loop
return {std::move(tf.source), std::move(actors)};
}
@@ -247,4 +316,5 @@ private:
std::unique_ptr<ISimilarityEngine> sim_engine_;
TrackGallery track_gallery_;
std::shared_ptr<TrackRegistry> registry_;
};
src/nodes/result_sink_node.hpp
+86 -34
View File
@@ -1,6 +1,8 @@
#pragma once
/// TRACES: IR-001 | SR-003
#include "types.hpp"
#include "config.hpp"
#include "track_registry.hpp"
#include <nlohmann/json.hpp>
#include <algorithm>
@@ -9,6 +11,8 @@
#include <fstream>
#include <iostream>
#include <map>
#include <mutex>
#include <functional>
#include <string>
#include <vector>
@@ -42,10 +46,26 @@ using json = nlohmann::json;
struct ResultSinkFunc {
static constexpr std::string_view label() { return "result_sink"; }
/// TRACES: AR-012, AR-017 | IR-002 | SR-002, SR-003
/// A finished presence claim from the registry. Called from inside the
/// registry's reap while it holds its own lock, so this must stay a cheap
/// push and must never re-enter the registry.
void add_claim(const DeadTrack& d) {
if (d.actor_idx < 0) return; // never owned: nothing to claim
std::lock_guard<std::mutex> g(claims_mu_);
claims_.push_back(d);
}
ResultSinkFunc(const Config& cfg, std::atomic<bool>& done)
: cfg_(cfg), done_(done)
{}
/// TRACES: AR-016 | SR-002
/// Runs immediately before the output is written, with the last timestamp
/// seen. Used to flush tracks still live at EOF, which have not timed out
/// and would otherwise never be emitted.
void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }
void operator()(SceneAnnotation sa) {
if (sa.eof) {
flush();
@@ -58,12 +78,20 @@ struct ResultSinkFunc {
<< " unknowns=" << count_unknown(sa.visible_actors)
<< std::flush;
for (const auto& ia : sa.visible_actors) {
if (ia.actor_idx < 0) continue;
auto& m = actor_meta_[ia.actor_idx];
if (m.name.empty())
m = {ia.name, ia.imdb_id, ia.tmdb_id, ia.jellyfin_id};
}
last_ts_ = sa.timestamp_sec;
frames_.push_back(std::move(sa));
}
// Write accumulated results and signal done. Safe to call more than once.
void flush() {
if (written_.exchange(true)) return;
if (pre_write_) pre_write_(last_ts_);
write_output();
done_.store(true, std::memory_order_release);
}
@@ -71,7 +99,7 @@ struct ResultSinkFunc {
private:
// Bump when the minimal/standard output JSON structure changes in a way
// the Jellyfin plugin needs to detect.
static constexpr int kSchemaVersion = 1;
static constexpr int kSchemaVersion = 2; // SR-003 coordinated bump
static int count_known(const std::vector<IdentifiedActor>& v) {
int n = 0;
@@ -91,10 +119,19 @@ private:
if (cfg_.verbosity == Verbosity::xray) {
root = build_xray();
} else {
/// TRACES: IR-002 | SR-003
/// schema_version 2, per jRay/SPEC.md JR-002. anneal_sec is REMOVED
/// rather than zeroed: a field naming a mechanism the pipeline no
/// longer has is actively misleading, and would outlive everyone who
/// remembers why it reads 0. extinction_sec succeeds it as the
/// parameter that actually shapes window extent.
root["schema_version"] = kSchemaVersion;
root["movie"] = cfg_.movie_path;
root["sample_fps"] = cfg_.sample_fps;
root["anneal_sec"] = cfg_.anneal_sec;
root["movie"] = cfg_.movie_path;
root["extraction"] = {
{"sample_fps", cfg_.sample_fps},
{"extinction_sec", cfg_.track_extinction_sec},
{"gallery_scope", cfg_.gallery_scope},
};
root["actors"] = build_epochs();
if (cfg_.verbosity == Verbosity::standard)
root["frames"] = build_standard();
@@ -109,42 +146,45 @@ private:
std::cerr << "[result_sink] done.\n";
}
struct Window {
double start{0.0};
double end{0.0};
float belief{0.f}; ///< the posterior that justified the claim (AR-017)
};
struct ActorWindow {
std::string name, imdb_id, tmdb_id, jellyfin_id;
std::vector<std::pair<double, double>> scenes; // [start_sec, end_sec]
std::vector<Window> scenes;
};
struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };
// Core logic: merge per-frame detections into annealed [start, end] windows.
/// TRACES: AR-012 | IR-002 | SR-002
/// A claim already IS a window — `[first_seen, last_seen]` of a track the
/// actor owned. There is no annealing pass: `anneal_sec` existed to bridge
/// gaps between isolated accepted frames, and a track that survives its own
/// gaps leaves it nothing to do (see the AR-012 withdrawal note).
std::vector<ActorWindow> build_actor_windows() {
struct Info { std::string name, imdb_id, tmdb_id, jellyfin_id; };
std::map<int, Info> actor_info;
std::map<int, std::vector<double>> timestamps;
std::lock_guard<std::mutex> g(claims_mu_);
for (const auto& frame : frames_) {
for (const auto& ia : frame.visible_actors) {
if (ia.actor_idx < 0) continue;
actor_info[ia.actor_idx] = {ia.name, ia.imdb_id, ia.tmdb_id, ia.jellyfin_id};
timestamps[ia.actor_idx].push_back(frame.timestamp_sec);
std::map<int, ActorWindow> by_actor;
for (const auto& c : claims_) {
auto& aw = by_actor[c.actor_idx];
if (aw.name.empty()) {
auto it = actor_meta_.find(c.actor_idx);
if (it != actor_meta_.end()) {
aw.name = it->second.name;
aw.imdb_id = it->second.imdb_id;
aw.tmdb_id = it->second.tmdb_id;
aw.jellyfin_id = it->second.jellyfin_id;
}
}
aw.scenes.push_back({c.first_seen, c.last_seen, c.belief});
}
std::vector<ActorWindow> result;
for (auto& [idx, ts_vec] : timestamps) {
ActorWindow aw;
aw.name = actor_info[idx].name;
aw.imdb_id = actor_info[idx].imdb_id;
aw.tmdb_id = actor_info[idx].tmdb_id;
aw.jellyfin_id = actor_info[idx].jellyfin_id;
double win_start = ts_vec[0], win_end = ts_vec[0];
for (size_t i = 1; i < ts_vec.size(); ++i) {
if (ts_vec[i] - win_end > cfg_.anneal_sec) {
aw.scenes.push_back({win_start, win_end});
win_start = ts_vec[i];
}
win_end = ts_vec[i];
}
aw.scenes.push_back({win_start, win_end});
for (auto& [idx, aw] : by_actor) {
std::sort(aw.scenes.begin(), aw.scenes.end(),
[](const Window& a, const Window& b) { return a.start < b.start; });
result.push_back(std::move(aw));
}
return result;
@@ -153,9 +193,16 @@ private:
json build_epochs() {
json actors = json::array();
for (const auto& aw : build_actor_windows()) {
// Objects, not float pairs: a window carries the belief that
// justified it and the route by which it was identified (AR-017),
// so a consumer can caveat or filter rather than treating every
// window as equally certain.
json windows = json::array();
for (const auto& [s, e] : aw.scenes)
windows.push_back({s, e});
for (const auto& w : aw.scenes)
windows.push_back({{"start", w.start},
{"end", w.end},
{"belief", w.belief},
{"route", "live"}});
json ja;
ja["name"] = aw.name;
ja["imdb_id"] = aw.imdb_id;
@@ -173,9 +220,9 @@ private:
json build_xray() {
std::map<int, std::vector<std::string>> xray;
for (const auto& aw : build_actor_windows()) {
for (const auto& [start, end] : aw.scenes) {
int t0 = static_cast<int>(std::floor(start));
int t1 = static_cast<int>(std::ceil(end));
for (const auto& w : aw.scenes) {
int t0 = static_cast<int>(std::floor(w.start));
int t1 = static_cast<int>(std::ceil(w.end));
for (int t = t0; t <= t1; ++t)
xray[t].push_back(aw.name);
}
@@ -226,4 +273,9 @@ private:
std::atomic<bool>& done_;
std::atomic<bool> written_{false};
std::vector<SceneAnnotation> frames_;
std::function<void(double)> pre_write_;
double last_ts_{0.0};
std::mutex claims_mu_;
std::vector<DeadTrack> claims_;
std::map<int, ActorMeta> actor_meta_; ///< actor_idx → identity keys
};
src/nodes/scene_boundary_annotator_node.hpp
+67
View File
@@ -0,0 +1,67 @@
#pragma once
/// TRACES: AR-010 | SR-002
///
/// SceneBoundaryAnnotatorFunc — the join of the decode butterfly.
///
/// `source` fans out to two branches: dense frames to TransNetV2, sampled frames
/// to face detection. A boundary found on the first has to reach the second, and
/// cannot ride along in the frame because the branches run in parallel.
///
/// This node sits on the sampled branch and stamps `Frame::is_scene_boundary`
/// from the detector's published verdict.
///
/// **It only works because the sampled branch lags.** TransNetV2 buffers
/// `kWindow` frames before it can score any of them, so this node must not reach
/// a frame before the detector has an opinion about it. Channel depth creates
/// that lag: with backpressure (AR-004) the fanout blocks on the slower branch,
/// so a deep channel here lets the detector run ahead by its window instead of
/// anything being dropped.
///
/// When the lag is insufficient the node **counts it** rather than guessing.
/// Annotating an unscored frame as boundary-free is indistinguishable from a
/// genuine "no boundary here", and that is the failure that makes a downstream
/// test pass while verifying nothing.
#include "scene_boundaries.hpp"
#include "types.hpp"
#include <memory>
#include <string_view>
#include <utility>
struct SceneBoundaryAnnotatorFunc {
static constexpr std::string_view label() { return "scene_annotate"; }
/// `tol` is half a sample interval. The branches sample at different rates,
/// so a boundary found on a dense frame rarely lands exactly on a sampled
/// one; half an interval attributes it to the nearest sampled frame and no
/// further.
SceneBoundaryAnnotatorFunc(std::shared_ptr<SceneBoundaries> b, double tol)
: bounds_(std::move(b)), tol_(tol) {}
Frame operator()(Frame f) {
if (f.eof || !bounds_) return f;
// Wait for the detector's verdict to cover this frame. Channel depth
// alone cannot provide the lag: it holds frames back only when the
// consumer is slower, and this branch is orders of magnitude faster per
// frame than TransNetV2. Blocking here is what makes the join real.
//
// Safe under backpressure because the branches are independent: this
// node stalling does not stop the detector consuming dense frames, and
// the fanout keeps feeding it.
if (!bounds_->wait_until_scored(f.timestamp_sec)) {
// The detector finished without covering this frame — the tail after
// its last full window. Unknown, not negative; counted so it cannot
// pass for "no boundary here".
bounds_->note_outran();
return f;
}
f.is_scene_boundary = bounds_->is_boundary(f.timestamp_sec, tol_);
return f;
}
private:
std::shared_ptr<SceneBoundaries> bounds_;
double tol_{0.0};
};
src/nodes/scene_detector_node.hpp
+35 -2
View File
@@ -1,6 +1,9 @@
#pragma once
#include "types.hpp"
#include "config.hpp"
#include "scene_boundaries.hpp"
#include <memory>
#include "inference/scene_detector.hpp"
#include <nlohmann/json.hpp>
@@ -30,6 +33,11 @@
struct SceneDetectorFunc {
static constexpr std::string_view label() { return "scene_detector"; }
/// TRACES: AR-010 | SR-002
/// Publish each window's verdict as it is scored, so the face branch — held
/// back by channel depth — can consult it for frames it has not reached yet.
void set_boundaries(std::shared_ptr<SceneBoundaries> b) { shared_ = std::move(b); }
SceneDetectorFunc(const Config& cfg, std::atomic<bool>& done)
: detector_(make_scene_detector(cfg))
, threshold_(cfg.scene_threshold)
@@ -51,6 +59,9 @@ struct SceneDetectorFunc {
void operator()(Frame f) {
if (f.eof) {
flush_remaining();
// Release anyone waiting on the join: the tail frames after the last
// full window will never be covered, so waiting for them would hang.
if (shared_) shared_->finish();
write_output();
done_.store(true, std::memory_order_release);
return;
@@ -82,6 +93,7 @@ private:
// otherwise skip the leading guard already covered by the previous window.
const int lo = (window_base_ == 0) ? 0 : guard_;
const int hi = ISceneDetector::kWindow - guard_;
std::vector<double> fresh;
for (int i = lo; i < hi; ++i) {
if (probs[i] <= threshold_) continue;
// Local maximum → the boundary frame (avoid a run of high scores
@@ -89,9 +101,19 @@ private:
const bool peak =
(i == 0 || probs[i] >= probs[i-1]) &&
(i == kLast_() || probs[i] >= probs[i+1]);
if (peak)
if (peak) {
boundaries_.push_back({times_[i], probs[i]});
fresh.push_back(times_[i]);
}
}
/// TRACES: AR-010 | SR-002
// Publish with a watermark: everything up to times_[hi-1] now has a
// final verdict. The face branch consults this for frames it has not
// reached yet, and the watermark is what lets it tell "no boundary
// here" from "not scored yet".
if (shared_ && hi > lo)
shared_->publish(fresh, times_[hi - 1]);
}
// At EOF the tail (< kWindow frames) never formed a full window. Pad it out
@@ -107,14 +129,24 @@ private:
std::vector<float> probs = detector_->detect_window(win);
const int lo = (window_base_ == 0) ? 0 : guard_;
std::vector<double> fresh;
for (int i = lo; i < n; ++i) { // only real (non-padded) frames
if (probs[i] <= threshold_) continue;
const bool peak =
(i == 0 || probs[i] >= probs[i-1]) &&
(i == n - 1 || probs[i] >= probs[i+1]);
if (peak)
if (peak) {
boundaries_.push_back({times_[i], probs[i]});
fresh.push_back(times_[i]);
}
}
/// TRACES: AR-010 | SR-002
// Publish the tail too. Without this the final frames — everything after
// the last full window — reach the join with no verdict and are treated
// as boundary-free without evidence, which is precisely the ambiguity
// the watermark exists to prevent.
if (shared_ && n > 0) shared_->publish(fresh, times_[n - 1]);
}
void write_output() {
@@ -176,4 +208,5 @@ private:
int64_t window_base_{0}; // frame index of images_.front()
std::vector<Boundary> boundaries_;
bool written_{false};
std::shared_ptr<SceneBoundaries> shared_; ///< AR-010 join point
};
src/python_bindings.cpp
+259 -3
View File
@@ -3,17 +3,78 @@
// Loads both ONNX sessions once per FaceEmbedder instance, then embeds many
// images via repeated embed() calls — avoiding the per-process model-load
// cost of the embed_faces CLI when embedding a large gallery.
//
// Beyond whole-image embed(), the individual pipeline stages are exposed —
// detect(), align_face(), embed_crop() — plus the gallery calibration. A study
// that needs to step between stages (a different landmark source, a degraded
// crop) drives the shipped C++ from Python rather than re-implementing
// detection, alignment, the ArcFace warp or the Platt fit in numpy. Those
// re-implementations drift from what ships, and the calibration is the one
// that must not: AR-024 requires every similarity to pass through
// GalleryCalibration::probability, never a bare cosine.
#include "face_embedder_engine.hpp"
#include "gallery/gallery_calibration.hpp"
#include "gallery/gallery_store.hpp"
#include "quality.hpp"
#include <nanobind/nanobind.h>
#include <nanobind/ndarray.h>
#include <nanobind/stl/optional.h>
#include <nanobind/stl/string.h>
#include <nanobind/stl/vector.h>
#include <array>
#include <cstring>
#include <stdexcept>
namespace nb = nanobind;
using namespace nb::literals;
namespace {
using ImageArray = nb::ndarray<const uint8_t, nb::ndim<3>, nb::c_contig, nb::device::cpu>;
// numpy HxWx3 uint8 (BGR, as cv::imread yields) → cv::Mat sharing that buffer.
// The Mat is a view: it must not outlive the caller's array, so every use here
// copies or consumes it before returning.
cv::Mat as_mat(const ImageArray& a) {
if (a.shape(2) != 3)
throw std::invalid_argument("expected an HxWx3 uint8 BGR image");
return cv::Mat(static_cast<int>(a.shape(0)), static_cast<int>(a.shape(1)),
CV_8UC3, const_cast<uint8_t*>(a.data()));
}
// cv::Mat → freshly-allocated numpy array (owns its buffer).
nb::ndarray<nb::numpy, uint8_t> mat_to_numpy(const cv::Mat& m) {
cv::Mat c = m.isContinuous() ? m : m.clone();
auto* buf = new uint8_t[c.total() * c.elemSize()];
std::memcpy(buf, c.data, c.total() * c.elemSize());
nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast<uint8_t*>(p); });
size_t shape[3] = {static_cast<size_t>(c.rows), static_cast<size_t>(c.cols),
static_cast<size_t>(c.channels())};
return nb::ndarray<nb::numpy, uint8_t>(buf, 3, shape, owner);
}
nb::ndarray<nb::numpy, float> vec_to_numpy(std::vector<float>&& v) {
auto* buf = new float[v.size()];
std::memcpy(buf, v.data(), v.size() * sizeof(float));
nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast<float*>(p); });
size_t shape[1] = {v.size()};
return nb::ndarray<nb::numpy, float>(buf, 1, shape, owner);
}
// numpy (5,2) float32 → the landmark array align_face expects. Order is
// types.hpp:60 — [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth.
std::array<cv::Point2f, 5> as_landmarks(
const nb::ndarray<const float, nb::shape<5, 2>, nb::c_contig, nb::device::cpu>& a) {
std::array<cv::Point2f, 5> lm;
for (int i = 0; i < 5; ++i) lm[i] = {a(i, 0), a(i, 1)};
return lm;
}
} // namespace
NB_MODULE(sae_embed, m) {
m.doc() = "SCRFD + ArcFace face embedding, models loaded once per FaceEmbedder";
@@ -27,14 +88,209 @@ NB_MODULE(sae_embed, m) {
})
.def_prop_ro("bbox", [](const FaceEmbedResult& r) {
return std::vector<float>{r.bbox[0], r.bbox[1], r.bbox[2], r.bbox[3]};
})
.def_prop_ro("landmarks", [](const FaceEmbedResult& r) {
std::vector<float> v;
for (const auto& p : r.landmarks) { v.push_back(p.x); v.push_back(p.y); }
return v;
});
nb::class_<DetectedFace>(m, "Detection")
.def_ro("confidence", &DetectedFace::confidence)
.def_prop_ro("bbox", [](const DetectedFace& d) {
return std::vector<float>{d.bbox.x, d.bbox.y, d.bbox.width, d.bbox.height};
})
.def_prop_ro("landmarks", [](const DetectedFace& d) {
// (5,2): [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth
std::vector<float> v;
for (const auto& p : d.landmarks) { v.push_back(p.x); v.push_back(p.y); }
return v;
});
nb::class_<FaceEmbedderEngine>(m, "FaceEmbedder")
.def(nb::init<std::string, std::string, float, float, int>(),
.def(nb::init<std::string, std::string, float, float, int,
std::string, std::string>(),
"detector_model"_a, "arcface_model"_a,
"conf"_a = 0.5f, "nms"_a = 0.4f, "max_side"_a = 500)
"conf"_a = 0.5f, "nms"_a = 0.4f, "max_side"_a = 500,
"detector_engine"_a = "", "arcface_engine"_a = "")
.def("embed", &FaceEmbedderEngine::embed_path, "path"_a,
nb::call_guard<nb::gil_scoped_release>(),
"Detect the highest-confidence face in the image, align it, and "
"return a FaceResult with its 512-d ArcFace embedding.");
"return a FaceResult with its 512-d ArcFace embedding.")
.def("embed_mat", [](FaceEmbedderEngine& e, ImageArray img) {
return e.embed_mat(as_mat(img).clone());
}, "image"_a,
"As embed(), on an in-memory HxWx3 uint8 BGR array.")
.def("detect", [](FaceEmbedderEngine& e, ImageArray img) {
return e.detect(as_mat(img));
}, "image"_a,
"Run the configured detector. Returns every Detection, unfiltered — "
"min_face_px is applied downstream in face_detector_node.")
.def("embed_crop", [](FaceEmbedderEngine& e, ImageArray crop) {
cv::Mat c = as_mat(crop);
if (c.rows != 112 || c.cols != 112)
throw std::invalid_argument("embed_crop expects a 112x112 aligned crop");
Embedding emb = e.embed_crop(c);
return vec_to_numpy(std::vector<float>(emb.begin(), emb.end()));
}, "crop"_a,
"Embed a caller-supplied 112x112 aligned BGR crop. The stage-level "
"entry point for studies that degrade or re-align a crop themselves.")
.def("embed_crops", [](FaceEmbedderEngine& e,
nb::ndarray<const uint8_t, nb::ndim<4>, nb::c_contig,
nb::device::cpu> crops) {
if (crops.shape(1) != 112 || crops.shape(2) != 112 || crops.shape(3) != 3)
throw std::invalid_argument("embed_crops expects (N,112,112,3) uint8 BGR");
const size_t n = crops.shape(0);
std::vector<cv::Mat> mats;
mats.reserve(n);
for (size_t i = 0; i < n; ++i)
mats.emplace_back(112, 112, CV_8UC3,
const_cast<uint8_t*>(crops.data()) + i * 112 * 112 * 3);
std::vector<Embedding> out = e.embed_crops(mats);
auto* buf = new float[n * 512];
for (size_t i = 0; i < n; ++i)
std::memcpy(buf + i * 512, out[i].data(), 512 * sizeof(float));
nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast<float*>(p); });
size_t shape[2] = {n, 512};
return nb::ndarray<nb::numpy, float>(buf, 2, shape, owner);
}, "crops"_a,
"Batched embed_crop: (N,112,112,3) uint8 BGR in, (N,512) float32 out. "
"The backend batches internally, so this avoids paying per-call "
"overhead once per crop across a large study.")
.def_prop_ro("max_batch", [](FaceEmbedderEngine& e) { return e.max_batch(); });
m.def("align_face", [](ImageArray img,
nb::ndarray<const float, nb::shape<5, 2>, nb::c_contig,
nb::device::cpu> landmarks)
-> std::optional<nb::ndarray<nb::numpy, uint8_t>> {
cv::Mat crop = ::align_face(as_mat(img), as_landmarks(landmarks));
if (crop.empty()) return std::nullopt; // degenerate fit
return mat_to_numpy(crop);
}, "image"_a, "landmarks"_a,
"The ArcFace 5-point similarity transform (face_utils.hpp, AR-005). "
"Returns a 112x112 BGR crop, or None if the affine fit is degenerate. "
"Landmark order is types.hpp:60 — right-eye, left-eye, nose, "
"right-mouth, left-mouth.");
m.def("enhance_for_retry", [](ImageArray img) {
return mat_to_numpy(::enhance_for_retry(as_mat(img)));
}, "image"_a,
"Border-replicate pad by 50% and CLAHE, for a detector second try.");
// ── Quality (AR-028 … AR-030) ────────────────────────────────────────────
// Exposed for the same reason the calibration is: VR-012 has to select
// among the AR-029 candidates, and the measure it selects must be the one
// that ships. A numpy copy scored during the study would leave the shipped
// measure unmeasured, which is precisely the failure the whole quality axis
// exists to prevent.
nb::class_<SharpnessScores>(m, "SharpnessScores")
.def_ro("var_laplacian", &SharpnessScores::var_laplacian)
.def_ro("norm_var_laplacian", &SharpnessScores::norm_var_laplacian)
.def_ro("tenengrad", &SharpnessScores::tenengrad)
.def_ro("hf_energy_ratio", &SharpnessScores::hf_energy_ratio)
.def_ro("dir_min_tenengrad", &SharpnessScores::dir_min_tenengrad)
.def_ro("ok", &SharpnessScores::ok)
.def("__repr__", [](const SharpnessScores& s) {
return "<SharpnessScores varlap=" + std::to_string(s.var_laplacian) +
" normvarlap=" + std::to_string(s.norm_var_laplacian) +
" tenengrad=" + std::to_string(s.tenengrad) +
" hf=" + std::to_string(s.hf_energy_ratio) +
(s.ok ? ">" : " NOT-OK>");
});
m.def("assess_sharpness", [](ImageArray crop) {
return ::assess_sharpness(as_mat(crop));
}, "crop"_a,
"All four AR-029 sharpness candidates for a 112x112 aligned crop "
"(quality.hpp). Higher is sharper for every measure; scales are not "
"comparable between measures. Scored over a fixed 64x64 window on the "
"face interior, so background bokeh and hairstyle do not enter.");
m.def("sharpness_window", [] {
const cv::Rect w = ::sharpness_window();
return std::vector<int>{w.x, w.y, w.width, w.height};
},
"The (x, y, w, h) canonical-pixel window every sharpness measure is "
"taken over, so a study can show the pixels a score came from.");
m.def("alignment_residual", [](nb::ndarray<const float, nb::shape<5, 2>,
nb::c_contig, nb::device::cpu> landmarks)
-> std::optional<float> {
const Alignment a = ::estimate_alignment(as_landmarks(landmarks));
if (!a.ok) return std::nullopt; // degenerate landmarks
return a.residual;
}, "landmarks"_a,
"The AR-030 visibility measure: RMS landmark error in canonical "
"112x112 px left over after the best similarity fit onto the ArcFace "
"template (face_utils.hpp). None when the landmarks are degenerate. "
"Exposed so VR-012 can check whether blur leaks into the pose axis — "
"if it does, discounting on both would double-count one cause.");
// ── Calibration ──────────────────────────────────────────────────────────
// AR-024: the pipeline reasons in one probability space. Exposed so Python
// scores through the same sigmoid the C++ matcher uses, rather than a numpy
// copy of it that can silently disagree.
nb::class_<GalleryCalibration>(m, "GalleryCalibration")
.def_ro("a", &GalleryCalibration::a)
.def_ro("b", &GalleryCalibration::b)
.def_ro("valid", &GalleryCalibration::valid)
.def("probability", &GalleryCalibration::probability,
"similarity"_a, "log_prior_odds"_a = 0.f,
"P(match | sim) = sigma(a*sim + b + log_prior_odds). Pass "
"log_prior_odds = log(p0/(1-p0)) for a base-rate prior p0; leave it "
"at 0 for association (is this one person), which is what the "
"balanced fit answers — see gallery_calibration.hpp:63.")
.def("boundary_at", &GalleryCalibration::boundary_at,
"p"_a = 0.5f, "log_prior_odds"_a = 0.f,
"The similarity at which P(match) == p. Diagnostic only — decisions "
"threshold the probability, not this.")
.def("__repr__", [](const GalleryCalibration& c) {
return "<GalleryCalibration a=" + std::to_string(c.a) +
" b=" + std::to_string(c.b) +
(c.valid ? " valid>" : " INVALID>");
});
m.def("gallery_calibration", [](const std::string& gallery_path) {
ActorGallery g = load_gallery(gallery_path);
if (g.calib_valid) {
std::cerr << "[calibration] " << gallery_path << ": cached fit"
<< " over " << g.actors.size() << " actors\n";
return GalleryCalibration{g.calib_a, g.calib_b, true};
}
// Legacy JSON galleries carry no stored fit; compute it over the
// whole gallery, which is the point — the calibration must come
// from the production actor population, not a handful of people.
std::cerr << "[calibration] " << gallery_path
<< ": no cached fit, computing over " << g.actors.size()
<< " actors\n";
std::vector<Embedding> flat;
std::vector<int> actor;
for (size_t a = 0; a < g.actors.size(); ++a)
for (const auto& e : g.actors[a].embeddings) {
flat.push_back(e);
actor.push_back(static_cast<int>(a));
}
return ::calibrate_gallery(flat, actor);
}, "gallery_path"_a,
"The production gallery's calibration — the global fit over every "
"actor in it. Use this to score, not a fit over a handful of people: "
"a sigmoid fitted on a few identities saturates, so its probabilities "
"mean nothing. Reads the cached fit stored in an HDF5 gallery, or "
"computes it over the whole gallery for a legacy JSON one.");
m.def("calibrate_gallery", [](nb::ndarray<const float, nb::shape<-1, 512>, nb::c_contig,
nb::device::cpu> emb,
std::vector<int> actor) {
const size_t n = emb.shape(0);
if (actor.size() != n)
throw std::invalid_argument("embeddings and actor ids differ in length");
std::vector<Embedding> flat(n);
for (size_t i = 0; i < n; ++i)
std::memcpy(flat[i].data(), &emb(i, 0), 512 * sizeof(float));
return ::calibrate_gallery(flat, actor);
}, "embeddings"_a, "actor_ids"_a,
"Fit the Platt sigmoid from intra/inter-class pairs — the same fit the "
"gallery build performs (gallery_calibration.hpp:85). embeddings is "
"(N,512) L2-normalised float32; actor_ids is a length-N list of "
"0-based actor indices.");
}
src/quality.hpp
+256
View File
@@ -0,0 +1,256 @@
#pragma once
/// TRACES: AR-028, AR-029 | SR-002
///
/// Sharpness of the aligned crop — candidate measures for AR-029.
///
/// Motion blur and soft focus destroy the high-frequency detail the embedder
/// keys on, and unlike face size they leave the bounding box looking perfectly
/// healthy. An embedder handed such a face does not fail: it returns a
/// confident, plausible, wrong vector that then competes on equal terms with
/// every good one in the gallery.
///
/// **Why four measures and not one.** AR-029's threshold has to be *located*,
/// the way VR-005 located the size floor, not chosen. Locating it means letting
/// a study rank candidates by how well each predicts real identity loss, so all
/// four ship and VR-012 picks the winner. Until that study reports, none of
/// these is "the" sharpness measure.
///
/// **They are computed on the 112×112 aligned crop**, never the raw box. The
/// crop is geometrically scale-normalised, so a measure taken there cannot
/// re-express face size the way a raw-pixel one would.
///
/// That normalisation is geometric, not informational, and the distinction
/// matters: a 40 px face upscaled into the canonical frame genuinely carries
/// less high-frequency detail than a 400 px one downscaled into it, so every
/// measure here *does* respond to source face size. It reads **effective
/// resolution in canonical space**, which is the union of "was small" and "was
/// blurred", not blur alone. Whether that makes a sharpness discount a
/// double-count against AR-002's size gate is VR-012's joint size×sigma grid to
/// settle: if identity loss is a function of the measure alone, one axis
/// suffices; if a small-but-sharp and a large-but-blurred face at equal measure
/// lose different amounts, the axes are genuinely separate. The unit test
/// `sharpness falls under downscale-upscale as well as under blur` pins this as
/// known behaviour rather than leaving it to be discovered as a surprise.
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <cmath>
// ── The measurement window ────────────────────────────────────────────────────
// All four measures see the same pixels, so a comparison between them is about
// the operator and not about the window each happened to pick.
//
// A 64×64 region centred on the face interior, not the whole crop. Under the
// ArcFace template the landmarks span x ∈ [38.3, 73.5], y ∈ [51.5, 92.4]; this
// window covers that plus the surrounding cheeks, brow and chin while excluding
// the corners.
//
// The corners are excluded because they are where the background lives, and
// studio headshots — the gallery's entire population — are very often shot at a
// wide aperture with a deliberately blurred background. Measured over the full
// crop, that bokeh drags the score down on exactly the sharpest, most
// cooperative images in the set, which would put the measure's response
// backwards on the population used to calibrate it. Hair is excluded for the
// weaker version of the same reason: its high-frequency content varies with
// hairstyle rather than with capture quality.
//
// 64 is also a power of two, so the DFT below gets its natural size.
inline constexpr int kSharpWindow = 64;
inline constexpr int kSharpWindowX = 24; // (24,36) … (88,100) in canonical px
inline constexpr int kSharpWindowY = 36;
/// Every candidate, computed in one pass over the window.
///
/// Higher is sharper for all four, so a discount curve has the same orientation
/// whichever one VR-012 selects. Scales are *not* comparable between measures —
/// only within one.
struct SharpnessScores {
/// Variance of the Laplacian. The textbook measure, included as the
/// baseline every other candidate has to beat. Second derivatives amplify
/// sensor noise, and the value scales with image contrast, so a
/// low-contrast sharp face reads as blurred. Expected to lose; it should
/// lose on the record rather than by assertion.
float var_laplacian{0.f};
/// Variance of the Laplacian over the variance of the intensity. Divides
/// out the first-order contrast dependence that var_laplacian carries,
/// which is the single confound most likely to matter on a gallery drawn
/// from thousands of different cameras, lighting setups and JPEG pipelines.
float norm_var_laplacian{0.f};
/// Tenengrad: mean squared Sobel gradient magnitude. A first derivative, so
/// markedly less noise-amplifying than the Laplacian, at the cost of
/// responding to coarser structure. Still contrast-dependent.
float tenengrad{0.f};
/// Fraction of spectral energy above a quarter of Nyquist, DC excluded.
/// A ratio, so contrast divides out by construction rather than by an
/// explicit correction, and it is the most direct statement of "how much
/// fine detail is actually present". Bounded in [0,1], which makes it the
/// easiest of the four to turn into a discount.
float hf_energy_ratio{0.f};
/// The worse of the two Sobel axes, normalised by a low-frequency contrast
/// estimate. The only candidate here that satisfies both requirements at
/// once, and it exists because the other four do not.
///
/// Two independent fixes, each answering a measured failure of the four
/// above (numbers from the T1 ladders in tests/test_quality.cpp):
///
/// - **Normalise by low frequencies, not by total energy.** Dividing by
/// the whole intensity variance puts the detail being measured into the
/// denominator as well as the numerator, so a blur shrinks both and the
/// quotient barely moves. A Gaussian at sigma 4 canonical px keeps
/// illumination and coarse facial structure and discards detail, giving
/// a contrast estimate that blur leaves alone.
/// - **Take the minimum over direction, not the sum.** Motion blur is
/// directional: a horizontal smear destroys horizontal detail and
/// leaves vertical detail untouched. Summing the two axes (as
/// Tenengrad does) lets the surviving axis mask the destroyed one — the
/// reason both ratio measures are U-shaped in blur length, scoring a
/// 21 px smear about as sharp as a 3 px one. The minimum tracks the
/// axis that was ruined, which is the one the embedder suffers from.
///
/// Falls 510 → 12 monotonically across that same motion-blur ladder, stays
/// monotone under Gaussian blur and resampling, and moves 0.4% when
/// contrast is halved.
float dir_min_tenengrad{0.f};
/// False when the crop was the wrong size or degenerate (flat). Scored,
/// never silently dropped: a face whose sharpness cannot be computed is a
/// fact the dump should record, not an absence.
bool ok{false};
};
/// The window every measure is taken over. Exposed so a study can show the
/// pixels a score was computed from rather than trusting the constants.
inline cv::Rect sharpness_window() {
return {kSharpWindowX, kSharpWindowY, kSharpWindow, kSharpWindow};
}
namespace detail {
/// Fraction of spectral energy above `cutoff` × Nyquist, DC bin excluded.
///
/// A Hann window is applied first. Without it the DFT sees the region's edges
/// as a step discontinuity, and that step is broadband: it deposits energy at
/// every frequency including the high band being measured, so a uniformly
/// blurry crop still scores a substantial high-frequency fraction and the
/// measure's dynamic range collapses.
///
/// **The mean is removed before the window, not after.** Windowing a signal
/// that still carries its DC offset multiplies that constant by the Hann taper,
/// and the taper's own spectrum is not a single bin — the offset smears across
/// the low-frequency neighbourhood, where dropping bin (0,0) no longer removes
/// it. The leaked energy lands in the denominator without scaling with image
/// contrast, so the "ratio" silently becomes a function of absolute brightness:
/// on the synthetic crop, a 20/255 brightening moved it 23% and halving the
/// contrast moved it by a factor of 3.6. Subtracting the mean first restores
/// the invariance the ratio form is supposed to provide for free.
inline float hf_ratio(const cv::Mat& gray32, float cutoff = 0.25f) {
static const cv::Mat hann = [] {
cv::Mat w(kSharpWindow, kSharpWindow, CV_32F);
for (int y = 0; y < kSharpWindow; ++y) {
const float wy = 0.5f * (1.f - std::cos(2.f * float(CV_PI) * y / (kSharpWindow - 1)));
for (int x = 0; x < kSharpWindow; ++x) {
const float wx = 0.5f * (1.f - std::cos(2.f * float(CV_PI) * x / (kSharpWindow - 1)));
w.at<float>(y, x) = wx * wy;
}
}
return w;
}();
cv::Mat centred;
cv::subtract(gray32, cv::mean(gray32), centred);
cv::Mat windowed;
cv::multiply(centred, hann, windowed);
cv::Mat spectrum;
cv::dft(windowed, spectrum, cv::DFT_COMPLEX_OUTPUT);
// Quadrants are wrapped: frequency index n maps to the signed frequency
// n - N for n > N/2, so the radius has to be computed on the wrapped index.
const int N = kSharpWindow;
const float nyquist = N / 2.f;
const float r_cut = cutoff * nyquist;
double total = 0.0, high = 0.0;
for (int y = 0; y < N; ++y) {
const float fy = (y <= N / 2) ? float(y) : float(y - N);
for (int x = 0; x < N; ++x) {
if (x == 0 && y == 0) continue; // DC carries no detail
const float fx = (x <= N / 2) ? float(x) : float(x - N);
const auto& c = spectrum.at<cv::Vec2f>(y, x);
const double e = double(c[0]) * c[0] + double(c[1]) * c[1];
total += e;
if (std::sqrt(fx * fx + fy * fy) > r_cut) high += e;
}
}
if (total < 1e-12) return 0.f; // flat region
return static_cast<float>(high / total);
}
} // namespace detail
/// Score a 112×112 aligned BGR (or single-channel) crop on all four candidates.
///
/// Costs one colour conversion and three small convolutions over a 64×64 window
/// — negligible beside the embedder inference it guards.
inline SharpnessScores assess_sharpness(const cv::Mat& crop) {
SharpnessScores s;
const cv::Rect win = sharpness_window();
if (crop.empty() ||
win.x + win.width > crop.cols || win.y + win.height > crop.rows)
return s;
cv::Mat gray;
if (crop.channels() == 3) cv::cvtColor(crop(win), gray, cv::COLOR_BGR2GRAY);
else gray = crop(win).clone();
// Scale into [0,1] so a score does not depend on the 8-bit convention, and
// so the two contrast-normalised measures are comparable across builds.
cv::Mat g32;
gray.convertTo(g32, CV_32F, 1.0 / 255.0);
cv::Scalar mu, sigma;
cv::meanStdDev(g32, mu, sigma);
const double var_img = sigma[0] * sigma[0];
cv::Mat lap;
cv::Laplacian(g32, lap, CV_32F, 3);
cv::Scalar lmu, lsigma;
cv::meanStdDev(lap, lmu, lsigma);
const double var_lap = lsigma[0] * lsigma[0];
cv::Mat gx, gy;
cv::Sobel(g32, gx, CV_32F, 1, 0, 3);
cv::Sobel(g32, gy, CV_32F, 0, 1, 3);
cv::Mat gx2, gy2;
cv::multiply(gx, gx, gx2);
cv::multiply(gy, gy, gy2);
cv::Mat mag2 = gx2 + gy2;
// Contrast from low frequencies only — see dir_min_tenengrad. Blur leaves
// this denominator alone, which is exactly what the other two normalised
// measures lack.
cv::Mat lf;
cv::GaussianBlur(g32, lf, cv::Size(0, 0), 4.0);
cv::Scalar lfmu, lfsigma;
cv::meanStdDev(lf, lfmu, lfsigma);
const double var_lf = lfsigma[0] * lfsigma[0];
s.var_laplacian = static_cast<float>(var_lap);
// A flat window has no contrast to normalise by. Reporting 0 (rather than a
// huge quotient) keeps "less sharp" pointing the same way for a degenerate
// input as for a blurred one.
s.norm_var_laplacian = var_img > 1e-9 ? static_cast<float>(var_lap / var_img) : 0.f;
s.tenengrad = static_cast<float>(cv::mean(mag2)[0]);
s.hf_energy_ratio = detail::hf_ratio(g32);
s.dir_min_tenengrad = var_lf > 1e-9
? static_cast<float>(std::min(cv::mean(gx2)[0], cv::mean(gy2)[0]) / var_lf)
: 0.f;
s.ok = var_img > 1e-9;
return s;
}
src/scene_boundaries.hpp
+126
View File
@@ -0,0 +1,126 @@
#pragma once
/// TRACES: AR-010 | SR-002
///
/// SceneBoundaries — the join point of the decode butterfly.
///
/// The topology forks after decode: one branch runs TransNetV2 over dense
/// frames, the other runs face detection over the sampled cadence. Boundaries
/// found on the first branch have to reach the second, and they cannot be
/// carried in the frames themselves because the branches are parallel.
///
/// **Why this needs a watermark.** TransNetV2 buffers `kWindow` frames before it
/// can score any of them, so at any instant the detector has an opinion about
/// everything up to some time T and nothing after it. Without recording T, a
/// consumer asking "is there a boundary at t?" cannot distinguish *no* from
/// *not yet* — and those demand opposite behaviour. Silently treating unscored
/// frames as boundary-free is exactly the class of failure that makes a
/// verification pass vacuously.
///
/// The consumer is held back by channel depth (see main.cpp) so that by the time
/// it pulls a frame, the detector has already scored past it. `scored_through()`
/// is what lets that assumption be *checked* rather than assumed.
#include <algorithm>
#include <condition_variable>
#include <mutex>
#include <vector>
class SceneBoundaries {
public:
/// Peaks closer than this are one boundary. Matches the dedup scenes.json
/// applies, so the two views agree.
static constexpr double kMergeSec = 0.04;
/// Called by the scene detector as each window is scored. `through` is the
/// timestamp up to which its verdict is now final.
void publish(const std::vector<double>& ts, double through) {
{
std::lock_guard<std::mutex> g(mu_);
// Dedup on insert, matching what scenes.json does at write time. A run
// of adjacent high-scoring frames is one boundary, not several, and
// leaving them raw made this view report 357 where the file said 13 —
// the same event counted many times. Harmless for is_boundary(), which
// absorbs them in its tolerance, but a count nobody can reconcile with
// the output file is a bad diagnostic.
bounds_.insert(bounds_.end(), ts.begin(), ts.end());
std::sort(bounds_.begin(), bounds_.end());
bounds_.erase(std::unique(bounds_.begin(), bounds_.end(),
[](double a, double b) { return b - a < kMergeSec; }),
bounds_.end());
scored_through_ = std::max(scored_through_, through);
}
cv_.notify_all();
}
/// True if a boundary falls within `tol` of `t`.
///
/// `tol` exists because the two branches sample at different rates: a
/// boundary found on a dense frame rarely lands exactly on a sampled one.
/// Half a sample interval is the natural width — it attributes the boundary
/// to the nearest sampled frame and no further.
bool is_boundary(double t, double tol) const {
std::lock_guard<std::mutex> g(mu_);
auto it = std::lower_bound(bounds_.begin(), bounds_.end(), t - tol);
return it != bounds_.end() && *it <= t + tol;
}
/// The timestamp through which the detector's verdict is final. A consumer
/// past this point is asking about frames nobody has looked at yet.
double scored_through() const {
std::lock_guard<std::mutex> g(mu_);
return scored_through_;
}
/// Block until the detector's verdict covers `t`, or it finishes.
///
/// Channel depth alone does NOT create the required lag: it only holds
/// frames back when the consumer is slower, and the face branch is roughly
/// four orders of magnitude faster per frame than TransNetV2. So the join
/// has to wait explicitly.
///
/// Returns false if the detector finished without ever covering `t`, which
/// happens for the tail frames after its last full window. The caller must
/// distinguish that from a genuine "no boundary" rather than assuming.
bool wait_until_scored(double t) const {
std::unique_lock<std::mutex> lk(mu_);
cv_.wait(lk, [&] { return finished_ || scored_through_ >= t; });
return scored_through_ >= t;
}
/// Called when the detector will publish nothing further. Without this the
/// join would deadlock on the tail: those frames are never covered by a full
/// window, so waiting for them would wait forever.
void finish() {
{
std::lock_guard<std::mutex> g(mu_);
finished_ = true;
}
cv_.notify_all();
}
std::size_t count() const {
std::lock_guard<std::mutex> g(mu_);
return bounds_.size();
}
/// Consumers that outran the detector. Nonzero means the face branch is not
/// buffered deeply enough for the detector's window, so some frames were
/// annotated from an incomplete verdict — a real misconfiguration, and one
/// that would otherwise be invisible.
void note_outran() const {
std::lock_guard<std::mutex> g(mu_);
++outran_;
}
std::size_t outran() const {
std::lock_guard<std::mutex> g(mu_);
return outran_;
}
private:
mutable std::mutex mu_;
mutable std::condition_variable cv_;
bool finished_{false};
std::vector<double> bounds_;
double scored_through_{-1.0};
mutable std::size_t outran_{0};
};
src/scene_preview.cpp
+8 -1
View File
@@ -21,6 +21,7 @@
#include "config.hpp"
#include "types.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/frame_source_node.hpp"
#include "nodes/camera_position_change_detector_node.hpp"
@@ -76,6 +77,7 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--detector-engine")) cfg.detector_engine = next();
else if (arg("--arcface")) cfg.arcface_model = next();
else if (arg("--arcface-engine")) cfg.arcface_engine = next();
else if (arg("--require-gallery-stamp")) cfg.require_gallery_stamp = true;
else if (arg("--conf")) cfg.detector_conf = std::stof(next());
else if (arg("--max-faces")) cfg.max_faces = std::stoi(next());
else if (arg("--min-face-px")) cfg.min_face_px = std::stof(next());
@@ -126,7 +128,12 @@ int main(int argc, char** argv) {
}
ActorGallery gallery;
try { gallery = load_gallery(cfg.gallery_path); }
try {
gallery = load_gallery(cfg.gallery_path);
/// TRACES: GR-004 | SR-001
verify_gallery_embedder(gallery, cfg.gallery_path, cfg.arcface_model,
cfg.require_gallery_stamp);
}
catch (const std::exception& e) {
std::cerr << "Gallery error: " << e.what() << "\n";
return 1;
src/track_registry.hpp
+355
View File
@@ -0,0 +1,355 @@
#pragma once
/// TRACES: AR-012, AR-013, AR-014, AR-015, AR-016, AR-017 | SR-002
///
/// TrackRegistry — the single owner of track state and of presence.
///
/// Presence follows **track extent**, not per-frame recognition (AR-012): a
/// window is `[first_seen, last_seen]` of a track an actor owns, so it starts
/// when the actor appeared rather than when the recogniser first succeeded.
///
/// `last_seen` carries the entire liveness state (AR-013):
///
/// unset → on screen now
/// set → went off screen at that timestamp, still revivable
/// reaped → emitted to the aggregator and erased
///
/// There is no missing-frame counter and no expired flag; the optional *is* the
/// state machine, and it subsumes what was previously a two-pool split in the
/// tracker (active vs. parked-across-a-cut).
///
/// **Interior gaps are claimed, the trailing cool-down is not.** A face lost at
/// t1 and re-associated at t2 within the timeout never closed its track, so the
/// actor is present across [t1, t2] — correct, since someone briefly occluded or
/// off-camera has not left the scene. But a track that dies ends its window at
/// `last_seen`, never at the moment of death. That asymmetry is what removes the
/// over-claim the retired `extinction_sec` keep-alive produced.
///
/// The registry is created in `main` and shared by `shared_ptr`; it is *not* a
/// KPN node. Ownership is not a stage in the stream — it is state several stages
/// read and write, whose final answer is only known when a track dies.
#include "types.hpp"
#include "evidence_discount.hpp"
#include <cmath>
#include <cstdint>
#include <functional>
#include <map>
#include <mutex>
#include <optional>
#include <string>
#include <vector>
// ── DeadTrack ────────────────────────────────────────────────────────────────
// A finished presence claim, emitted exactly once when a track is reaped or
// flushed. Immutable by construction: it carries everything needed to justify
// itself (AR-017), with no back-reference into registry state.
struct DeadTrack {
int track_id{-1};
double first_seen{0.0};
double last_seen{0.0}; ///< always the last sighting, never the death time
int actor_idx{-1}; ///< -1 when the track was never owned
float belief{0.0f}; ///< accumulated posterior for actor_idx
int observations{0}; ///< evidence updates that landed on this track
float effective_obs{0.f}; ///< sum of discounted weights — the evidence that counted
};
// ── Track ────────────────────────────────────────────────────────────────────
struct Track {
int id{-1};
double first_seen{0.0};
std::optional<double> last_seen; ///< unset ⇒ on screen
std::optional<int> actor; ///< set once a posterior crosses
/// actor_idx → accumulated log(1 − P). Lazy-OR (noisy-OR) accumulation:
/// each frame is new evidence that this track is that actor, and the
/// combined belief is the probability that *at least one* sighting was
/// right. Stored as log(1−P) because that makes the update additive and
/// keeps precision where it matters — as P approaches 1, (1−P) is the
/// quantity with the significant digits.
std::map<int, float> belief;
Embedding mean{}; ///< running directional mean
std::vector<Embedding> views; ///< distinct looks, for AR-025 discounting
float discounted_weight{0.f}; ///< sum of applied weights
int n_obs{0};
bool on_screen() const { return !last_seen.has_value(); }
};
// ── TrackRegistry ────────────────────────────────────────────────────────────
class TrackRegistry {
public:
using DeadTrackFn = std::function<void(const DeadTrack&)>;
struct Config {
double extinction_sec{5.0}; ///< how long a lost track stays revivable
float ownership_logodds{2.0f}; ///< belief needed to own a track (~0.88 posterior)
};
/// The discounter is a constructor argument rather than an option: there is
/// no correct way to accumulate per-frame evidence without it.
TrackRegistry(Config cfg, EvidenceDiscounter discounter)
: cfg_(cfg), discounter_(std::move(discounter)) {}
void on_track_dead(DeadTrackFn fn) { on_dead_ = std::move(fn); }
// ── Frame scope ──────────────────────────────────────────────────────────
// The tracker mutates registry state across a whole association pass, so
// that pass must be atomic as a unit — per-call locking would let another
// thread observe a half-updated frame. FrameScope holds the lock for its
// lifetime and exposes the mutating operations without re-locking.
class FrameScope {
public:
FrameScope(TrackRegistry& reg, double now)
: reg_(reg), lock_(reg.mu_) { reg_.tick_locked(now); }
/// All live tracks — **one pool**. `last_seen` tells the caller whether
/// IoU is meaningful; a dormant track is matched on embedding alone.
/// There is no separate revival path (AR-008).
std::vector<Track*> candidates() {
std::vector<Track*> out;
out.reserve(reg_.tracks_.size());
for (auto& [id, t] : reg_.tracks_) out.push_back(&t);
return out;
}
int create(double t, const Embedding& e) { return reg_.create_locked(t, e); }
void mark_seen(int id, double t, const Embedding& e){ reg_.mark_seen_locked(id, t, e); }
void mark_lost(int id, double last_on_screen) { reg_.mark_lost_locked(id, last_on_screen); }
private:
TrackRegistry& reg_;
std::unique_lock<std::mutex> lock_;
};
FrameScope begin_frame(double now) { return FrameScope(*this, now); }
/// Advance the clock and reap. Called every sampled frame **whether or not
/// it had detections** — without it a track only dies when some other face
/// happens to appear, and a film ending mid-track never closes.
void tick(double now) { std::lock_guard g(mu_); tick_locked(now); }
// ── Evidence ─────────────────────────────────────────────────────────────
/// Fold one observation into a track's belief (AR-025).
///
/// `posterior` is a **calibrated probability**, never a raw cosine
/// (AR-024) — the registry converts it to log-odds itself, so the
/// accumulation cannot be fed an uncalibrated number by a careless caller.
///
/// Correlation discounting is applied **here**, not by the caller.
/// Consecutive frames of one track are near-identical, and accumulating
/// them as independent evidence drives the posterior to certainty on what
/// is effectively a single measurement. Leaving that to callers would mean
/// a forgotten or doubly-applied discount produces confident wrong answers
/// silently; the registry is the one place all evidence converges, so it is
/// the one place the correction belongs.
///
/// A vote for a track that has already been reaped is dropped and counted:
/// a nonzero `dropped_votes()` means the timeout is shorter than the
/// matcher's lag, which is a real misconfiguration and must not be silent.
void observe(int track_id, int actor_idx, float posterior, const Embedding& e) {
std::lock_guard g(mu_);
auto it = tracks_.find(track_id);
if (it == tracks_.end()) { ++dropped_votes_; return; }
Track& t = it->second;
const float w = discounter_.weight(t.views, t.n_obs, e);
// Weighted lazy-OR: P_new = 1 − (1 − P_old)·(1 − p)^w, which in log
// space is a plain sum. w is the discounted evidence (AR-025), so a
// repeated view still advances the belief but by a fraction of what a
// genuinely new look would.
const float p = std::min(1.f - 1e-6f, std::max(1e-6f, posterior));
t.belief[actor_idx] += w * std::log(1.f - p);
t.discounted_weight += w;
++t.n_obs;
const int best = argmax_belief(t);
const float best_p = 1.f - std::exp(t.belief[best]);
if (best_p < own_threshold()) return;
if (!t.actor.has_value()) {
claim_locked(t, best);
return;
}
if (*t.actor != best) {
// AR-014 — belief swapped A→B. Not a correction: a track_id almost
// certainly carried across a viewpoint change onto a different
// person. Two non-twins both clearing the threshold on one face is
// not realistic; a track spanning two people is. Continuing would
// emit one window blending both, so close here and start afresh.
split_locked(t, best);
}
}
/// Snapshot read: tally and verdict under one lock. Reading them separately
/// would let a track be both unowned and owned within a single promotion
/// decision, since the matcher may be voting concurrently.
std::optional<int> owner(int track_id) const {
std::lock_guard g(mu_);
auto it = tracks_.find(track_id);
return it == tracks_.end() ? std::nullopt : it->second.actor;
}
// ── Termination ──────────────────────────────────────────────────────────
/// Emit every still-live track and empty the registry (AR-016). A film ends
/// with faces on screen and those tracks have not timed out, so without this
/// the closing scene's cast is silently never emitted — a loss that presents
/// as a recognition miss rather than a bookkeeping bug.
///
/// Idempotent: calling it twice emits nothing the second time.
void flush(double final_ts) {
std::lock_guard g(mu_);
for (auto& [id, t] : tracks_) emit_locked(t, t.last_seen.value_or(final_ts));
tracks_.clear();
}
// ── Diagnostics ──────────────────────────────────────────────────────────
// These measure how often tracking is silently wrong, which nothing in the
// pipeline currently reveals.
int dropped_votes() const { std::lock_guard g(mu_); return dropped_votes_; }
int belief_swaps() const { std::lock_guard g(mu_); return belief_swaps_; }
int actor_conflicts() const { std::lock_guard g(mu_); return actor_conflicts_; }
std::size_t live() const { std::lock_guard g(mu_); return tracks_.size(); }
private:
// ── Locked internals ─────────────────────────────────────────────────────
void tick_locked(double now) {
for (auto it = tracks_.begin(); it != tracks_.end(); ) {
const auto& ls = it->second.last_seen;
if (ls && (now - *ls) > cfg_.extinction_sec) {
emit_locked(it->second, *ls);
it = tracks_.erase(it);
} else {
++it;
}
}
}
int create_locked(double t, const Embedding& e) {
const int id = next_id_++;
Track tr;
tr.id = id;
tr.first_seen = t;
tr.mean = e;
tr.n_obs = 0;
tracks_.emplace(id, std::move(tr));
return id;
}
void mark_seen_locked(int id, double t, const Embedding& e) {
auto it = tracks_.find(id);
if (it == tracks_.end()) return;
Track& tr = it->second;
tr.last_seen.reset(); // back on screen; the gap is absorbed
update_mean(tr, e);
(void)t;
}
void mark_lost_locked(int id, double last_on_screen) {
auto it = tracks_.find(id);
if (it == tracks_.end()) return;
it->second.last_seen = last_on_screen;
}
void claim_locked(Track& t, int actor) {
// AR-015 — if another live track already owns this actor, at least one
// is wrong: a person cannot be in two places at once. The cause is the
// same as a belief swap — a missed camera or scene change. Detected on
// the update that causes it via the reverse index, not by scanning.
auto seen = owner_index_.find(actor);
if (seen != owner_index_.end() && seen->second != t.id
&& tracks_.count(seen->second)) {
++actor_conflicts_;
}
t.actor = actor;
owner_index_[actor] = t.id;
}
void split_locked(Track& t, int new_actor) {
++belief_swaps_;
const double boundary = t.last_seen.value_or(t.first_seen);
emit_locked(t, boundary);
// The successor inherits the embedding and the belief that caused the
// swap, and starts at the swap frame — so the two windows abut without
// overlapping and neither blends the two people.
Track next;
next.id = next_id_++;
next.first_seen = boundary;
next.mean = t.mean;
next.belief[new_actor] = t.belief[new_actor];
next.n_obs = 1;
const int old_id = t.id;
Track stash = std::move(next);
tracks_.erase(old_id);
const int nid = stash.id;
tracks_.emplace(nid, std::move(stash));
claim_locked(tracks_.at(nid), new_actor);
}
void emit_locked(Track& t, double end_ts) {
if (!on_dead_) return;
DeadTrack d;
d.track_id = t.id;
d.first_seen = t.first_seen;
d.last_seen = end_ts;
d.observations = t.n_obs;
d.effective_obs = t.discounted_weight;
if (t.actor) {
d.actor_idx = *t.actor;
d.belief = 1.f - std::exp(t.belief[*t.actor]);
auto oi = owner_index_.find(*t.actor);
if (oi != owner_index_.end() && oi->second == t.id) owner_index_.erase(oi);
}
on_dead_(d);
}
/// Most-believed actor. belief holds log(1 − P), so the strongest claim is
/// the *most negative* entry, not the largest.
static int argmax_belief(const Track& t) {
int best = -1;
float lo = 1e30f;
for (const auto& [a, v] : t.belief) if (v < lo) { lo = v; best = a; }
return best;
}
/// Ownership expressed as a probability. Config still carries log-odds so
/// the knob keeps its meaning across this change.
float own_threshold() const {
return 1.f / (1.f + std::exp(-cfg_.ownership_logodds));
}
static void update_mean(Track& t, const Embedding& e) {
// Directional mean: accumulate then re-normalise to the unit sphere, so
// cosine against it stays a plain dot product.
double norm = 0.0;
for (int i = 0; i < 512; ++i) {
t.mean[i] = t.mean[i] * static_cast<float>(t.n_obs ? t.n_obs : 1) + e[i];
norm += static_cast<double>(t.mean[i]) * t.mean[i];
}
norm = norm > 0 ? std::sqrt(norm) : 1.0;
for (int i = 0; i < 512; ++i) t.mean[i] = static_cast<float>(t.mean[i] / norm);
}
static float logistic(float z) {
return z >= 0 ? 1.f / (1.f + std::exp(-z))
: std::exp(z) / (1.f + std::exp(z));
}
static float logit(float p) {
const float eps = 1e-6f;
p = std::min(1.f - eps, std::max(eps, p));
return std::log(p / (1.f - p));
}
Config cfg_;
EvidenceDiscounter discounter_;
mutable std::mutex mu_;
std::map<int, Track> tracks_;
std::map<int, int> owner_index_; ///< actor_idx → live track_id (AR-015)
DeadTrackFn on_dead_;
int next_id_{0};
int dropped_votes_{0};
int belief_swaps_{0};
int actor_conflicts_{0};
};
src/types.hpp
+14
View File
@@ -6,6 +6,8 @@
#include <opencv2/core.hpp>
#include "gallery/embedder_stamp.hpp"
// ── Embedding ─────────────────────────────────────────────────────────────────
// 512-dim L2-normalised ArcFace embedding
using Embedding = std::array<float, 512>;
@@ -61,6 +63,13 @@ struct DetectedFace {
cv::Rect2f bbox;
std::array<cv::Point2f, 5> landmarks;
float confidence{0.f};
// AR-030 visibility: RMS landmark misfit, in canonical 112×112 pixels, left
// over after the best similarity fit to the ArcFace template. Rises with
// out-of-plane pose and with occlusion; blind to in-plane roll and to face
// size, both of which the fit absorbs. Set by the aligner, which is where
// the transform is computed; -1 until then.
float alignment_residual{-1.f};
};
// ── Pipeline messages ─────────────────────────────────────────────────────────
@@ -136,6 +145,11 @@ struct ActorGallery {
};
std::vector<Actor> actors;
/// TRACES: GR-004 | SR-001
// Which embedder produced every embedding above. Empty == the file predates
// model binding; see gallery/embedder_stamp.hpp for what is checked and why.
EmbedderStamp embedder;
// Cached Platt-sigmoid calibration (see gallery/gallery_calibration.hpp),
// stored alongside the gallery in HDF5 so it never needs recomputing
// unless the reference embeddings actually change. calib_valid=false and
tests/CMakeLists.txt
+26 -1
View File
@@ -19,23 +19,48 @@ add_executable(sae_tests
test_calibration.cpp
test_gallery_store.cpp
test_face_utils.cpp
test_quality.cpp
test_track_gallery.cpp
test_face_tracker.cpp
test_track_registry.cpp
test_replay_fixtures.cpp
test_audio_signature.cpp
${CMAKE_SOURCE_DIR}/src/backends/gemm_backend.cpp
${CMAKE_SOURCE_DIR}/src/gallery/gallery_store.cpp
${CMAKE_SOURCE_DIR}/src/audio_signature.cpp
${CMAKE_SOURCE_DIR}/src/gallery/embedder_stamp.cpp
)
target_include_directories(sae_tests PRIVATE ${CMAKE_SOURCE_DIR}/src)
# SAE_GEMM_CPU: build the CPU reference GEMM regardless of the main backend.
# SAE_MODELS_DIR: config.hpp (pulled in by track_gallery.hpp) bakes model paths.
# SAE_TEST_FIXTURES_DIR: the audio golden vector is read from the source tree,
# not copied, so the file the plugin repo shares is the file under test.
# AR-026/AR-027: exercise the same kernel CI actually runs. Without this the
# suite compiles the scalar fallback while the CPU builder image links OpenBLAS,
# so the tested path and the shipped path would differ.
find_package(PkgConfig QUIET)
if(PkgConfig_FOUND)
pkg_check_modules(OPENBLAS_T QUIET openblas)
endif()
if(OPENBLAS_T_FOUND)
target_include_directories(sae_tests PRIVATE ${OPENBLAS_T_INCLUDE_DIRS})
target_link_libraries(sae_tests PRIVATE ${OPENBLAS_T_LINK_LIBRARIES})
endif()
target_compile_definitions(sae_tests PRIVATE
$<$<BOOL:${OPENBLAS_T_FOUND}>:SAE_GEMM_CBLAS>
SAE_GEMM_CPU
SAE_MODELS_DIR="${SAE_MODELS_DIR}")
SAE_MODELS_DIR="${SAE_MODELS_DIR}"
SAE_TEST_FIXTURES_DIR="${CMAKE_CURRENT_SOURCE_DIR}/fixtures")
# gallery_store.cpp + gallery_calibration.hpp use nlohmann/json and HDF5
# (galleries are HDF5-native, see src/gallery/gallery_store.cpp); face_utils.hpp
# and the calibration GEMM pull in OpenCV (calib3d/imgproc/core) via types.hpp.
# ffmpeg_libs: audio_signature.cpp decodes the golden fixture (avformat/avcodec/
# avutil/swresample). Still GPU-free — the audio path is pure CPU.
target_link_libraries(sae_tests PRIVATE
Catch2::Catch2WithMain
nlohmann_json::nlohmann_json
ffmpeg_libs
${OpenCV_LIBS}
${HDF5_CXX_LIBRARIES})
target_include_directories(sae_tests PRIVATE ${HDF5_INCLUDE_DIRS})
tests/fixtures/audio/jray_audio_v1_golden.json
Vendored
+194
View File
@@ -0,0 +1,194 @@
{
"_": "Golden vector for the JRay v1 audio signature (JRay-public-server SPEC.md \u00a73). Shared verbatim between scene-actor-extraction (C++) and the jRay Jellyfin plugin (C#) so the two implementations can be proven bit-identical. IR-004, IR-005, IR-007, IR-008.",
"version": "v1",
"signature": "v1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAeHx8eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh8fHzk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5V1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dycnJycnJycnJycnJycnJycnJycnJycnJycnJycnMPDgwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKytFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRWNjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2Njfn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5/GxoZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGTc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3UlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU1JsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbAoLCwoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCwsLJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSVDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ15eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eX19eeXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXkXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFzExMTExMTExMTExMTExMTExMTExMTExMTExMTExT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09qampqampqampqampqampqampqampqampqampqamsHBwUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyM+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4/PlhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYd3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3ExMRERERERERERERERERERERERERERERERERERERES8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpLS0plZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZQMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0eHh44ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4OFdXV1ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWV1dXcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXEPDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDyoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKysqRERERERERERERERERERERERERERERERERERERERjY2NjY2NjY2NjYw==",
"frame_count": 1288,
"media": {
"file": "jray_audio_v1_tone.flac",
"generator": "make_fixture.py",
"container": "FLAC (lossless \u2014 decodes to exactly the PCM make_fixture.py emits)",
"duration_sec": 120.0,
"sample_rate": 11025,
"channels": 1,
"sample_format": "s16",
"sha256": "912ecd426cd426dccb37753e0249694227619c701cb9f533502b37da0fbe8096",
"bytes": 585142
},
"decoded_window": {
"_": "Checksums of the 120 s centre window after downmix to mono and resample to 11025 Hz, i.e. exactly the stream `ffmpeg -ss <mid-60> -t 120 -i <file> -vn -ac 1 -ar 11025 -f f32le -` produces. Check these first: a mismatch here is a decode problem, not a DSP one.",
"samples": 1323000,
"f32le_fnv1a64": "0x1ef7899cd4d12662",
"s16le_fnv1a64": "0xf824fa56f125c0dc"
},
"params": {
"window_sec": 120.0,
"window_centre": "runtime/2, i.e. samples from runtime/2 - 60 s; truncated to exactly 1323000 samples",
"min_duration_sec": 120.0,
"min_duration_rule": "IR-007 \u2014 below this emit NO signature and apply no sync offset",
"sample_rate": 11025,
"channels": 1,
"arithmetic": "IEEE-754 double throughout; float32 is not sufficient",
"sample_scale": "s16 * (1/32768), FFmpeg's native s16->flt",
"frame_size": 4096,
"hop_size": 1024,
"frame_count_rule": "1 + (n_samples - 4096) / 1024, integer division; whole frames only",
"window_fn": "Hann, PERIODIC: w[n] = 0.5 * (1 - cos(2*pi*n/4096))",
"transform": "radix-2 DIT complex FFT over the 4096 real samples (imag=0), no normalisation",
"magnitude": "sqrt(re^2 + im^2), linear",
"band_lo_hz": 300.0,
"band_hi_hz": 3000.0,
"num_bands": 32,
"band_edges": "edge[b] = 300 * (3000/300)^(b/32), b = 0..32",
"band_bins": "band b owns FFT bins [k_lo[b], k_lo[b+1]) with k_lo[b] = ceil(edge[b] * 4096 / 11025); see band_fft_bins",
"band_value": "MEAN of the linear magnitudes in the band (not sum, not max)",
"peak_bin": "argmax over the 32 band values; ties resolve to the LOWEST index",
"energy_metric": "E = mean magnitude over all FFT bins 112..1114, i.e. the whole 300-3000 Hz band",
"energy_reference": "upper median of E over all frames: sorted[n/2], no averaging of the two middle values",
"energy_ratio": "r = log10((E + 1e-12) / (E_ref + 1e-12))",
"energy_class_edges": [
-0.6,
-0.2,
0.2
],
"energy_class": "0 if r < -0.6, 1 if r < -0.2, 2 if r < 0.2, else 3",
"byte_layout": "bit7 = 0 (reserved), bits6..2 = 5-bit band index, bits1..0 = 2-bit energy class; byte = (band << 2) | class",
"base64": "standard alphabet A-Za-z0-9+/ with '=' padding",
"prefix": "v1:"
},
"band_fft_bins": [
[
112,
120
],
[
120,
129
],
[
129,
139
],
[
139,
149
],
[
149,
160
],
[
160,
172
],
[
172,
185
],
[
185,
199
],
[
199,
213
],
[
213,
229
],
[
229,
246
],
[
246,
265
],
[
265,
285
],
[
285,
306
],
[
306,
328
],
[
328,
353
],
[
353,
379
],
[
379,
408
],
[
408,
438
],
[
438,
471
],
[
471,
506
],
[
506,
543
],
[
543,
584
],
[
584,
627
],
[
627,
674
],
[
674,
724
],
[
724,
778
],
[
778,
836
],
[
836,
899
],
[
899,
966
],
[
966,
1038
],
[
1038,
1115
]
],
"notes": [
"The server spec fixes the window, rate, STFT geometry, band and the 5+2 bit packing. Everything under params beyond that (Hann periodicity, band aggregation, the energy-class definition, tie-breaking, base64 alphabet) is pinned HERE for v1 \u2014 the spec does not constrain it, and two implementations that guess differently produce non-matching signatures.",
"Decision margins on this fixture: the two strongest bands are within 1.3% on the closest frame, and the closest frame to an energy-class edge is 3.6e-3 away in log10. Both are many orders of magnitude above double-precision FFT differences, so any two correct double- precision implementations agree; a float32 implementation is not guaranteed to.",
"Coverage: all 32 bands and all 4 energy classes appear in the golden signature.",
"Robustness observed on this fixture: identical peak-bin sequence after a stereo/44100 Hz round trip and after AAC 128 kbit/s re-encoding."
]
}
tests/fixtures/audio/jray_audio_v1_tone.flac
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/audio/make_fixture.py
Vendored
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""Regenerate the JRay audio-signature golden fixture.
python3 make_fixture.py # writes jray_audio_v1_tone.flac here
This is the *source of truth* for the fixture media: `jray_audio_v1_tone.flac`
is a lossless FLAC encoding of exactly the PCM this script emits, so any repo
that wants to check its own audio-signature implementation against the golden
vector in `jray_audio_v1_golden.json` can regenerate the input from scratch and
confirm it is byte-identical (the golden file records `pcm_fnv1a64`, a hash of
the decoded 16-bit samples).
Deliberately dependency-free (no numpy) and written in plain arithmetic so it
ports to any language in ~20 lines.
Signal — 120.000 s, mono, 11025 Hz, 16-bit signed PCM:
* split into segments of 32768 samples (~2.97 s), 40.4 segments in total;
* segment `s` carries one sine at the geometric centre of log-band
`(s * 7) mod 32` of the 300-3000 Hz band, so all 32 bands are exercised;
* its amplitude walks a golden-ratio low-discrepancy sequence over
[10^-1.55, 10^-0.02] so frame energies spread continuously across ~1.5
decades and all four energy classes are exercised, without a dense cluster
of frames sitting on a class boundary;
* phase is carried across segment boundaries (no clicks);
* a constant, far quieter 777 Hz tone sits underneath so no frame is
degenerate;
* samples are quantised with floor(x * 32767 + 0.5).
Why FLAC and not WAV: 120 s of 11025 Hz 16-bit PCM is 2.6 MB and does not
compress in git. FLAC is lossless — FFmpeg decodes it to exactly the PCM
written here — and is ~3.5x smaller. `--wav` writes the uncompressed original
if you want to diff it.
"""
import math
import struct
import subprocess
import sys
import os
SAMPLE_RATE = 11025
DURATION_SEC = 120.0
SEGMENT = 32768 # samples per tone segment
BAND_STRIDE = 7 # coprime with 32 -> visits every band
BAND_LO_HZ = 300.0
BAND_HI_HZ = 3000.0
NUM_BANDS = 32
AMP_LOG_MIN = -1.55 # 10^-1.55 ~= 0.028
AMP_LOG_SPAN = 1.53 # up to 10^-0.02 ~= 0.955
PHI_FRAC = 0.6180339887498949
BG_HZ = 777.0
BG_AMP = 0.004
OUT_FLAC = "jray_audio_v1_tone.flac"
OUT_WAV = "jray_audio_v1_tone.wav"
def generate():
"""Return the 120 s signal as a list of int16 sample values."""
n = int(round(SAMPLE_RATE * DURATION_SEC))
out = [0] * n
phase = 0.0
two_pi = 2.0 * math.pi
for start in range(0, n, SEGMENT):
s = start // SEGMENT
end = min(n, start + SEGMENT)
band = (s * BAND_STRIDE) % NUM_BANDS
# geometric centre of log-band `band`
freq = BAND_LO_HZ * (BAND_HI_HZ / BAND_LO_HZ) ** ((band + 0.5) / NUM_BANDS)
amp = 10.0 ** (AMP_LOG_MIN + AMP_LOG_SPAN * ((s * PHI_FRAC) % 1.0))
step = two_pi * freq / SAMPLE_RATE
for k in range(end - start):
i = start + k
x = amp * math.sin(phase + step * k)
x += BG_AMP * math.sin(two_pi * BG_HZ * i / SAMPLE_RATE)
if x > 1.0:
x = 1.0
elif x < -1.0:
x = -1.0
out[i] = int(math.floor(x * 32767.0 + 0.5))
phase = (phase + step * (end - start)) % two_pi
return out
def write_wav(path, samples):
data = struct.pack("<%dh" % len(samples), *samples)
hdr = b"RIFF" + struct.pack("<I", 36 + len(data)) + b"WAVE"
hdr += b"fmt " + struct.pack("<IHHIIHH", 16, 1, 1, SAMPLE_RATE,
SAMPLE_RATE * 2, 2, 16)
hdr += b"data" + struct.pack("<I", len(data))
with open(path, "wb") as fh:
fh.write(hdr + data)
def main():
here = os.path.dirname(os.path.abspath(__file__))
samples = generate()
wav = os.path.join(here, OUT_WAV)
write_wav(wav, samples)
if "--wav" in sys.argv:
print("wrote", wav)
return
flac = os.path.join(here, OUT_FLAC)
# -compression_level 12 is deterministic for a given libFLAC/ffmpeg build;
# only the container bytes vary, never the decoded PCM.
subprocess.run(["ffmpeg", "-nostdin", "-v", "error", "-y", "-i", wav,
"-c:a", "flac", "-compression_level", "12", flac],
check=True)
os.remove(wav)
print("wrote", flac)
if __name__ == "__main__":
main()
tests/fixtures/audio/make_offset_fixture.sh
Vendored Executable
+60
View File
@@ -0,0 +1,60 @@
#!/bin/sh
#
# Regenerate bali_offset_200s.flac — the real-audio fixture behind VR-014, the
# audio-signature offset-recovery validation.
#
# sh make_offset_fixture.sh /path/to/clips
#
# Why real audio and not a second synthetic tone: jray_audio_v1_tone.flac pins
# the *arithmetic* (IR-005) and is deliberately built so every band and every
# energy class appears. It cannot answer the question VR-014 asks — whether the
# peak-bin sequence of ordinary film audio is distinctive enough that sliding
# one signature against another finds the true alignment and only the true
# alignment. Tones are pathologically easy for that; dialogue and score are not.
#
# Source: five scene clips from "Road to Bali" (1952), the public-domain corpus
# this repo already uses for the replay fixtures — tests/fixtures/dumps/bali_*.h5
# are dumps of these same clips. Each is under the 120 s window on its own
# (29-77 s), so they are concatenated in scene order to make a source long
# enough that a 120 s window can slide inside it.
#
# 200 s is chosen, not arbitrary: the window is 120 s and the match search is
# capped at +/-600 frames (~55.7 s), so a source of 120 + 56 s is the shortest
# one that can place two windows at the edge of the cap. The 200 s here leaves
# room to go past it as well, which is what lets the test check that an
# out-of-range offset is declined rather than guessed.
#
# Encoded mono at 11025 Hz, 16-bit, which is exactly what the signature decodes
# to anyway. That keeps a 200 s fixture at ~2.4 MB instead of ~20 MB, and makes
# every trim below sample-exact — the test measures offset recovery, not the
# resampler, which tests/test_audio_signature.cpp already covers (UT-103).
#
# FLAC because it is lossless: the decoded PCM is the same on every machine, so
# a signature computed from this file is reproducible. A lossy fixture would
# make the measurement depend on the decoder version.
#
# sha256 of the committed file:
# 4a952e46a090a9acd9eae56996250ec03e08e0d04ee139ac0a42f1690a536c83
# A regenerated file that hashes differently means the source clips or the
# encoder changed, and VR-014's recorded numbers should be re-measured — the
# offsets will still be exact, but the scores are this audio's.
set -eu
CLIPS="${1:-../../../../bali}"
OUT="$(dirname "$0")/bali_offset_200s.flac"
LIST="$(mktemp)"
trap 'rm -f "$LIST"' EXIT
for scene in 13 27 28 31 46; do
clip="$CLIPS/Road_To_Bali-$scene.webm"
[ -f "$clip" ] || { echo "missing clip: $clip" >&2; exit 1; }
echo "file '$(cd "$(dirname "$clip")" && pwd)/$(basename "$clip")'" >> "$LIST"
done
ffmpeg -nostdin -v error -y -f concat -safe 0 -i "$LIST" \
-vn -t 200 -ac 1 -ar 11025 -sample_fmt s16 \
-c:a flac -compression_level 12 "$OUT"
echo "wrote $OUT"
sha256sum "$OUT" 2>/dev/null || shasum -a 256 "$OUT"
tests/test_audio_signature.cpp
+359
View File
@@ -0,0 +1,359 @@
// Unit tests for the JRay v1 audio signature (src/audio_signature.*).
//
/// TRACES: UT-101, UT-102, UT-103, UT-104 | IR-004, IR-005, IR-007, IR-008
//
// The headline test is the golden vector: a deterministic tone fixture checked
// into tests/fixtures/audio/ together with the signature it must produce. That
// fixture is the artefact shared with the jRay plugin repo, and it is what
// makes "both producers agree bit-for-bit" a checked claim rather than an
// assertion (IR-005).
//
// GPU-free, model-free, no network. Pure CPU DSP plus an FFmpeg decode of a
// 585 KB file — which is precisely why this is the right cross-repo check: it
// runs anywhere, including the N100 CI host.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "audio_signature.hpp"
#include <nlohmann/json.hpp>
#include <cmath>
#include <cstdint>
#include <filesystem>
#include <fstream>
#include <stdexcept>
#include <string>
#include <vector>
using namespace sae::audio;
namespace fs = std::filesystem;
namespace {
const std::string kFixtureDir = SAE_TEST_FIXTURES_DIR "/audio";
const std::string kGoldenPath = kFixtureDir + "/jray_audio_v1_golden.json";
const std::string kMediaPath = kFixtureDir + "/jray_audio_v1_tone.flac";
const nlohmann::json& golden() {
static const nlohmann::json j = [] {
std::ifstream in(kGoldenPath);
if (!in.good())
throw std::runtime_error("golden fixture not found: " + kGoldenPath);
nlohmann::json parsed;
in >> parsed;
return parsed;
}();
return j;
}
std::uint64_t hex64(const std::string& s) {
return std::stoull(s, nullptr, 16);
}
// The fixture is 120 s of audio: decoding and signing it is the expensive part
// of this file, so both results are computed once and shared. Every test below
// still asserts against the on-disk golden values, not against each other.
const std::optional<std::vector<float>>& fixture_window() {
static const std::optional<std::vector<float>> w = decode_centre_window(kMediaPath);
return w;
}
const std::optional<std::string>& fixture_signature() {
static const std::optional<std::string> s = compute_signature(kMediaPath);
return s;
}
// ── Minimal WAV writer, so the short-media and resample cases need no fixture ─
// 16-bit PCM, interleaved.
struct TempWav {
fs::path path;
explicit TempWav(const std::string& name)
: path(fs::temp_directory_path() / ("sae_audio_test_" + name + ".wav")) {}
~TempWav() { std::error_code ec; fs::remove(path, ec); }
void write(const std::vector<std::int16_t>& samples, int rate, int channels) const {
const std::uint32_t bytes = static_cast<std::uint32_t>(samples.size() * 2);
const std::uint32_t byte_rate = static_cast<std::uint32_t>(rate * channels * 2);
std::ofstream out(path, std::ios::binary);
auto u32 = [&](std::uint32_t v) { out.write(reinterpret_cast<const char*>(&v), 4); };
auto u16 = [&](std::uint16_t v) { out.write(reinterpret_cast<const char*>(&v), 2); };
out.write("RIFF", 4); u32(36 + bytes); out.write("WAVE", 4);
out.write("fmt ", 4); u32(16); u16(1); u16(static_cast<std::uint16_t>(channels));
u32(static_cast<std::uint32_t>(rate)); u32(byte_rate);
u16(static_cast<std::uint16_t>(channels * 2)); u16(16);
out.write("data", 4); u32(bytes);
out.write(reinterpret_cast<const char*>(samples.data()), bytes);
}
};
// A plain 1 kHz tone, mono, at the signature's own rate.
std::vector<std::int16_t> tone(double seconds, int rate = kSampleRate) {
const std::size_t n = static_cast<std::size_t>(std::llround(seconds * rate));
std::vector<std::int16_t> s(n);
for (std::size_t i = 0; i < n; ++i)
s[i] = static_cast<std::int16_t>(std::llround(
20000.0 * std::sin(2.0 * 3.14159265358979323846 * 1000.0 * double(i) / rate)));
return s;
}
std::vector<std::uint8_t> base64_decode(const std::string& in) {
auto val = [](char c) -> int {
if (c >= 'A' && c <= 'Z') return c - 'A';
if (c >= 'a' && c <= 'z') return c - 'a' + 26;
if (c >= '0' && c <= '9') return c - '0' + 52;
if (c == '+') return 62;
if (c == '/') return 63;
return -1;
};
std::vector<std::uint8_t> out;
std::uint32_t acc = 0;
int bits = 0;
for (char c : in) {
const int v = val(c);
if (v < 0) continue; // '=' padding
acc = (acc << 6) | static_cast<std::uint32_t>(v);
bits += 6;
if (bits >= 8) {
bits -= 8;
out.push_back(static_cast<std::uint8_t>((acc >> bits) & 0xFF));
}
}
return out;
}
} // namespace
// ── UT-101 — the golden vector ──────────────────────────────────────────────
/// TRACES: UT-101 | IR-004, IR-005, IR-008
TEST_CASE("signature of the golden fixture matches the recorded value exactly",
"[audio_signature][golden]") {
REQUIRE(fs::exists(kMediaPath));
const std::optional<std::string>& sig = fixture_signature();
REQUIRE(sig.has_value());
CHECK(*sig == golden()["signature"].get<std::string>());
}
/// TRACES: UT-101 | IR-005
TEST_CASE("decoded centre window matches the recorded PCM checksum",
"[audio_signature][golden]") {
// Checked separately from the signature so a codec-level difference is
// distinguishable from a DSP-level one: if this passes and the signature
// test fails, the DSP diverged; if this fails, the decode did.
const std::optional<std::vector<float>>& mono = fixture_window();
REQUIRE(mono.has_value());
CHECK(mono->size() == golden()["decoded_window"]["samples"].get<std::size_t>());
CHECK(fnv1a64(mono->data(), mono->size() * sizeof(float)) ==
hex64(golden()["decoded_window"]["f32le_fnv1a64"].get<std::string>()));
}
/// TRACES: UT-101 | IR-004
TEST_CASE("log-spaced band table matches the recorded one", "[audio_signature][golden]") {
// The band->FFT-bin table is the part of the construction most likely to
// drift between two implementations, so it is pinned independently of the
// signature it produces.
const auto& tbl = band_fft_bins();
const auto& want = golden()["band_fft_bins"];
REQUIRE(want.size() == tbl.size());
for (std::size_t b = 0; b < tbl.size(); ++b) {
CHECK(tbl[b].first == want[b][0].get<int>());
CHECK(tbl[b].second == want[b][1].get<int>());
CHECK(tbl[b].second > tbl[b].first); // no empty band
if (b) CHECK(tbl[b].first == tbl[b - 1].second); // contiguous, no overlap
}
}
/// TRACES: UT-101 | IR-004, IR-008
TEST_CASE("signature is well-formed: v1 prefix, 1288 frames, structural bytes",
"[audio_signature][golden]") {
const std::optional<std::string>& sig = fixture_signature();
REQUIRE(sig.has_value());
// IR-008 — the signature carries its own version, separate from
// schema_version, so a future DSP change is detectable rather than silently
// producing non-matching signatures.
REQUIRE(sig->rfind(kVersionPrefix, 0) == 0);
const std::vector<std::uint8_t> bytes = base64_decode(sig->substr(3));
CHECK(bytes.size() == kExpectedFrames);
CHECK(bytes.size() == golden()["frame_count"].get<std::size_t>());
// The server validates this structure on upload (server SPEC §3): each byte
// is a 5-bit band index plus a 2-bit energy class, so bit 7 is always clear
// and arbitrary bytes are invalid. That is what keeps the field from being
// a payload channel.
bool bands_seen[kNumBands] = {};
bool classes_seen[4] = {};
for (std::uint8_t b : bytes) {
REQUIRE((b & 0x80) == 0);
bands_seen[(b >> 2) & 0x1F] = true;
classes_seen[b & 0x03] = true;
}
// The fixture is built to exercise the whole output alphabet — if it ever
// stops doing so, the golden vector has become a weaker check than it looks.
for (bool seen : bands_seen) CHECK(seen);
for (bool seen : classes_seen) CHECK(seen);
}
// ── UT-102 — IR-007, media shorter than the window ──────────────────────────
/// TRACES: UT-102 | IR-007
TEST_CASE("media shorter than 120 s emits no signature", "[audio_signature][short]") {
// The window runtime/2 ± 60 s underflows, so there is no signature and no
// sync offset downstream. Both producers must apply the identical rule or
// they diverge on exactly the short items most likely to be misidentified.
SECTION("30 s") {
TempWav w("short30");
w.write(tone(30.0), kSampleRate, 1);
CHECK_FALSE(compute_signature(w.path.string()).has_value());
CHECK_FALSE(decode_centre_window(w.path.string()).has_value());
}
SECTION("just under the boundary") {
TempWav w("short11999");
w.write(tone(119.99), kSampleRate, 1);
CHECK_FALSE(compute_signature(w.path.string()).has_value());
}
}
/// TRACES: UT-102 | IR-007
TEST_CASE("media of exactly 120 s emits a full-length signature",
"[audio_signature][short]") {
TempWav w("exact120");
w.write(tone(120.0), kSampleRate, 1);
const std::optional<std::string> sig = compute_signature(w.path.string());
REQUIRE(sig.has_value());
CHECK(base64_decode(sig->substr(3)).size() == kExpectedFrames);
}
/// TRACES: UT-102 | IR-007
TEST_CASE("unreadable media degrades to no signature rather than failing",
"[audio_signature][short]") {
// UR-9 is an enhancement and must never be able to break a fetch.
CHECK_FALSE(compute_signature("/nonexistent/definitely-not-here.mkv").has_value());
}
/// TRACES: UT-102 | IR-004
TEST_CASE("the window is taken from the centre, not the head",
"[audio_signature][centre]") {
// Sampling from the centre is the whole reason the construction avoids the
// head and tail (logos, cold opens, credits), so it needs its own check:
// wrap the fixture's own 120 s in 90 s of silence either side and the
// signature of the 300 s file must be the golden value, byte for byte.
// Nothing else pins the seek offset — a head-anchored window would pass
// every other test in this file.
const std::optional<std::vector<float>>& mono = fixture_window();
REQUIRE(mono.has_value());
const std::size_t pad = 90 * kSampleRate;
std::vector<std::int16_t> padded(pad * 2 + mono->size(), 0);
for (std::size_t i = 0; i < mono->size(); ++i)
padded[pad + i] = static_cast<std::int16_t>(std::llround(double((*mono)[i]) * 32768.0));
TempWav w("centred300");
w.write(padded, kSampleRate, 1);
const std::optional<std::string> sig = compute_signature(w.path.string());
REQUIRE(sig.has_value());
CHECK(*sig == golden()["signature"].get<std::string>());
}
// ── UT-103 — downmix and resample ───────────────────────────────────────────
/// TRACES: UT-103 | IR-004
TEST_CASE("stereo, non-native sample rate yields the same peak-bin sequence",
"[audio_signature][resample]") {
// The golden fixture is already mono at 11025 Hz so the golden vector does
// not depend on the resampler's version. This case exercises the path that
// real media takes — downmix plus resample — by rebuilding the fixture's own
// audio as 22050 Hz stereo and checking the peak bins survive it.
const std::optional<std::vector<float>>& mono = fixture_window();
REQUIRE(mono.has_value());
std::vector<std::int16_t> stereo;
stereo.reserve(mono->size() * 4);
for (float f : *mono) {
const auto s = static_cast<std::int16_t>(std::llround(double(f) * 32768.0));
stereo.push_back(s); stereo.push_back(s); // sample 1, L/R
stereo.push_back(s); stereo.push_back(s); // sample 2 (zero-order hold)
}
TempWav w("stereo22050");
w.write(stereo, 2 * kSampleRate, 2);
const std::optional<std::string> sig = compute_signature(w.path.string());
REQUIRE(sig.has_value());
const std::vector<std::uint8_t> got = base64_decode(sig->substr(3));
const std::vector<std::uint8_t> want =
base64_decode(golden()["signature"].get<std::string>().substr(3));
REQUIRE(got.size() == want.size());
std::size_t agree = 0;
for (std::size_t i = 0; i < got.size(); ++i)
agree += ((got[i] >> 2) == (want[i] >> 2)) ? 1 : 0;
// The server treats ≥ 0.85 as the `audio` match tier; this path scores 1.0
// in practice, and the margin is left for libswresample version drift.
CHECK(double(agree) / double(got.size()) >= 0.85);
}
// ── UT-104 — the pure DSP surface ───────────────────────────────────────────
/// TRACES: UT-104 | IR-004
TEST_CASE("pack_frames uses whole frames only", "[audio_signature][dsp]") {
CHECK(pack_frames(std::vector<float>(kFrameSize - 1, 0.f)).empty());
CHECK(pack_frames(std::vector<float>(kFrameSize, 0.f)).size() == 1);
CHECK(pack_frames(std::vector<float>(kFrameSize + kHopSize - 1, 0.f)).size() == 1);
CHECK(pack_frames(std::vector<float>(kFrameSize + kHopSize, 0.f)).size() == 2);
// The full 120 s window is 1288 frames — asserted as a constant rather than
// by running the DSP over 1.3 M zeros, which is the same claim for free.
CHECK(kWindowSamples == 1323000u);
CHECK(kExpectedFrames == 1288u);
CHECK_FALSE(signature_from_mono(std::vector<float>(kFrameSize - 1, 0.f)).has_value());
}
/// TRACES: UT-104 | IR-004
TEST_CASE("a pure tone lands in the band that contains it", "[audio_signature][dsp]") {
// 1000 Hz sits in log-band floor(32 * log10(1000/300)) = 16.
const int expect = static_cast<int>(std::floor(
kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));
std::vector<float> mono(kWindowSamples / 100);
for (std::size_t i = 0; i < mono.size(); ++i)
mono[i] = static_cast<float>(0.5 * std::sin(
2.0 * 3.14159265358979323846 * 1000.0 * double(i) / kSampleRate));
const std::vector<std::uint8_t> packed = pack_frames(mono);
REQUIRE_FALSE(packed.empty());
for (std::uint8_t b : packed) CHECK(((b >> 2) & 0x1F) == expect);
}
/// TRACES: UT-104 | IR-004
TEST_CASE("signature is invariant to overall gain", "[audio_signature][dsp]") {
// Loudness normalisation between two releases of the same cut must not
// change the signature — that is why the energy class is relative.
std::vector<float> a(kWindowSamples / 50);
for (std::size_t i = 0; i < a.size(); ++i) {
const double t = double(i) / kSampleRate;
a[i] = static_cast<float>(0.4 * std::sin(2.0 * 3.14159265358979323846 * 640.0 * t) +
0.2 * std::sin(2.0 * 3.14159265358979323846 * 1900.0 * t) *
std::sin(2.0 * 3.14159265358979323846 * 0.7 * t));
}
std::vector<float> b(a.size());
for (std::size_t i = 0; i < a.size(); ++i) b[i] = a[i] * 0.25f;
CHECK(pack_frames(a) == pack_frames(b));
}
/// TRACES: UT-104 | IR-004
TEST_CASE("base64 encoder matches the standard alphabet and padding",
"[audio_signature][dsp]") {
auto enc = [](const std::string& s) {
return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());
};
CHECK(enc("") == "");
CHECK(enc("f") == "Zg==");
CHECK(enc("fo") == "Zm8=");
CHECK(enc("foo") == "Zm9v");
CHECK(enc("foob") == "Zm9vYg==");
CHECK(enc("fooba") == "Zm9vYmE=");
CHECK(enc("foobar") == "Zm9vYmFy");
const std::uint8_t all[] = {0xFB, 0xFF, 0xBF}; // exercises '+' and '/'
CHECK(base64_encode(all, 3) == "+/+/");
}
tests/test_calibration.cpp
+93
View File
@@ -168,3 +168,96 @@ TEST_CASE("calibrate_gallery_cached treats hash=0 as always-recompute", "[calibr
CHECK(recomputed);
CHECK(cal.valid);
}
// ── GR-003 — the build report ────────────────────────────────────────────────
#include "gallery/gallery_report.hpp"
namespace {
// A unit vector on one axis. Distinct axes are orthogonal, which is unrealistic
// as a same-actor cluster but irrelevant here: these tests count actors, they do
// not assess fit quality.
Embedding unit_axis(int slot) {
Embedding e{};
e[slot % 512] = 1.0f;
return e;
}
} // namespace
TEST_CASE("report surfaces actors that can never be recognised", "[report][GR-003]") {
// An actor with no usable image is a silent recall ceiling: the pipeline
// will never name them, and nothing in the gallery says why. This is the
// single most useful number in the report.
ActorGallery g;
for (int a = 0; a < 3; ++a) {
ActorGallery::Actor act;
act.name = "actor" + std::to_string(a);
if (a != 1) // actor1 gets nothing
for (int i = 0; i < 6; ++i) act.embeddings.push_back(unit_axis(a * 10 + i));
g.actors.push_back(std::move(act));
}
std::vector<Embedding> flat;
std::vector<int> flat_actor;
for (int a = 0; a < static_cast<int>(g.actors.size()); ++a)
for (const auto& e : g.actors[a].embeddings) { flat.push_back(e); flat_actor.push_back(a); }
GalleryCalibrationStats stats;
GalleryCalibration cal = calibrate_gallery(flat, flat_actor, &stats);
GalleryReport r = build_gallery_report(g, cal, stats);
// An actor present in the gallery with no embeddings is counted as
// in-gallery but contributes nothing; the zero-usable list is populated
// from the build audit, which a stored gallery cannot supply.
CHECK(r.actors_in_gallery == 3);
CHECK(r.actors[1].references == 0);
}
TEST_CASE("report surfaces actors too thin to calibrate on", "[report][GR-003]") {
// Below the positive-pair threshold an actor contributes nothing to the
// intra-class side of the fit. They are not broken, so nothing complains —
// they just quietly weaken every threshold downstream.
ActorGallery g;
for (int a = 0; a < 2; ++a) {
ActorGallery::Actor act;
act.name = "actor" + std::to_string(a);
const int n = (a == 0) ? 6 : 2; // actor1 is under-referenced
for (int i = 0; i < n; ++i) act.embeddings.push_back(unit_axis(a * 10 + i));
g.actors.push_back(std::move(act));
}
std::vector<Embedding> flat;
std::vector<int> flat_actor;
for (int a = 0; a < static_cast<int>(g.actors.size()); ++a)
for (const auto& e : g.actors[a].embeddings) { flat.push_back(e); flat_actor.push_back(a); }
GalleryCalibrationStats stats;
GalleryCalibration cal = calibrate_gallery(flat, flat_actor, &stats);
GalleryReport r = build_gallery_report(g, cal, stats);
CHECK(r.actors_below_positive_threshold >= 1);
}
TEST_CASE("report round-trips", "[report][GR-003]") {
ActorGallery g;
ActorGallery::Actor act;
act.name = "solo";
for (int i = 0; i < 6; ++i) act.embeddings.push_back(unit_axis(i));
g.actors.push_back(std::move(act));
std::vector<Embedding> flat;
std::vector<int> flat_actor;
for (const auto& e : g.actors[0].embeddings) { flat.push_back(e); flat_actor.push_back(0); }
GalleryCalibrationStats stats;
GalleryCalibration cal = calibrate_gallery(flat, flat_actor, &stats);
GalleryReport r = build_gallery_report(g, cal, stats);
const std::string path = "/tmp/gr003_roundtrip.report.json";
save_gallery_report(path, r);
GalleryReport back = load_gallery_report(path);
CHECK(back.actors_in_gallery == r.actors_in_gallery);
CHECK(back.actors_below_positive_threshold == r.actors_below_positive_threshold);
CHECK(back.calib_a == r.calib_a);
std::remove(path.c_str());
}
tests/test_face_tracker.cpp
+117 -53
View File
@@ -13,8 +13,12 @@
#include "config.hpp"
#include "nodes/face_tracker_node.hpp"
#include "types.hpp"
#include "track_registry.hpp"
#include "evidence_discount.hpp"
#include <algorithm>
#include <cmath>
#include <memory>
namespace {
@@ -55,85 +59,145 @@ EmbeddedSceneFrame frame(double t, float x, float y, const Embedding& emb,
return ef;
}
Config tracker_cfg() {
Config cfg;
cfg.cut_revive_sim = 0.50f;
cfg.cut_inactive_max_frames = 5;
return cfg;
}
// Build a tracker over a fresh registry. The registry IS the tracker's state
// now (AR-008), so a test constructs both together and can inspect either.
struct Rig {
std::shared_ptr<TrackRegistry> reg;
FaceTrackerFunc ft;
explicit Rig(double extinction = 30.0, float assoc_min_prob = 0.5f)
: reg(std::make_shared<TrackRegistry>(
[extinction] {
TrackRegistry::Config c;
c.extinction_sec = extinction;
return c;
}(),
EvidenceDiscounter([](float cos) { return std::max(0.f, cos); })))
, ft([&] {
Config c;
c.track_assoc_min_prob = assoc_min_prob;
return c;
}(),
reg,
// Trivial calibration: cosine passed through as P(same). Real runs use
// the fit belonging to the active embedder (AR-023/AR-024).
[](float cos) { return std::max(0.f, cos); })
{}
int track_of(EmbeddedSceneFrame f) { return ft(std::move(f)).track_ids[0]; }
};
} // namespace
TEST_CASE("track id is stable across ordinary frames", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
// ── AR-008 — one pool, ordinary association ──────────────────────────────────
TEST_CASE("track id is stable across ordinary frames", "[face_tracker][AR-008]") {
Rig r;
Embedding e = axis(0);
int id0 = ft(frame(0.0, 10, 10, e)).track_ids[0];
int id1 = ft(frame(1.0, 11, 10, e)).track_ids[0]; // overlaps → same track
int id0 = r.track_of(frame(0.0, 10, 10, e));
int id1 = r.track_of(frame(1.0, 11, 10, e)); // overlaps → same track
CHECK(id0 >= 0);
CHECK(id1 == id0);
}
TEST_CASE("cut revives the same track id for a matching identity", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
TEST_CASE("a face lost across a cut and re-associated is the SAME track",
"[face_tracker][AR-008]") {
// Previously this was a distinct "revival" path guarded by a raw-cosine
// constant. There is no such path now: a dormant track is an ordinary
// association candidate, and continuity falls out of the embedding match.
Rig r;
// Pre-cut: establish a track for a person whose embedding is near-identical
// across the cut (sim well above cut_revive_sim), but whose box jumps so IoU
// is 0 — the ordinary spatial path cannot re-link it.
Embedding pre = at_sim(0, 1, 0.99f);
int id_pre = ft(frame(0.0, 10, 10, pre)).track_ids[0];
int id_pre = r.track_of(frame(0.0, 10, 10, pre));
REQUIRE(id_pre >= 0);
Embedding post = at_sim(0, 1, 0.98f); // cos(diff) ≈ 0.9997 > 0.50
auto out = ft(frame(1.0, 300, 300, post, /*is_cut=*/true));
CHECK(out.track_ids[0] == id_pre); // revived, not a fresh id
// Box jumps so IoU is zero — only the embedding can link it.
Embedding post = at_sim(0, 1, 0.98f);
CHECK(r.track_of(frame(1.0, 300, 300, post, /*is_cut=*/true)) == id_pre);
}
TEST_CASE("cut starts a fresh track when identity does not match", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
int id_pre = ft(frame(0.0, 10, 10, axis(0))).track_ids[0];
TEST_CASE("a cut starts a fresh track when identity does not match",
"[face_tracker][AR-008]") {
Rig r;
int id_pre = r.track_of(frame(0.0, 10, 10, axis(0)));
REQUIRE(id_pre >= 0);
// Post-cut face is orthogonal (sim 0 < cut_revive_sim) and spatially disjoint
// → no revival, brand-new id.
auto out = ft(frame(1.0, 300, 300, axis(5), /*is_cut=*/true));
CHECK(out.track_ids[0] != id_pre);
CHECK(out.track_ids[0] >= 0);
// Orthogonal embedding and disjoint box: nothing links them.
int id_post = r.track_of(frame(1.0, 300, 300, axis(5), /*is_cut=*/true));
CHECK(id_post != id_pre);
CHECK(id_post >= 0);
}
TEST_CASE("parked track expires after cut_inactive_max_frames", "[face_tracker]") {
Config cfg = tracker_cfg();
cfg.cut_inactive_max_frames = 2;
FaceTrackerFunc ft(cfg);
// ── AR-007 — a cut makes association ignore position ─────────────────────────
TEST_CASE("on a cut, identity follows the embedding rather than the box",
"[face_tracker][AR-007]") {
// Two people swap screen positions across a cut while keeping their faces.
// If IoU still carried weight the ids would follow the boxes and swap; with
// alpha driven to embedding-only on a cut, they must follow the faces.
Rig r;
Embedding a = at_sim(0, 1, 0.99f);
Embedding b = at_sim(2, 3, 0.99f);
EmbeddedSceneFrame f0;
f0.source.timestamp_sec = 0.0;
f0.faces = {face_at(10, 10), face_at(300, 300)};
f0.crops = {cv::Mat(), cv::Mat()};
f0.embeddings = {a, b};
auto out0 = r.ft(std::move(f0));
const int id_a = out0.track_ids[0];
const int id_b = out0.track_ids[1];
REQUIRE(id_a >= 0);
REQUIRE(id_b >= 0);
REQUIRE(id_a != id_b);
// Same two people, positions exchanged, on a cut frame.
EmbeddedSceneFrame f1;
f1.source.timestamp_sec = 1.0;
f1.source.is_cut = true;
f1.faces = {face_at(300, 300), face_at(10, 10)};
f1.crops = {cv::Mat(), cv::Mat()};
f1.embeddings = {a, b};
auto out1 = r.ft(std::move(f1));
CHECK(out1.track_ids[0] == id_a); // A kept its id despite moving to B's box
CHECK(out1.track_ids[1] == id_b);
}
// ── AR-013 — extinction replaces the parked-pool frame counter ───────────────
TEST_CASE("a track past the extinction window is gone, not revived",
"[face_tracker][AR-013]") {
// The old design aged a parked pool in frames, which silently changed
// meaning with sample_fps. Extinction is in seconds and lives in the
// registry, so the tracker no longer counts anything.
Rig r(/*extinction=*/2.0);
Embedding person = at_sim(0, 1, 0.99f);
int id_pre = ft(frame(0.0, 10, 10, person)).track_ids[0];
int id_pre = r.track_of(frame(0.0, 10, 10, person));
REQUIRE(id_pre >= 0);
// Cut with an unrelated face parks id_pre; then let the pool age past its
// limit with more unrelated, spatially-disjoint faces (each ages the pool by
// one). By the time the person returns, id_pre must be gone.
ft(frame(1.0, 300, 300, axis(7), /*is_cut=*/true)); // park (age 1)
ft(frame(2.0, 300, 300, axis(7))); // age 2
ft(frame(3.0, 300, 300, axis(7))); // age 3 → id_pre dropped
// Unrelated faces elsewhere while the clock runs well past extinction.
r.track_of(frame(1.0, 300, 300, axis(7), /*is_cut=*/true));
r.track_of(frame(10.0, 300, 300, axis(7)));
auto out = ft(frame(4.0, 10, 10, person)); // same identity returns
CHECK(out.track_ids[0] != id_pre); // too late — fresh id
CHECK(r.track_of(frame(11.0, 10, 10, person)) != id_pre);
}
TEST_CASE("eof clears active and parked tracks", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
Embedding person = at_sim(0, 1, 0.99f);
int id_pre = ft(frame(0.0, 10, 10, person)).track_ids[0];
ft(frame(1.0, 300, 300, axis(7), /*is_cut=*/true)); // park id_pre
TEST_CASE("a track within the extinction window is still a candidate",
"[face_tracker][AR-013]") {
Rig r(/*extinction=*/30.0);
Embedding person = at_sim(0, 1, 0.99f);
int id_pre = r.track_of(frame(0.0, 10, 10, person));
r.track_of(frame(1.0, 300, 300, axis(7), /*is_cut=*/true));
// Back inside the window: the same person continues the same track, so the
// gap is absorbed into one window rather than splitting it.
CHECK(r.track_of(frame(3.0, 10, 10, person)) == id_pre);
}
TEST_CASE("eof is forwarded", "[face_tracker]") {
Rig r;
EmbeddedSceneFrame eof;
eof.source.eof = true;
auto out = ft(std::move(eof));
CHECK(out.source.eof);
// After eof the pools are empty: the returning identity must get a fresh id,
// not the parked one.
auto out2 = ft(frame(2.0, 10, 10, person));
CHECK(out2.track_ids[0] != id_pre);
CHECK(r.ft(std::move(eof)).source.eof);
}
tests/test_face_utils.cpp
+107 -2
View File
@@ -1,6 +1,8 @@
// TRACES: AR-005, AR-030 | SR-002
//
// Unit tests for the geometric/numeric helpers in types.hpp and face_utils.hpp:
// cosine_similarity and the ArcFace 5-point alignment transform. GPU-free,
// model-free.
// cosine_similarity, the ArcFace 5-point alignment transform, and the alignment
// residual that AR-030 reads as its visibility measure. GPU-free, model-free.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
@@ -74,3 +76,106 @@ TEST_CASE("align_face returns empty on degenerate (collinear) landmarks", "[face
cv::Mat crop = align_face(img, lm);
CHECK(crop.empty());
}
// ── AR-030: the alignment residual as a visibility measure ────────────────────
// These assert the *properties* the measure is relied on for, not a magic value.
// Each would fail under a RANSAC fit, which buys a small residual by discarding
// the very landmarks that carry the signal.
namespace {
std::array<cv::Point2f, 5> canonical() {
std::array<cv::Point2f, 5> lm;
for (int i = 0; i < 5; ++i) lm[i] = {kArcFaceRef[i][0], kArcFaceRef[i][1]};
return lm;
}
// Rotate by `deg` in-plane, scale uniformly, translate — i.e. exactly the 4 DoF
// the similarity transform models.
std::array<cv::Point2f, 5> similarity(const std::array<cv::Point2f, 5>& in,
float deg, float s, float tx, float ty) {
const float r = deg * 3.14159265358979f / 180.f;
const float c = std::cos(r), sn = std::sin(r);
std::array<cv::Point2f, 5> out;
for (int i = 0; i < 5; ++i)
out[i] = {s * (c * in[i].x - sn * in[i].y) + tx,
s * (sn * in[i].x + c * in[i].y) + ty};
return out;
}
// Squash x about the centroid by `k`: the anisotropic deformation an out-of-plane
// yaw produces, and the one a similarity provably cannot absorb.
std::array<cv::Point2f, 5> foreshorten(const std::array<cv::Point2f, 5>& in, float k) {
float cx = 0.f;
for (const auto& p : in) cx += p.x;
cx /= 5.f;
std::array<cv::Point2f, 5> out = in;
for (auto& p : out) p.x = cx + (p.x - cx) * k;
return out;
}
} // namespace
TEST_CASE("residual is zero for a face in canonical pose", "[face_utils][AR-030]") {
const Alignment a = estimate_alignment(canonical());
REQUIRE(a.ok);
CHECK_THAT(a.residual, WithinAbs(0.0f, 1e-3f));
}
TEST_CASE("residual ignores in-plane roll, scale and translation", "[face_utils][AR-030]") {
// The structural claim behind AR-030: the fit absorbs all four similarity
// DoF exactly, so what remains is only the deformation a similarity cannot
// explain. A rolled head must not read as a turned one.
for (float deg : {-40.f, -12.f, 0.f, 17.f, 65.f}) {
const Alignment a = estimate_alignment(similarity(canonical(), deg, 3.5f, 220.f, -40.f));
REQUIRE(a.ok);
CHECK_THAT(a.residual, WithinAbs(0.0f, 1e-3f));
}
}
TEST_CASE("residual rises monotonically with foreshortening", "[face_utils][AR-030]") {
float prev = -1.f;
for (float k : {1.0f, 0.9f, 0.75f, 0.5f, 0.3f}) {
const Alignment a = estimate_alignment(foreshorten(canonical(), k));
REQUIRE(a.ok);
CHECK(a.residual > prev);
prev = a.residual;
}
}
TEST_CASE("residual is independent of face size", "[face_utils][AR-030]") {
// The measure must not silently re-express face size — that is AR-002's job,
// and double-counting it would make a small frontal face look occluded.
// Same deformation, two very different face sizes, one answer.
const auto small = similarity(foreshorten(canonical(), 0.7f), 20.f, 1.0f, 0.f, 0.f);
const auto large = similarity(foreshorten(canonical(), 0.7f), 20.f, 12.0f, 500.f, 300.f);
const Alignment a = estimate_alignment(small);
const Alignment b = estimate_alignment(large);
REQUIRE(a.ok);
REQUIRE(b.ok);
CHECK_THAT(b.residual, WithinAbs(a.residual, 1e-2f));
}
TEST_CASE("the fit never mirrors the face", "[face_utils][AR-030]") {
// SVD will happily return an orientation-reversing solution; a similarity
// transform may rotate but never reflect. Without the determinant guard a
// mirrored landmark set fits "perfectly" as a reflection.
const auto mirrored = foreshorten(canonical(), -1.f);
const Alignment a = estimate_alignment(mirrored);
REQUIRE(a.ok);
const double det = a.M.at<double>(0,0) * a.M.at<double>(1,1)
- a.M.at<double>(0,1) * a.M.at<double>(1,0);
CHECK(det > 0.0);
CHECK(a.residual > 1.0f); // and the mirroring shows up as misfit
}
TEST_CASE("degenerate landmarks report not-ok rather than a residual", "[face_utils][AR-030]") {
std::array<cv::Point2f, 5> lm;
for (auto& p : lm) p = {50.f, 50.f};
const Alignment a = estimate_alignment(lm);
CHECK_FALSE(a.ok);
CHECK(a.M.empty());
}
tests/test_gallery_store.cpp
+251 -1
View File
@@ -1,13 +1,17 @@
// Unit tests for gallery (de)serialisation: HDF5 round-trip fidelity (the only
// format save_gallery writes), legacy JSON read back-compat (optional field
// defaults, the legacy "jellyfin_person_id" fallback). GPU-free, model-free.
// defaults, the legacy "jellyfin_person_id" fallback), and the GR-004 embedder
// stamp. GPU-free, model-free — the stamp tests exercise the comparison logic
// with synthetic stamps and never load an ONNX, so they run on CI's Intel N100.
#include <catch2/catch_test_macros.hpp>
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "types.hpp"
#include <cstdio>
#include <fstream>
#include <stdexcept>
#include <string>
#include <nlohmann/json.hpp>
@@ -27,6 +31,22 @@ Embedding make_embedding(float base) {
return e;
}
// A stamp built by hand — no ONNX is read, so these tests never need a model.
EmbedderStamp stamp(const std::string& name, const std::string& sha, int32_t dim = 512) {
EmbedderStamp s;
s.model_name = name;
s.model_sha256 = sha;
s.embed_dim = dim;
return s;
}
const std::string kShaA(64, 'a');
const std::string kShaB(64, 'b');
bool mentions(const std::string& haystack, const std::string& needle) {
return haystack.find(needle) != std::string::npos;
}
} // namespace
TEST_CASE("gallery save/load round-trips actors and embeddings", "[gallery]") {
@@ -153,3 +173,233 @@ TEST_CASE("load_gallery reads the legacy jellyfin_person_id key", "[gallery]") {
TEST_CASE("load_gallery throws on a missing file", "[gallery]") {
CHECK_THROWS(load_gallery("/nonexistent/path/gallery.json"));
}
// ── GR-004: gallery ↔ embedder binding ───────────────────────────────────────
// Verification plan row GR-004/T1: "Mismatched embedder → hard startup error;
// error names both sides." The comparison is a pure function over two stamps, so
// none of this needs a GPU, an ONNX, or even a file.
/// TRACES: GR-004 | SR-001
TEST_CASE("gallery save/load round-trips the embedder stamp", "[gallery][GR-004]") {
ActorGallery g;
ActorGallery::Actor a;
a.name = "Stamped Actor";
a.embeddings = {make_embedding(0.3f)};
g.actors.push_back(a);
g.embedder = stamp("LVFace-B_Glint360K.onnx", kShaA);
TempFile tf("gallery_stamped.h5");
save_gallery(tf.path, g);
ActorGallery loaded = load_gallery(tf.path);
CHECK(loaded.embedder.model_name == "LVFace-B_Glint360K.onnx");
CHECK(loaded.embedder.model_sha256 == kShaA);
CHECK(loaded.embedder.embed_dim == 512);
CHECK_FALSE(loaded.embedder.empty());
}
/// TRACES: GR-004 | SR-001
TEST_CASE("a gallery written without a stamp loads as unstamped", "[gallery][GR-004]") {
// The back-compat case: pre-GR-004 files have no /embedder group at all. The
// absence must survive the round trip as an absence — a stamp naming no model
// would read as "checked and fine" to every consumer.
ActorGallery g;
ActorGallery::Actor a;
a.name = "Legacy Actor";
a.embeddings = {make_embedding(0.f)};
g.actors.push_back(a);
TempFile tf("gallery_unstamped.h5");
save_gallery(tf.path, g);
ActorGallery loaded = load_gallery(tf.path);
CHECK(loaded.embedder.empty());
}
/// TRACES: GR-004 | SR-001
TEST_CASE("legacy JSON galleries carry an optional embedder stamp", "[gallery][GR-004]") {
nlohmann::json j;
j["embedder"] = {{"model_name", "arcface_w600k_r50.onnx"},
{"model_sha256", kShaB},
{"embed_dim", 512}};
j["actors"] = nlohmann::json::array();
nlohmann::json ja;
ja["name"] = "JSON Actor";
ja["embeddings"] = nlohmann::json::array();
ja["embeddings"].push_back(std::vector<float>(512, 0.1f));
j["actors"].push_back(ja);
TempFile tf("gallery_json_stamp.json");
{ std::ofstream out(tf.path); out << j.dump(); }
ActorGallery g = load_gallery(tf.path);
CHECK(g.embedder.model_name == "arcface_w600k_r50.onnx");
CHECK(g.embedder.model_sha256 == kShaB);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("matching embedder stamps pass", "[gallery][GR-004]") {
auto chk = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaA));
CHECK(chk.verdict == StampVerdict::match);
CHECK_FALSE(chk.fatal(false));
CHECK_FALSE(chk.fatal(true)); // a proven match is never fatal, even in strict mode
CHECK_NOTHROW(enforce_embedder_stamp(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaA),
"g.h5", "model.onnx", true));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("the hash decides, not the filename", "[gallery][GR-004]") {
// Same bytes under a different filename is the SAME model — a renamed or
// relocated file must not be treated as a different one.
auto same = compare_embedder_stamps(stamp("lvface.onnx", kShaA),
stamp("LVFace-B_Glint360K.onnx", kShaA));
CHECK(same.verdict == StampVerdict::match);
// Different bytes under the SAME filename is a DIFFERENT model — this is the
// in-place re-export a name-only stamp would miss entirely, and the reason the
// stamp carries a hash at all.
auto differ = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaB));
CHECK(differ.verdict == StampVerdict::mismatch);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("mismatched embedder is fatal and names both sides", "[gallery][GR-004]") {
const auto built = stamp("LVFace-B_Glint360K.onnx", kShaA);
const auto loaded = stamp("arcface_w600k_r50.onnx", kShaB);
auto chk = compare_embedder_stamps(built, loaded, "cast.h5", "models/r50.onnx");
REQUIRE(chk.verdict == StampVerdict::mismatch);
CHECK(chk.fatal(false)); // no bypass: a mismatch is fatal in every mode
CHECK(chk.fatal(true));
// Both sides must be identifiable from the message alone.
CHECK(mentions(chk.message, "LVFace-B_Glint360K.onnx"));
CHECK(mentions(chk.message, "arcface_w600k_r50.onnx"));
CHECK(mentions(chk.message, kShaA));
CHECK(mentions(chk.message, kShaB));
CHECK(mentions(chk.message, "cast.h5"));
CHECK(mentions(chk.message, "models/r50.onnx"));
// ...and it must reach the caller as an error, not a log line.
CHECK_THROWS_AS(enforce_embedder_stamp(built, loaded, "cast.h5",
"models/r50.onnx", false),
std::runtime_error);
try {
enforce_embedder_stamp(built, loaded, "cast.h5", "models/r50.onnx", false);
FAIL("mismatch must throw");
} catch (const std::runtime_error& e) {
const std::string what = e.what();
CHECK(mentions(what, "LVFace-B_Glint360K.onnx"));
CHECK(mentions(what, "arcface_w600k_r50.onnx"));
}
}
/// TRACES: GR-004 | SR-001
TEST_CASE("differing embedding width is a mismatch", "[gallery][GR-004]") {
auto chk = compare_embedder_stamps(stamp("a.onnx", kShaA, 512),
stamp("a.onnx", kShaA, 256));
CHECK(chk.verdict == StampVerdict::mismatch);
CHECK(mentions(chk.message, "512"));
CHECK(mentions(chk.message, "256"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("an unstamped gallery warns by default and fails under strict",
"[gallery][GR-004]") {
// Decision recorded in src/gallery/embedder_stamp.hpp: unstamped is UNKNOWN,
// not known-bad, and every pre-GR-004 gallery is unstamped. Hard-failing them
// all would make the check something people disable rather than trust; so it
// warns loudly, names the risk, and is promotable to fatal for measurement runs.
EmbedderStamp none;
auto chk = compare_embedder_stamps(none, stamp("model.onnx", kShaA), "old.h5");
REQUIRE(chk.verdict == StampVerdict::unstamped);
CHECK_FALSE(chk.fatal(false));
CHECK(chk.fatal(true));
CHECK(mentions(chk.message, "old.h5"));
CHECK(mentions(chk.message, "model.onnx")); // the loaded side is still named
CHECK(mentions(chk.message, "UNKNOWN")); // ...and the gallery side is honest
CHECK_NOTHROW(enforce_embedder_stamp(none, stamp("model.onnx", kShaA),
"old.h5", "model.onnx", false));
CHECK_THROWS_AS(enforce_embedder_stamp(none, stamp("model.onnx", kShaA),
"old.h5", "model.onnx", true),
std::runtime_error);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("an unidentifiable embedder against a stamped gallery is not silent",
"[gallery][GR-004]") {
// e.g. a replay whose dump predates GR-004: we know what built the gallery but
// not what produced the vectors being fed in. Unverifiable, so it must not
// report success.
auto chk = compare_embedder_stamps(stamp("model.onnx", kShaA), EmbedderStamp{},
"g.h5", "old dump.h5");
CHECK(chk.verdict == StampVerdict::unknown_embedder);
CHECK_FALSE(chk.fatal(false));
CHECK(chk.fatal(true));
CHECK(mentions(chk.message, "model.onnx"));
CHECK(mentions(chk.message, "old dump.h5"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("name-only agreement is a weak match, not a clean pass", "[gallery][GR-004]") {
// A TRT deployment can run from a prebuilt .engine with the .onnx absent, so
// no hash is computable. Names agreeing is evidence, not proof.
auto weak = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", ""));
CHECK(weak.verdict == StampVerdict::weak_match);
CHECK_FALSE(weak.fatal(false));
CHECK(weak.fatal(true));
// Names disagreeing with no hash available is still a mismatch — the weaker
// evidence is enough to convict, just not to acquit.
auto bad = compare_embedder_stamps(stamp("lvface.onnx", ""),
stamp("arcface.onnx", ""));
CHECK(bad.verdict == StampVerdict::mismatch);
CHECK(mentions(bad.message, "lvface.onnx"));
CHECK(mentions(bad.message, "arcface.onnx"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("sha256 matches the published vectors", "[gallery][GR-004]") {
// Pins the in-tree FIPS 180-4 implementation against the standard vectors.
// This is what guarantees the C++ stamp and the Python (hashlib) stamp in
// scripts/sae_gallery.py agree on the same model file — without it the two
// halves of GR-004 could silently diverge and every check would be a mismatch.
CHECK(sha256_hex("") ==
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855");
CHECK(sha256_hex("abc") ==
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad");
CHECK(sha256_hex("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq") ==
"248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1");
// Multi-block input, exercising the length-padding path past 64 bytes.
CHECK(sha256_hex(std::string(1000, 'a')) ==
"41edece42d63e8d9bf515a9ba6932e1c20cbc9f5a5d134645adb5db1b9737ea3");
}
/// TRACES: GR-004 | SR-001
TEST_CASE("make_embedder_stamp hashes a real file and degrades gracefully",
"[gallery][GR-004]") {
// Stands in for an ONNX: the stamp does not care what the bytes mean.
TempFile tf("fake_model.onnx");
{ std::ofstream out(tf.path, std::ios::binary); out << "abc"; }
EmbedderStamp s = make_embedder_stamp(tf.path);
CHECK(s.model_sha256 ==
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad");
CHECK_FALSE(s.model_name.empty());
CHECK(s.model_name.find('/') == std::string::npos); // basename, not full path
// A model path that does not exist still yields a comparable name-only stamp
// rather than an empty one, which is what keeps engine-only deployments usable.
EmbedderStamp missing = make_embedder_stamp("/nonexistent/models/foo.onnx");
CHECK(missing.model_name == "foo.onnx");
CHECK(missing.model_sha256.empty());
CHECK_FALSE(missing.empty());
CHECK(make_embedder_stamp("").empty());
}
tests/test_quality.cpp
+302
View File
@@ -0,0 +1,302 @@
// TRACES: AR-029 | SR-002
//
// T1 for the AR-029 sharpness candidates: the properties that have to hold
// before a study is allowed to pick between them. GPU-free, model-free.
//
// The register's acceptance criterion is "synthetic blur ladder ->
// monotonically falling sharpness; Gaussian vs motion blur; small sharp face vs
// large soft one — size must not leak into this axis". The last clause needs
// care, and the tests below split it in two:
//
// - What must NOT leak is *geometric* scale. The measure is taken in the
// canonical frame, so changing how big the face was in the source while
// preserving its detail must not move the score. That is structural: the
// window is fixed at 64x64 canonical px.
// - What DOES legitimately move the score is lost *detail*. A face that was
// 40 px before being warped up to 112 really does carry less
// high-frequency content than one that was 400 px, and a measure blind to
// that would be blind to the thing it exists to catch.
//
// So "size must not leak" cannot mean "invariant to the source face size", and
// the ladder test below asserts the opposite on purpose. What it buys is that
// the overlap with AR-002 is a recorded property with a test naming it, rather
// than a surprise VR-012 discovers when the two axes turn out to be correlated.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "quality.hpp"
#include "types.hpp" // kArcFaceRef, for the window-placement test
#include <algorithm>
#include <cmath>
#include <utility>
#include <vector>
using Catch::Matchers::WithinAbs;
using Catch::Matchers::WithinRel;
namespace {
// A deterministic 112x112 stand-in for a face crop.
//
// **Broadband, not a sum of a few sinusoids.** An earlier version of this
// fixture used three discrete spatial frequencies, and the resampling ladder
// below was non-monotone for hf_energy_ratio because of it: a period-7
// component downsampled to 32 px lands exactly at Nyquist and aliases, so the
// ratio rose at one rung instead of falling. That is a property of a
// three-tone test pattern meeting a resampler, not of the measure or of any
// face — a real crop has energy spread across the band, where such a
// resonance averages out. Deterministic value noise, smoothed to give the
// roughly 1/f falloff of a photograph, exercises the whole band at once.
//
// Mid-grey base with bounded amplitude, so scaling the contrast in the tests
// below does not clip.
cv::Mat synthetic_crop() {
// Fixed LCG rather than cv::randu: the suite must not depend on OpenCV's
// RNG state, which other tests share.
uint32_t seed = 0x5eed1234u;
auto next = [&seed] {
seed = seed * 1664525u + 1013904223u;
return (seed >> 16) & 0xffffu;
};
cv::Mat noise(112, 112, CV_32F);
for (int y = 0; y < 112; ++y)
for (int x = 0; x < 112; ++x)
noise.at<float>(y, x) = float(next()) / 65535.f - 0.5f;
// Mild smoothing: white noise is flat to Nyquist, which no lens produces
// and which would make the sharpest rung of every ladder unrealistic.
cv::Mat smooth;
cv::GaussianBlur(noise, smooth, cv::Size(0, 0), 0.8);
cv::normalize(smooth, smooth, -1.0, 1.0, cv::NORM_MINMAX);
cv::Mat img(112, 112, CV_8UC3);
for (int y = 0; y < 112; ++y) {
for (int x = 0; x < 112; ++x) {
double v = 128.0 + 70.0 * smooth.at<float>(y, x);
const auto b = static_cast<uchar>(std::clamp(v, 0.0, 255.0));
img.at<cv::Vec3b>(y, x) = {b, b, b};
}
}
return img;
}
cv::Mat gaussian(const cv::Mat& src, double sigma) {
cv::Mat out;
cv::GaussianBlur(src, out, cv::Size(0, 0), sigma, sigma);
return out;
}
// Horizontal box blur — the camera-pan case, and the one an isotropic measure
// could in principle miss.
cv::Mat motion(const cv::Mat& src, int len) {
cv::Mat kernel = cv::Mat::zeros(1, len, CV_32F);
kernel.setTo(1.0f / len);
cv::Mat out;
cv::filter2D(src, out, -1, kernel);
return out;
}
// Throw away detail a face detected at size x size never had, then warp back up
// to the 112x112 the embedder is fed — the VR-005 degradation.
cv::Mat rescale(const cv::Mat& src, int size) {
if (size == 112) return src.clone();
cv::Mat small, out;
cv::resize(src, small, {size, size}, 0, 0, cv::INTER_AREA);
cv::resize(small, out, {112, 112}, 0, 0, cv::INTER_LINEAR);
return out;
}
std::vector<float> field(const std::vector<SharpnessScores>& s,
float SharpnessScores::* m) {
std::vector<float> v;
v.reserve(s.size());
for (const auto& x : s) v.push_back(x.*m);
return v;
}
void check_strictly_falling(const std::vector<float>& v, const char* what) {
INFO(what);
for (size_t i = 1; i < v.size(); ++i) {
INFO("step " << i << ": " << v[i - 1] << " -> " << v[i]);
CHECK(v[i] < v[i - 1]);
}
}
const std::vector<std::pair<const char*, float SharpnessScores::*>> kMeasures{
{"var_laplacian", &SharpnessScores::var_laplacian},
{"norm_var_laplacian", &SharpnessScores::norm_var_laplacian},
{"tenengrad", &SharpnessScores::tenengrad},
{"hf_energy_ratio", &SharpnessScores::hf_energy_ratio},
{"dir_min_tenengrad", &SharpnessScores::dir_min_tenengrad},
};
} // namespace
TEST_CASE("every candidate falls monotonically along a Gaussian blur ladder",
"[quality][AR-029]") {
const cv::Mat base = synthetic_crop();
std::vector<SharpnessScores> ladder;
for (double sigma : {0.0, 0.5, 1.0, 1.5, 2.0, 3.0})
ladder.push_back(assess_sharpness(sigma == 0.0 ? base : gaussian(base, sigma)));
for (const auto& [name, m] : kMeasures) {
REQUIRE(ladder.front().ok);
check_strictly_falling(field(ladder, m), name);
}
}
TEST_CASE("only the absolute and directional measures survive motion blur",
"[quality][AR-029]") {
// Motion blur is the commonest way a film frame is unusable, and it is
// where the candidates separate. A horizontal smear destroys horizontal
// detail and leaves vertical detail untouched, so what a measure does here
// depends on whether it can be fooled by the surviving axis.
const cv::Mat base = synthetic_crop();
std::vector<SharpnessScores> ladder{assess_sharpness(base)};
for (int len : {3, 5, 9, 15, 21})
ladder.push_back(assess_sharpness(motion(base, len)));
// Total gradient/Laplacian energy keeps falling: nothing replaces what the
// smear removed.
check_strictly_falling(field(ladder, &SharpnessScores::var_laplacian),
"var_laplacian");
check_strictly_falling(field(ladder, &SharpnessScores::tenengrad),
"tenengrad");
// The fix for the two below: low-frequency denominator, and the worse of
// the two axes rather than their sum.
check_strictly_falling(field(ladder, &SharpnessScores::dir_min_tenengrad),
"dir_min_tenengrad");
// The disqualifying behaviour, pinned rather than hidden. Both measures
// normalise by a quantity that contains the detail they are measuring, so
// once the horizontal band is gone the quotient climbs back toward its
// unblurred value: each is U-shaped in blur length, and a single score
// maps to two very different amounts of blur. A 21 px smear scores about
// as sharp as a 3 px one.
for (const auto& [name, m] : {
std::pair{"norm_var_laplacian", &SharpnessScores::norm_var_laplacian},
std::pair{"hf_energy_ratio", &SharpnessScores::hf_energy_ratio}}) {
const std::vector<float> v = field(ladder, m);
INFO(name);
const auto trough = std::min_element(v.begin(), v.end());
CHECK(trough != v.begin()); // it does fall at first …
CHECK(trough != v.end() - 1); // … then turns back up
CHECK(v.back() > 0.8f * v[1]); // recovering most of one rung
}
}
TEST_CASE("sharpness falls under downscale-upscale as well as under blur",
"[quality][AR-029]") {
// The overlap with AR-002, asserted rather than assumed. Losing resolution
// and losing focus are the same loss of high-frequency content, so every
// candidate reads a small upscaled face as less sharp. VR-012's joint
// size x sigma grid decides whether that makes a sharpness discount a
// double-count against the size gate, or whether the two axes carry
// separable information.
const cv::Mat base = synthetic_crop();
std::vector<SharpnessScores> ladder;
for (int size : {112, 64, 48, 32, 24, 16})
ladder.push_back(assess_sharpness(rescale(base, size)));
for (const auto& [name, m] : kMeasures)
check_strictly_falling(field(ladder, m), name);
}
TEST_CASE("the ratio measures are contrast-free and the raw ones are not",
"[quality][AR-029]") {
// The confound that decides the bake-off. A gallery drawn from thousands of
// cameras, lighting setups and JPEG pipelines varies enormously in
// contrast, and a measure that reads a low-contrast sharp face as blurred
// would discount it for the photographer's choices rather than for anything
// the embedder cares about.
const cv::Mat base = synthetic_crop();
// Halve the contrast about mid-grey, leaving spatial structure untouched.
cv::Mat low;
base.convertTo(low, CV_8UC3, 0.5, 64.0);
const auto s_hi = assess_sharpness(base);
const auto s_lo = assess_sharpness(low);
REQUIRE(s_hi.ok);
REQUIRE(s_lo.ok);
// Invariant by construction: both are ratios in which the contrast factor
// cancels.
CHECK_THAT(s_lo.norm_var_laplacian,
WithinRel(s_hi.norm_var_laplacian, 0.02f));
CHECK_THAT(s_lo.hf_energy_ratio, WithinRel(s_hi.hf_energy_ratio, 0.02f));
// Not invariant: both scale with the square of the contrast factor, so
// halving the contrast quarters them. This is the disqualifying behaviour,
// pinned so that a change making them contrast-free is a deliberate one.
CHECK_THAT(s_lo.var_laplacian, WithinRel(0.25f * s_hi.var_laplacian, 0.05f));
CHECK_THAT(s_lo.tenengrad, WithinRel(0.25f * s_hi.tenengrad, 0.05f));
}
TEST_CASE("brightness alone moves nothing", "[quality][AR-029]") {
const cv::Mat base = synthetic_crop();
cv::Mat bright;
base.convertTo(bright, CV_8UC3, 1.0, 20.0);
const auto a = assess_sharpness(base);
const auto b = assess_sharpness(bright);
for (const auto& [name, m] : kMeasures) {
INFO(name);
CHECK_THAT(b.*m, WithinRel(a.*m, 0.02f));
}
}
TEST_CASE("a flat crop is scored not-ok rather than given a number",
"[quality][AR-029]") {
// A face whose sharpness cannot be computed is a fact to record, not an
// absence — the same rule AR-030 follows for degenerate landmarks.
const cv::Mat flat(112, 112, CV_8UC3, cv::Scalar(128, 128, 128));
const auto s = assess_sharpness(flat);
CHECK_FALSE(s.ok);
for (const auto& [name, m] : kMeasures) {
INFO(name);
CHECK_THAT(s.*m, WithinAbs(0.0f, 1e-6f));
CHECK_FALSE(std::isnan(s.*m));
}
}
TEST_CASE("a crop smaller than the measurement window is scored not-ok",
"[quality][AR-029]") {
const cv::Mat small(64, 64, CV_8UC3, cv::Scalar(40, 90, 160));
CHECK_FALSE(assess_sharpness(small).ok);
CHECK_FALSE(assess_sharpness(cv::Mat()).ok);
}
TEST_CASE("the measurement window covers the face interior of the crop",
"[quality][AR-029]") {
// The landmarks the ArcFace template pins must all fall inside the window,
// or the measure is scoring background and hair rather than the face.
const cv::Rect w = sharpness_window();
CHECK(w.x >= 0);
CHECK(w.y >= 0);
CHECK(w.x + w.width <= 112);
CHECK(w.y + w.height <= 112);
for (int i = 0; i < 5; ++i) {
INFO("landmark " << i);
CHECK(w.contains(cv::Point(static_cast<int>(kArcFaceRef[i][0]),
static_cast<int>(kArcFaceRef[i][1]))));
}
}
TEST_CASE("a single-channel crop scores the same as its BGR equivalent",
"[quality][AR-029]") {
// The dump replays crops; nothing should depend on whether they arrived as
// three identical channels or one.
const cv::Mat base = synthetic_crop();
cv::Mat gray;
cv::cvtColor(base, gray, cv::COLOR_BGR2GRAY);
const auto a = assess_sharpness(base);
const auto b = assess_sharpness(gray);
for (const auto& [name, m] : kMeasures) {
INFO(name);
CHECK_THAT(b.*m, WithinRel(a.*m, 1e-3f));
}
}
tests/test_replay_fixtures.cpp
+258
View File
@@ -0,0 +1,258 @@
// Replay tests — the real tracker and registry driven from committed fixtures.
//
// TRACES: AR-004, AR-012, AR-013 | VR-001, VR-002 | IT-001
//
// Tier T2: composition, not units. The registry tests construct awkward states
// directly; these check that the pieces behave when wired together and fed real
// footage — 480x360 public-domain clips at 5 fps, with the cuts, gaps and
// crowded frames that actual film produces and synthetic input does not.
//
// No GPU and no model: the fixtures are HDF5 dumps taken after embedding, so
// everything here is CPU maths. That is what lets this run on the CI host at
// all (see docs/requirements.md, "CI never calls a model").
//
// Driving the node functors directly rather than through a KPN network is
// deliberate: functors are plain objects, so there are no threads, no channels
// and no scheduling — the same input gives the same output every time, which is
// exactly what a fixture-based test needs.
#include <catch2/catch_test_macros.hpp>
#include "config.hpp"
#include "evidence_discount.hpp"
#include "nodes/face_tracker_node.hpp"
#include "track_registry.hpp"
#include "types.hpp"
#include <H5Cpp.h>
#include <algorithm>
#include <cmath>
#include <memory>
#include <string>
#include <vector>
namespace {
// ── Fixture reader ───────────────────────────────────────────────────────────
// The flat/ragged layout of scripts/optimizer/SCHEMA.md: per-face arrays
// concatenated, with a per-frame index table pointing into them.
struct Dump {
std::vector<double> ts;
std::vector<uint8_t> is_cut;
std::vector<int64_t> face_offset;
std::vector<int32_t> face_count;
std::vector<Embedding> emb;
std::vector<float> bbox; // 4 per face
std::string embedder;
std::size_t frames() const { return ts.size(); }
std::size_t faces() const { return emb.size(); }
};
template <typename T>
std::vector<T> read1d(H5::Group& g, const char* name, const H5::DataType& dt) {
H5::DataSet ds = g.openDataSet(name);
hsize_t n = 0;
ds.getSpace().getSimpleExtentDims(&n, nullptr);
std::vector<T> out(n);
if (n) ds.read(out.data(), dt);
return out;
}
Dump load(const std::string& path) {
H5::H5File f(path, H5F_ACC_RDONLY);
H5::Group frames = f.openGroup("frames");
H5::Group faces = f.openGroup("faces");
Dump d;
d.ts = read1d<double>(frames, "timestamp_sec", H5::PredType::NATIVE_DOUBLE);
d.is_cut = read1d<uint8_t>(frames, "is_cut", H5::PredType::NATIVE_UINT8);
d.face_offset = read1d<int64_t>(frames, "face_offset", H5::PredType::NATIVE_INT64);
d.face_count = read1d<int32_t>(frames, "face_count", H5::PredType::NATIVE_INT32);
H5::DataSet e = faces.openDataSet("embedding");
hsize_t dims[2]{0, 0};
e.getSpace().getSimpleExtentDims(dims, nullptr);
std::vector<float> flat(dims[0] * dims[1]);
if (!flat.empty()) e.read(flat.data(), H5::PredType::NATIVE_FLOAT);
d.emb.resize(dims[0]);
for (hsize_t i = 0; i < dims[0]; ++i)
std::copy_n(flat.begin() + i * dims[1], 512, d.emb[i].begin());
// bbox is 2-D [N,4]; reading it with the 1-D helper would size the buffer
// from the first extent only and then read four times that many floats.
{
H5::DataSet bs = faces.openDataSet("bbox");
hsize_t bd[2]{0, 0};
bs.getSpace().getSimpleExtentDims(bd, nullptr);
d.bbox.resize(bd[0] * bd[1]);
if (!d.bbox.empty()) bs.read(d.bbox.data(), H5::PredType::NATIVE_FLOAT);
}
// GR-004: the dump records which embedder produced it, so a replay cannot
// be silently scored against a gallery from a different model.
if (f.attrExists("embedder_model")) {
// Written as a variable-length string (embedding_dump_node.hpp:99), so
// the read must name the same type explicitly.
H5::StrType vlen(H5::PredType::C_S1, H5T_VARIABLE);
f.openAttribute("embedder_model").read(vlen, d.embedder);
}
return d;
}
std::string fixture(const char* name) {
return std::string(SAE_TEST_FIXTURES_DIR) + "/dumps/" + name;
}
// ── Harness ──────────────────────────────────────────────────────────────────
struct Replay {
std::vector<DeadTrack> claims;
std::vector<int> track_ids; // per face, in fixture order
std::size_t faces_seen{0};
};
Replay run(const Dump& d, double extinction = 10.0) {
Replay r;
TrackRegistry::Config rc;
rc.extinction_sec = extinction;
auto cal = [](float cos) { return std::max(0.f, cos); };
auto reg = std::make_shared<TrackRegistry>(rc, EvidenceDiscounter(cal));
reg->on_track_dead([&r](const DeadTrack& t) { r.claims.push_back(t); });
Config cfg;
cfg.track_assoc_min_prob = 0.5f;
FaceTrackerFunc ft(cfg, reg, cal);
for (std::size_t i = 0; i < d.frames(); ++i) {
EmbeddedSceneFrame ef;
ef.source.timestamp_sec = d.ts[i];
ef.source.is_cut = d.is_cut[i] != 0;
const int64_t off = d.face_offset[i];
const int32_t n = d.face_count[i];
for (int32_t k = 0; k < n; ++k) {
DetectedFace face;
const float* b = &d.bbox[(off + k) * 4];
face.bbox = cv::Rect2f(b[0], b[1], b[2], b[3]);
face.confidence = 1.0f;
ef.faces.push_back(face);
ef.crops.push_back(cv::Mat());
ef.embeddings.push_back(d.emb[off + k]);
}
r.faces_seen += static_cast<std::size_t>(n);
auto out = ft(std::move(ef));
for (int id : out.track_ids) r.track_ids.push_back(id);
}
reg->flush(d.ts.empty() ? 0.0 : d.ts.back());
return r;
}
} // namespace
// ── AR-004 / VR-001 — the fixtures are intact and self-describing ────────────
TEST_CASE("fixtures are complete and carry their embedder identity",
"[replay][AR-004][VR-001]") {
// Frame counts are exact rather than approximate. Before node outputs
// blocked on a full channel, generation lost most of a clip and what it
// lost depended on timing — these numbers could not have been asserted.
struct Expect { const char* file; std::size_t frames, faces; };
const Expect all[] = {
{"bali_13.h5", 385, 693},
{"bali_27.h5", 335, 335},
{"bali_28.h5", 345, 368},
{"bali_31.h5", 145, 203},
{"bali_46.h5", 385, 140},
};
for (const auto& x : all) {
INFO(x.file);
Dump d = load(fixture(x.file));
CHECK(d.frames() == x.frames);
CHECK(d.faces() == x.faces);
CHECK(d.embedder == "LVFace-B_Glint360K.onnx");
// face_offset must be contiguous: a gap means faces went missing
// between frames, which no consumer could detect.
int64_t running = 0;
for (std::size_t i = 0; i < d.frames(); ++i) {
REQUIRE(d.face_offset[i] == running);
running += d.face_count[i];
}
CHECK(static_cast<std::size_t>(running) == d.faces());
}
}
// ── VR-002 — replay is deterministic ─────────────────────────────────────────
TEST_CASE("replaying a fixture twice gives identical tracks", "[replay][VR-002]") {
// The property the whole fixture strategy rests on. If this fails, every
// golden output derived from a fixture is unreliable and the CI replay
// tier is worthless.
Dump d = load(fixture("bali_28.h5"));
Replay a = run(d);
Replay b = run(d);
REQUIRE(a.track_ids.size() == b.track_ids.size());
CHECK(a.track_ids == b.track_ids);
REQUIRE(a.claims.size() == b.claims.size());
for (std::size_t i = 0; i < a.claims.size(); ++i) {
CHECK(a.claims[i].first_seen == b.claims[i].first_seen);
CHECK(a.claims[i].last_seen == b.claims[i].last_seen);
}
}
// ── AR-012 / AR-013 — window invariants on real footage ──────────────────────
TEST_CASE("every face is assigned a track and every track closes",
"[replay][AR-012]") {
Dump d = load(fixture("bali_13.h5"));
Replay r = run(d);
CHECK(r.track_ids.size() == r.faces_seen);
for (int id : r.track_ids) CHECK(id >= 0); // nothing silently unassigned
// flush() must leave nothing behind: a track still open at EOF would be a
// window that never reaches the output.
CHECK(r.claims.size() > 0);
}
TEST_CASE("windows are well-formed and inside the clip", "[replay][AR-013]") {
for (const char* f : {"bali_13.h5", "bali_27.h5", "bali_28.h5",
"bali_31.h5", "bali_46.h5"}) {
INFO(f);
Dump d = load(fixture(f));
Replay r = run(d);
const double t0 = d.ts.front(), t1 = d.ts.back();
for (const auto& c : r.claims) {
// A window ends at the last sighting, never after it — so it can
// never extend past the footage that produced it.
CHECK(c.first_seen <= c.last_seen);
CHECK(c.first_seen >= t0);
CHECK(c.last_seen <= t1);
}
}
}
TEST_CASE("a longer extinction window yields fewer, longer tracks",
"[replay][AR-013]") {
// The timeout decides whether a gap is absorbed into one window or splits
// it in two, so lengthening it must merge tracks rather than multiply them.
// On sparse footage this is the difference the constant actually makes.
Dump d = load(fixture("bali_46.h5")); // 140 faces over 385 frames
Replay tight = run(d, /*extinction=*/1.0);
Replay loose = run(d, /*extinction=*/30.0);
CHECK(loose.claims.size() <= tight.claims.size());
}
// ── AR-007 — cuts are exercised by the corpus, not just by construction ──────
TEST_CASE("the cut-heavy fixture actually contains cuts", "[replay][AR-007]") {
// Guards the corpus rather than the code: if a regeneration produced a
// fixture with no cuts, the association tests above would still pass while
// silently testing nothing about viewpoint changes.
Dump d = load(fixture("bali_28.h5"));
const int cuts = std::count(d.is_cut.begin(), d.is_cut.end(), uint8_t{1});
CHECK(cuts >= 5);
}
tests/test_track_gallery.cpp
+15 -5
View File
@@ -83,15 +83,25 @@ TEST_CASE("novelty gate skips views the gallery already covers", "[track_gallery
CHECK(tg.annex().empty());
}
TEST_CASE("spread gate rejects a two-person track", "[track_gallery]") {
TEST_CASE("a two-person track never poisons the annex", "[track_gallery][AR-018]") {
TrackGallery tg(expand_cfg());
// Two orthogonal identities under one track ID: pairwise sim 0 → spread 1.0
// > spread_max 0.60. Whole track rejected, annex stays empty even though
// frames are accepted and gallery-far.
// Two orthogonal identities under one track ID — a track-ID collision.
//
// The banded admission (AR-018) now catches this EARLIER than the spread
// gate did: an embedding unlike everything already on the track falls below
// the band's lower bound and is refused entry, so the buffer never becomes
// two-person in the first place. The spread gate remains as a second line
// for a track that drifts gradually rather than jumping.
//
// The assertion is on the outcome, not the mechanism: whichever gate fires,
// the outsider must not reach the actor's annex.
tg.observe(3, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(3, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(3, one_hot(400), 0, 0.30f, true, kNoCrop); // orthogonal outlier
CHECK(tg.annex().empty());
CHECK(tg.band_rejected() > 0); // refused at the door
for (const auto& e : tg.annex())
CHECK(cosine_similarity(e.emb, one_hot(400)) < 0.5f);
}
TEST_CASE("unconfirmed track (too few accepts) does not promote", "[track_gallery]") {

Some files were not shown because too many files have changed in this diff Show More