dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dtourolle/scene-actor-extraction:e7dfd20a1a47b00a9ced02e35663d4c2a65475e8
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
..
compare: dtourolle/scene-actor-extraction:feature/quality-knee
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
25 Commits
e7dfd20a1a .. feature/quality-knee
Author SHA1 Message Date
dtourolle 889018aa34 docs(VR-012): the quality knee, and what it retires 
Sharpness is not a sufficient statistic for identity loss. Six cells at
effectively identical measured sharpness span 15.3% to 91.0% TPI,
ordered entirely by source size, because a scalar keyed on
high-frequency energy cannot separate attenuated high frequencies from
destroyed spatial sampling. AR-028's "kept separate, not collapsed into
one scalar" now rests on a measurement rather than an argument -- and
the reasoning it used to rest on, that the aligned crop is
scale-normalised so a measure there cannot re-measure size, was wrong
and is corrected in place.

Variance of Laplacian -- the most widely used blur metric there is -- is
anti-predictive at fixed degradation on all three blur families. The
decile it calls sharpest is 2.6x less identifiable than the decile it
calls blurriest, monotone across ten bins, because within a cell its
residual variance is native contrast rather than detail, and hard
shadows and JPEG ringing raise it while making a face harder to match.
Gating on it would preferentially discard the more identifiable faces.

Blur breaks confidence, not identity: rank-1 holds at 80.2% where TPI is
15.3%, and FPI never left 0.1% in any of the 108 cells. Degradation
produces abstention, never a wrong name. That is also why sharpness
fails as a compute gate -- even a visually destroyed face stays 46.9%
identifiable, so a gate discards recoverable evidence at three times the
cost of the free size filter. Discount, do not gate; the rule AR-028
already stated now has evidence for why it is right rather than merely
cautious.

Records the shape a discount must have (flat, then a cliff between sigma
2 and 3), that its cost scales with proximity to the decision boundary
rather than with blur, and that the pose half of VR-012 has not been
run.

TRACES: VR-012, AR-028, AR-029, AR-030 | SR-002
 2026-07-31 22:29:47 +02:00
dtourolle 26de01b2e3 study(VR-012): quality knee over a joint size x blur grid, three blur families 
Extends the VR-005 protocol -- hold out one mugshot per actor, degrade
only the probe, match against a gallery held at native resolution,
decide through the Platt calibration -- from one axis to two, over 1670
actors rather than 100.

Joint rather than separable, because the interaction is the question: a
16 px face upscaled to 112 has already lost its high frequencies, so
further blur costs it almost nothing, while the same blur at full
resolution is expensive. Sweeping the axes independently would measure
each with the other implicitly at its best and miss that entirely.

Three blur families, compared at matched per-axis PSF spread rather than
at equal raw parameter. Optical defocus is a uniform disc whose transfer
function is a jinc with exact zeros, not a Gaussian that merely rolls
off, and it is also how a face ends up large and useless -- the case a
size filter cannot catch. Sweeping Gaussian alone, as the first version
did, understates real lens blur by a factor of five in error rate.

Every candidate measure is scored on every degraded crop and the
candidates are ranked by how well each predicts the pipeline's actual
decision, not by how smooth its synthetic ladder looks. Both a pooled
and a within-cell AUC are reported: they answer different questions and
the candidates rank differently under each.

Runs through sae_embed throughout. Stages gains optional engine paths so
the same study can drive a TRT build, which is what makes the full grid
five minutes rather than four and a half hours.

TRACES: VR-012, AR-028, AR-029 | SR-002
 2026-07-31 22:28:06 +02:00
dtourolle 41d30395da fix(ort): CUDA detection was gated behind the TensorRT-EP build flag 
detect_ort_provider() only tested for CUDAExecutionProvider inside
#ifdef SAE_ORT_WITH_TRT_EP, so any build that did not also opt into the
TensorRT execution provider could never select CUDA and fell straight
through to the CPU. The two are independent: the TRT EP needs the
headers and profile plumbing and is rightly an opt-in, CUDA is a plain
ORT provider and is not.

The failure is silent rather than loud, which is why it survived --
inference runs on the CPU and every answer is still correct, just far
slower. Measured on the VR-012 study: 0.32 s/crop against 0.0021 s/crop
once a GPU backend is actually used, with the card sitting at 212 MiB
and 0% utilisation throughout.

Only the TensorrtExecutionProvider line stays inside the guard.
 2026-07-31 22:25:00 +02:00
dtourolle 1ae88376e1 feat(quality): five AR-029 sharpness candidates on the aligned crop 
assess_sharpness() scores a 112x112 crop on variance-of-Laplacian, a
contrast-normalised variant, Tenengrad, a spectral high-frequency ratio
and dir_min_tenengrad, over a fixed 64x64 window on the face interior.
The window excludes the corners because studio headshots are routinely
shot at a wide aperture, and background bokeh measured over the whole
crop would drag the score down on the sharpest images in the set.

Five rather than one because AR-029's threshold has to be located, not
chosen: VR-012 ranks them by how well each predicts real identity loss.

The T1 ladders drove two corrections during development. The spectral
ratio applied its Hann window before removing the mean, so the DC term
smeared into the low-frequency bins and the "ratio" tracked absolute
brightness (a 20/255 brightening moved it 23%). And no measure taken
from the literature survived directional blur: normalising by total
energy divides out the loss being measured, so both ratio measures are
U-shaped in motion-blur length and score a 21 px smear about as sharp as
a 3 px one. dir_min_tenengrad exists to fix that -- a low-frequency
contrast denominator that blur leaves alone, and the worse of the two
Sobel axes rather than their sum.

The tests pin the disqualifying behaviours as well as the desirable
ones, so a change that makes var_laplacian contrast-free is a deliberate
act rather than an accident. They also record that every candidate falls
under downscale-upscale as well as under blur: the aligned crop is
scale-normalised geometrically, not informationally.

Exposed through sae_embed alongside the AR-030 alignment residual, so a
study scores through shipped code rather than a numpy copy -- the same
argument that already applies to the calibration.

TRACES: AR-028, AR-029 | SR-002
 2026-07-31 22:24:39 +02:00
dtourolle 5f6daefc40 Merge branch 'feature/quality-knee' into feature/opencv5 
# Conflicts:
#	docs/requirements.md
 2026-07-31 16:54:09 +02:00
dtourolle 629d698ad9 Merge branch 'feature/dump-provenance' into feature/opencv5 2026-07-31 16:53:58 +02:00
dtourolleandClaude Opus 5 71354e862a docs: VR-013 and VR-014 results; AR-002 raised to 40px 
Records study results and the requirement change that follows from them.

VR-013 measures minimum face size end to end — gallery from one recording,
probes from another — rather than by degrading an already-aligned crop. Holding
90% of the plateau needs ~50 px that way against VR-005's ~22 px, the gap being
detection and landmark error rather than the embedder. AR-002 therefore takes
40 px, not 32: VR-005 isolates the embedder and is an upper bound, and 32 admits
faces in the falling region. FPI stayed 0.0% at every scale, and the ceiling is
cross-view rather than resolution.

VR-014 exercises audio-signature offset recovery on real film audio instead of
the synthetic golden tone. Forty random in-cap offsets, every one recovered to
the nearest frame, worst error 46 ms against a 500 ms budget — and 46 ms is the
quantisation floor rather than a result, since offsets land on whole 92.88 ms
frames. The runtime/2 anchor is confirmed through head-trimmed files.

The soft spot VR-014 found is tier labelling, not accuracy: the score drops with
sub-frame misalignment, so 27 of 40 correct alignments were demoted to `loose`.
One frame of slack in the score restores all forty to `audio` with false matches
unmoved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-002, VR-005, VR-013, VR-014 | SR-002, SR-003
 2026-07-31 16:53:58 +02:00
dtourolleandClaude Opus 5 2a8ee3660b feat(audio): bind the v1 signature and validate offset recovery on real content 
sae_audio exposes the shipped signature to Python. It compiles
audio_signature.cpp directly against FFmpeg rather than linking
sae_gallery: the signature needs no model, no OpenCV and no HDF5, so a
module that dragged those in would make `import sae_audio` depend on a
GPU-capable build of a path that is pure CPU DSP.

The point of binding rather than porting is that a fingerprint is only
useful if every implementation agrees byte for byte. A numpy port would
be a third implementation, and the one nobody checks against the golden
vector.

VR-014 then recovers a known trim from real film audio rather than from
the synthetic tone: 40 random in-cap offsets, every one recovered to the
nearest frame, worst error 46 ms against a 500 ms budget — and 46 ms is
the quantisation floor, not a result, since offsets land on whole
92.88 ms frames.

The soft spot is tier labelling rather than accuracy. Sub-frame
misalignment drags the score down (0.94-0.99 near a frame boundary,
0.69-0.73 at half a frame), demoting 27 of 40 correct alignments to
`loose`. Allowing +/-1 frame of slack in the score fixes it: all 40 back
to `audio` at min 0.906, false matches unmoved at 0.12-0.16, for 81 ms
of the budget.

The module stops at the producer's edge. Sliding one signature against
another is the consumer's algorithm (server SPEC §3, and the jRay
plugin implements it), so a caller writing that slide in numpy is not
duplicating anything this repo owns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: IR-004, IR-005 | VR-014 | UT-105, UT-106, UT-107, UT-108 | SR-003
 2026-07-31 16:52:11 +02:00
dtourolleandClaude Opus 5 b4318f8d9e fix: belief accumulates across frames (lazy-OR), not once 
A track recognised on 318 of 385 frames was owned on none, so the truth file
named nobody while the matcher was accepting almost continuously.

The correlation discount was an annihilator rather than an attenuator. Weight
was 1 - P(same view), so once a track had one stored view every later frame of
that same face scored ~0.01 and the belief stopped moving. One observation just
over the accept threshold is logit(0.78) ~ 1.27, under the ownership bar — hence
recognised always, owned never.

Two changes, in the order they were found.

Correlated evidence is now attenuated by effective sample size,
n_eff = n / (1 + (n-1)·rho), each frame contributing the marginal gain. That has
the right shape at both ends: uncorrelated evidence accumulates linearly, and a
held pose converges on 1/rho rather than growing without bound. A constant floor
was tried first and rejected — it grows linearly forever, so a long shot could
out-argue genuinely varied evidence purely by lasting longer.

Combination is now weighted lazy-OR: P = 1 - (1-P_old)·(1-p)^w, stored as
log(1-P) so the update is additive and precision stays where it matters as P
approaches 1. Each frame is new evidence that this track is that actor, and the
belief is the probability that at least one sighting was right. It converges
faster than summing log-odds at the same effective count — 2.98 vs 2.53 after
two observations at p=0.78 — which is what a real clip needs.

Note that summing log-odds was already a correct sequential Bayesian update:
the matcher fits with prior 0.5, so logit(p) IS the per-frame log-likelihood
ratio and the running sum carries the prior forward. It was not wrong, it was
slow. What blocked ownership was the discount, not the combination rule.

Also fixes a real correctness bug: the observation count lived on the
discounter, which is shared by every track, so tracks pooled into one effective
sample and each was discounted by how many others happened to be on screen. It
is now a per-track parameter.

The registry's frame scope holds its lock for its lifetime and the mutex is not
recursive, so calling observe() inside a scope self-deadlocks. The pipeline
never does — separate nodes — but the test did, and hung rather than failing.
Documented at the call site.

Verified end to end: the same clip that produced zero actors now identifies
Bing Crosby and Dorothy Lamour with belief 0.97.

Suite: 96 cases, 6142 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-025 | SR-002
 2026-07-31 16:51:08 +02:00
dtourolle af5208035e docs: the RANSAC aligner was a defect, measured 
AR-005 replaced cv::estimateAffinePartial2D(..., RANSAC, 3.0) with Umeyama
least squares over all five points — the estimator InsightFace aligns with, and
so the one the ArcFace/LVFace training crops were produced by.

The first note here assumed the two agree wherever RANSAC keeps all five points,
leaving a small divergence on non-frontal faces. Measured on 400 gallery
headshots with the model held fixed, that was wrong: the crops disagree by a
median 17 source px and 83.5% embed below cos 0.99 of their Umeyama counterpart.
A 4-DoF similarity is exactly determined by two points, so every minimal sample
fits its own pair perfectly and is scored on the other three; real landmarks sit
a median 2.74 canonical px from any similarity fit, so a landmark outside the
3 px band is the common case and RANSAC returns an under-determined transform.

How much that cost in accuracy is a separate question, and the honest answer is
less than those numbers suggest. Rebuilding the full gallery moved the
intra/inter separation the AR-023 calibration is fitted from by 0.583 to 0.590:
the old warp was wrong but self-consistent, gallery and probe both went through
it, and the embedder tolerates framing variation. The sharper evidence is
duplicate detection — the rebuild dropped 1614 near-duplicates against the
original build's ~100, because unstable two-point fits gave near-identical
images visibly different vectors. That instability, not a headline accuracy
delta, is what a tracker accumulating evidence across frames was paying for.

Also records the AR-030 residual's real-data floor: on the most cooperative
images the pipeline sees, it runs a median 2.74 px, so landmark noise occupies
the first few pixels and the synthetic foreshortening ladder is optimistic about
the low end. Any discount curve has to treat that range as uninformative rather
than as mild pose, and VR-012 must set thresholds against the measured
distribution.

Tests carry the tag they verify: the residual's roll/scale invariance and
monotonicity under foreshortening are what make it a pose measure rather than a
pose-and-everything-else measure.

TRACES: AR-005, AR-030 | SR-002
 2026-07-31 16:39:12 +02:00
dtourolleandClaude Opus 5 d3ab598434 feat(artifacts): push and pull the VR-013 corpus 
The cross-source study needs two 4K recordings and a hand-sorted set of
face crops, neither of which belongs in git. Adds an xsource target to
both artifact scripts.

Push uploads the clips as-is (already compressed) and zips labelling/.
Pull fetches both and regenerates frames with ffmpeg rather than
downloading them: ~320 MB of PNG that is deterministic from the clips.
The extraction settings are pinned in the script, not left to the
caller, because the manifests key on frame filenames and on detection
order within each frame — verify_labels.py runs afterwards and fails
loudly if they drift.

Pull refuses to overwrite an existing labelling/. It is human ground
truth: somebody looked at 167 crops and placed each one, and silently
replacing that with a remote copy would destroy the expensive half of
the study.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-013
 2026-07-31 15:58:24 +02:00
dtourolle 81ec77625c Merge branch 'feature/gallery-report' into feature/opencv5 2026-07-31 15:31:07 +02:00
dtourolleandClaude Opus 5 1dfd6fea11 feat: gallery build report 
GR-003 — the calibration fit already computed per-actor dedup counts, how many
actors are eligible for positive pairs, and a 200-bin histogram of the intra and
inter distributions, then discarded all of it to stderr. Nothing persisted, so
nobody could audit whether a gallery was any good.

The report is written alongside the gallery at build time. That is the right
moment: the matcher fits the same sigmoid at analysis time, but by then the
answer is per-run and nobody is looking, whereas build time is when a gallery's
quality is actually decided.

What it surfaces, in order of usefulness:
- actors with no usable image — a silent recall ceiling, since the pipeline can
  never name them and nothing else says why
- actors below the positive-pair threshold — not broken, so nothing complains;
  they just quietly weaken every threshold downstream
- near-duplicate references removed, per actor and total
- the fitted calibration AND the two distributions behind it

That last one is the point. Every threshold in the pipeline is expressed in the
probability space this sigmoid defines, so if the distributions overlap heavily
the calibration is weak and every downstream decision inherits it — while the
gallery still looks fine from the outside.

The gallery-derived prior, intra/(intra+inter), is computed and reported but the
shipped default of 0.5 is deliberately left alone. The spec records these as
disagreeing; now the real value is visible, so the decision can be made on
evidence rather than argument.

Three tests: a zero-image actor is visible in the report, an under-referenced
actor is counted, and the report round-trips through JSON.

Suite: 95 cases, 6142 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: GR-003 | SR-001
 2026-07-31 15:30:46 +02:00
dtourolleandClaude Opus 5 6aabeb9897 feat: provenance attributes on the embedding dump 
VR-010 — a dump made with one detector/embedder pair was byte-indistinguishable
from one made with another, except for the two attributes GR-004 added. Replayed
against a gallery from a different model, cosine similarities are meaningless
but look entirely plausible. The register states the principle directly: a
fixture whose provenance is unknown is worse than no fixture, because it will be
trusted.

Sixteen attributes now record everything that determines the dump's content:
detector model and thresholds, min_face_px, max_faces, cut_threshold,
dense_scale, bbox_upscale, start/end, track_assoc_min_prob, and scene_detect.

scene_detect is the one that matters most. is_scene_boundary is all-zero both
when the detector found nothing and when it never ran, and those mean completely
different things to a consumer — without the flag they are indistinguishable.

No schema_version bump: new root attributes are additive and replay.py already
reads attributes with a default, so older dumps stay readable and the committed
fixtures — which predate this — still load.

Also corrects SCHEMA.md, which claimed bbox was already mapped to original
resolution at dump time. It is not; the upscale is applied downstream in the
matcher, after the dump tap. Harmless while dense_scale is 1 and silently wrong
otherwise, so bbox_upscale is now recorded and the doc says what the code does.

Verified end to end: all sixteen attributes present and correct on a freshly
generated dump.

Suite: 92 cases, 6136 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-010, VR-001 | PR-002
 2026-07-31 15:24:06 +02:00
dtourolleandClaude Opus 5 01d7ead1e7 study(VR-013): cross-source identification probe over input resolution 
Gallery from one recording, probes from another, sweeping the probe's
input resolution end to end. VR-005 asked the same question over gallery
mugshots but degraded an already-aligned 112x112 crop with alignment held
perfect, so it isolates the embedder. Here the whole frame is downscaled
before the detector, so detection and landmark regression degrade with
it — which is most of the difference.

Corpus is two 4096x2160 clips of one shoot, four people, hand-sorted.
Ground truth is sorted by hand and gated by verify_labels.py; labels
carried down the scales geometrically by box position, never by
embedding similarity, which would keep only the faces the embedder
already gets right and drop the ones the sweep exists to find.

Findings, all scored through the production gallery sigmoid at
prob_threshold 0.754 — never a raw cosine:

- Holding 90% of the plateau needs ~50 px end to end, against VR-005's
  ~22 px. min_face_px at 40 looks right; 32 would admit faces in the
  falling region.
- FPI is 0.0% at every scale. Resolution loss goes entirely to TBI.
- The ceiling is cross-view, not resolution: everyone matches themselves
  within a recording (0.55-0.85) and collapses across two (0.14-0.45,
  threshold 0.335). Only the subject with frontal *gallery* references
  identified reliably, whatever their probe pose — so the lever is
  gallery pose coverage, not a better landmark source.
- Averaging SCRFD's overlapping detections instead of discarding them at
  NMS lifts cross-recording TPI 41% -> 49%, for one forward pass and no
  extra model.

Four identities and one shoot, so the shape is the result and the
absolute rates are not. Both clips contain all four people, so there is
no out-of-gallery class and the 10x-weighted out-of-cast misID is
untested here.

Clips, frames, hand-sorted crops and results are gitignored and belong
in the artifact registry — the sorting is human ground truth and
expensive to redo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-013 | AR-002, AR-005, AR-024
 2026-07-31 15:20:18 +02:00
dtourolleandClaude Opus 5 9fc2763096 feat: expansion promotion gated on all three discontinuity signals 
AR-019 — promotion may only borrow same-identity evidence from a span where
identity is certain, so every discontinuity signal now clears the buffers rather
than just the histogram cut.

is_scene_boundary was already named in the gate but never set by anything, so
that half of it was dead until AR-010 gave it a producer. It now does what the
spec always said. The third signal, an identity contradiction, needs no code
here: AR-015 closes a track whose belief swapped, so it can no longer promote.

Ownership now comes from the registry rather than a second tally. TrackGallery
was computing its own plurality vote over accepted frames, which meant two
different answers to "who is this track" could coexist in one run — and the
expansion one ignored the Bayesian accumulation entirely, weighting thirty
near-identical looks the same as thirty distinct ones. The local tally survives
only as a fallback for callers with no registry attached, which is the unit
tests and the replay harness.

Suite: 92 cases, 6133 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-019, AR-010, AR-015 | SR-005
 2026-07-31 15:11:56 +02:00
dtourolleandClaude Opus 5 c843c4abe3 feat: banded admission for the per-subject embedding store 
AR-018 — an embedding joins a track's store only if its similarity to something
already there falls inside a band, rather than merely being far from the gallery.

Above the upper bound it is redundant: another look at a pose the store already
covers, teaching the annex nothing while costing a slot a novel view could have
used. Below the lower bound it is suspect: within one track every face is the
same person by construction, so an embedding unlike everything else on the track
is evidence that construction failed — a track-ID collision or a bad detection.
Admitting it is exactly how an actor's annex gets poisoned with someone else's
face.

The old gate had only the upper half of that idea, expressed as a raw cosine
against the gallery. Both bounds are now calibrated probabilities (AR-024), so
the same number means the same thing here as in association and evidence
weighting rather than three different things.

This catches track-ID collisions EARLIER than the spread gate did — at the door
rather than at promotion — so the buffer never becomes two-person in the first
place. The spread gate stays as a second line for a track that drifts gradually
instead of jumping. The existing test was asserting the mechanism rather than
the outcome, so it was rewritten to assert what actually matters: whichever gate
fires, the outsider must not reach the annex.

Rejections are counted. A store that admits nothing is as broken as one that
admits everything, and neither is visible otherwise.

Band defaults 0.90-0.95 are working values pending VR-007; the two bounds fail in
opposite directions and must be swept separately.

Suite: 92 cases, 6133 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-018, AR-024 | SR-005
 2026-07-31 15:10:31 +02:00
dtourolleandClaude Opus 5 cc1bed92d8 perf: back the CPU similarity GEMM with OpenBLAS 
The CPU path was a scalar triple loop. It is the correctness oracle for the GPU
backends, but it is also what CI runs — there is no GPU on the N100 host — and
since AR-003 removed the per-frame face cap, a crowded frame now scores many
faces against a library-scale gallery. Scoring one face against 5000 embeddings
is 2.6 MFLOP; in scalar that does not hold up (AR-027).

S(g,f) viewed as row-major [n_faces x n_gallery] is exactly query * gallery^T,
so the loop nest collapses into a single cblas_sgemm.

OpenBLAS is optional in the build: found via pkg-config, and the scalar path
remains when it is absent so no hard dependency is added and the two can be
diffed when a similarity looks wrong. The configure step warns rather than
failing, since a developer without it should still get a working tree.

The test target links it too. Without that the suite compiles the scalar
fallback while the builder image ships CBLAS, so CI would be verifying a kernel
that is not the one running in production — the same class of mistake as testing
a path the gate never executes.

Recorded as required (not optional) in the DP-007 image, for the same reason.

Suite: 92 cases, 6136 assertions, with CBLAS compiled in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-026, AR-027, DP-007 | SR-001
 2026-07-31 15:04:29 +02:00
dtourolleandClaude Opus 5 13bdc27566 feat: join the decode butterfly so scene boundaries reach the face branch 
AR-010 — is_scene_boundary had no producer: SceneDetectorFunc was a terminal
sink writing scenes.json and never annotating the frames flowing to face
detection. The flag was permanently false, so the boundary half of AR-007's
frame-dependent association was dead code that a test could still exercise
synthetically and appear to verify.

The topology already forks after decode — dense frames to TransNetV2, sampled
frames to face detection — so this is a fork-join. SceneBoundaries is the join:
the detector publishes each window's verdict with a watermark, and an annotator
on the sampled branch stamps the flag.

The watermark is the part that matters. TransNetV2 buffers 100 frames before it
can score any of them, so at any instant it has an opinion up to some time T and
none after. Without recording T a consumer cannot tell "no boundary" from "not
scored yet", and those demand opposite behaviour — treating unscored frames as
boundary-free is exactly what makes a downstream check pass while verifying
nothing.

Buffering alone does not work, which was my first attempt. Channel depth creates
lag only when the consumer is slower, and the face branch runs four orders of
magnitude faster per frame than TransNetV2 (0.01ms vs 400ms), so its channels
drain instantly and no lag accumulates. Measured: 106 of 364 frames outran the
detector. The annotator therefore waits on the watermark explicitly. The
detector signals completion so the tail cannot deadlock, and publishes from
flush_remaining too — without that the final frames arrive with no verdict.

Boundaries are deduped on publish, matching what scenes.json does at write time.
A run of adjacent high-scoring frames is one boundary, not several; leaving them
raw made this view report 357 where the file said 13. Now the two agree exactly.

Frames past the detector's last scored window remain unverified and are counted
as such rather than silently marked boundary-free.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-007, AR-010 | SR-002
 2026-07-31 14:56:38 +02:00
dtourolleandClaude Opus 5 fa1c494825 docs: AR-010 is blocked on a design decision, not an implementation gap 
Making SceneDetectorFunc a pass-through does not work. TransNetV2 buffers 100
dense frames before it can score any of them and trusts only each window's
centre, so a boundary at time T is not known until roughly 3.3s after T at
30 fps. The face pipeline runs on a parallel branch and has long since passed T.
An association hint that arrives after the association is worthless.

Three options recorded with their costs: two-pass (correct, doubles the decode
that already dominates runtime), delaying the face branch (couples the two
branches' timing, which invites heisenbugs under backpressure), or leaving it
unwired.

Leaving it unwired costs less than it looks, which is what makes this a decision
rather than a defect. The redesign made cuts and boundaries do the same thing —
both say "spatial continuity is broken, associate on embedding" — so TransNetV2
adds nothing over the histogram except on transitions the histogram cannot see:
slow dissolves and fades. That gap is real but narrow.

Where TransNetV2 still earns its cost is AR-019, whose promotion gate wants a
span free of cuts and boundaries. A late answer is fine there, because promotion
happens on track confirmation rather than per frame — so it can be wired
offline against the collected boundary list, off the hot path entirely.

Recommendation: leave the association path on is_cut alone, wire boundaries into
AR-019, and revisit if dissolve-heavy material shows association failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-010, AR-019 | SR-002
 2026-07-31 14:27:05 +02:00
dtourolleandClaude Opus 5 f99f1c5ccc feat: no fixed cap on faces per frame 
AR-003 — max_faces defaults to 0, meaning no cap. A fixed cap discards the
SMALLEST faces first, which are exactly the background cast X-Ray still credits
with scene membership, so the pipeline was systematically losing the people it
is supposed to find in crowded scenes.

This is only safe now that AR-004 landed. Previously an uncapped frame would
have pushed more work into channels that dropped on overflow, trading a visible
cap for silent loss. With backpressure the producer slows instead, so per-frame
cost is contained rather than discarded.

The matcher's kMaxFaces used to throw above 32, which made it an accidental
second cap. It sizes the similarity engine's preallocated buffer, so it bounds
memory rather than face count — the frame is now scored in batches of that size.
Memory stays bounded; faces do not.

Largest-first ordering is kept even without the cap, and the comment now says
why: the Hungarian solver tie-breaks on index order, so that ordering is
load-bearing for the replay determinism test rather than a leftover of the cap.

Verified end to end on a real clip: identical output to the capped run (385
frames, 693 faces), which is expected since that footage peaks at 4 faces per
frame — the point is the absence of a regression. The committed fixtures remain
byte-identical and valid.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-003 | SR-002
 2026-07-31 14:22:20 +02:00
dtourolleandClaude Opus 5 3605b8da78 fix: point the KPN submodule at the merged commit 
The recorded pointer was be6e922 — the backpressure fix as originally committed,
before it was rebased onto KPN master. That commit exists on no pushed branch,
so a fresh clone of this branch could not fetch the submodule at all.

Now 6595e6e, the same change on KPN master.

Worth noting for next time: rebasing a submodule commit after the superproject
has already recorded it silently invalidates the pointer. Nothing in the
superproject's status shows it, because the submodule working tree is clean and
at a valid commit — just not the one recorded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004 | SR-002
 2026-07-31 11:53:27 +02:00
dtourolleandClaude Opus 5 61e487fbee test: replay the real tracker and registry from committed fixtures 
Tier T2 — composition rather than units. The registry tests construct awkward
states directly; these feed the pieces real 480x360 footage with the cuts, gaps
and crowded frames that synthetic input does not produce.

Six cases:
- fixture integrity: exact frame and face counts, contiguous face_offset, and
  the embedder identity each dump carries (GR-004). The counts are asserted
  exactly rather than approximately, which was impossible before AR-004 — what
  a lossy run dropped depended on timing.
- determinism: replaying a fixture twice gives identical track ids and windows.
  This is the property the whole fixture strategy rests on; without it every
  golden output derived from a fixture is unreliable and the CI replay tier is
  worthless.
- every face is assigned a track, and flush leaves nothing open — a track still
  live at EOF is a window that never reaches the output.
- windows are well-formed and inside the clip. A window ends at the last
  sighting, so it can never extend past the footage that produced it.
- a longer extinction window yields fewer, longer tracks. On the sparse fixture
  (140 faces over 385 frames) that is the difference the constant actually
  makes: absorbing a gap versus splitting a window.
- the cut-heavy fixture still contains cuts. This guards the corpus, not the
  code: a regeneration that produced cut-free fixtures would leave the
  association tests passing while silently testing nothing.

Driving the functors directly rather than through a KPN network is deliberate —
no threads, no channels, no scheduling, so the same input gives the same output.

Suite: 86 cases, 6106 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004, AR-012, AR-013, VR-001, VR-002 | SR-002
 2026-07-31 11:23:17 +02:00
dtourolleandClaude Opus 5 c12838b9fd fix: dropped frames fail the run instead of printing a footer 
A drop was reported to stderr and the process exited 0, so a run that discarded
320 frames "succeeded" and produced a truth file that looked complete. The
output in that case is a claim about footage that was never analysed, and
nothing in the file says so.

Now exits 2 and says why. Distinct from 1 (node crash) because the failures are
different: a crash produced no output, a drop produced output that cannot be
trusted.

This is also the regression test for AR-004 that otherwise did not exist. The
backpressure fix is one line in the KPN submodule — easy to lose in an update —
and with data pushes blocking, a drop can no longer occur on the data path. So
any drop now means either that fix regressed or a channel was disabled mid-run,
and both are worth stopping for.

Verified: a clean run still exits 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004 | SR-002
 2026-07-31 10:45:39 +02:00
dtourolleandClaude Opus 5 d31526cfaf test: committed replay fixtures from the public-domain corpus 
Five HDF5 embedding dumps from bali/ — Road to Bali (1952) — 3.6 MB total,
generated at 5 fps with a 32 px minimum face. CI never calls a model, so
inference happens on a GPU host and CI replays these as data; everything
downstream of embedding is cheap CPU maths.

Public domain is the reason this corpus rather than a convenient one: derived
fixtures can be committed, where anything cut from a copyrighted title could not
live in the repository at all.

The set covers distinct behaviours rather than being five of the same thing:
bali_28 has 9 cuts, so it exercises shot/reverse-shot association (AR-007);
bali_46 is sparse at 140 faces over 385 frames, so it exercises gaps and
extinction (AR-013); bali_13 is the busiest at 4 faces per frame; bali_31 is
short at 29s. All five recorded zero drops.

Both pinned parameters are consequences of measurements, not defaults: 5 fps
because 1 fps over a 77s clip is 77 frames, too thin for an extinction window
measured in tens of seconds; 32 px because that is the VR-005 floor, and the
corpus is 480x360 so a stricter value would reject most of what is there.

make_fixtures.sh regenerates them. Reproducibility is the requirement — a
fixture whose provenance is unknown is worse than none, because it will be
trusted. These are byte-reproducible only because of AR-004: before node
outputs blocked rather than dropped, the same command produced different dumps
run to run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-001 | PR-002
 2026-07-31 10:41:51 +02:00
55 changed files with 2002 additions and 1537 deletions
Expand all files Collapse all files
Dockerfile.builder-cpu
-342
View File
@@ -1,342 +0,0 @@
# sae-builder-cpu — the CI build image
#
# TRACES: DP-007 | PR-004
#
# Build/push: scripts/ci/build_builder_image.sh --push
# Consumed by: .gitea/workflows/unit-tests.yml (pinned by tag, never :latest)
# Docs: docs/ci-image.md
#
# This is the CPU corner of the DP-008 builder matrix and the DP-007 CI image at
# the same time — one artifact, two uses. The CUDA and ROCm siblings differ only
# in the accelerator stack layered on top of this dependency set.
#
# CI runs on an Intel N100 with no discrete GPU. Everything here is chosen so
# that `-DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=CPU -DSAE_BUILD_TESTS=ON`
# configures, builds and runs without a GPU, without a model, and without
# reaching GitHub.
# ─── Base image ──────────────────────────────────────────────────────────────
#
# Chosen for the OLDEST glibc to be supported, not for recency. A binary built
# in a container runs against the *host's* glibc; glibc is backward compatible
# but not forward, so the build base sets the floor for every machine DP-008's
# binaries can ever run on. Building on a newer base than the oldest supported
# host produces the classic `GLIBC_2.xx not found` failure at load time.
#
# Debian 12 "bookworm" = glibc 2.36 (Aug 2022). What that floor covers:
#
# Distro glibc Covered?
# Arch / CachyOS (rolling) 2.41+ yes
# Fedora 37 and later 2.36+ yes ← DP-005's targets are Fedora+Arch
# Debian 12 / 13 2.36+ yes
# Ubuntu 24.04 LTS 2.39 yes
# Ubuntu 22.04 LTS 2.35 NO
# RHEL / Rocky / Alma 9 2.34 NO
# Debian 11 2.31 NO
#
# The three misses are accepted deliberately: DP-005 puts Debian/Ubuntu out of
# installer scope and names Fedora + Arch as the supported distros, and every
# supported Fedora is 2.36 or newer. Going lower costs the toolchain rather than
# buying reach — Debian 11 ships GCC 10 (incomplete C++20) and Python 3.9, which
# has no `tomllib` and therefore cannot read the traceability gate's
# traceability.toml.
#
# Escape hatch, recorded now so it is not rediscovered under pressure: if the
# floor must drop to glibc 2.28 (RHEL 8 / manylinux_2_28 — the same baseline the
# ONNX Runtime and PyTorch wheels target), the move is a Rocky 8 base plus
# gcc-toolset-13, and OpenCV/FFmpeg/HDF5 all leave apt for source or
# EPEL/RPM Fusion. That is a different image, not a flag on this one.
#
# Not a glibc problem but worth stating: the binaries this image produces also
# link OpenCV, FFmpeg and HDF5 shared objects by soname. Making a *portable*
# release binary (DP-008) is a separate question from the glibc floor, and is
# answered by static linking or bundling, not by the base image.
FROM debian:12-slim
# Pins. Every version this image installs from source is an ARG so a rebuild is
# a one-line diff and `docker history` records what a given tag actually holds.
#
# ORT 1.28.0 and OpenCV 5.0.0 match the developer machine, so CI and local
# builds exercise the same libraries rather than merely similar ones.
# Catch2 / nlohmann_json / nanobind match the FetchContent pins in
# CMakeLists.txt:248 and tests/CMakeLists.txt:12 exactly — a vendored copy at a
# different version would be a silent divergence, not a convenience.
ARG ORT_VERSION=1.28.0
ARG OPENCV_VERSION=5.0.0
ARG CATCH2_VERSION=v3.5.3
ARG NLOHMANN_JSON_VERSION=v3.11.3
ARG NANOBIND_VERSION=v2.4.0
# Stamped so a build can prove which image it ran in, and so a green tick can be
# traced back to a specific dependency set. See the "Confirm the builder image"
# step in .gitea/workflows/unit-tests.yml.
ARG IMAGE_TAG=dev
ENV SAE_BUILDER=cpu \
SAE_BUILDER_VERSION=${IMAGE_TAG} \
SAE_ORT_VERSION=${ORT_VERSION} \
SAE_OPENCV_VERSION=${OPENCV_VERSION} \
DEBIAN_FRONTEND=noninteractive
# ─── System dependencies ─────────────────────────────────────────────────────
#
# One layer, ordered by why it is here rather than alphabetically.
RUN apt-get update && apt-get install -y --no-install-recommends \
# Toolchain. bookworm's default gcc is 12.2 — enough for the C++20 the
# project sets unconditionally (CMakeLists.txt:4). cmake is 3.25, above the
# 3.21 minimum. Ninja because the N100 has four cores and every second of
# build scheduling shows.
build-essential \
cmake \
ninja-build \
pkg-config \
git \
ca-certificates \
curl \
# Gitea's act_runner executes JS actions (actions/checkout, upload-artifact)
# with the `node` found *inside* the container. Without this the job cannot
# even check the repository out. Same reason as the kpnpp-builder image.
nodejs \
# HDF5 with the C++ API: galleries are HDF5-native and it is also the VR-001
# dump format. find_package(HDF5 COMPONENTS CXX) at CMakeLists.txt:273.
libhdf5-dev \
# FFmpeg decode. swresample is on this list deliberately: the audio
# signature (IR-004) downmixes to mono and resamples to 11025 Hz, and
# tests/test_audio_signature.cpp decodes the golden FLAC fixture, so the
# test build needs it as much as the main build does.
libavformat-dev \
libavcodec-dev \
libavutil-dev \
libswscale-dev \
libswresample-dev \
# OpenBLAS — required here, not optional. CI has no GPU, so SAE_GEMM_BACKEND
# =CPU is the only path it ever exercises, and without OpenBLAS the CPU GEMM
# falls back to a scalar loop that does not scale against a library-sized
# gallery (AR-027). The build only *warns* when it is missing so a developer
# without it still gets a working tree; the image must never be that case.
# Both the main build (CMakeLists.txt:162) and the test target
# (tests/CMakeLists.txt:42) discover it through pkg-config `openblas`.
libopenblas-dev \
# Python: the build itself needs the interpreter and headers
# (find_package(Python COMPONENTS Interpreter Development.Module) at
# CMakeLists.txt:254, for the nanobind modules). numpy/h5py/scipy are for
# the Python-side tooling — fixture generation, replay, validation scripts.
# From apt rather than pip: bookworm marks the environment externally
# managed (PEP 668), and apt's h5py is already linked against the same
# libhdf5 installed above. bookworm's python3 is 3.11, which has tomllib —
# the traceability gate needs it to read traceability.toml.
python3 \
python3-dev \
python3-numpy \
python3-h5py \
python3-scipy \
# Image codecs for the OpenCV build below. Without these OpenCV silently
# builds an imgcodecs that cannot read a JPEG, which fails at run time in a
# gallery build rather than at compile time here.
libjpeg62-turbo-dev \
libpng-dev \
libtiff-dev \
libwebp-dev \
libopenjp2-7-dev \
zlib1g-dev \
&& rm -rf /var/lib/apt/lists/*
# Fail the image build, not the CI run, if OpenBLAS or swresample are not
# discoverable the way CMakeLists.txt discovers them. An image that ships
# libopenblas but no openblas.pc would compile the scalar fallback in silence.
RUN set -eux; \
pkg-config --exists openblas; \
echo "openblas $(pkg-config --modversion openblas)"; \
pkg-config --exists libswresample; \
echo "swresample $(pkg-config --modversion libswresample)"
# ─── ONNX Runtime, CPU provider only ─────────────────────────────────────────
#
# The official prebuilt linux-x64 tarball is the CPU build: no CUDA, no
# TensorRT, no ROCm execution providers. That is the whole requirement here —
# excluding the GPU providers is not a size optimisation, it is the point.
#
# Verified against the 1.28.0 tarball: the shared object's highest versioned
# symbol requirement is GLIBC_2.27 / GLIBCXX_3.4.21, well under this base's
# 2.36, so ORT does not raise the floor set above.
#
# Installed to /usr/local/{lib,include/onnxruntime} because CMakeLists.txt
# includes <onnxruntime/onnxruntime_cxx_api.h> and needs the *parent* of that
# directory on the include path (CMakeLists.txt:108-113).
#
# CI never calls a model — the embedder measures ~930 ms/frame on this CPU
# provider — so ORT is present to satisfy the link, not to run inference.
RUN set -eux; \
curl -fsSL -o /tmp/ort.tgz \
"https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VERSION}/onnxruntime-linux-x64-${ORT_VERSION}.tgz"; \
mkdir -p /tmp/ort; \
tar -xzf /tmp/ort.tgz -C /tmp/ort --strip-components=1; \
cp -a /tmp/ort/lib/libonnxruntime.so* /usr/local/lib/; \
mkdir -p /usr/local/include/onnxruntime; \
cp -a /tmp/ort/include/. /usr/local/include/onnxruntime/; \
ldconfig; \
rm -rf /tmp/ort /tmp/ort.tgz; \
test -f /usr/local/include/onnxruntime/onnxruntime_cxx_api.h
# ─── OpenCV 5, from source ───────────────────────────────────────────────────
#
# This is the reason the image is prebuilt at all. CMakeLists.txt:25 probes for
# OpenCV 5 first and falls back to 4; the branch targets 5, which no Debian
# release ships (bookworm has 4.6), and building it inside every CI run would
# dominate the run on an N100.
#
# BUILD_LIST is exactly the seven components find_package asks for
# (CMakeLists.txt:25-29) — OpenCV resolves their internal dependencies itself.
# Everything else is off: tests, samples, Java/Python bindings, and the apps.
#
# No GUI backend. highgui still builds (find_package REQUIREs the component) but
# with a stub — CI never calls imshow, and pulling GTK/Qt into a headless build
# image buys nothing. scene_preview is a developer tool, not a CI target.
#
# CUDA/cuDNN explicitly off: DP-007 excludes the GPU stack outright.
#
# The source tree and build tree are removed in the same layer, so the ~3 GB of
# intermediates cost nothing in the published image.
RUN set -eux; \
curl -fsSL -o /tmp/opencv.tar.gz \
"https://github.com/opencv/opencv/archive/refs/tags/${OPENCV_VERSION}.tar.gz"; \
mkdir -p /tmp/opencv-src; \
tar -xzf /tmp/opencv.tar.gz -C /tmp/opencv-src --strip-components=1; \
cmake -S /tmp/opencv-src -B /tmp/opencv-build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DBUILD_LIST=core,imgproc,imgcodecs,videoio,dnn,objdetect,highgui \
-DBUILD_SHARED_LIBS=ON \
-DBUILD_TESTS=OFF \
-DBUILD_PERF_TESTS=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_DOCS=OFF \
-DBUILD_opencv_apps=OFF \
-DBUILD_JAVA=OFF \
-DBUILD_opencv_python3=OFF \
-DWITH_FFMPEG=ON \
-DWITH_GTK=OFF \
-DWITH_QT=OFF \
-DWITH_OPENGL=OFF \
-DWITH_CUDA=OFF \
-DWITH_CUDNN=OFF \
-DOPENCV_GENERATE_PKGCONFIG=ON \
-DCMAKE_INSTALL_RPATH=/usr/local/lib; \
cmake --build /tmp/opencv-build --parallel; \
cmake --install /tmp/opencv-build; \
ldconfig; \
rm -rf /tmp/opencv-src /tmp/opencv-build /tmp/opencv.tar.gz
# ─── Vendored dependencies: Catch2, nlohmann/json, nanobind ──────────────────
#
# All three are FetchContent'ed by the build today, which makes every CI run
# depend on GitHub being reachable — a network outage would present as a code
# failure. Baking them in removes that dependency entirely.
#
# Catch2 is *installed*, so tests/CMakeLists.txt:6 `find_package(Catch2 3 QUIET)`
# succeeds and the FetchContent fallback is never reached. Its source is kept as
# well so the override below can cover the case where find_package somehow does
# not fire.
#
# nanobind must be cloned with submodules: its `ext/robin_map` is a git
# submodule, and a GitHub source tarball does not contain it. This is the one
# dependency where "download the tarball" produces a tree that configures and
# then fails to compile.
RUN set -eux; \
mkdir -p /opt/vendor; \
git clone --depth 1 --branch "${NLOHMANN_JSON_VERSION}" \
https://github.com/nlohmann/json.git /opt/vendor/nlohmann_json; \
git clone --depth 1 --branch "${NANOBIND_VERSION}" --recurse-submodules \
https://github.com/wjakob/nanobind.git /opt/vendor/nanobind; \
git clone --depth 1 --branch "${CATCH2_VERSION}" \
https://github.com/catchorg/Catch2.git /opt/vendor/Catch2; \
cmake -S /opt/vendor/Catch2 -B /tmp/catch2-build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DBUILD_TESTING=OFF; \
cmake --build /tmp/catch2-build --parallel; \
cmake --install /tmp/catch2-build; \
rm -rf /tmp/catch2-build; \
find /opt/vendor -maxdepth 2 -name .git -exec rm -rf {} +; \
ldconfig
# The initial-cache script the build is configured with. It lives in the image,
# not in the workflow, so the vendor paths have exactly one owner: move a
# directory here and no consumer needs editing.
#
# FETCHCONTENT_FULLY_DISCONNECTED=ON is the load-bearing line. With it, any
# FetchContent dependency that is *not* covered by an override above is a hard
# configure error instead of a silent download — so "this build does not touch
# GitHub" is enforced by the build system rather than asserted in a comment.
RUN set -eux; \
printf '%s\n' \
'# Baked into sae-builder-cpu. Use with: cmake -C /opt/vendor/vendored-deps.cmake ...' \
'# TRACES: DP-007' \
'set(FETCHCONTENT_SOURCE_DIR_NLOHMANN_JSON "/opt/vendor/nlohmann_json" CACHE PATH "vendored in the CI image")' \
'set(FETCHCONTENT_SOURCE_DIR_NANOBIND "/opt/vendor/nanobind" CACHE PATH "vendored in the CI image")' \
'set(FETCHCONTENT_SOURCE_DIR_CATCH2 "/opt/vendor/Catch2" CACHE PATH "vendored in the CI image")' \
'set(FETCHCONTENT_FULLY_DISCONNECTED ON CACHE BOOL "no CI build may fetch from the network")' \
> /opt/vendor/vendored-deps.cmake; \
cat /opt/vendor/vendored-deps.cmake
# ─── Self-check ──────────────────────────────────────────────────────────────
#
# Run the project's own dependency discovery — the same find_package and
# pkg_check_modules calls CMakeLists.txt makes — against this image, at image
# build time. An image that cannot satisfy them should fail here, loudly, once,
# rather than in every CI run that pulls it.
#
# Deliberately not a build of the project: the image must be buildable without
# the repository, and the repository's own configure step is what CI is for.
RUN set -eux; \
mkdir -p /tmp/selfcheck; \
printf '%s\n' \
'cmake_minimum_required(VERSION 3.21)' \
'project(sae_image_selfcheck LANGUAGES CXX)' \
'set(CMAKE_CXX_STANDARD 20)' \
'set(CMAKE_CXX_STANDARD_REQUIRED ON)' \
'find_package(OpenCV 5 REQUIRED COMPONENTS core imgproc imgcodecs videoio dnn objdetect highgui)' \
'message(STATUS "OpenCV ${OpenCV_VERSION}")' \
'find_package(HDF5 REQUIRED COMPONENTS CXX)' \
'message(STATUS "HDF5 ${HDF5_VERSION}")' \
'find_package(Catch2 3 REQUIRED)' \
'message(STATUS "Catch2 ${Catch2_VERSION}")' \
'find_package(Python 3.8 REQUIRED COMPONENTS Interpreter Development.Module)' \
'find_package(PkgConfig REQUIRED)' \
'pkg_check_modules(AVFORMAT REQUIRED libavformat)' \
'pkg_check_modules(AVCODEC REQUIRED libavcodec)' \
'pkg_check_modules(AVUTIL REQUIRED libavutil)' \
'pkg_check_modules(SWSCALE REQUIRED libswscale)' \
'pkg_check_modules(SWRESAMPLE REQUIRED libswresample)' \
'pkg_check_modules(OPENBLAS REQUIRED openblas)' \
'find_library(ORT_LIB onnxruntime REQUIRED HINTS /usr/lib /usr/local/lib)' \
'find_path(ORT_INCLUDE onnxruntime_cxx_api.h PATH_SUFFIXES onnxruntime' \
' HINTS /usr/include/onnxruntime /usr/local/include/onnxruntime /usr/local/include REQUIRED)' \
'message(STATUS "ORT ${ORT_LIB} / ${ORT_INCLUDE}")' \
> /tmp/selfcheck/CMakeLists.txt; \
cmake -S /tmp/selfcheck -B /tmp/selfcheck/build -G Ninja; \
rm -rf /tmp/selfcheck
# Python-side tooling the fixture and validation scripts import. Checked here so
# a missing wheel is an image failure rather than a mid-run traceback.
RUN python3 -c "import numpy, h5py, scipy; print('numpy', numpy.__version__, 'h5py', h5py.__version__, 'scipy', scipy.__version__)"
# ─── What is deliberately NOT here ───────────────────────────────────────────
#
# CUDA, TensorRT, ROCm, and the ORT GPU execution providers
# No GPU to use them. They belong to the sae-builder-cuda and
# sae-builder-rocm siblings (DP-008).
#
# The ONNX models
# Seven files, ~725 MB, in Git LFS. T1/T2 tests are model-free by design
# (tests/CMakeLists.txt:1-4), so the CI image needs none of them, and
# baking them in would inflate the image roughly tenfold to serve the T3
# smoke tests alone. Those pull the model they need via LFS in a separate
# job. The CI workflow checks out with LFS off for the same reason.
#
# The repository
# Nothing from the source tree is COPYed in. The image is a toolchain, and
# a toolchain that embeds the code it builds has to be rebuilt whenever the
# code changes — which is exactly the per-run cost this image exists to
# avoid.
WORKDIR /src
docs/SPEC.md
+86 -41
View File
@@ -248,11 +248,29 @@ the same response.
- **Size** — already AR-002, floor at 40×40 px in original resolution, measured - **Size** — already AR-002, floor at 40×40 px in original resolution, measured
end to end by VR-013. It is the precedent for the other two: the end to end by VR-013. It is the precedent for the other two: the
threshold was *located*, not chosen. threshold was *located*, not chosen.
- **Sharpness** — motion blur and soft focus destroy the high-frequency detail - **Sharpness** — motion blur and optical defocus destroy the high-frequency
the embedder keys on, and unlike size they leave the bounding box looking detail the embedder keys on, and unlike size they leave the bounding box
perfectly healthy. Measured on the **112×112 aligned crop**, not the raw box: looking perfectly healthy. Measured on the **112×112 aligned crop**, not the
the crop is already scale-normalised, so a measure taken there cannot silently raw box.
re-measure face size and double-count it against AR-002.
An earlier version of this clause argued the crop is scale-normalised and so a
measure taken there "cannot re-measure face size and double-count it against
AR-002". **That reasoning is wrong and VR-012 measured it wrong.** The
normalisation is geometric, not informational: a 40 px face upscaled into the
canonical frame genuinely carries less high-frequency content than a 400 px
one downscaled into it, so every candidate measure *does* respond to source
size. What the crop yields is **effective resolution in canonical space** —
the union of "was small" and "was blurred", not blur alone.
The conclusion survives, for a better reason. VR-012 sorted its grid by
measured sharpness and found the six cells at effectively identical sharpness
(0.0003–0.0005) spanning **15.3% to 91.0% TPI**, ordered entirely by source
size. Sharpness is therefore not a sufficient statistic for identity loss: a
scalar keyed on high-frequency energy cannot separate *attenuated* high
frequencies from *destroyed* spatial sampling, because blur preserves
mid-frequency facial geometry exactly while downsampling destroys it. The two
axes are not redundant and neither substitutes for the other — which is what
"not collapsed into one scalar" above now rests on.
- **Visibility** — extreme pose or occlusion means the face presents fewer of the - **Visibility** — extreme pose or occlusion means the face presents fewer of the
features the embedding assumes are present. The measure is the **residual of features the embedding assumes are present. The measure is the **residual of
the AR-005 alignment fit**: the RMS landmark error, in canonical 112×112 the AR-005 alignment fit**: the RMS landmark error, in canonical 112×112
@@ -333,18 +351,63 @@ hand-chosen cutoff on an uncalibrated measure is the same unfalsifiable magic
number AR-024 retired for similarity, and it would fail the same way: meaning number AR-024 retired for similarity, and it would fail the same way: meaning
something different for every detector, every embedder and every film. something different for every detector, every embedder and every film.
**A discount curve on sharpness must be flat, then steep.** VR-012 measured the
response as a cliff rather than a gradient: Gaussian sigma up to 1.5 costs under
1.5 points of TPI in every cell — at 16 px it is very slightly *positive*,
smoothing upscale artifacts — sigma 2 costs 1–3, and the 2→3 step costs 7–19. A
linear or sigmoid discount over the measure would penalise the whole flat region
where blur demonstrably costs nothing.
**Which blur is modelled is a first-order decision, not a detail.** VR-012 swept
three families at matched per-axis PSF spread, and at σ=3 px on a 112 px face
they cost 9%, 18% and **53%** error for Gaussian, motion and optical defocus
respectively. Defocus is the destructive one because its disc PSF has a jinc
transfer function with **exact zeros** — bands annihilated rather than
attenuated — where a Gaussian merely rolls off. It is also the case AR-002
cannot catch, since a defocused face is large and confidently detected. Any
future study that sweeps blur states its family and its justification; a
Gaussian-only sweep understated the effect by a factor of five and would have
retired this axis as not worth its cost.
**The cost of blur is proportional to proximity to the decision boundary, not to
blur itself.** Sigma 3 costs −22.5 points at 24 px, but only −7.9 at 112 px
(margin to spare) and −8.3 at 16 px (already below threshold). This is why the
axes must combine multiplicatively in `EvidenceDiscounter` rather than each
gating independently.
**Sharpness discounts; it must never gate.** VR-012 tried the gate directly, as
a compute saving: skipping the embed below a sharpness threshold costs 15.1% of
true identifications to save 20% of the work, against the size filter's 4.7% at
16.7% — three times the damage, from a measure that needs the warped crop plus a
DFT where size is a bbox dimension available for free. The reason is a ceiling
no measure can beat: **at 112 px with defocus radius 6 — visually destroyed —
46.9% of faces still identify correctly, and rank-1 is still 94.8%.** Apparent
blur does not determine the outcome. The size filter wins only because smallness
destroys identity more completely than blur does (16 px succeeds 23.5% of the
time), and that asymmetry is the measured justification for the rule above:
**failing sharpness discounts the observation, failing size may drop it.**
**Current:** visibility is measured and carried — `estimate_alignment()` in **Current:** visibility is measured and carried — `estimate_alignment()` in
`src/face_utils.hpp` returns the residual alongside the transform, and `src/face_utils.hpp` returns the residual alongside the transform, and
`FaceAlignerFunc` writes it to `DetectedFace::alignment_residual`. Size is `FaceAlignerFunc` writes it to `DetectedFace::alignment_residual`. Sharpness is
`min_face_px` (40, decoded-frame space — AR-002 still open). Sharpness is measured: `assess_sharpness()` in `src/quality.hpp` returns five AR-029
unmeasured. Nothing yet *consumes* any of it: no discount is applied, and candidates over a fixed 64×64 window on the face interior, and VR-012 has ranked
`align_face()` still drops the degenerate-fit case without counting it. them — `var_laplacian` and `tenengrad` are disqualified as discounts (see
AR-029), leaving `hf_energy_ratio` as the only correctly-signed survivor. Size
is `min_face_px` (40, decoded-frame space — AR-002 still open). All three are
exposed to studies through `sae_embed`. Nothing yet *consumes* any of it: no
discount is applied, and `align_face()` still drops the degenerate-fit case
without counting it.
**Gap:** AR-029 entirely. For AR-030, the measure exists but the discount does **Gap:** the discount itself, on every axis. Neither sharpness nor the residual
not — it must reach `EvidenceDiscounter` as the reliability term. For AR-028, the reaches `EvidenceDiscounter`, whose weight remains pure novelty — so a profile
residual does not yet reach the VR-001 dump, which is what VR-012 needs to run or defocused view still moves a track's belief hardest when it deserves the
from fixtures; that is the next step, since it unblocks the study that sets least trust. Neither reaches the VR-001 dump either, so VR-012 must still re-run
every remaining behaviour. video rather than replay fixtures. VR-012's **pose half is not started**: the
AR-030 residual has no arm in the grid, so whether the 5-point proxy suffices or
a dedicated landmark model is needed remains open. And the sharpness result is
weak enough (best within-cell AUC 0.530) that whether AR-029 earns a discount at
all is still a judgement, not a measurement.
## AR-007, AR-008 — Tracking ## AR-007, AR-008 — Tracking
@@ -666,30 +729,12 @@ An embedding is admitted only if its similarity to one already in the store fall
admitting it risks poisoning the store. admitting it risks poisoning the store.
A starting band of roughly **0.90–0.95** is the working estimate, to be tuned A starting band of roughly **0.90–0.95** is the working estimate, to be tuned
(VR-007). Note this is deliberately conservative compared to the retired (VR-007). Note this is deliberately conservative compared to the current
`expand_novelty_sim` (0.55), which promoted embeddings *far* from the gallery — `expand_novelty_sim` (0.55), which promotes embeddings *far* from the gallery —
much more aggressive, and much more exposed to admitting the wrong person. much more aggressive, and much more exposed to admitting the wrong person.
Both bounds must be expressed as calibrated probabilities, not raw cosines (AR-024). Both bounds must be expressed as calibrated probabilities, not raw cosines (AR-024).
The lower bound is asked twice. `admit` compares a newcomer against its
*closest* existing member, which a gradually drifting track can chain past: every
step inside the band while the endpoints are strangers — the shape a track-ID
collision takes over a slow pan. So the same bound is re-applied across **every
pair** in the store before promotion. One bound, two enforcement points; not a
second constant.
Novelty is deliberately **not** a threshold. The store's eviction policy orders
its members by similarity to the actor's existing references and drops the
best-recognised one, so novelty-seeking is a ranking with nothing to tune, and
the band's upper bound already refuses the redundant views at the door.
**Current:** implemented in `gallery/track_gallery.hpp` — `admit` at the door,
`store_coherence` at promotion, both bounds from `Config::expand_band_lo/hi`.
Refusals are counted (`band_rejected`).
**Gap:** the bounds themselves are unswept working values (VR-007).
### AR-019 — Expansion of known actors ### AR-019 — Expansion of known actors
When a track is owned (AR-012), its store is promoted into a **per-film, in-memory When a track is owned (AR-012), its store is promoted into a **per-film, in-memory
@@ -790,14 +835,14 @@ natural unit for anonymous presence, should that be adopted (AR-012, TBD).
open (VR-007). open (VR-007).
**Current:** `src/gallery/track_gallery.hpp` implements a per-track diversity **Current:** `src/gallery/track_gallery.hpp` implements a per-track diversity
buffer with eviction biased to gallery-far poses, admission and promotion both buffer with eviction biased to gallery-far poses, promotion gated on
gated on the AR-018 band in probability space, cleared on `is_cut`. Wired at `expand_novelty_sim` / `expand_track_spread_max`, cleared on `is_cut`. Wired at
`identity_matcher_node.hpp:227`, cleared at `:126`; the calibration is handed `identity_matcher_node.hpp:227`, cleared at `:126`.
over at `:114`.
**Gap:** all three quiet-signal conditions rather than only `is_cut`; and the **Gap:** the band rule of AR-018 replacing the current novelty/spread gates; all
whole of AR-020 — the TBI queue, the deferred pass, and deferring output until it three quiet-signal conditions rather than only `is_cut`; probability space
completes. throughout (AR-024); and the whole of AR-020 — the TBI queue, the deferred pass, and
deferring output until it completes.
## AR-022 — Unidentified-track capture ## AR-022 — Unidentified-track capture
docs/benchmark.md
-186
View File
@@ -1,186 +0,0 @@
# Benchmark — SuperHero
The reference film for end-to-end accuracy. Replaces Road to Bali, which was
withdrawn for the reason in [Why not Road to Bali](#why-not-road-to-bali).
TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
---
## The film
SuperHero, from the [NIST TRECVID Deep Video Understanding development
set](https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset/).
14 films are asserted Creative Commons and need no data agreement; only the 5
KinoLorber test films are gated.
| | |
|---|---|
| Runtime | 1025.5 s (17.1 min), 10 scenes |
| Resolution | 640×360 |
| Ground truth | Per-scene presence, from the scene knowledge graphs |
| Gallery | 5 characters, 14 references |
The DVU set is what makes this workable: it ships **character** face crops cut
from the film itself, so ground truth and gallery are both in character space
and scoring needs no actor→character mapping.
**Licence caveat.** NIST links licence evidence for only 4 of the 14 films, and
SuperHero is not one of them — its end credits carry no copyright or CC notice,
list a "Temporary Musical Score" and a SAG cast, and it has no traceable online
release. Fine for internal benchmarking; do not redistribute frames from it.
Valkaama is the one film with an independently documented licence (CC BY-SA 3.0)
if provenance ever has to be defended.
---
## Reproducing it
```sh
# 1. Annotations, character mugshots, scene segmentation.
# NIST names the same film three different ways, hence the overrides.
KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero ../dvu-hero
# 2. Scene clips (movie.shots), then fuse them into one stream.
# Fusing matters — see "Run it as one film" below.
# SuperHero-1.webm … SuperHero-10.webm from
# <dataset>/movie.shots/, then:
ffmpeg -f concat -safe 0 -i concat.txt -c copy SuperHero_full.webm
# 3. Gallery, with the face-size floor that keeps references in distribution.
./build/build_gallery --root ../dvu-hero/root \
--output ../dvu-hero/hero66.h5 --min-face-px 66
# 4. Run, on the GPU path (see "Check you are on the GPU").
./build/scene_analyze --movie hero/SuperHero_full.webm \
--gallery ../dvu-hero/hero66.h5 \
--detector-engine trt_cache/scrfd.scrfd_500m_bnkps.640.fp16.engine \
--arcface-engine trt_cache/arcface.LVFace-B_Glint360K.b4.fp16.engine \
--fps 5 --min-face-px 32 --expand-gallery \
--output pred.json
```
Nothing here is in git: the clips are ~130 MB and the annotations are
regenerable. Replay fixtures derived from the run ship through the artifact
registry instead:
```sh
scripts/artifacts/push_artifacts.sh replay-fixtures
scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
```
The gallery travels in the same archive as the dumps deliberately — a dump only
replays meaningfully against the gallery it was produced with, and pairing one
with a different gallery silently changes every identity decision in it.
---
## Results
Measured on the fused film, gallery expansion on.
| Metric | Value |
|---|---|
| Precision | **1.00** |
| Recall | 0.65 |
| F1 | 0.79 |
| True positives | 13 |
| False positives | **0** |
| False negatives | 7 |
Six of ten scenes scored exactly right, including the three-character scenes 4
and 5.
**Zero false positives is the result worth keeping.** Every out-of-gallery
character — Beast, Mighty Celestial, Ms. Johnson, Doctor, two Masked Persons —
was declined rather than forced onto a nearest match. That is the calibrated
probability (AR-024) doing its job, and it is the right failure direction for an
X-Ray overlay: a miss is a gap, an invention is a lie.
**The misses have a shape.** Scenes 1, 2, 3 and 8 were missed, and 1–3 are the
three shortest scenes in the film (14 s, 38 s, 27 s). That is consistent with
per-track Bayesian accumulation (AR-025) needing enough sightings before belief
crosses threshold. Scene 8 is 65 s and does not fit that story — it is the one
to look at first when improving recall.
Running the same scenes as isolated clips did *not* do better, so cross-scene
gallery expansion is not currently compensating for short scenes.
### Run it as one film, not as clips
Per-scene clips defeat per-film gallery expansion (AR-019), which grows a
temporary gallery from track continuity across the whole film and re-assesses
unknown tracks at the end. Ten isolated clips give it nothing to work with, and
pay model and gallery load ten times over.
Fusing also makes presence windows cross real scene boundaries, which is how
SR-002's scene-scoped question is asked in production. Note the joins are
artificial cuts — consecutive scenes were never contiguous footage — so presence
bleeding across a boundary may be the join rather than a tracking fault.
---
## Throughput
| Path | Realtime factor | Sampled fps | 17-min film |
|---|---|---|---|
| `build/` (TensorRT) | **2.0×** | ~10 | ~8 min |
| `build-ort/` (ORT) | 0.54× | 2.7 | ~32 min |
Throughput varies strongly with face density; a sparse stretch measured 8×
realtime, so quote the whole-film average, not a window.
### Check you are on the GPU
ORT's CUDA execution provider fails to load on this machine and **silently falls
back to CPU**:
```
Failed to load library libonnxruntime_providers_cuda.so:
undefined symbol: cudnnGetConvolutionBackwardDataAlgorithm_v7
```
That symbol was removed in cuDNN 9; the packaged ORT is built against cuDNN 8.
ORT logs this once at startup and then runs happily on CPU, so a `build-ort`
timing is a CPU number wearing a GPU label — a 15× error with no symptom other
than a figure you have no baseline for. Grep the log for `Failed to load
library` before trusting any throughput measurement.
The TensorRT path (`build/`) needs prebuilt engines from
`scripts/build_trt_engines.sh` and reports what it loaded:
```
[TrtScrfd] loaded: … [TrtArcFace] loaded: … max_batch=4
[similarity] cuBLAS/CUDA engine: gallery resident on GPU
```
---
## Why not Road to Bali
Bali was chosen because DVU ships character mugshots for it. It was withdrawn on
**face scale**, measured on its own reference crops:
| | Bali | SuperHero |
|---|---|---|
| Median detected face | 27 px | **69 px** |
| Maximum detected face | 69 px | **241 px** |
| References ≥66 px | 2 of 69 | 14 of 27 |
The DVU images are scene crops, not mugshots, so the crop dimensions say nothing
about face scale — the face has to be detected and measured. Bali's median
reference was being upscaled roughly 4× to reach ArcFace's 112×112, and the
worst 7×, which violates AR-011: every model gets the input it was trained for.
A model run off-distribution returns confident, plausible, wrong output.
In a gallery that error is permanent. A bad frame costs one frame; a poisoned
reference corrupts every future match against that identity.
No threshold rescued it. At 66 px only 2 of 69 references survived — the largest
face in the entire set is 69 px — so there was no cut that both kept references
in distribution and left enough of them to calibrate. SuperHero's gallery builds
at a 66 px floor and calibrates on its own (`a=15.2867 b=-4.98633`, 100 % train
accuracy) rather than borrowing constants.
Any accuracy figure recorded against Bali predates this and should be treated as
measuring upscaling artifacts as much as the pipeline.
docs/index.md
+8
View File
@@ -53,6 +53,14 @@ every finding below.
A training-set effect that did not reproduce on 5 held-out films once A training-set effect that did not reproduce on 5 held-out films once
two methodology bugs in the comparison harness were found and fixed. two methodology bugs in the comparison harness were found and fixed.
- :material-blur:{ .lg .middle } **[What does blur cost?](quality-knee.md)**
---
Sharpness is not a sufficient statistic for identity loss, blur breaks
confidence rather than ranking, and variance-of-Laplacian is
anti-predictive at fixed resolution.
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)** - :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
--- ---
docs/plan.md
+2 -2
View File
@@ -363,8 +363,8 @@ one identity out of the gallery would fix that.
thing measured is the thing that ships. thing measured is the thing that ships.
`scripts/validation/test_audio_offset.py` over `scripts/validation/test_audio_offset.py` over
`tests/fixtures/audio/superhero_offset_200s.flac`: 200 s of public-domain film audio `tests/fixtures/audio/bali_offset_200s.flac`: 200 s of public-domain film audio
(the same SuperHero clips the replay fixtures use), long enough for a 120 s (the same Road to Bali clips the replay fixtures use), long enough for a 120 s
window to slide past the ±600-frame search cap. The slide itself is numpy here window to slide past the ±600-frame search cap. The slide itself is numpy here
on purpose — matching belongs to the consumer, so writing it out keeps this a on purpose — matching belongs to the consumer, so writing it out keeps this a
test of the signature rather than of somebody's matcher. test of the signature rather than of somebody's matcher.
docs/quality-knee.md
+308
View File
@@ -0,0 +1,308 @@
# Quality knee: what does a blurred or small face cost, and can a measure predict it?
VR-012. Companion to the minimum-face-size studies VR-005 and VR-013 (see the
[requirement register](requirements.md)), which located the size floor at 40 px;
this asks the same question for **sharpness**, and asks whether any cheap
measure taken on the aligned crop can be acted on at inference.
Run by
[`scripts/validation/quality_knee.py`](https://REPOLINK/scripts/validation/quality_knee.py)
through the `sae_embed` bindings — detection, the ArcFace warp, the embedder,
the five candidate measures and the Platt calibration are all the shipped C++.
## Protocol
1670 gallery actors with 3 or more mugshots (of 2456 total), one image held out
per actor as a probe, the remaining 10326 embeddings staying in the gallery at
native resolution. Only the probe degrades — reference mugshots are clean and
the face coming out of the video is not.
Each probe passes through a **joint grid**: downscale to *S*×*S* and back to
112 (the sampling loss), then blur at level *L* in canonical pixels. Three blur
families, 36 cells each, 60120 probe-cell records per family:
| family | models | parameter |
|---|---|---|
| Gaussian | soft focus, a generic stand-in | sigma 0 … 3 |
| **Disc** | **real optical defocus** — the circle of confusion | radius 0 … 6 |
| Motion | camera pan or moving subject | length 0 … 21 px |
The three are not interchangeable, and sweeping only the first was the original
design error — one that would have produced a wrong answer, not merely an
incomplete one (Result 3). A defocused lens spreads a point into a **uniform
disc**, whose transfer function is a jinc — `2·J1(x)/x` — that crosses zero and
goes negative, annihilating whole frequency bands and returning the ones beyond
each zero phase-reversed. A Gaussian MTF is strictly positive and monotone and
does neither. More practically: defocus and motion are how a face ends up
**large and useless**, while Gaussian blur as swept here mostly co-occurs with
small faces. That difference decides whether sharpness carries anything the size
filter does not.
Families are compared at matched **per-axis PSF standard deviation** (σ for a
Gaussian, R/2 for a disc, L/√12 for a linear smear), never at equal raw
parameter, which would compare different amounts of damage.
Identification is the pipeline's own decision: per-actor best-of-N cosine →
Platt sigmoid → accept above `prob_threshold` 0.754. Never a raw cosine
(AR-024).
## Result 1 — sharpness is not a sufficient statistic
Sorting the 36 Gaussian cells by `hf_energy_ratio`, the six sigma-3 cells land
at effectively identical measured sharpness:
| size | sigma | hf_energy_ratio | TPI |
|---|---|---|---|
| 16 | 3 | 0.0003 | **15.3%** |
| 24 | 3 | 0.0003 | 63.2% |
| 32 | 3 | 0.0003 | 79.4% |
| 48 | 3 | 0.0003 | 86.6% |
| 64 | 3 | 0.0004 | 88.4% |
| 112 | 3 | 0.0005 | **91.0%** |
Same measured sharpness, a **76-point spread in identification**. It inverts
too: 16 px unblurred measures 0.0033 and scores 23.5%, while 48 px at sigma 2
measures *lower* at 0.0021 and scores 96.6%.
A canonical-frame sharpness scalar cannot separate *attenuated* high
frequencies from *destroyed* spatial sampling. Blur suppresses the high band
while preserving mid-frequency facial geometry exactly; downsampling to 16 px
destroys that geometry outright. Both look alike to any measure keyed on
high-frequency energy.
This is the measured basis for AR-028's rule that the axes are **kept separate
and not collapsed into one scalar**, and it settles the double-counting
question: size and sharpness are not redundant, and neither substitutes for the
other.
## Result 2 — blur is a cliff, and it breaks confidence, not identity
TPI % by size (rows) against Gaussian sigma (columns):
| size | 0 | 0.5 | 1 | 1.5 | 2 | 3 |
|---|---|---|---|---|---|---|
| 16 | 23.5 | 24.0 | 25.0 | 24.6 | 23.9 | 15.3 |
| 24 | 85.7 | 85.1 | 85.6 | 85.9 | 82.6 | 63.2 |
| 32 | 95.9 | 95.9 | 96.0 | 95.5 | 93.7 | 79.4 |
| 48 | 98.7 | 98.7 | 98.4 | 98.1 | 96.6 | 86.6 |
| 64 | 98.6 | 98.6 | 98.8 | 98.4 | 97.5 | 88.4 |
| 112 | 98.9 | 98.9 | 98.8 | 98.6 | 98.0 | 91.0 |
Three regimes: **sigma ≤ 1.5 is free** (every cell moves under 1.5 points, sign
flipping at random — at 16 px it slightly *improves*, smoothing upscale
artifacts); sigma 2 costs 1–3 points; the 2→3 step costs 7–19. A smooth
discount curve is therefore the wrong shape — the response is flat, then falls
off a cliff.
**The cost peaks at the size knee, not at full resolution.** Sigma 3 costs
−22.5 points at 24 px but only −7.9 at 112 px and −8.3 at 16 px. Blur has no
intrinsic cost; it costs in proportion to how close the observation already sits
to the decision boundary. At 112 px there is margin to spare, at 16 px the probe
is already below threshold, and at 24 px it sits exactly on the knee.
**What blur destroys is confidence, not ranking.** Rank-1 barely moves: 99.3% →
99.2% at 112 px across the whole sigma range. The extreme case is 16 px at sigma
3, where rank-1 is **80.2%** while TPI is **15.3%** — 65 points of probes have
the correct actor ranked first and are rejected anyway for falling under the
probability threshold.
That is why **FPI never left 0.1% in any of the 108 cells across all three
families**. Degradation produces TBI, never a wrong name. The calibration
degrades gracefully, which is what SR-002 needs.
## Result 3 — the blur *family* matters more than the blur *amount*
Comparing families by their raw parameter is meaningless — sigma, radius and
length are different units. They are matched here by the **per-axis standard
deviation of the PSF**, which puts them on one scale:
| family | per-axis σ | level giving σ = 3 px |
|---|---|---|
| Gaussian σ | σ | 3 |
| Disc radius R | R/2 | 6 |
| Motion length L | L/√12 | 10.4 |
For reference the ArcFace template places the eyes 35.2 canonical px apart, so
σ = 3 px is 9% of the inter-ocular distance.
TPI at matched severity, interpolated within each family:
| size | σ=3 Gaussian | σ=3 Motion | σ=3 **Defocus** | defocus penalty |
|---|---|---|---|---|
| 16 | 15.3 | 15.1 | 11.0 | +4.3 |
| 24 | 63.2 | 61.1 | 41.4 | +21.9 |
| 32 | 79.4 | 76.1 | 50.4 | +29.0 |
| 48 | 86.6 | 80.9 | 52.6 | +34.0 |
| 64 | 88.4 | 81.7 | 51.0 | +37.4 |
| 112 | 91.0 | 82.1 | **46.9** | **+44.0** |
**Optical defocus is up to 44 points more destructive than a Gaussian of
identical spread**, and the ordering is defocus ≫ motion > Gaussian throughout.
At σ=1 the three families are indistinguishable, and at σ=2 they differ by under
5 points; the divergence appears only when both the blur is severe *and* the face
is large.
That pattern is physically consistent. At 16 px the resampling has already
removed the high frequencies, so the PSF's shape has nothing left to act on and
all three agree. At 112 px the full spectrum is present and shape decides: a
Gaussian MTF rolls off gently and always leaves *some* energy at every
frequency, so the embedder receives a merely attenuated signal, while a disc MTF
is a jinc that **hits exact zeros** — whole frequency bands annihilated rather
than attenuated, with the bands beyond each zero returning phase-reversed.
Motion sits between them because it ruins one axis and leaves the perpendicular
one untouched.
**The methodological consequence is the important one.** This study originally
swept Gaussian blur alone and concluded blur was a minor effect. On the family
that actually occurs in film, the same nominal severity costs **53% error
instead of 9%** at full resolution. A threshold set from the Gaussian arm would
have been wrong by a factor of five in error rate, and the axis would probably
have been dropped as not worth its cost.
**Defocus is also the case a size gate cannot catch.** Every one of those 112 px
faces is large and confidently detected, and sails through AR-002 untouched.
That, not the Gaussian result, is what justifies a sharpness axis existing at
all.
## Result 4 — variance of Laplacian is anti-predictive at fixed degradation
Pooled across all cells, every candidate scores AUC 0.76–0.80 for predicting
correct identification, with textbook `var_laplacian` top. That number is close
to worthless: it rewards a measure for detecting *how degraded the crop is*,
which all five do. The question a per-observation discount needs is whether, at
a **fixed** degradation, the measure predicts which faces fail:
| measure | Gaussian | Defocus | Motion |
|---|---|---|---|
| `hf_energy_ratio` | **0.530** | **0.521** | **0.557** |
| `norm_var_laplacian` | 0.520 | 0.507 | 0.539 |
| `dir_min_tenengrad` | 0.524 | 0.512 | 0.506 |
| `tenengrad` | 0.433 | 0.437 | 0.457 |
| `var_laplacian` | 0.423 | 0.422 | 0.473 |
Best is 0.557 — barely above chance, and `hf_energy_ratio` wins on all three
families. `var_laplacian` is anti-predictive on all three too, so that finding
does not depend on the blur model.
**The two metrics measure different jobs, and the candidates split along that
line.** On the motion arm `dir_min_tenengrad` has the best *pooled* AUC by a
wide margin — **0.854** against 0.792 for the next — exactly as its synthetic
directional-blur ladder predicted, yet its within-cell AUC there is 0.506. It is
an excellent detector of *how badly smeared a crop is* and no guide at all to
*which face will be recognised*. Pooled AUC is the right metric for a
gross-degradation flag; within-cell AUC is the right one for a per-observation
discount; a measure can be strong at one and useless at the other.
Deciles within the 16 px Gaussian cell, where 1277 failures give the test real
power:
| `var_laplacian` decile | TPI |
|---|---|
| 0.00071–0.00192 (blurriest) | **37.1%** |
| 0.00242–0.00278 | 22.8% |
| 0.00397–0.00447 | 25.7% |
| 0.00625–0.01445 (sharpest) | **14.4%** |
The faces the measure calls sharpest are **2.6x less identifiable** than those
it calls blurriest, monotone across ten bins of 167. Within a cell every crop
received identical degradation, so the residual variance is *native contrast*,
not native detail — and hard shadows, high-contrast lighting, sharpening halos
and JPEG ringing all raise Laplacian variance while making a face harder to
match. The measure reads photographic style and encoding artifacts and calls
them sharpness.
`hf_energy_ratio` is the only candidate with a correctly-signed within-cell
trend (16.2% → 35.3% across the same deciles), being a pure ratio in which the
contrast factor cancels.
**Consequence:** a per-face quality *discount* keyed on variance of Laplacian —
the most widely used blur metric in production vision pipelines — would
systematically down-weight the *more* identifiable faces. It is worse than no
discount.
## Result 5 — as a compute gate, sharpness loses to the size filter
Skipping the embed for crops below a threshold, measured as compute saved
against true identifications lost:
| gate | skipped | true IDs lost | of skipped, doomed anyway |
|---|---|---|---|
| `hf_energy_ratio` < 0.00023 | 10.0% | 7.6% | 37.9% |
| `hf_energy_ratio` < 0.00051 | 20.0% | 15.1% | 38.7% |
| **source size < 24 px** | **16.7%** | **4.7%** | **77.3%** |
At a comparable skip rate the size filter loses **4.7% against sharpness's
15.1%** — three times less damage — and it is free, being a bbox dimension
available before alignment or embedding, where sharpness needs the warped crop
plus a colour convert, three convolutions and a 64×64 DFT.
Restricting to large faces (≥64 px) on the **defocus** arm, where the size
filter is blind, improves the gate's precision 3.5x (37% of skipped crops doomed
versus 10.7% on the Gaussian arm) but not its trade: skip 10%, lose 7.0%.
A hard ceiling explains why. **At 112 px with defocus radius 6 — visually
destroyed — 46.9% of faces still identify correctly and rank-1 is still 94.8%.**
Blur does not determine the outcome, so any gate keyed on apparent blur is
predicting a coin flip. The size filter wins not because size is better
measured, but because *smallness destroys identity more completely than blur
does*: 16 px faces succeed only 23.5% of the time, so discarding them is cheap.
## What this means for the requirements
**Do not gate on sharpness; discount on it.** Heavily defocused faces remain
~47% identifiable, so a gate destroys recoverable evidence. This is the first
hard evidence that AR-028's "**discounts the observation, never deletes the
detection**" is right on the merits rather than merely cautious. Since ranking
survives where confidence does not, the per-track accumulation (AR-025) should
recover much of what a single-frame threshold rejects — which is also the
argument for the discount living in `EvidenceDiscounter` rather than in a filter.
**`var_laplacian` and `tenengrad` are disqualified as discounts** by Result 4,
on all three blur families. They remain usable as coarse *gross-degradation*
detectors, the role in which their pooled AUC is real — the same role the size
filter plays — but they must never weight a per-observation belief.
**`hf_energy_ratio` is the only surviving discount candidate**, best on all
three families, and its within-cell signal (0.52–0.56) is weak enough that
shipping a discount on it needs justification beyond this study.
**`dir_min_tenengrad` earns a different job.** Its pooled 0.854 on the motion arm
makes it the best available detector of gross directional smear — useful as a
per-frame "this shot is unusable" flag, which is a decision about a *frame*, not
a weighting of an *observation*. If AR-029 ships two measures for two roles, this
is the second one, and it must not be confused with the first.
**Model the blur family, not just its amount.** Result 3 makes the choice of
degradation model a first-order design decision rather than a detail: the same
matched severity costs 9% or 53% error depending on the PSF. Any future study
that sweeps blur must state which family it used and why.
**Any discount curve must be flat then steep**, not linear or sigmoid over the
measure. Blur costs nothing until it costs a great deal.
## Limitations
- **Cooperative population.** Gallery mugshots are frontal and well-lit;
within-cell failures are likely dominated by cross-view mismatch, which no
sharpness measure can predict. Read the ~chance within-cell AUCs as "sharpness
does not predict the dominant failure mode *here*", not as "sharpness is
meaningless".
- **Uniform grid, not a natural distribution.** Sizes and blur levels are
sampled evenly, so "skip 16.7%" is exactly the 16 px row. The gate comparisons
are like-for-like on identical records, but the absolute savings are not what
a film would show.
- **TensorRT fp16.** A different realisation of the embedder from the fp32 ONNX
reference — VR-005 measured ~0.85 cosine agreement with separation intact.
Gallery and probes share one session so the study is internally consistent,
but the absolute knee belongs to the fp16 space.
- **Blur is applied in the canonical frame**, after resampling, so its width is
independent of the cell's size. Real optics blur before sampling.
- **The top motion rung is an anchor, not an operating point.** Length 21 is a
per-axis σ of 6.1 — 17% of the inter-ocular distance, a streak rather than a
face — and it is swept to bound the curve, not because a frame like that is
worth reasoning about. Its 3.4% TPI at 112 px should not be quoted as a
headline. The same caution applies less severely to defocus radius 6 (σ = 3).
- **Per-axis σ equates spread, not perceptual damage.** It is the fairest single
scalar for comparing PSFs, but Result 3 is precisely the finding that equal
spread does *not* mean equal harm, so the matched-severity tables compare
like-for-like inputs, not like-for-like severity as a face would experience it.
docs/requirements.md
+8 -9
View File
@@ -31,7 +31,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done | | AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
| AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned | | AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing | | AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Mostly** — node outputs *park* on a full channel: the value is held, the worker released, and a channel space-callback resumes the node. Replaces `push_blocking`, which parked a scheduler worker inside the push and, with one thread per node, stopped that node draining its own input. Verified: 385/385 frames, 0 drops. **Gap:** a rare hang survives, ~1 run in 20 at a 300 s timeout (was: every run). `FanoutNode` still drops on overflow (`fanout.hpp:129`) rather than parking, so the AR-010 scene join sheds frames exactly when the dense branch falls behind | | AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far | | AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far |
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done | | AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks | | AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks |
@@ -45,18 +45,18 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | **Done** — reverse index detects it on the causing update; counted | | AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | **Done** — reverse index detects it on the causing update; counted |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | **Done** — `flush()`, idempotent, closes at last sighting or final tick | | AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | **Done** — `flush()`, idempotent, closes at last sighting or final tick |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief and observation count | | AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief and observation count |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space, bounds from `expand_band_lo/hi`; the lower bound re-asked pairwise at promotion, since `admit` compares only against the nearest member and a drifting track can chain past it. Retires `expand_novelty_sim` and `expand_track_spread_max` — novelty is now the eviction ordering, not a threshold. Rejections counted. Bounds unswept (VR-007) | | AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space; replaces `expand_novelty_sim`. Rejections counted |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | **Done** — all three discontinuity signals clear the buffers; ownership comes from the registry, not a second local tally | | AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | **Done** — all three discontinuity signals clear the buffers; ownership comes from the registry, not a second local tally |
| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned | | AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned |
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned | | AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned | | AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done | | AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association, accumulation and expansion all in probability space; `track_max_embed_dist`, `cut_revive_sim`, `expand_novelty_sim`, `expand_track_spread_max` retired | | AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association and accumulation both in probability space; `track_max_embed_dist`, `cut_revive_sim` retired |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` | | AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress | | AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned | | AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | Planned | | AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | Planned |
| AR-029 | Sharpness measure on the **aligned crop** (scale-normalised, so it cannot re-measure size) | SR-002 | Medium | Planned | | AR-029 | Sharpness measure on the **aligned crop**, consumed as a discount and **never as a gate** | SR-002 | Medium | **In Progress** — five candidates implemented (`src/quality.hpp`) and ranked by VR-012 over three blur families. `var_laplacian` and `tenengrad` are **disqualified as discounts**: within a fixed degradation they are anti-predictive on *all three* families (AUC 0.42–0.47; the decile the measure calls sharpest is 2.6× *less* identifiable), since their residual variance is native contrast, not detail. `hf_energy_ratio` is the only correctly-signed survivor, best on all three, and weak (0.52–0.56). `dir_min_tenengrad` is the best *gross-smear detector* (pooled AUC 0.854 on motion) but ~chance within-cell, so it serves a per-frame flag, not a per-observation weight. The parenthetical this row used to carry — "scale-normalised, so it cannot re-measure size" — was wrong: every candidate responds to source size, and the axes are separable for a different reason (see AR-028) |
| AR-030 | Visibility measure from the AR-001 5-point landmarks — extreme pose or occlusion **discounts the observation, never deletes the detection** | SR-002 | Medium | **In Progress** — measure is the AR-005 alignment residual (`estimate_alignment()`), carried on `DetectedFace`; roll/scale invariance and monotonicity under foreshortening asserted. Nothing consumes it as a discount yet | | AR-030 | Visibility measure from the AR-001 5-point landmarks — extreme pose or occlusion **discounts the observation, never deletes the detection** | SR-002 | Medium | **In Progress** — measure is the AR-005 alignment residual (`estimate_alignment()`), carried on `DetectedFace`; roll/scale invariance and monotonicity under foreshortening asserted. Nothing consumes it as a discount yet |
## Deployment (DP) ## Deployment (DP)
@@ -91,7 +91,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|---|---|---|---|---| |---|---|---|---|---|
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done | | GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done | | GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | **Done** — `gallery/gallery_report.hpp`, written next to the gallery by `build_gallery`. Zero-usable-image actors come from the build audit, which a stored gallery cannot reconstruct; also `distinct_references`, `duplicates_removed`, and the intra/inter distributions the calibration fits and would otherwise discard | | GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | **Done** — basename + SHA-256 + `embed_dim`; mismatch fatal with no bypass, unstamped warns unless `--require-gallery-stamp`; `scripts/stamp_gallery.py` migrates in place | | GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | **Done** — basename + SHA-256 + `embed_dim`; mismatch fatal with no bypass, unstamped warns unless `--require-gallery-stamp`; `scripts/stamp_gallery.py` migrates in place |
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done | | GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned | | GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
@@ -114,7 +114,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned | | VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned | | VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned |
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned | | VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned |
| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | Planned | | VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | **In Progress** — sharpness half done ([`docs/quality-knee.md`](quality-knee.md)): 1670 actors, joint size×blur grid over three blur families (Gaussian, disc defocus, linear motion), 60120 probe-cell records each. Sharpness is **not a sufficient statistic** (equal measured sharpness spans 15.3–91.0% TPI, ordered by source size); **the blur family matters more than its amount** — at matched per-axis σ=3 on a 112 px face, Gaussian/motion/defocus cost 9/18/**53**% error, so a Gaussian-only sweep understates real lens blur fivefold; blur breaks **confidence, not ranking** (rank-1 80.2% where TPI is 15.3%), so FPI never left 0.1% in any of the 108 cells; a sharpness **gate** loses 3× more true presence than the free size filter at equal saving, because even destroyed faces stay 46.9% identifiable. **Pose half not started** — the AR-030 residual is exposed via `sae_embed.alignment_residual` but no pose arm has been run, so the dedicated-landmark-model question is still open |
| VR-014 | Audio-signature **offset recovery on real content** — a known trim recovered from film audio, not from the synthetic golden tone | PR-002 | Medium | **Done** — 40 random in-cap offsets, every one recovered to the nearest frame: **worst error 46 ms against a 500 ms budget**, and 46 ms is the floor rather than a result, since the offset is quantised to whole 92.88 ms frames. The `runtime/2` anchor confirmed through real head-trimmed files (a `delta` trim moves the window by `delta/2`). The one soft spot is **tier labelling, not accuracy**: the score falls with sub-frame misalignment (0.94–0.99 near a frame boundary, 0.69–0.73 at half a frame), so 27/40 correct alignments were demoted to `loose`. ±1 frame of slack in the *score* fixes it — measured, all 40 back to `audio` (min 0.906), false matches unmoved at 0.12–0.16, costing 81 ms of the budget | | VR-014 | Audio-signature **offset recovery on real content** — a known trim recovered from film audio, not from the synthetic golden tone | PR-002 | Medium | **Done** — 40 random in-cap offsets, every one recovered to the nearest frame: **worst error 46 ms against a 500 ms budget**, and 46 ms is the floor rather than a result, since the offset is quantised to whole 92.88 ms frames. The `runtime/2` anchor confirmed through real head-trimmed files (a `delta` trim moves the window by `delta/2`). The one soft spot is **tier labelling, not accuracy**: the score falls with sub-frame misalignment (0.94–0.99 near a frame boundary, 0.69–0.73 at half a frame), so 27/40 correct alignments were demoted to `loose`. ±1 frame of slack in the *score* fixes it — measured, all 40 back to `audio` (min 0.906), false matches unmoved at 0.12–0.16, costing 81 ms of the budget |
| VR-013 | Cross-source identification probe — gallery from one recording, probes from another, swept over input resolution end to end | PR-002 | Medium | **In Progress** — holding 90% of the plateau needs ~50 px end to end against VR-005's ~22 px, the gap being detection and landmark error; **`min_face_px` 40, since 32 admits faces in the falling region** (AR-002). FPI 0.0% at every scale. Ceiling is cross-view, not resolution | | VR-013 | Cross-source identification probe — gallery from one recording, probes from another, swept over input resolution end to end | PR-002 | Medium | **In Progress** — holding 90% of the plateau needs ~50 px end to end against VR-005's ~22 px, the gap being detection and landmark error; **`min_face_px` 40, since 32 admits faces in the falling region** (AR-002). FPI 0.0% at every scale. Ceiling is cross-view, not resolution |
@@ -213,7 +213,6 @@ as such rather than counted as covered.
| GR-001 … GR-005 | T1 + T3 | Gallery assembly is I/O and bookkeeping; embedding is T3 smoke | | GR-001 … GR-005 | T1 + T3 | Gallery assembly is I/O and bookkeeping; embedding is T3 smoke |
| GR-006 … GR-008 | T1 | Tiering and outlier detection operate on stored embeddings | | GR-006 … GR-008 | T1 | Tiering and outlier detection operate on stored embeddings |
| VR-* | Out of CI | Studies are run deliberately and their results committed as documents | | VR-* | Out of CI | Studies are run deliberately and their results committed as documents |
| VR-014 | **T2** | The exception, and the reason the blanket row above is not the whole story: its fixture is committed and its signature is CPU-only DSP, so the study *is* a test a CI host can run — not a measurement someone has to remember to repeat |
**One consequence worth stating:** AR-027 (arbitrary gallery scale) is **One consequence worth stating:** AR-027 (arbitrary gallery scale) is
structurally unverifiable on the CI host. It needs a GPU host and a synthetic structurally unverifiable on the CI host. It needs a GPU host and a synthetic
@@ -231,9 +230,9 @@ time**, and CI consumes the result as data.
This is what makes the T1/T2 split load-bearing rather than a preference: T1 and This is what makes the T1/T2 split load-bearing rather than a preference: T1 and
T2 are the only tiers that can exist in CI at all. T2 are the only tiers that can exist in CI at all.
### Fixture corpus — `hero/` ### Fixture corpus — `bali/`
Five clips of **SuperHero (1952)**, ~77 s each, 480×360, 30 fps, 42 MB total. Five clips of **Road to Bali (1952)**, ~77 s each, 480×360, 30 fps, 42 MB total.
Public domain, and that is the reason to use it rather than a convenience: Public domain, and that is the reason to use it rather than a convenience:
**derived fixtures — dumps, crops, golden outputs — can be committed without the **derived fixtures — dumps, crops, golden outputs — can be committed without the
docs/traceability.md
+55 -107
View File
@@ -3,7 +3,7 @@
<!-- GENERATED FILE - do not edit by hand. --> <!-- GENERATED FILE - do not edit by hand. -->
<!-- Regenerate: scripts/traceability/traceability-gate.sh --> <!-- Regenerate: scripts/traceability/traceability-gate.sh -->
**Generated:** 2026-07-31T20:39:35+00:00 **Generated:** 2026-07-31T14:44:58+00:00
Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`). Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`).
@@ -11,14 +11,14 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev
| Metric | Value | | Metric | Value |
|---|---| |---|---|
| Source files scanned | 112 | | Source files scanned | 111 |
| TRACES tags found | 148 | | TRACES tags found | 132 |
| EXCEPTION tags found | 0 | | EXCEPTION tags found | 0 |
| Requirements defined | 69 | | Requirements defined | 69 |
| Requirements covered | 38 | | Requirements covered | 38 |
| **Coverage** | **55.1%** (38/69) | | **Coverage** | **55.1%** (38/69) |
| Coverage of CI-executable scope | 67.9% (38/56) | | Coverage of CI-executable scope | 67.9% (38/56) |
| Tagged but unexecuted in CI | 8 | | Tagged but unexecuted in CI | 5 |
| Orphan tags | 0 | | Orphan tags | 0 |
### By type ### By type
@@ -29,7 +29,7 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev
| DP | 2 | 0 | 8 | | DP | 2 | 0 | 8 |
| IR | 8 | 0 | 8 | | IR | 8 | 0 | 8 |
| GR | 5 | 0 | 9 | | GR | 5 | 0 | 9 |
| VR | 1 | 7 | 14 | | VR | 1 | 4 | 14 |
- **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104, UT-105, UT-106, UT-107, UT-108 - **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104, UT-105, UT-106, UT-107, UT-108
- **IT** tags present (separate taxonomy, not counted in coverage): IT-001 - **IT** tags present (separate taxonomy, not counted in coverage): IT-001
@@ -47,16 +47,16 @@ These requirements have no verification tier this repo's CI host can run, so a t
| VR-002 | out-of-ci | yes | Replay drives the **real** KPN nodes, not a reimplementation | | VR-002 | out-of-ci | yes | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | out-of-ci | yes | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… | | VR-003 | out-of-ci | yes | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | out-of-ci | yes | Reproducible validation corpus with ground truth | | VR-004 | out-of-ci | yes | Reproducible validation corpus with ground truth |
| VR-005 | out-of-ci | yes | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… | | VR-005 | out-of-ci | no | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | out-of-ci | no | Re-tune `scene_threshold` once native-rate decode lands | | VR-006 | out-of-ci | no | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | out-of-ci | no | Expansion band, clustering threshold, and deferred-pass ablation | | VR-007 | out-of-ci | no | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | out-of-ci | no | Gallery scaling benchmark — throughput vs gallery size | | VR-008 | out-of-ci | no | Gallery scaling benchmark — throughput vs gallery size |
| VR-010 | out-of-ci | yes | Dump provenance attributes — embedder model, detector settings, `dens… | | VR-010 | out-of-ci | no | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | out-of-ci | no | Rewrite the replay harness for the post-AR-012 output contract | | VR-011 | out-of-ci | no | Rewrite the replay harness for the post-AR-012 output contract |
| VR-012 | T4, out-of-ci | no | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … | | VR-012 | T4, out-of-ci | no | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … |
| VR-013 | T4, out-of-ci | yes | Cross-source identification probe — gallery from one recording, probe… | | VR-013 | T4, out-of-ci | no | Cross-source identification probe — gallery from one recording, probe… |
**Tagged but unexecuted:** AR-027, VR-001, VR-002, VR-003, VR-004, VR-005, VR-010, VR-013 — a test exists and is tagged, but this CI host cannot run it. Report those runs separately. **Tagged but unexecuted:** AR-027, VR-001, VR-002, VR-003, VR-004 — a test exists and is tagged, but this CI host cannot run it. Report those runs separately.
## Orphan tags ## Orphan tags
@@ -84,10 +84,10 @@ _None._
| AR-002 | Planned | T2 | SR-002 | untagged | - | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's … | | AR-002 | Planned | T2 | SR-002 | untagged | - | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's … |
| AR-003 | **Done** — `max_fac… | T1, T2, T4 | SR-002 | covered | `src/config.hpp`, `src/nodes/face_detector_node.hpp`, `src/nodes/identity_matcher_node.hpp` | No fixed per-frame face cap — crowd scenes must not lose background c… | | AR-003 | **Done** — `max_fac… | T1, T2, T4 | SR-002 | covered | `src/config.hpp`, `src/nodes/face_detector_node.hpp`, `src/nodes/identity_matcher_node.hpp` | No fixed per-frame face cap — crowd scenes must not lose background c… |
| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_replay_fixtures.cpp` | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… | | AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_replay_fixtures.cpp` | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… |
| AR-005 | **Done** — `umeyama… | T1, T3 | SR-002 | covered | `src/face_utils.hpp`, `src/nodes/face_aligner_node.hpp`, `tests/test_face_utils.cpp` | Align to 112×112 via ArcFace 5-point similarity transform, fitted by … | | AR-005 | **Done** — `umeyama… | T1, T3 | SR-002 | covered | `src/face_utils.hpp`, `tests/test_face_utils.cpp` | Align to 112×112 via ArcFace 5-point similarity transform, fitted by … |
| AR-006 | Done | T3 | SR-002 | covered | `src/nodes/embedder_node.hpp` | 512-d L2-normalised embeddings, batched | | AR-006 | Done | T3 | SR-002 | covered | `src/nodes/embedder_node.hpp` | 512-d L2-normalised embeddings, batched |
| AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp`, `tests/test_face_tracker.cpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… | | AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… |
| AR-008 | **Done** — one pool… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp`, `tests/test_face_tracker.cpp` | One track pool keyed on `last_seen`; no separate revival path | | AR-008 | **Done** — one pool… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | One track pool keyed on `last_seen`; no separate revival path |
| AR-009 | Done | T2 | SR-002 | covered | `src/nodes/camera_position_change_detector_node.hpp` | Camera-cut detection (histogram) as an association hint | | AR-009 | Done | T2 | SR-002 | covered | `src/nodes/camera_position_change_detector_node.hpp` | Camera-cut detection (histogram) as an association hint |
| AR-010 | **Done** — decode b… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/scene_boundary_annotator_node.hpp`, `src/nodes/scene_detector_node.hpp`, `src/scene_boundaries.hpp` | Scene-boundary detection (TransNetV2) as an association hint | | AR-010 | **Done** — decode b… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/scene_boundary_annotator_node.hpp`, `src/nodes/scene_detector_node.hpp`, `src/scene_boundaries.hpp` | Scene-boundary detection (TransNetV2) as an association hint |
| AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… | | AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… |
@@ -97,19 +97,19 @@ _None._
| AR-015 | **Done** — reverse … | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Two live tracks owned by one actor ⇒ treat as a detected cut, re-asso… | | AR-015 | **Done** — reverse … | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Two live tracks owned by one actor ⇒ treat as a detected cut, re-asso… |
| AR-016 | **Done** — `flush()… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | All tracks closed at EOF — a film ends with faces on screen | | AR-016 | **Done** — `flush()… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | All tracks closed at EOF — a film ends with faces on screen |
| AR-017 | **Done** — `DeadTra… | T1, T2 | SR-002 | covered | `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Every presence claim carries its belief and identification route | | AR-017 | **Done** — `DeadTra… | T1, T2 | SR-002 | covered | `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Every presence claim carries its belief and identification route |
| AR-018 | **Done** — banded a… | T1, T2 | SR-005 | covered | `src/config.hpp`, `src/gallery/track_gallery.hpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_track_gallery.cpp` | Per-subject embedding store with banded admission (novel enough, safe… | | AR-018 | **Done** — banded a… | T1, T2 | SR-005 | covered | `src/config.hpp`, `src/gallery/track_gallery.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-subject embedding store with banded admission (novel enough, safe… |
| AR-019 | **Done** — all thre… | T2 | SR-005 | covered | `src/gallery/track_gallery.hpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_track_gallery.cpp` | Per-film gallery annex from owned tracks; acquires the non-frontal vi… | | AR-019 | **Done** — all thre… | T2 | SR-005 | covered | `src/gallery/track_gallery.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-film gallery annex from owned tracks; acquires the non-frontal vi… |
| AR-020 | Planned | T2 | SR-005 | untagged | - | Deferred re-identification of unknown tracks against the final expand… | | AR-020 | Planned | T2 | SR-005 | untagged | - | Deferred re-identification of unknown tracks against the final expand… |
| AR-021 | Planned | T2 | SR-005 | untagged | - | Cluster unknown tracks into one entity per person, under temporal can… | | AR-021 | Planned | T2 | SR-005 | untagged | - | Cluster unknown tracks into one entity per person, under temporal can… |
| AR-022 | Planned | T1, T2 | §4 | untagged | - | Capture still-unidentified tracks: embeddings, metadata, **context cr… | | AR-022 | Planned | T1, T2 | §4 | untagged | - | Capture still-unidentified tracks: embeddings, metadata, **context cr… |
| AR-023 | Done | T1 | SR-002 | covered | `src/gallery/gallery_calibration.hpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_calibration.cpp` | Fit sigmoid calibration from intra/inter similarity distributions | | AR-023 | Done | T1 | SR-002 | covered | `src/gallery/gallery_calibration.hpp`, `src/nodes/identity_matcher_node.hpp` | Fit sigmoid calibration from intra/inter similarity distributions |
| AR-024 | **Done** — associat… | T1, static | SR-002 | covered | `src/config.hpp`, `src/evidence_discount.hpp`, `src/gallery/gallery_calibration.hpp`, `src/gallery/track_gallery.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_track_gallery.cpp` | **Always the calibrated probability, never a raw cosine** — exception… | | AR-024 | **Done** — associat… | T1, static | SR-002 | covered | `src/config.hpp`, `src/evidence_discount.hpp`, `src/gallery/gallery_calibration.hpp`, `src/gallery/track_gallery.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp`, `src/nodes/identity_matcher_node.hpp` | **Always the calibrated probability, never a raw cosine** — exception… |
| AR-025 | **Done** — log-odds… | T1 | SR-002 | covered | `src/evidence_discount.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-track Bayesian accumulation in log-odds, with correlated-observat… | | AR-025 | **Done** — log-odds… | T1 | SR-002 | covered | `src/evidence_discount.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-track Bayesian accumulation in log-odds, with correlated-observat… |
| AR-026 | In Progress | T1, T4 | SR-001 | covered | `src/backends/gemm_backend.cpp`, `tests/test_similarity.cpp` | All similarity computed as GEMM, including annex and deferred pass | | AR-026 | In Progress | T1, T4 | SR-001 | covered | `src/backends/gemm_backend.cpp` | All similarity computed as GEMM, including annex and deferred pass |
| AR-027 | Planned | T4 | SR-001 | tagged, unexecuted | `src/backends/gemm_backend.cpp` | Throughput acceptable for **arbitrary** gallery size | | AR-027 | Planned | T4 | SR-001 | tagged, unexecuted | `src/backends/gemm_backend.cpp` | Throughput acceptable for **arbitrary** gallery size |
| AR-028 | Planned | T2 | SR-002 | untagged | - | **Embedding input quality assessed and carried** — every face scored … | | AR-028 | Planned | T2 | SR-002 | untagged | - | **Embedding input quality assessed and carried** — every face scored … |
| AR-029 | Planned | T1 | SR-002 | untagged | - | Sharpness measure on the **aligned crop** (scale-normalised, so it ca… | | AR-029 | Planned | T1 | SR-002 | untagged | - | Sharpness measure on the **aligned crop** (scale-normalised, so it ca… |
| AR-030 | **In Progress** — m… | T1 | SR-002 | covered | `src/face_utils.hpp`, `src/nodes/face_aligner_node.hpp`, `tests/test_face_utils.cpp` | Visibility measure from the AR-001 5-point landmarks — extreme pose o… | | AR-030 | **In Progress** — m… | T1 | SR-002 | covered | `src/face_utils.hpp`, `tests/test_face_utils.cpp` | Visibility measure from the AR-001 5-point landmarks — extreme pose o… |
| DP-001 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | One analysis core; modes are front-ends and must not fork pipeline lo… | | DP-001 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | One analysis core; modes are front-ends and must not fork pipeline lo… |
| DP-002 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | Batch CLI over one title | | DP-002 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | Batch CLI over one title |
| DP-003 | Planned | T1, manual | PR-004 | untagged | - | On-demand resident service with bounded, observable queue | | DP-003 | Planned | T1, manual | PR-004 | untagged | - | On-demand resident service with bounded, observable queue |
@@ -128,7 +128,7 @@ _None._
| IR-008 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Emit and honour the signature's own `v1:` version prefix | | IR-008 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Emit and honour the signature's own `v1:` version prefix |
| GR-001 | Done | T1, T3 | SR-001, SR-005 | covered | `scripts/make_jellyfin_gallery.py` | Build gallery from Jellyfin library cast, TMDB profile fallback | | GR-001 | Done | T1, T3 | SR-001, SR-005 | covered | `scripts/make_jellyfin_gallery.py` | Build gallery from Jellyfin library cast, TMDB profile fallback |
| GR-002 | Done | T1, T3 | PR-003 | covered | `scripts/make_jellyfin_gallery.py` | Incremental `--merge` refresh without re-embedding known actors | | GR-002 | Done | T1, T3 | PR-003 | covered | `scripts/make_jellyfin_gallery.py` | Incremental `--merge` refresh without re-embedding known actors |
| GR-003 | **Done** — `gallery… | T1, T3 | SR-001 | covered | `src/build_gallery.cpp`, `src/gallery/gallery_calibration.hpp`, `src/gallery/gallery_report.hpp`, `tests/test_calibration.cpp` | Report coverage: zero-image actors, under-referenced actors, dedup, c… | | GR-003 | Planned | T1, T3 | SR-001 | covered | `src/build_gallery.cpp`, `src/gallery/gallery_calibration.hpp`, `src/gallery/gallery_report.hpp` | Report coverage: zero-image actors, under-referenced actors, dedup, c… |
| GR-004 | **Done** — basename… | T1, T3 | SR-001 | covered | `scripts/filter_gallery.py`, `scripts/make_gallery.py`, `scripts/make_jellyfin_gallery.py`, `scripts/movienet_eval.py`, `scripts/optimizer/fetch_missing_actors.py`, `scripts/optimizer/optimize.py`, `scripts/optimizer/reembed_gallery.py`, `scripts/optimizer/replay.py`, `scripts/sae_embed_loader.py`, `scripts/sae_gallery.py`, `scripts/sae_stamp.py`, `scripts/stamp_gallery.py`, `src/config.hpp`, `src/gallery/embedder_stamp.cpp`, `src/gallery/embedder_stamp.hpp`, `src/gallery/gallery_builder.cpp`, `src/gallery/gallery_store.cpp`, `src/kpn_bindings.cpp`, `src/main.cpp`, `src/nodes/embedding_dump_node.hpp`, `src/scene_preview.cpp`, `src/types.hpp`, `tests/test_gallery_store.cpp` | Stamp embedder identity into the gallery; **hard startup error** on m… | | GR-004 | **Done** — basename… | T1, T3 | SR-001 | covered | `scripts/filter_gallery.py`, `scripts/make_gallery.py`, `scripts/make_jellyfin_gallery.py`, `scripts/movienet_eval.py`, `scripts/optimizer/fetch_missing_actors.py`, `scripts/optimizer/optimize.py`, `scripts/optimizer/reembed_gallery.py`, `scripts/optimizer/replay.py`, `scripts/sae_embed_loader.py`, `scripts/sae_gallery.py`, `scripts/sae_stamp.py`, `scripts/stamp_gallery.py`, `src/config.hpp`, `src/gallery/embedder_stamp.cpp`, `src/gallery/embedder_stamp.hpp`, `src/gallery/gallery_builder.cpp`, `src/gallery/gallery_store.cpp`, `src/kpn_bindings.cpp`, `src/main.cpp`, `src/nodes/embedding_dump_node.hpp`, `src/scene_preview.cpp`, `src/types.hpp`, `tests/test_gallery_store.cpp` | Stamp embedder identity into the gallery; **hard startup error** on m… |
| GR-005 | Done | T1, T3 | **SR-005** | covered | `src/gallery/gallery_store.hpp` | Gallery data never leaves the instance | | GR-005 | Done | T1, T3 | **SR-005** | covered | `src/gallery/gallery_store.hpp` | Gallery data never leaves the instance |
| GR-006 | Planned | T1 | SR-005 | untagged | - | Provenance tiers: baked / harvested / confirmed, distinguishable per … | | GR-006 | Planned | T1 | SR-005 | untagged | - | Provenance tiers: baked / harvested / confirmed, distinguishable per … |
@@ -139,15 +139,15 @@ _None._
| VR-002 | **Done** — replay d… | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py`, `tests/test_replay_fixtures.cpp` | Replay drives the **real** KPN nodes, not a reimplementation | | VR-002 | **Done** — replay d… | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py`, `tests/test_replay_fixtures.cpp` | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/second_score.py` | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… | | VR-003 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/second_score.py` | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/validation/ground_truth.py` | Reproducible validation corpus with ground truth | | VR-004 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/validation/ground_truth.py` | Reproducible validation corpus with ground truth |
| VR-005 | **Done** — knee at … | out-of-ci | PR-002 | tagged, unexecuted | `scripts/validation/min_face_size.py` | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… | | VR-005 | **Done** — knee at … | out-of-ci | PR-002 | untagged | - | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | Planned | out-of-ci | PR-002 | untagged | - | Re-tune `scene_threshold` once native-rate decode lands | | VR-006 | Planned | out-of-ci | PR-002 | untagged | - | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | Planned | out-of-ci | PR-002 | untagged | - | Expansion band, clustering threshold, and deferred-pass ablation | | VR-007 | Planned | out-of-ci | PR-002 | untagged | - | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | Planned | out-of-ci | PR-002 | untagged | - | Gallery scaling benchmark — throughput vs gallery size | | VR-008 | Planned | out-of-ci | PR-002 | untagged | - | Gallery scaling benchmark — throughput vs gallery size |
| VR-009 | Planned | T1, out-of-ci | PR-002 | untagged | - | Verify accumulated posteriors are calibrated against held-out tracks | | VR-009 | Planned | T1, out-of-ci | PR-002 | untagged | - | Verify accumulated posteriors are calibrated against held-out tracks |
| VR-010 | Planned | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp` | Dump provenance attributes — embedder model, detector settings, `dens… | | VR-010 | Planned | out-of-ci | PR-002 | untagged | - | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | Planned | out-of-ci | PR-002 | untagged | - | Rewrite the replay harness for the post-AR-012 output contract | | VR-011 | Planned | out-of-ci | PR-002 | untagged | - | Rewrite the replay harness for the post-AR-012 output contract |
| VR-012 | Planned | T4, out-of-ci | PR-002 | untagged | - | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … | | VR-012 | Planned | T4, out-of-ci | PR-002 | untagged | - | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … |
| VR-013 | **In Progress** — h… | T4, out-of-ci | PR-002 | tagged, unexecuted | `experiments/xsource/resolution_sweep.py`, `experiments/xsource/verify_labels.py` | Cross-source identification probe — gallery from one recording, probe… | | VR-013 | **In Progress** — h… | T4, out-of-ci | PR-002 | untagged | - | Cross-source identification probe — gallery from one recording, probe… |
| VR-014 | **Done** — 40 rando… | T2, out-of-ci | PR-002 | covered | `scripts/validation/test_audio_offset.py` | Audio-signature **offset recovery on real content** — a known trim re… | | VR-014 | **Done** — 40 rando… | T2, out-of-ci | PR-002 | covered | `scripts/validation/test_audio_offset.py` | Audio-signature **offset recovery on real content** — a known trim re… |
## Detailed mapping ## Detailed mapping
@@ -171,16 +171,15 @@ _None._
**Locations:** 4 **Locations:** 4
- [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;` - [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;`
- [`src/main.cpp:319`](../src/main.cpp#L319) — `Unknown` - [`src/main.cpp:299`](../src/main.cpp#L299) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:166`](../src/nodes/identity_matcher_node.hpp#L166) — `std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);` - [`src/nodes/identity_matcher_node.hpp:166`](../src/nodes/identity_matcher_node.hpp#L166) — `std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);`
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` - [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
### AR-005 ### AR-005
**Locations:** 3 **Locations:** 2
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown` - [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown`
- [`src/nodes/face_aligner_node.hpp:7`](../src/nodes/face_aligner_node.hpp#L7) — `struct FaceAlignerFunc`
- [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown` - [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown`
### AR-006 ### AR-006
@@ -191,21 +190,19 @@ _None._
### AR-007 ### AR-007
**Locations:** 4 **Locations:** 3
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown` - [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));` - [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown` - [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`tests/test_face_tracker.cpp:1`](../tests/test_face_tracker.cpp#L1) — `Unknown`
### AR-008 ### AR-008
**Locations:** 4 **Locations:** 3
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown` - [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));` - [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown` - [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`tests/test_face_tracker.cpp:1`](../tests/test_face_tracker.cpp#L1) — `Unknown`
### AR-009 ### AR-009
@@ -218,9 +215,9 @@ _None._
**Locations:** 9 **Locations:** 9
- [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;` - [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;`
- [`src/main.cpp:329`](../src/main.cpp#L329) — `Unknown` - [`src/main.cpp:309`](../src/main.cpp#L309) — `Unknown`
- [`src/main.cpp:399`](../src/main.cpp#L399) — `return run_net(std::move(net));` - [`src/main.cpp:379`](../src/main.cpp#L379) — `return run_net(std::move(net));`
- [`src/main.cpp:431`](../src/main.cpp#L431) — `Unknown` - [`src/main.cpp:411`](../src/main.cpp#L411) — `Unknown`
- [`src/nodes/scene_boundary_annotator_node.hpp:2`](../src/nodes/scene_boundary_annotator_node.hpp#L2) — `Unknown` - [`src/nodes/scene_boundary_annotator_node.hpp:2`](../src/nodes/scene_boundary_annotator_node.hpp#L2) — `Unknown`
- [`src/nodes/scene_detector_node.hpp:36`](../src/nodes/scene_detector_node.hpp#L36) — `static constexpr std::string_view label() { return "scene_detector"; }` - [`src/nodes/scene_detector_node.hpp:36`](../src/nodes/scene_detector_node.hpp#L36) — `static constexpr std::string_view label() { return "scene_detector"; }`
- [`src/nodes/scene_detector_node.hpp:110`](../src/nodes/scene_detector_node.hpp#L110) — `void flush_remaining()` - [`src/nodes/scene_detector_node.hpp:110`](../src/nodes/scene_detector_node.hpp#L110) — `void flush_remaining()`
@@ -282,48 +279,42 @@ _None._
### AR-018 ### AR-018
**Locations:** 5 **Locations:** 3
- [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown` - [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown`
- [`src/gallery/track_gallery.hpp:164`](../src/gallery/track_gallery.hpp#L164) — `struct TrackState` - [`src/gallery/track_gallery.hpp:161`](../src/gallery/track_gallery.hpp#L161) — `struct TrackState`
- [`src/gallery/track_gallery.hpp:274`](../src/gallery/track_gallery.hpp#L274) — `static int plurality_actor(const TrackState& ts)`
- [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);` - [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);`
- [`tests/test_track_gallery.cpp:1`](../tests/test_track_gallery.cpp#L1) — `Unknown`
### AR-019 ### AR-019
**Locations:** 4 **Locations:** 3
- [`src/gallery/track_gallery.hpp:123`](../src/gallery/track_gallery.hpp#L123) — `void forget(int track_id) { tracks_.erase(track_id); }` - [`src/gallery/track_gallery.hpp:122`](../src/gallery/track_gallery.hpp#L122) — `void forget(int track_id) { tracks_.erase(track_id); }`
- [`src/nodes/identity_matcher_node.hpp:147`](../src/nodes/identity_matcher_node.hpp#L147) — `MatchedSceneFrame operator()(TrackedSceneFrame tf)` - [`src/nodes/identity_matcher_node.hpp:147`](../src/nodes/identity_matcher_node.hpp#L147) — `MatchedSceneFrame operator()(TrackedSceneFrame tf)`
- [`src/nodes/identity_matcher_node.hpp:285`](../src/nodes/identity_matcher_node.hpp#L285) — `Unknown` - [`src/nodes/identity_matcher_node.hpp:285`](../src/nodes/identity_matcher_node.hpp#L285) — `Unknown`
- [`tests/test_track_gallery.cpp:1`](../tests/test_track_gallery.cpp#L1) — `Unknown`
### AR-023 ### AR-023
**Locations:** 4 **Locations:** 3
- [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown` - [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const` - [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }` - [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`tests/test_calibration.cpp:1`](../tests/test_calibration.cpp#L1) — `Unknown`
### AR-024 ### AR-024
**Locations:** 12 **Locations:** 10
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown` - [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown` - [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown`
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown` - [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const` - [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/gallery/track_gallery.hpp:133`](../src/gallery/track_gallery.hpp#L133) — `void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }` - [`src/gallery/track_gallery.hpp:132`](../src/gallery/track_gallery.hpp#L132) — `void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }`
- [`src/gallery/track_gallery.hpp:164`](../src/gallery/track_gallery.hpp#L164) — `struct TrackState` - [`src/gallery/track_gallery.hpp:161`](../src/gallery/track_gallery.hpp#L161) — `struct TrackState`
- [`src/gallery/track_gallery.hpp:274`](../src/gallery/track_gallery.hpp#L274) — `static int plurality_actor(const TrackState& ts)`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));` - [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown` - [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);` - [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);`
- [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }` - [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`tests/test_track_gallery.cpp:1`](../tests/test_track_gallery.cpp#L1) — `Unknown`
### AR-025 ### AR-025
@@ -335,10 +326,9 @@ _None._
### AR-026 ### AR-026
**Locations:** 2 **Locations:** 1
- [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;` - [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;`
- [`tests/test_similarity.cpp:1`](../tests/test_similarity.cpp#L1) — `Unknown`
### AR-027 ### AR-027
@@ -348,10 +338,9 @@ _None._
### AR-030 ### AR-030
**Locations:** 3 **Locations:** 2
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown` - [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown`
- [`src/nodes/face_aligner_node.hpp:7`](../src/nodes/face_aligner_node.hpp#L7) — `struct FaceAlignerFunc`
- [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown` - [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown`
### DP-001 ### DP-001
@@ -380,7 +369,7 @@ _None._
### GR-003 ### GR-003
**Locations:** 16 **Locations:** 13
- [`src/build_gallery.cpp:83`](../src/build_gallery.cpp#L83) — `Unknown` - [`src/build_gallery.cpp:83`](../src/build_gallery.cpp#L83) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:80`](../src/gallery/gallery_calibration.hpp#L80) — `struct GalleryCalibrationStats` - [`src/gallery/gallery_calibration.hpp:80`](../src/gallery/gallery_calibration.hpp#L80) — `struct GalleryCalibrationStats`
@@ -395,9 +384,6 @@ _None._
- [`src/gallery/gallery_report.hpp:446`](../src/gallery/gallery_report.hpp#L446) — `inline void save_gallery_report(const std::string& path, const GalleryReport& r)` - [`src/gallery/gallery_report.hpp:446`](../src/gallery/gallery_report.hpp#L446) — `inline void save_gallery_report(const std::string& path, const GalleryReport& r)`
- [`src/gallery/gallery_report.hpp:454`](../src/gallery/gallery_report.hpp#L454) — `inline GalleryReport load_gallery_report(const std::string& path)` - [`src/gallery/gallery_report.hpp:454`](../src/gallery/gallery_report.hpp#L454) — `inline GalleryReport load_gallery_report(const std::string& path)`
- [`src/gallery/gallery_report.hpp:464`](../src/gallery/gallery_report.hpp#L464) — `return gallery_report_from_json(j);` - [`src/gallery/gallery_report.hpp:464`](../src/gallery/gallery_report.hpp#L464) — `return gallery_report_from_json(j);`
- [`tests/test_calibration.cpp:192`](../tests/test_calibration.cpp#L192) — `Embedding unit_axis(int slot)`
- [`tests/test_calibration.cpp:222`](../tests/test_calibration.cpp#L222) — `Unknown`
- [`tests/test_calibration.cpp:248`](../tests/test_calibration.cpp#L248) — `Unknown`
### GR-004 ### GR-004
@@ -413,8 +399,8 @@ _None._
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown` - [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown` - [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:188`](../src/main.cpp#L188) — `Unknown` - [`src/main.cpp:188`](../src/main.cpp#L188) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:127`](../src/nodes/embedding_dump_node.hpp#L127) — `static constexpr std::string_view label() { return "embedding_dump"; }` - [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:238`](../src/nodes/embedding_dump_node.hpp#L238) — `H5::H5File file(path_, H5F_ACC_TRUNC);` - [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)` - [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:148`](../src/types.hpp#L148) — `struct Actor` - [`src/types.hpp:148`](../src/types.hpp#L148) — `struct Actor`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");` - [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
@@ -549,18 +535,12 @@ _None._
### PR-002 ### PR-002
**Locations:** 10 **Locations:** 4
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown` - [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:18`](../src/nodes/embedding_dump_node.hpp#L18) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:133`](../src/nodes/embedding_dump_node.hpp#L133) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:159`](../src/nodes/embedding_dump_node.hpp#L159) — `void operator()(EmbeddedSceneFrame ef)`
- [`src/nodes/embedding_dump_node.hpp:242`](../src/nodes/embedding_dump_node.hpp#L242) — `H5::H5File file(path_, H5F_ACC_TRUNC);`
- [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an` - [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an`
- [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown` - [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown`
- [`scripts/validation/ground_truth.py:24`](../scripts/validation/ground_truth.py#L24) — `Unknown` - [`scripts/validation/ground_truth.py:24`](../scripts/validation/ground_truth.py#L24) — `Unknown`
- [`experiments/xsource/resolution_sweep.py:4`](../experiments/xsource/resolution_sweep.py#L4) — `Unknown`
- [`experiments/xsource/verify_labels.py:4`](../experiments/xsource/verify_labels.py#L4) — `Unknown`
### PR-004 ### PR-004
@@ -570,7 +550,7 @@ _None._
### SR-001 ### SR-001
**Locations:** 64 **Locations:** 60
- [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;` - [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;`
- [`src/build_gallery.cpp:83`](../src/build_gallery.cpp#L83) — `Unknown` - [`src/build_gallery.cpp:83`](../src/build_gallery.cpp#L83) — `Unknown`
@@ -596,13 +576,10 @@ _None._
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown` - [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown` - [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:188`](../src/main.cpp#L188) — `Unknown` - [`src/main.cpp:188`](../src/main.cpp#L188) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:127`](../src/nodes/embedding_dump_node.hpp#L127) — `static constexpr std::string_view label() { return "embedding_dump"; }` - [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:238`](../src/nodes/embedding_dump_node.hpp#L238) — `H5::H5File file(path_, H5F_ACC_TRUNC);` - [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)` - [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:148`](../src/types.hpp#L148) — `struct Actor` - [`src/types.hpp:148`](../src/types.hpp#L148) — `struct Actor`
- [`tests/test_calibration.cpp:192`](../tests/test_calibration.cpp#L192) — `Embedding unit_axis(int slot)`
- [`tests/test_calibration.cpp:222`](../tests/test_calibration.cpp#L222) — `Unknown`
- [`tests/test_calibration.cpp:248`](../tests/test_calibration.cpp#L248) — `Unknown`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");` - [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");` - [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");` - [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");`
@@ -615,7 +592,6 @@ _None._
- [`tests/test_gallery_store.cpp:348`](../tests/test_gallery_store.cpp#L348) — `Unknown` - [`tests/test_gallery_store.cpp:348`](../tests/test_gallery_store.cpp#L348) — `Unknown`
- [`tests/test_gallery_store.cpp:367`](../tests/test_gallery_store.cpp#L367) — `Unknown` - [`tests/test_gallery_store.cpp:367`](../tests/test_gallery_store.cpp#L367) — `Unknown`
- [`tests/test_gallery_store.cpp:384`](../tests/test_gallery_store.cpp#L384) — `TempFile tf("fake_model.onnx");` - [`tests/test_gallery_store.cpp:384`](../tests/test_gallery_store.cpp#L384) — `TempFile tf("fake_model.onnx");`
- [`tests/test_similarity.cpp:1`](../tests/test_similarity.cpp#L1) — `Unknown`
- [`scripts/filter_gallery.py:80`](../scripts/filter_gallery.py#L80) — `if actor_jellyfin_id(a) in cast_ids]` - [`scripts/filter_gallery.py:80`](../scripts/filter_gallery.py#L80) — `if actor_jellyfin_id(a) in cast_ids]`
- [`scripts/make_gallery.py:181`](../scripts/make_gallery.py#L181) — `Unknown` - [`scripts/make_gallery.py:181`](../scripts/make_gallery.py#L181) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown` - [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
@@ -639,7 +615,7 @@ _None._
### SR-002 ### SR-002
**Locations:** 35 **Locations:** 32
- [`src/config.hpp:44`](../src/config.hpp#L44) — `Unknown` - [`src/config.hpp:44`](../src/config.hpp#L44) — `Unknown`
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown` - [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
@@ -650,13 +626,12 @@ _None._
- [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;` - [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));` - [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));` - [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:319`](../src/main.cpp#L319) — `Unknown` - [`src/main.cpp:299`](../src/main.cpp#L299) — `Unknown`
- [`src/main.cpp:329`](../src/main.cpp#L329) — `Unknown` - [`src/main.cpp:309`](../src/main.cpp#L309) — `Unknown`
- [`src/main.cpp:399`](../src/main.cpp#L399) — `return run_net(std::move(net));` - [`src/main.cpp:379`](../src/main.cpp#L379) — `return run_net(std::move(net));`
- [`src/main.cpp:431`](../src/main.cpp#L431) — `Unknown` - [`src/main.cpp:411`](../src/main.cpp#L411) — `Unknown`
- [`src/nodes/camera_position_change_detector_node.hpp:30`](../src/nodes/camera_position_change_detector_node.hpp#L30) — `struct CameraPositionChangeDetectorFunc` - [`src/nodes/camera_position_change_detector_node.hpp:30`](../src/nodes/camera_position_change_detector_node.hpp#L30) — `struct CameraPositionChangeDetectorFunc`
- [`src/nodes/embedder_node.hpp:21`](../src/nodes/embedder_node.hpp#L21) — `struct EmbedderFunc` - [`src/nodes/embedder_node.hpp:21`](../src/nodes/embedder_node.hpp#L21) — `struct EmbedderFunc`
- [`src/nodes/face_aligner_node.hpp:7`](../src/nodes/face_aligner_node.hpp#L7) — `struct FaceAlignerFunc`
- [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown` - [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown`
- [`src/nodes/face_detector_node.hpp:47`](../src/nodes/face_detector_node.hpp#L47) — `private:` - [`src/nodes/face_detector_node.hpp:47`](../src/nodes/face_detector_node.hpp#L47) — `private:`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown` - [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
@@ -673,8 +648,6 @@ _None._
- [`src/nodes/scene_detector_node.hpp:144`](../src/nodes/scene_detector_node.hpp#L144) — `void write_output()` - [`src/nodes/scene_detector_node.hpp:144`](../src/nodes/scene_detector_node.hpp#L144) — `void write_output()`
- [`src/scene_boundaries.hpp:2`](../src/scene_boundaries.hpp#L2) — `Unknown` - [`src/scene_boundaries.hpp:2`](../src/scene_boundaries.hpp#L2) — `Unknown`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown` - [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_calibration.cpp:1`](../tests/test_calibration.cpp#L1) — `Unknown`
- [`tests/test_face_tracker.cpp:1`](../tests/test_face_tracker.cpp#L1) — `Unknown`
- [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown` - [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown`
### SR-003 ### SR-003
@@ -691,18 +664,16 @@ _None._
### SR-005 ### SR-005
**Locations:** 11 **Locations:** 9
- [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown` - [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown`
- [`src/gallery/gallery_store.hpp:15`](../src/gallery/gallery_store.hpp#L15) — `Unknown` - [`src/gallery/gallery_store.hpp:15`](../src/gallery/gallery_store.hpp#L15) — `Unknown`
- [`src/gallery/track_gallery.hpp:123`](../src/gallery/track_gallery.hpp#L123) — `void forget(int track_id) { tracks_.erase(track_id); }` - [`src/gallery/track_gallery.hpp:122`](../src/gallery/track_gallery.hpp#L122) — `void forget(int track_id) { tracks_.erase(track_id); }`
- [`src/gallery/track_gallery.hpp:133`](../src/gallery/track_gallery.hpp#L133) — `void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }` - [`src/gallery/track_gallery.hpp:132`](../src/gallery/track_gallery.hpp#L132) — `void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }`
- [`src/gallery/track_gallery.hpp:164`](../src/gallery/track_gallery.hpp#L164) — `struct TrackState` - [`src/gallery/track_gallery.hpp:161`](../src/gallery/track_gallery.hpp#L161) — `struct TrackState`
- [`src/gallery/track_gallery.hpp:274`](../src/gallery/track_gallery.hpp#L274) — `static int plurality_actor(const TrackState& ts)`
- [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);` - [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);`
- [`src/nodes/identity_matcher_node.hpp:147`](../src/nodes/identity_matcher_node.hpp#L147) — `MatchedSceneFrame operator()(TrackedSceneFrame tf)` - [`src/nodes/identity_matcher_node.hpp:147`](../src/nodes/identity_matcher_node.hpp#L147) — `MatchedSceneFrame operator()(TrackedSceneFrame tf)`
- [`src/nodes/identity_matcher_node.hpp:285`](../src/nodes/identity_matcher_node.hpp#L285) — `Unknown` - [`src/nodes/identity_matcher_node.hpp:285`](../src/nodes/identity_matcher_node.hpp#L285) — `Unknown`
- [`tests/test_track_gallery.cpp:1`](../tests/test_track_gallery.cpp#L1) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown` - [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### UT-001 ### UT-001
@@ -798,29 +769,6 @@ _None._
- [`scripts/validation/ground_truth.py:24`](../scripts/validation/ground_truth.py#L24) — `Unknown` - [`scripts/validation/ground_truth.py:24`](../scripts/validation/ground_truth.py#L24) — `Unknown`
### VR-005
**Locations:** 1
- [`scripts/validation/min_face_size.py:5`](../scripts/validation/min_face_size.py#L5) — `Unknown`
### VR-010
**Locations:** 5
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:18`](../src/nodes/embedding_dump_node.hpp#L18) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:133`](../src/nodes/embedding_dump_node.hpp#L133) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:159`](../src/nodes/embedding_dump_node.hpp#L159) — `void operator()(EmbeddedSceneFrame ef)`
- [`src/nodes/embedding_dump_node.hpp:242`](../src/nodes/embedding_dump_node.hpp#L242) — `H5::H5File file(path_, H5F_ACC_TRUNC);`
### VR-013
**Locations:** 2
- [`experiments/xsource/resolution_sweep.py:4`](../experiments/xsource/resolution_sweep.py#L4) — `Unknown`
- [`experiments/xsource/verify_labels.py:4`](../experiments/xsource/verify_labels.py#L4) — `Unknown`
### VR-014 ### VR-014
**Locations:** 1 **Locations:** 1
experiments/xsource/landmark_voting.py
+7 -65
View File
@@ -30,8 +30,6 @@ import sae_embed # before cv2 — see alignment_compare.py
import numpy as np import numpy as np
import cv2 import cv2
import argparse
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/" ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/" M = ROOT + "models/"
CLIPS = ["5157344", "5157339"] CLIPS = ["5157344", "5157339"]
@@ -39,31 +37,16 @@ PROB_THRESHOLD = 0.754
GROUP_IOU = 0.55 # detections overlapping this much are the same face GROUP_IOU = 0.55 # detections overlapping this much are the same face
MATCH_IOU = 0.35 # tie a detection to the hand-labelled face MATCH_IOU = 0.35 # tie a detection to the hand-labelled face
_ap = argparse.ArgumentParser() base_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
_ap.add_argument("--detector", default="scrfd_500m_bnkps.onnx",
help="detector under models/. SCRFD sizes 500m / 2.5g / 10g come "
"from InsightFace's buffalo_sc / buffalo_m / buffalo_l packs")
_ap.add_argument("--vote-conf", type=float, default=None,
help="confidence floor for the voting pass. Omit to auto-tune "
"it to --target-votes")
_ap.add_argument("--target-votes", type=int, default=3,
help="votes per face to tune --vote-conf towards, so detectors "
"are compared at equal redundancy rather than equal settings")
_args = _ap.parse_args()
base_eng = sae_embed.FaceEmbedder(detector_model=M + _args.detector,
arcface_model=M + "LVFace-B_Glint360K.onnx", arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.5, nms=0.4, max_side=0) conf=0.5, nms=0.4, max_side=0)
# Same models, looser suppression: keep the duplicates NMS would have removed.
vote_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
def _make_vote_engine(conf):
# Same models, looser suppression: keep the duplicates NMS would have removed.
return sae_embed.FaceEmbedder(detector_model=M + _args.detector,
arcface_model=M + "LVFace-B_Glint360K.onnx", arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=conf, nms=0.9, max_side=0) conf=0.3, nms=0.9, max_side=0)
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5") cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
def iou(a, b): def iou(a, b):
ax, ay, aw, ah = a; bx, by, bw, bh = b ax, ay, aw, ah = a; bx, by, bw, bh = b
x0, y0 = max(ax, bx), max(ay, by) x0, y0 = max(ax, bx), max(ay, by)
@@ -96,46 +79,6 @@ def vote(dets):
return out return out
def tune_vote_conf(target, sample=6):
"""Pick the confidence floor giving ~target detections per face to average.
A larger SCRFD is more confident and suppresses harder, so at a fixed floor
it emits fewer overlapping anchors — median 2 against 500m's 3. Comparing
detectors at equal SETTINGS therefore also compares them at unequal
redundancy, and the voting arm is handicapped for the bigger models. Tuning
each to the same votes-per-face isolates landmark quality from how much
there was to average.
"""
frames = sorted(glob.glob(f"frames/d{CLIPS[0]}_*.png"))[:sample]
imgs = [cv2.imread(f) for f in frames]
best = (None, None, 1e9)
for conf in (0.30, 0.20, 0.12, 0.07, 0.04, 0.02, 0.01):
eng = _make_vote_engine(conf)
sizes = [n for img in imgs for _, _, _, n in vote(eng.detect(img))]
if not sizes:
continue
med = float(np.median(sizes))
if abs(med - target) < best[2]:
best = (conf, eng, abs(med - target))
print(f"[tune] conf={conf:.2f} -> median {med:.0f} votes/face", file=sys.stderr)
if med >= target:
break
if best[1] is None:
print(f"[tune] no confidence floor reached {target} votes/face; "
f"falling back to 0.30", file=sys.stderr)
return 0.30, _make_vote_engine(0.30)
print(f"[tune] chose conf={best[0]:.2f} for ~{target} votes/face", file=sys.stderr)
return best[0], best[1]
if _args.vote_conf is not None:
VOTE_CONF, vote_eng = _args.vote_conf, _make_vote_engine(_args.vote_conf)
else:
VOTE_CONF, vote_eng = tune_vote_conf(_args.target_votes)
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
def collect(clip): def collect(clip):
lab = {os.path.basename(p): os.path.basename(os.path.dirname(p)) lab = {os.path.basename(p): os.path.basename(os.path.dirname(p))
for p in glob.glob(f"labelling/{clip}/*/*.jpg") for p in glob.glob(f"labelling/{clip}/*/*.jpg")
@@ -181,8 +124,7 @@ print(f"[voting] group size: median {np.median(allv):.0f}, "
file=sys.stderr) file=sys.stderr)
GAL, PRB = "5157344", "5157339" GAL, PRB = "5157344", "5157339"
print(f"\ndetector={_args.detector} vote_conf={VOTE_CONF:.2f} " print(f"\ngallery {GAL} -> probe {PRB}, P>{PROB_THRESHOLD}\n")
f"gallery {GAL} -> probe {PRB}, P>{PROB_THRESHOLD}\n")
print(f"{'align':>8}{'person':>8}{'n_gal':>7}{'n_prb':>7}" print(f"{'align':>8}{'person':>8}{'n_gal':>7}{'n_prb':>7}"
f"{'within-clip':>13}{'cross-clip':>12}{'hit rate':>10}") f"{'within-clip':>13}{'cross-clip':>12}{'hit rate':>10}")
summary = {} summary = {}
experiments/xsource/resolution_sweep.py
-2
View File
@@ -1,8 +1,6 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Impact of input resolution on cross-source identification. """Impact of input resolution on cross-source identification.
TRACES: VR-013 | PR-002
Gallery is built from one clip at NATIVE resolution. Probes come from the other Gallery is built from one clip at NATIVE resolution. Probes come from the other
clip with the WHOLE FRAME downscaled before it reaches the detector, so clip with the WHOLE FRAME downscaled before it reaches the detector, so
detection and landmark regression degrade together with the pixels. That is the detection and landmark regression degrade together with the pixels. That is the
experiments/xsource/verify_labels.py
-8
View File
@@ -1,14 +1,6 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Integrity check on the labelled set, before it is used as ground truth. """Integrity check on the labelled set, before it is used as ground truth.
TRACES: VR-013 | PR-002
VR-013's ground truth is hand-sorted rather than propagated by embedding
similarity, because propagation would keep only the faces the embedder already
gets right and silently drop the ones the sweep exists to find. This script is
what makes that claim checkable, so it is part of the requirement rather than a
helper of it.
Checks, loudest failure first: Checks, loudest failure first:
1. INDEX INTEGRITY. Each crop's embedding is taken by re-detecting its source 1. INDEX INTEGRITY. Each crop's embedding is taken by re-detecting its source
external/KPN
Vendored
+1 -1
Submodule external/KPN updated: 454f72c167...6595e6e925
mkdocs.yml
+1 -1
View File
@@ -35,11 +35,11 @@ extra_css:
nav: nav:
- Home: index.md - Home: index.md
- How We Score Against X-Ray: methodology.md - How We Score Against X-Ray: methodology.md
- Benchmark — SuperHero: benchmark.md
- Findings: - Findings:
- Best Model: best-model.md - Best Model: best-model.md
- Gallery Scope (Full vs. Limited): gallery-scope.md - Gallery Scope (Full vs. Limited): gallery-scope.md
- Pose Expansion: pose-expansion.md - Pose Expansion: pose-expansion.md
- Quality Knee (Blur and Size): quality-knee.md
- LVFace Deep Dive: lvface-deep-dive.md - LVFace Deep Dive: lvface-deep-dive.md
- Full Experiment Log: model-bakeoff.md - Full Experiment Log: model-bakeoff.md
- Service Conversion (proposal): service-conversion.md - Service Conversion (proposal): service-conversion.md
models/2d106det.onnx
LFS
BIN
View File
Binary file not shown.
models/scrfd_10g_bnkps.onnx
LFS
BIN
View File
Binary file not shown.
models/scrfd_2.5g_bnkps.onnx
LFS
BIN
View File
Binary file not shown.
scripts/artifacts/pull_artifacts.sh
+1 -23
View File
@@ -10,7 +10,6 @@
# scripts/artifacts/pull_artifacts.sh galleries [version] # scripts/artifacts/pull_artifacts.sh galleries [version]
# scripts/artifacts/pull_artifacts.sh montage-frames <film-slug> [version] # scripts/artifacts/pull_artifacts.sh montage-frames <film-slug> [version]
# scripts/artifacts/pull_artifacts.sh experiment-data [version] # scripts/artifacts/pull_artifacts.sh experiment-data [version]
# scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
# scripts/artifacts/pull_artifacts.sh report-highlights <name> [version] # scripts/artifacts/pull_artifacts.sh report-highlights <name> [version]
# scripts/artifacts/pull_artifacts.sh xsource [version] # scripts/artifacts/pull_artifacts.sh xsource [version]
# version defaults to "latest" (newest uploaded version, by created_at). # version defaults to "latest" (newest uploaded version, by created_at).
@@ -43,22 +42,6 @@ print(matches[-1]['version'])
" "
} }
pull_replay_fixtures() {
local version="$1"
local dest="${REPO_ROOT}/tests/fixtures/dumps"
mkdir -p "$dest"
echo "=== replay-fixtures (version ${version}) ==="
local tmp; tmp="$(mktemp -d)"
if curl -sf "${DL_BASE}/generic/replay-fixtures/${version}/replay-fixtures.zip" \
-o "${tmp}/f.zip"; then
unzip -qo "${tmp}/f.zip" -d "$dest"
echo " restored: $(ls "$dest" | wc -l) files into tests/fixtures/dumps/"
else
echo " [warn] replay-fixtures.zip not found at version ${version}" >&2
fi
rm -rf "$tmp"
}
pull_galleries() { pull_galleries() {
local version="$1" local version="$1"
local dest="${REPO_ROOT}/experiments/galleries" local dest="${REPO_ROOT}/experiments/galleries"
@@ -198,13 +181,8 @@ case "$TARGET" in
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version xsource)" [ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version xsource)"
pull_xsource "$VERSION" pull_xsource "$VERSION"
;; ;;
replay-fixtures)
VERSION="${2:-latest}"
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version replay-fixtures)"
pull_replay_fixtures "$VERSION"
;;
*) *)
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, xsource, or replay-fixtures)" >&2 echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2
exit 1 exit 1
;; ;;
esac esac
scripts/artifacts/push_artifacts.sh
+1 -27
View File
@@ -13,12 +13,10 @@
# scripts/artifacts/push_artifacts.sh experiment-data # scripts/artifacts/push_artifacts.sh experiment-data
# scripts/artifacts/push_artifacts.sh report-highlights # scripts/artifacts/push_artifacts.sh report-highlights
# scripts/artifacts/push_artifacts.sh xsource # scripts/artifacts/push_artifacts.sh xsource
# scripts/artifacts/push_artifacts.sh replay-fixtures
# scripts/artifacts/push_artifacts.sh galleries montage-frames experiment-data report-highlights # scripts/artifacts/push_artifacts.sh galleries montage-frames experiment-data report-highlights
# #
# Package layout (owner=dtourolle, repo=scene-actor-extraction): # Package layout (owner=dtourolle, repo=scene-actor-extraction):
# generic/galleries/<version>/gallery_<model>.h5 (one file per model) # generic/galleries/<version>/gallery_<model>.h5 (one file per model)
# generic/replay-fixtures/<version>/replay-fixtures.zip (T2 dumps + their gallery)
# generic/montage-frames/<version>/<film-slug>.zip (zipped per-film frames) # generic/montage-frames/<version>/<film-slug>.zip (zipped per-film frames)
# generic/experiment-data/<version>/experiment-data.zip (manifests/trajectories/results) # generic/experiment-data/<version>/experiment-data.zip (manifests/trajectories/results)
# generic/report-highlights/<version>/<name>.jpg (individual, hand-picked # generic/report-highlights/<version>/<name>.jpg (individual, hand-picked
@@ -52,29 +50,6 @@ upload() {
-o /dev/null -w " HTTP %{http_code}\n" -o /dev/null -w " HTTP %{http_code}\n"
} }
push_replay_fixtures() {
echo "=== replay-fixtures (version ${VERSION}) ==="
# T2 replay fixtures: per-frame detections, landmarks and embeddings dumped
# from a real run, so the tracker and identity stages can be replayed on CPU
# with no GPU, no models and no film. Too large for git (superhero.h5 alone
# is ~9 MB) and regenerating them needs the film plus a GPU, which CI has
# neither of — so they ship as artifacts and CI pulls them.
#
# The gallery travels with them: a dump replays against the gallery it was
# produced with, and pairing a dump with a different gallery silently
# changes every identity decision in it.
local dir="${REPO_ROOT}/tests/fixtures/dumps"
if [ ! -d "$dir" ]; then
echo " no tests/fixtures/dumps dir, skipping" >&2
return
fi
local tmp
tmp="$(mktemp -d)"
( cd "$dir" && zip -qr "$tmp/replay-fixtures.zip" . )
upload "replay-fixtures" "replay-fixtures.zip" "$tmp/replay-fixtures.zip"
rm -rf "$tmp"
}
push_galleries() { push_galleries() {
echo "=== galleries (version ${VERSION}) ===" echo "=== galleries (version ${VERSION}) ==="
local dir="${REPO_ROOT}/experiments/galleries" local dir="${REPO_ROOT}/experiments/galleries"
@@ -174,8 +149,7 @@ for target in "$@"; do
experiment-data) push_experiment_data ;; experiment-data) push_experiment_data ;;
report-highlights) push_report_highlights ;; report-highlights) push_report_highlights ;;
xsource) push_xsource ;; xsource) push_xsource ;;
replay-fixtures) push_replay_fixtures ;; *) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2; exit 1 ;;
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, xsource, or replay-fixtures)" >&2; exit 1 ;;
esac esac
done done
scripts/ci/build_builder_image.sh
-118
View File
@@ -1,118 +0,0 @@
#!/bin/bash
# build_builder_image.sh — build and publish the DP-007 CI builder image to the
# Gitea container registry.
#
# TRACES: DP-007 | PR-004
#
# Usage:
# scripts/ci/build_builder_image.sh # build only, tag v1
# scripts/ci/build_builder_image.sh --push # build and push
# scripts/ci/build_builder_image.sh --tag v2 --push # bump the pinned tag
# scripts/ci/build_builder_image.sh --no-cache # force a clean rebuild
#
# The tag is the contract with CI. .gitea/workflows/unit-tests.yml names an
# explicit tag in its `container:` block and never `latest`, so that rebuilding
# the image cannot silently change what a previous green build meant. Bumping
# the dependency set means bumping the tag AND editing the workflow — the two
# edits landing in the same commit is the point, not an inconvenience.
#
# Registry auth: this script does not log in. Do it once, out of band:
# docker login gitea.tourolle.paris
# The CI host is already authenticated this way (its cached credentials in
# ~/.docker/config.json are what the kpnpp-builder push relies on), so a
# workflow that calls this script needs no secret plumbing.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
REGISTRY="gitea.tourolle.paris"
OWNER="dtourolle"
IMAGE="sae-builder-cpu"
DOCKERFILE="Dockerfile.builder-cpu"
# The tag CI pins to today. Keep this in step with the `container.image` line in
# .gitea/workflows/unit-tests.yml; the workflow asserts at run time that the
# image it landed in reports this same version, so a drift shows up as a failed
# job rather than as a build against the wrong toolchain.
TAG="v1"
PUSH=0
EXTRA_ARGS=()
while [ $# -gt 0 ]; do
case "$1" in
--push) PUSH=1 ;;
--tag) TAG="${2:?--tag needs a value}"; shift ;;
--no-cache) EXTRA_ARGS+=(--no-cache) ;;
-h|--help) sed -n '2,30p' "${BASH_SOURCE[0]}"; exit 0 ;;
*) echo "error: unknown argument '$1'" >&2; exit 2 ;;
esac
shift
done
if [ "$TAG" = "latest" ]; then
echo "error: refusing to build the tag 'latest'." >&2
echo "DP-007 requires CI to pin an immutable tag. A moving 'latest' means a" >&2
echo "rebuild retroactively changes what every earlier green build proved." >&2
exit 2
fi
REF="${REGISTRY}/${OWNER}/${IMAGE}:${TAG}"
# A second tag carrying the commit that produced the image. The workflow pins
# the human-readable tag; this one is the audit trail — given any image you can
# recover the Dockerfile that built it.
SHA="$(git -C "$REPO_ROOT" rev-parse --short HEAD)"
REF_SHA="${REGISTRY}/${OWNER}/${IMAGE}:${TAG}-${SHA}"
# The Dockerfile COPYs nothing from the repository on purpose (see its closing
# comment), so the build context is an empty directory rather than the repo
# root. Sending ~1 GB of models, fixtures and experiment data to the daemon for
# a build that reads none of it is pure latency.
CONTEXT="$(mktemp -d)"
trap 'rm -rf "$CONTEXT"' EXIT
echo "=== building ${REF}"
echo " dockerfile: ${REPO_ROOT}/${DOCKERFILE}"
echo " context: (empty — the image embeds no repository content)"
echo
echo " Expect this to take a while: OpenCV 5 is compiled from source because"
echo " no Debian release ships it. That cost is paid once per image, which is"
echo " the entire reason DP-007 asks for a prebuilt image instead of"
echo " installing dependencies inside each CI run."
echo
docker build \
"${EXTRA_ARGS[@]}" \
--build-arg "IMAGE_TAG=${TAG}" \
-f "${REPO_ROOT}/${DOCKERFILE}" \
-t "${REF}" \
-t "${REF_SHA}" \
"${CONTEXT}"
echo
echo "=== built"
docker image inspect "${REF}" --format ' {{.RepoTags}} {{.Size}} bytes'
docker run --rm "${REF}" sh -c 'echo " SAE_BUILDER=$SAE_BUILDER version=$SAE_BUILDER_VERSION ort=$SAE_ORT_VERSION opencv=$SAE_OPENCV_VERSION"'
if [ "$PUSH" -eq 0 ]; then
echo
echo "Not pushed. Re-run with --push, or push by hand:"
echo " docker push ${REF}"
echo " docker push ${REF_SHA}"
exit 0
fi
echo
echo "=== pushing"
# No `latest` tag is pushed, by design. Publishing one invites a workflow to use
# it, and DP-007 exists to prevent exactly that.
docker push "${REF}"
docker push "${REF_SHA}"
echo
echo "=== published ${REF}"
echo "If this was a dependency-set change, bump the tag in"
echo " .gitea/workflows/unit-tests.yml (container.image)"
echo " scripts/ci/build_builder_image.sh (TAG, above)"
echo "in the same commit, so no run can build against an image the repository"
echo "does not describe."
scripts/fetch_dvu.sh
-98
View File
@@ -1,98 +0,0 @@
#!/bin/bash
# fetch_dvu.sh — pull one film's character mugshots and presence annotations from
# the NIST TRECVID Deep Video Understanding development set.
#
# The DVU dev set is the reason Road to Bali is our benchmark film: it ships
# 5-7 face crops per *character*, cut from the film itself, alongside
# scene-scoped presence annotations. That matches SR-002 directly — presence is
# per scene, not per frame — and it keeps ground truth in character space, so
# scoring needs no actor->character mapping.
#
# This exists as a script, rather than as ad hoc commands, because the first
# copy of this data lived in a temp directory and was lost to a /tmp wipe,
# taking the working gallery with it.
#
# 14 films are asserted Creative Commons and need no data agreement (only the
# 5 KinoLorber test films are gated).
#
# Usage:
# scripts/fetch_dvu.sh [film] [dest]
# film default Road_To_Bali
# dest default ./dvu
set -euo pipefail
BASE="https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset"
FILM="${1:-Road_To_Bali}"
DEST="${2:-dvu}"
mkdir -p "$DEST/images" "$DEST/scenes"
echo "[dvu] $FILM -> $DEST"
# Scene segmentation: start/end as HH:MM:SS. Note valkaama.csv line 38 carries a
# shift-key typo (01:!4:00) — parse defensively if you extend this to that film.
echo "[dvu] scene segmentation"
curl -fsSL "$BASE/scene.segmentation.reference/${FILM}.csv" \
-o "$DEST/${FILM}.csv" || echo " (missing: ${FILM}.csv)"
# Entity types: which entities are Person vs Location/Concept. Only Person rows
# become gallery identities — the images/ directory also holds Location and
# Concept crops (bedroom, boat, ...), which must not enter a face gallery.
#
# Directory and file naming are inconsistent with the film slug used elsewhere:
# the folder is Road_to_Bali (lowercase "to") while the entity file is
# RoadToBali.entity.types.txt. Both are derived here rather than assumed.
# NIST is inconsistent across all three axes, and not by a rule worth deriving:
# Road to Bali is Road_To_Bali.csv / Road_to_Bali/ / RoadToBali.entity.types.txt,
# while SuperHero is SuperHero.csv / superHero/ / superhero.entity.types.txt.
# Defaults cover the Bali shape; override per film rather than guessing.
# KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero dvu-hero
KG_DIR="${KG_DIR:-${FILM//_To_/_to_}}"
KG_FILE="${KG_FILE:-$(echo "$FILM" | sed -E 's/_([a-z])/\U\1/g; s/_//g')}"
echo "[dvu] entity types ($KG_DIR/$KG_FILE)"
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/${KG_FILE}.entity.types.txt" \
-o "$DEST/${FILM}.entity.types.txt" || echo " (missing: entity types)"
# Character face crops. Names are discovered from the directory listing rather
# than probed as <Character>_N, since the crop count varies per character and
# the listing is authoritative.
echo "[dvu] character mugshots"
PERSONS="$DEST/persons.txt"
if [ -f "$DEST/${FILM}.entity.types.txt" ]; then
grep -iE "person" "$DEST/${FILM}.entity.types.txt" \
| sed -E 's/[[:space:]]*[:,].*$//' | tr -d '\r' \
| awk '{print tolower($1)}' | sort -u > "$PERSONS"
fi
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/images/" 2>/dev/null \
| grep -oE 'href="[^"?/][^"]*\.png"' | sed -E 's/href="//; s/"//' | sort -u \
> "$DEST/all_images.txt"
while read -r img; do
[ -z "$img" ] && continue
# Strip the trailing _N to recover the entity name.
who="$(echo "$img" | sed -E 's/_[0-9]+\.png$//' | awk '{print tolower($0)}')"
if [ -s "$PERSONS" ] && ! grep -qx "$who" "$PERSONS"; then
continue # Location/Concept crop, not a face
fi
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/images/${img}" \
-o "$DEST/images/${img}" 2>/dev/null || rm -f "$DEST/images/${img}"
done < "$DEST/all_images.txt"
# Per-scene knowledge graphs. A Person->Location edge means that person was
# present for the whole scene. Some of these contain a stray ", ," that breaks
# strict JSON parsers.
echo "[dvu] scene graphs"
for n in $(seq 1 60); do
curl -fsSL "$BASE/scenes_knowledge_graphs/${FILM//_/ }-${n}.json" \
-o "$DEST/scenes/${FILM}-${n}.json" 2>/dev/null \
|| curl -fsSL "$BASE/scenes_knowledge_graphs/${FILM}-${n}.json" \
-o "$DEST/scenes/${FILM}-${n}.json" 2>/dev/null \
|| rm -f "$DEST/scenes/${FILM}-${n}.json"
done
echo "[dvu] done:"
echo " mugshots: $(ls "$DEST/images" 2>/dev/null | wc -l)"
echo " scenes: $(ls "$DEST/scenes" 2>/dev/null | wc -l)"
echo " csv: $([ -f "$DEST/${FILM}.csv" ] && echo yes || echo no)"
scripts/filter_gallery.py
+1 -2
View File
@@ -77,8 +77,7 @@ def main():
if missing > 0: if missing > 0:
print(f"[warn] {missing} cast member(s) not present in gallery (not yet embedded)", file=sys.stderr) print(f"[warn] {missing} cast member(s) not present in gallery (not yet embedded)", file=sys.stderr)
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — a filtered gallery holds the SAME vectors as its
# a filtered gallery holds the SAME vectors as its
# source, so it inherits the source's binding. Dropping the stamp here would # source, so it inherits the source's binding. Dropping the stamp here would
# silently launder a stamped gallery into an unstamped one. # silently launder a stamped gallery into an unstamped one.
save_gallery_hdf5({"actors": actors}, Path(args.output), save_gallery_hdf5({"actors": actors}, Path(args.output),
scripts/make_fixtures.sh
+7 -14
View File
@@ -19,16 +19,13 @@
# on a full channel (AR-004). Before that fix the same command produced # on a full channel (AR-004). Before that fix the same command produced
# different dumps run to run, since what got dropped depended on timing. # different dumps run to run, since what got dropped depended on timing.
# #
# Source: hero/ — SuperHero, from the TRECVID DVU development set. Chosen over # Source: bali/ — Road to Bali (1952), public domain. That matters: derived
# SuperHero on face scale: Bali reference crops had a median detected face of
# 27 px against a 69 px maximum, so every reference was upscaled far past what
# the embedder was trained for. SuperHero is 69 px median, 241 px max. That matters: derived
# fixtures can be committed, where anything cut from a copyrighted title could # fixtures can be committed, where anything cut from a copyrighted title could
# not live in the repository at all. # not live in the repository at all.
set -euo pipefail set -euo pipefail
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CLIPS="${CLIPS:-$REPO/../hero}" CLIPS="${CLIPS:-$REPO/../bali}"
GALLERY="${GALLERY:-$REPO/gallery_lvface.h5}" GALLERY="${GALLERY:-$REPO/gallery_lvface.h5}"
BIN="${BIN:-$REPO/build/scene_analyze}" BIN="${BIN:-$REPO/build/scene_analyze}"
OUT="$REPO/tests/fixtures/dumps" OUT="$REPO/tests/fixtures/dumps"
@@ -36,12 +33,8 @@ OUT="$REPO/tests/fixtures/dumps"
# Pinned. Changing either invalidates every committed fixture. # Pinned. Changing either invalidates every committed fixture.
# fps 5 — 1 fps over a 77 s clip is 77 frames, too thin to exercise an # fps 5 — 1 fps over a 77 s clip is 77 frames, too thin to exercise an
# extinction window measured in tens of seconds. # extinction window measured in tens of seconds.
# min-face — 32 px. This is a *fixture* setting, deliberately below AR-002's # min-face — 32 px, the VR-005 measured floor (98.1% TPI). The corpus is
# production floor of 40 px (VR-013, measured end to end): the # 480x360, so a stricter value would reject most faces present.
# corpus is 480x360, where faces run 40-80 px, so pinning at 40
# would thin the dumps for reasons unrelated to what they test.
# 32 px is where VR-005 still shows 98.1% TPI, so the faces kept
# are identifiable; it is not the threshold the pipeline ships.
FPS=5 FPS=5
MIN_FACE_PX=32 MIN_FACE_PX=32
@@ -51,12 +44,12 @@ MIN_FACE_PX=32
mkdir -p "$OUT" mkdir -p "$OUT"
for clip in "$CLIPS"/SuperHero-*.webm; do for clip in "$CLIPS"/Road_To_Bali-*.webm; do
n="$(basename "$clip" .webm)"; n="${n##*-}" n="$(basename "$clip" .webm)"; n="${n##*-}"
echo "── superhero_$n" echo "── bali_$n"
"$BIN" --movie "$clip" --gallery "$GALLERY" \ "$BIN" --movie "$clip" --gallery "$GALLERY" \
--fps "$FPS" --min-face-px "$MIN_FACE_PX" \ --fps "$FPS" --min-face-px "$MIN_FACE_PX" \
--dump-embeddings "$OUT/superhero_$n.h5" \ --dump-embeddings "$OUT/bali_$n.h5" \
--output /dev/null 2>&1 | grep -E "wrote|dropped" || true --output /dev/null 2>&1 | grep -E "wrote|dropped" || true
done done
scripts/make_gallery.py
+1 -2
View File
@@ -178,8 +178,7 @@ def main():
output = Path(args.output) output = Path(args.output)
image_root = Path(args.image_dir) if args.image_dir else output.parent / "images" image_root = Path(args.image_dir) if args.image_dir else output.parent / "images"
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — stamp with the model actually loaded, resolved
# stamp with the model actually loaded, resolved
# through the same helper load_embedder uses so the two cannot diverge. # through the same helper load_embedder uses so the two cannot diverge.
arcface_path = resolve_arcface(args.models_dir, args.arcface) arcface_path = resolve_arcface(args.models_dir, args.arcface)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface) embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
scripts/make_jellyfin_gallery.py
+1 -2
View File
@@ -453,8 +453,7 @@ def main():
existing_actors = {} existing_actors = {}
if args.merge and output.is_file(): if args.merge and output.is_file():
existing = load_gallery_hdf5(output) existing = load_gallery_hdf5(output)
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — --merge keeps the existing actors' vectors and
# --merge keeps the existing actors' vectors and
# embeds the new ones with THIS model. If they disagree, the result is one # embeds the new ones with THIS model. If they disagree, the result is one
# gallery holding two incompatible embedding spaces, which is worse than a # gallery holding two incompatible embedding spaces, which is worse than a
# mismatched gallery: no later check can separate them again. # mismatched gallery: no later check can separate them again.
scripts/movienet_eval.py
+1 -2
View File
@@ -62,8 +62,7 @@ def main():
args = p.parse_args() args = p.parse_args()
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface) embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — match() below is a bare dot product against the
# match() below is a bare dot product against the
# gallery's vectors; if the gallery came from another model those numbers are # gallery's vectors; if the gallery came from another model those numbers are
# noise wearing a similarity's clothes. # noise wearing a similarity's clothes.
verify_gallery_stamp(args.gallery, verify_gallery_stamp(args.gallery,
scripts/optimizer/fetch_missing_actors.py
+2 -4
View File
@@ -106,8 +106,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
f"(wiki={n_via_wikidata}) no_tmdb={n_no_tmdb} no_img={n_no_img} " f"(wiki={n_via_wikidata}) no_tmdb={n_no_tmdb} no_img={n_no_img} "
f"no_face={n_no_face}", file=sys.stderr) f"no_face={n_no_face}", file=sys.stderr)
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — the legacy JSON gallery carries the same stamp as
# the legacy JSON gallery carries the same stamp as
# the HDF5 one; src/gallery/gallery_store.cpp reads it from either. # the HDF5 one; src/gallery/gallery_store.cpp reads it from either.
Path(out_path).write_text(json.dumps({"embedder": stamp, "actors": actors}, indent=2)) Path(out_path).write_text(json.dumps({"embedder": stamp, "actors": actors}, indent=2))
n_emb = sum(len(a["embeddings"]) for a in actors) n_emb = sum(len(a["embeddings"]) for a in actors)
@@ -121,8 +120,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
def merge(base_path, add_path, out_path): def merge(base_path, add_path, out_path):
base = json.loads(Path(base_path).read_text()) base = json.loads(Path(base_path).read_text())
add = json.loads(Path(add_path).read_text()) add = json.loads(Path(add_path).read_text())
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — merging two galleries from different models makes
# merging two galleries from different models makes
# ONE file containing two incompatible embedding spaces. Nothing downstream can # ONE file containing two incompatible embedding spaces. Nothing downstream can
# ever untangle that, so this is the one place the check must run before, not # ever untangle that, so this is the one place the check must run before, not
# after, the write. # after, the write.
scripts/optimizer/optimize.py
+1 -2
View File
@@ -199,8 +199,7 @@ def main():
if not Path(f["dump"]).exists(): if not Path(f["dump"]).exists():
sys.exit(f"[opt] missing dump for {f['name']}: {f['dump']}") sys.exit(f"[opt] missing dump for {f['name']}: {f['dump']}")
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — every (dump, gallery) pair is checked ONCE here,
# every (dump, gallery) pair is checked ONCE here,
# before the first evaluation. A DE sweep is thousands of replays; discovering # before the first evaluation. A DE sweep is thousands of replays; discovering
# a cross-model pair at the end (or never) means every number it produced was # a cross-model pair at the end (or never) means every number it produced was
# noise. Each replay subprocess re-checks its own pair anyway. # noise. Each replay subprocess re-checks its own pair anyway.
scripts/optimizer/reembed_gallery.py
+1 -2
View File
@@ -59,8 +59,7 @@ def main():
ref = load_gallery_hdf5(Path(args.ref)) ref = load_gallery_hdf5(Path(args.ref))
images_root = Path(args.images) images_root = Path(args.images)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface) embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — this script exists to produce a gallery in a
# this script exists to produce a gallery in a
# DIFFERENT model's space from the reference. The output must therefore never # DIFFERENT model's space from the reference. The output must therefore never
# inherit the reference's stamp; it carries the stamp of --arcface, which is # inherit the reference's stamp; it carries the stamp of --arcface, which is
# the whole point of the bake-off being safe to run. # the whole point of the bake-off being safe to run.
scripts/optimizer/replay.py
+2 -4
View File
@@ -110,8 +110,7 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool =
sys.path.insert(0, build_dir) sys.path.insert(0, build_dir)
import sae_kpn import sae_kpn
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — checked here, before any network is built, so a
# checked here, before any network is built, so a
# cross-model replay dies with one readable error instead of producing a # cross-model replay dies with one readable error instead of producing a
# plausible-looking score. add_identity_matcher re-checks it C++-side below; # plausible-looking score. add_identity_matcher re-checks it C++-side below;
# that is the backstop for any other caller of the binding. # that is the backstop for any other caller of the binding.
@@ -250,8 +249,7 @@ def main():
# per-film gallery expansion: promotes pose-varied views of confidently-identified # per-film gallery expansion: promotes pose-varied views of confidently-identified
# actors into an in-memory annex, recovering ~+4 recall at no precision cost. # actors into an in-memory annex, recovering ~+4 recall at no precision cost.
p.add_argument("--expand-gallery", action="store_true") p.add_argument("--expand-gallery", action="store_true")
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — promote an unprovable gallery/dump binding from a
# promote an unprovable gallery/dump binding from a
# loud warning to a hard error. Measurement sweeps should set this (or # loud warning to a hard error. Measurement sweeps should set this (or
# SAE_REQUIRE_GALLERY_STAMP=1) so no number comes from an unbound pair. # SAE_REQUIRE_GALLERY_STAMP=1) so no number comes from an unbound pair.
p.add_argument("--require-gallery-stamp", action="store_true") p.add_argument("--require-gallery-stamp", action="store_true")
scripts/sae_embed_loader.py
+5 -16
View File
@@ -11,7 +11,6 @@ argument, which is often None.
""" """
import sys import sys
import os
from pathlib import Path from pathlib import Path
DEFAULT_ARCFACE = "arcface_w600k_r50.onnx" DEFAULT_ARCFACE = "arcface_w600k_r50.onnx"
@@ -20,10 +19,8 @@ DEFAULT_ARCFACE = "arcface_w600k_r50.onnx"
def resolve_arcface(models_dir: str, arcface: str | None = None) -> str: def resolve_arcface(models_dir: str, arcface: str | None = None) -> str:
"""The ArcFace/LVFace ONNX path load_embedder would use for these arguments. """The ArcFace/LVFace ONNX path load_embedder would use for these arguments.
TRACES: GR-004 | SR-001 TRACES: GR-004 | SR-001 — single source of truth for "which model is this",
so the stamp written into a gallery can never drift from the model loaded."""
Single source of truth for "which model is this", so the stamp written into
a gallery can never drift from the model loaded."""
return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE) return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)
@@ -52,21 +49,13 @@ def load_embedder(build_dir: str, models_dir: str, arcface: str | None = None,
sys.exit(f"{name} model not found: {model}\nRun: bash scripts/download_models.sh") sys.exit(f"{name} model not found: {model}\nRun: bash scripts/download_models.sh")
# A TRT-backend build cannot load .onnx; it needs pre-built engines from # A TRT-backend build cannot load .onnx; it needs pre-built engines from
# scripts/build_trt_engines.sh. # scripts/build_trt_engines.sh. Pass them when present (ignored by ORT).
#
# These are passed only on request. The old comment here claimed they were
# "ignored by ORT" — they are not. The ORT backend treats an engine path as
# an instruction and raises, which is the right behaviour (silently ignoring
# a requested engine would be worse), but it meant that merely HAVING a
# populated trt_cache/ broke every ORT gallery build in the repo, with an
# error naming a flag the caller never set.
use_engines = os.environ.get("SAE_USE_TRT_ENGINES", "") not in ("", "0", "false")
trt = Path(models_path).parent / "trt_cache" trt = Path(models_path).parent / "trt_cache"
det_engine = trt / "scrfd.scrfd_500m_bnkps.640.fp16.engine" det_engine = trt / "scrfd.scrfd_500m_bnkps.640.fp16.engine"
arc_engine = trt / f"arcface.{Path(arcface_path).stem}.b4.fp16.engine" arc_engine = trt / f"arcface.{Path(arcface_path).stem}.b4.fp16.engine"
return sae_embed.FaceEmbedder( return sae_embed.FaceEmbedder(
detector_path, arcface_path, conf, nms, max_side, detector_path, arcface_path, conf, nms, max_side,
str(det_engine) if (use_engines and det_engine.is_file()) else "", str(det_engine) if det_engine.is_file() else "",
str(arc_engine) if (use_engines and arc_engine.is_file()) else "", str(arc_engine) if arc_engine.is_file() else "",
) )
scripts/sae_gallery.py
+2 -4
View File
@@ -168,8 +168,7 @@ def save_gallery_hdf5(gallery: dict, output: Path, embedder: dict | None = None)
f.create_dataset("jellyfin_id", data=np.asarray(jf, dtype=object), dtype=str_t) f.create_dataset("jellyfin_id", data=np.asarray(jf, dtype=object), dtype=str_t)
f.create_dataset("name", data=np.asarray(name, dtype=object), dtype=str_t) f.create_dataset("name", data=np.asarray(name, dtype=object), dtype=str_t)
f.create_dataset("source_images", data=np.asarray(src_images, dtype=object), dtype=str_t) f.create_dataset("source_images", data=np.asarray(src_images, dtype=object), dtype=str_t)
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — omitted entirely when unknown, so "unstamped"
# omitted entirely when unknown, so "unstamped"
# round-trips as unstamped rather than as a stamp naming no model. # round-trips as unstamped rather than as a stamp naming no model.
if not _stamp_empty(embedder): if not _stamp_empty(embedder):
g = f.create_group("embedder") g = f.create_group("embedder")
@@ -197,8 +196,7 @@ def load_gallery_hdf5(path: Path) -> dict:
if "source_images" in f: if "source_images" in f:
src_images = [s.decode() if isinstance(s, bytes) else s src_images = [s.decode() if isinstance(s, bytes) else s
for s in f["source_images"][:]] for s in f["source_images"][:]]
# TRACES: GR-004 | SR-001 # TRACES: GR-004 | SR-001 — carried through so a derived gallery (filter,
# carried through so a derived gallery (filter,
# merge, cast-restrict) keeps the binding of the gallery it came from. # merge, cast-restrict) keeps the binding of the gallery it came from.
stamp = None stamp = None
if "embedder" in f: if "embedder" in f:
scripts/validation/ground_truth.py
-5
View File
@@ -17,11 +17,6 @@ X-Ray/MovieNet key on IMDb nm-ids — see [[per-scene-presence-eval-design]].
Two sources implemented: Two sources implemented:
* XRayGroundTruth — Zenodo scene-level Amazon X-Ray CSVs (cast-in-scene). * XRayGroundTruth — Zenodo scene-level Amazon X-Ray CSVs (cast-in-scene).
* MovieNetGroundTruth — MovieNet-PS per-shot face annotations (on-screen faces). * MovieNetGroundTruth — MovieNet-PS per-shot face annotations (on-screen faces).
Both are published corpora addressed by title, so a scoring run is reproducible
from the identifiers alone — no annotation of ours travels with the code.
TRACES: VR-004 | PR-002
""" """
from __future__ import annotations from __future__ import annotations
scripts/validation/min_face_size.py
+8 -2
View File
@@ -183,10 +183,16 @@ DEDUP_SIM = 1.0 - 1e-7
class Stages: class Stages:
"""Thin holder so the rest of the script has one object to call.""" """Thin holder so the rest of the script has one object to call."""
def __init__(self, detector: str, arcface: str, conf: float, nms: float): def __init__(self, detector: str, arcface: str, conf: float, nms: float,
detector_engine: str = "", arcface_engine: str = ""):
# The engine paths are only consulted by a TRT-backend build, where they
# are mandatory — that backend loads a pre-built .engine and will not
# fall back to reading the .onnx. An ORT build ignores them, so passing
# them unconditionally is safe and keeps one constructor for both.
self.engine = sae_embed.FaceEmbedder( self.engine = sae_embed.FaceEmbedder(
detector_model=detector, arcface_model=arcface, detector_model=detector, arcface_model=arcface,
conf=conf, nms=nms, max_side=0) conf=conf, nms=nms, max_side=0,
detector_engine=detector_engine, arcface_engine=arcface_engine)
def detect(self, img): def detect(self, img):
return self.engine.detect(img) return self.engine.detect(img)
scripts/validation/quality_knee.py
+667
View File
@@ -0,0 +1,667 @@
#!/usr/bin/env python3
"""
quality_knee.py — VR-012: what does a blurred or small face cost in identification,
and which sharpness measure predicts it?
TRACES: VR-012, AR-028, AR-029
VR-005 located the size floor by degrading held-out gallery mugshots and watching
TPI/FPI fall. This does the same over a **joint size x blur grid**, and adds the
part that makes the result usable at inference.
Why a joint grid and not two sweeps
-----------------------------------
A 16 px face upscaled to 112 has already lost its high frequencies, so additional
blur costs it far less than it costs a 112 px one. Sweeping the axes separately
measures each in the presence of an implicit "other axis at its best" and misses
that interaction entirely — and the interaction is the whole question, because
AR-002 already gates on size and AR-029 proposes to discount on sharpness. If
identity loss turns out to be a function of the sharpness measure alone, then one
axis carries the information and discounting on both double-counts. If a
small-but-sharp and a large-but-blurred probe at equal measure lose different
amounts, the axes are genuinely separate and both belong.
Why sigma is not the answer
---------------------------
Sigma is a lab variable. At inference nothing knows how blurred a face is, so a
knee expressed in sigma cannot be acted on. What AR-028/AR-030 can consume is
measure value -> expected identity reliability
so the controlled degradation exists to *select and calibrate the measure*, and
the measure is what ships. Every candidate is therefore scored on every degraded
crop, and the candidates are ranked by how well each predicts the identification
outcome (AUC over probe-cell records), not by how smooth its ladder looks.
Protocol (VR-005's, extended)
-----------------------------
1. Every gallery actor with at least `--min-images` mugshots. At the default 3,
holding one out still leaves two references per actor.
2. Hold out ONE image per actor as the probe; the rest stay in the gallery at
native resolution. Only the probe degrades — reference mugshots are clean and
the face coming out of the video is not, which is the production case.
3. For each (size, sigma) cell: downscale the probe crop to size x size and back
to 112 (the sampling loss), then Gaussian blur at sigma canonical px (the
optical/motion loss). Resolution first, then blur, so sigma always means the
same thing in the frame AR-029 measures in, whatever the cell's size.
4. Score all five AR-029 candidates on the degraded crop, through the C++
binding.
5. Embed, match against the whole gallery, record TPI/FPI/unidentified.
Decision rule is the pipeline's: per-actor best-of-N cosine -> Platt sigmoid ->
accept if P > prob_threshold. Never a raw cosine (CLAUDE.md invariant, AR-024).
Everything runs through `sae_embed` — detection, the ArcFace warp, the embedder,
the sharpness measures and the calibration are all the shipped C++. Nothing here
re-implements a pipeline stage in numpy; the analysis on top of the recorded
numbers (AUC, knee location) is analysis and is numpy's job.
CAVEAT — FPI IS RELATIVE, NOT ABSOLUTE
--------------------------------------
False positives grow with the number of actors competing. Read FPI as a curve
across cells, not as a production rate. This runs the whole eligible gallery
rather than VR-005's 100-actor sample, so the understatement is much smaller,
but a production library is larger still.
Usage
-----
python scripts/validation/quality_knee.py \
--images images --gallery gallery_lvface.h5 \
--arcface models/LVFace-B_Glint360K.onnx \
--min-images 3 --out experiments/results/vr012_quality_knee
"""
from __future__ import annotations
import argparse
import csv
import json
import random
import sys
import time
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
sys.path.insert(0, str(Path(__file__).resolve().parent))
# min_face_size owns the shared scaffolding — actor discovery, the sae_embed
# locator, the Stages wrapper, the calibration-through-the-binding and the house
# plot palette. Importing it keeps one copy of each; a second copy of the
# calibration path in particular is what AR-024 exists to prevent.
import min_face_size as vr005 # noqa: E402
from min_face_size import ( # noqa: E402
DEDUP_SIM, INTERP, Stages, calibrate_gallery, discover_actors,
gallery_keys, normalise_name, probability, err,
INK, MUTED, GRID, SURFACE, BLUE, GREEN, RED, AMBER,
)
import cv2 # noqa: E402
import numpy as np # noqa: E402
import sae_embed # noqa: E402
# The five AR-029 candidates, in quality.hpp's order. Names match the binding's
# attributes so the CSV columns and the C++ fields cannot drift apart.
MEASURES = ["var_laplacian", "norm_var_laplacian", "tenengrad",
"hf_energy_ratio", "dir_min_tenengrad"]
# ── Degradation ───────────────────────────────────────────────────────────────
def disc_kernel(radius: float) -> np.ndarray:
"""The circle-of-confusion PSF of a defocused lens.
Optical defocus is **not** Gaussian, and the difference is not cosmetic. A
lens out of focus spreads a point into a uniform disc, whose transfer
function is a jinc — `2·J1(x)/x` — which crosses zero and goes negative.
Defocus therefore reverses contrast at particular spatial frequencies and
can leave *more* energy in some high bands than a Gaussian of the same
nominal width. A Gaussian MTF is strictly positive and monotonically
decreasing and does neither.
That matters here beyond realism: defocus is how a face ends up **large and
useless**. A focus pull, a shallow depth of field, an actor stepping off the
focal plane — all leave a big, confidently-detected face carrying no usable
detail, and all sail straight through a size gate. Gaussian blur was the one
family that mostly co-occurs with small faces, which is precisely why
sharpness looked redundant against AR-002 on the first grid.
The disc is supersampled 8x before downsampling so its edge is
anti-aliased; a hard-edged binary disc at small radii is a poor circle and
its spectrum carries the staircase, not the optics.
"""
ss = 8
n = int(np.ceil(radius)) * 2 + 1
hi = np.zeros((n * ss, n * ss), np.float32)
c = (n * ss - 1) / 2.0
y, x = np.ogrid[:n * ss, :n * ss]
hi[((x - c) ** 2 + (y - c) ** 2) <= (radius * ss) ** 2] = 1.0
k = hi.reshape(n, ss, n, ss).mean(axis=(1, 3))
s = k.sum()
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
def motion_kernel(length: int, angle_deg: float) -> np.ndarray:
"""Linear motion blur — a camera pan or a moving subject.
Directional by construction: it destroys detail along one axis and leaves
the perpendicular axis untouched. That is the property that separates the
AR-029 candidates, since a measure normalising by total energy divides out
the loss and reads a heavy smear as mild (see tests/test_quality.cpp).
"""
k = np.zeros((length, length), np.float32)
k[length // 2, :] = 1.0
m = cv2.getRotationMatrix2D(((length - 1) / 2.0, (length - 1) / 2.0),
angle_deg, 1.0)
k = cv2.warpAffine(k, m, (length, length))
s = k.sum()
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
def degrade(crop: np.ndarray, size: int, level: float, kind: str,
down: int, up: int, angle: float = 0.0) -> np.ndarray:
"""Resolution loss, then blur of the requested family.
Order matters and this one is deliberate. Sampling happens in the source
frame, so the downscale/upscale pair models a face that was `size` px when
detected. The blur is then applied in the canonical frame, so `level` means
the same number of canonical pixels in every cell of the grid — which is what
lets the two axes be read independently. Blurring first would make the
effective width depend on the cell's size, and the grid would no longer be
factorial.
`level` is the family's natural parameter: Gaussian sigma, disc radius, or
motion length in canonical px. They are NOT equivalent at equal numbers —
matching families by parameter would compare different amounts of damage, so
the analysis matches them on measured effect instead.
"""
out = crop
if size != 112:
small = cv2.resize(out, (size, size), interpolation=down)
out = cv2.resize(small, (112, 112), interpolation=up)
if level > 0:
if kind == "gaussian":
out = cv2.GaussianBlur(out, (0, 0), level, level)
elif kind == "disc":
out = cv2.filter2D(out, -1, disc_kernel(level))
elif kind == "motion":
out = cv2.filter2D(out, -1, motion_kernel(int(round(level)), angle))
else:
raise ValueError(f"unknown blur kind: {kind}")
return out
def score_sharpness(crop: np.ndarray) -> dict:
"""All five candidates, from the shipped C++ (quality.hpp)."""
s = sae_embed.assess_sharpness(np.ascontiguousarray(crop))
d = {m: float(getattr(s, m)) for m in MEASURES}
d["ok"] = bool(s.ok)
return d
# ── Analysis ──────────────────────────────────────────────────────────────────
def auc(scores: np.ndarray, positive: np.ndarray) -> float:
"""Area under the ROC for `scores` predicting `positive`, by the rank
(Mann-Whitney U) identity. 0.5 is chance; 1.0 is a measure that orders every
correctly-identified probe above every failure.
This is the ranking criterion for AR-029. A measure earns the job by
predicting *the decision the pipeline makes*, not by having a tidy response
to synthetic blur — a candidate can be beautifully monotone in sigma and
still be a poor guide to whether this particular face will be recognised.
"""
pos = scores[positive]
neg = scores[~positive]
if pos.size == 0 or neg.size == 0:
return float("nan")
order = np.argsort(np.concatenate([pos, neg]), kind="mergesort")
ranks = np.empty(order.size, dtype=np.float64)
ranks[order] = np.arange(1, order.size + 1)
# Average ranks over ties, or a measure with many equal values is scored
# arbitrarily by input order.
vals = np.concatenate([pos, neg])
sv = vals[order]
i = 0
while i < sv.size:
j = i
while j + 1 < sv.size and sv[j + 1] == sv[i]:
j += 1
if j > i:
ranks[order[i:j + 1]] = ranks[order[i:j + 1]].mean()
i = j + 1
r_pos = ranks[:pos.size].sum()
return float((r_pos - pos.size * (pos.size + 1) / 2) / (pos.size * neg.size))
def knee_from_measure(records: list[dict], measure: str, retention: float,
n_bins: int = 20) -> dict:
"""Where on `measure`'s own scale does identification start to fall apart?
Bins the probe-cell records by measure value and reports the TPI rate in
each. The threshold is the lowest bin edge whose bin and every bin above it
retain `retention` of the undegraded control's TPI rate — a stated rule, so
changing the answer means changing the rule rather than picking a number.
"""
vals = np.array([r[measure] for r in records], dtype=np.float64)
tpi = np.array([r["outcome"] == "TPI" for r in records])
control = np.array([r["size_px"] == 112 and r["sigma"] == 0.0 for r in records])
if control.sum() == 0:
return {}
floor = retention * float(tpi[control].mean())
# Quantile edges: the measures have wildly different scales and heavy tails,
# so equal-width bins would put almost everything in one bucket.
edges = np.unique(np.quantile(vals, np.linspace(0, 1, n_bins + 1)))
if edges.size < 3:
return {}
idx = np.clip(np.digitize(vals, edges[1:-1]), 0, edges.size - 2)
bins = []
for b in range(edges.size - 1):
m = idx == b
if m.sum() == 0:
continue
bins.append({"lo": float(edges[b]), "hi": float(edges[b + 1]),
"n": int(m.sum()), "tpi_rate": float(tpi[m].mean()),
"fpi_rate": float(np.mean([r["outcome"] == "FPI"
for r, k in zip(records, m) if k]))})
# Walk down from the top; the threshold is where retention first breaks.
thr = None
for b in reversed(bins):
if b["tpi_rate"] < floor:
thr = b["hi"]
break
return {"measure": measure, "control_tpi": float(tpi[control].mean()),
"tpi_floor": floor, "threshold": thr, "bins": bins}
# ── Plot ──────────────────────────────────────────────────────────────────────
def write_plots(cells: list[dict], records: list[dict], ranking: list[dict],
out_png: Path, meta: dict) -> None:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
plt.rcParams.update({
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE,
"savefig.facecolor": SURFACE, "text.color": INK,
"axes.edgecolor": MUTED, "axes.labelcolor": INK,
"xtick.color": MUTED, "ytick.color": MUTED,
"axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.8,
"axes.spines.top": False, "axes.spines.right": False,
})
sizes = sorted({c["size_px"] for c in cells})
sigmas = sorted({c["sigma"] for c in cells})
fig, axes = plt.subplots(1, 3, figsize=(17, 5.4))
# (a) the joint grid as TPI heat map
grid = np.full((len(sigmas), len(sizes)), np.nan)
for c in cells:
grid[sigmas.index(c["sigma"]), sizes.index(c["size_px"])] = 100 * c["tpi_rate"]
im = axes[0].imshow(grid, origin="lower", aspect="auto", cmap="viridis",
vmin=0, vmax=100)
axes[0].set_xticks(range(len(sizes)), [str(s) for s in sizes])
axes[0].set_yticks(range(len(sigmas)), [f"{s:g}" for s in sigmas])
axes[0].set_xlabel("probe size before upscaling (px)")
axes[0].set_ylabel("Gaussian sigma (canonical px)")
axes[0].set_title("TPI % over the joint grid", fontsize=11, loc="left")
axes[0].grid(False)
fig.colorbar(im, ax=axes[0], fraction=0.046)
# (b) TPI against the winning measure — the curve a discount is built from
best = ranking[0]["measure"]
vals = np.array([r[best] for r in records])
tpi = np.array([r["outcome"] == "TPI" for r in records])
edges = np.unique(np.quantile(vals, np.linspace(0, 1, 21)))
centres, rates = [], []
for i in range(edges.size - 1):
m = (vals >= edges[i]) & (vals <= edges[i + 1])
if m.sum() > 20:
centres.append(0.5 * (edges[i] + edges[i + 1]))
rates.append(100 * tpi[m].mean())
axes[1].plot(centres, rates, "-o", color=GREEN, lw=2)
axes[1].set_xscale("log")
axes[1].set_xlabel(f"{best} (log scale)")
axes[1].set_ylabel("TPI %")
axes[1].set_title(f"identification vs the measure\nbest predictor: {best} "
f"(AUC {ranking[0]['auc']:.3f})", fontsize=11, loc="left")
# (c) how well each candidate predicts the decision
names = [r["measure"] for r in ranking]
aucs = [r["auc"] for r in ranking]
axes[2].barh(range(len(names)), aucs, color=BLUE)
axes[2].axvline(0.5, color=RED, lw=1.4, ls="--")
axes[2].set_yticks(range(len(names)), names, fontsize=9)
axes[2].set_xlim(0.4, 1.0)
axes[2].set_xlabel("AUC — predicts correct identification")
axes[2].set_title("AR-029 candidate ranking", fontsize=11, loc="left")
axes[2].invert_yaxis()
fig.suptitle(f"VR-012 — quality knee, {meta['model']}, {meta['n_actors']} actors, "
f"{meta['n_probes']} probes x {len(cells)} cells",
fontsize=12, x=0.01, ha="left")
fig.tight_layout(rect=(0, 0.02, 1, 0.97))
out_png.parent.mkdir(parents=True, exist_ok=True)
fig.savefig(out_png, dpi=150)
plt.close(fig)
# ── Main ──────────────────────────────────────────────────────────────────────
def main() -> int:
p = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--images", default=str(REPO / "images"))
p.add_argument("--gallery", default=str(REPO / "gallery_lvface.h5"))
p.add_argument("--out", default=str(REPO / "experiments/results/vr012_quality_knee"))
p.add_argument("--actors", type=int, default=0,
help="cap the actor pool (0 = every eligible actor, the default: "
"FPI is gallery-size dependent and the whole gallery is the "
"least understated estimate available)")
p.add_argument("--min-images", type=int, default=3,
help="minimum mugshots to be eligible (default 3, so holding one "
"out still leaves two references)")
p.add_argument("--seed", type=int, default=0)
p.add_argument("--sizes", default="16,24,32,48,64,112",
help="probe sizes before upscaling; 112 is undegraded")
p.add_argument("--sigmas", default="0,0.5,1,1.5,2,3",
help="blur level in canonical px; 0 is unblurred. Meaning "
"depends on --blur-kind: Gaussian sigma, disc radius, "
"or motion length")
p.add_argument("--blur-kind", default="gaussian",
choices=["gaussian", "disc", "motion"],
help="blur family. gaussian is a soft-focus stand-in; disc "
"is the circle-of-confusion PSF of real optical "
"defocus (non-Gaussian, jinc MTF with zero crossings); "
"motion is a linear smear. The last two are how a face "
"ends up large and useless, which a size gate cannot "
"catch")
p.add_argument("--motion-angle", type=float, default=0.0,
help="motion blur direction in degrees (--blur-kind motion)")
p.add_argument("--keep-duplicates", action="store_true")
p.add_argument("--models-dir", default=str(REPO / "models"))
p.add_argument("--arcface", default=None)
p.add_argument("--detector", default=None)
p.add_argument("--conf", type=float, default=0.5)
p.add_argument("--nms", type=float, default=0.4)
p.add_argument("--max-side", type=int, default=500)
# Required by a TRT-backend build, ignored by an ORT one. A TensorRT fp16
# run is a different realisation of the embedder — VR-005 measured ~0.85
# cosine agreement with the fp32 ONNX path on LVFace-B, with separation
# essentially intact — so a knee located here belongs to the fp16 space.
# The study stays internally consistent because gallery and probes are both
# embedded in this one session.
p.add_argument("--detector-engine", default="",
help="pre-built SCRFD .engine (TRT builds only)")
p.add_argument("--arcface-engine", default="",
help="pre-built ArcFace .engine (TRT builds only)")
p.add_argument("--prob-threshold", type=float, default=0.754)
p.add_argument("--match-prior", type=float, default=0.5)
p.add_argument("--tpi-retention", type=float, default=0.95)
p.add_argument("--down-interp", default="area", choices=sorted(INTERP))
p.add_argument("--up-interp", default="linear", choices=sorted(INTERP))
args = p.parse_args()
models_dir = Path(args.models_dir)
arcface = Path(args.arcface) if args.arcface else models_dir / "LVFace-B_Glint360K.onnx"
detector = Path(args.detector) if args.detector else models_dir / "scrfd_500m_bnkps.onnx"
for path, what in ((arcface, "embedder"), (detector, "detector")):
if not path.is_file():
return err(f"{what} model not found: {path}")
images_root = Path(args.images)
if not images_root.is_dir():
return err(f"image cache not found: {images_root}")
sizes = sorted({int(s) for s in args.sizes.split(",") if s.strip()})
sigmas = sorted({float(s) for s in args.sigmas.split(",") if s.strip()})
cv2.setRNGSeed(args.seed)
# ── actor pool ────────────────────────────────────────────────────────────
pool = discover_actors(images_root)
print(f"[select] {len(pool)} actor dirs under {images_root}", file=sys.stderr)
if args.gallery and Path(args.gallery).is_file():
ids, names = gallery_keys(Path(args.gallery))
pool = [a for a in pool
if (a["jellyfin_id"] and a["jellyfin_id"] in ids)
or normalise_name(a["name"]) in names]
print(f"[select] {len(pool)} are in {args.gallery}", file=sys.stderr)
eligible = [a for a in pool if len(a["images"]) >= args.min_images]
print(f"[select] {len(eligible)} have >= {args.min_images} mugshots",
file=sys.stderr)
if len(eligible) < 2:
return err(f"need at least 2 eligible actors; found {len(eligible)}")
rng = random.Random(args.seed)
selected = (sorted(rng.sample(eligible, min(args.actors, len(eligible))),
key=lambda a: a["dir"].name)
if args.actors else eligible)
# ── detect + align every mugshot once ─────────────────────────────────────
stages = Stages(str(detector), str(arcface), args.conf, args.nms,
args.detector_engine, args.arcface_engine)
print(f"[models] detector={detector.name} embedder={arcface.name} "
f"batch={stages.engine.max_batch}", file=sys.stderr)
t0 = time.time()
crops, rows, actors = [], [], []
n_nodetect = 0
for a in selected:
actor_crops, actor_paths = [], []
for img_path in a["images"]:
img = cv2.imread(str(img_path))
if img is None:
n_nodetect += 1
continue
if args.max_side > 0 and max(img.shape[:2]) > args.max_side:
s = args.max_side / max(img.shape[:2])
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
faces = stages.detect(img)
if not faces:
enhanced = stages.enhance(img)
faces = stages.detect(enhanced)
if faces:
img = enhanced
if not faces:
n_nodetect += 1
continue
best = max(faces, key=lambda f: f.confidence)
crop = stages.align(img, best.landmarks)
if crop is None:
n_nodetect += 1
continue
actor_crops.append(crop)
actor_paths.append(img_path)
if len(actor_crops) < 2:
continue
ai = len(actors)
actors.append({"name": a["name"], "jellyfin_id": a["jellyfin_id"],
"dir": a["dir"].name, "n_images": len(actor_crops)})
for crop, img_path in zip(actor_crops, actor_paths):
rows.append({"actor_idx": ai, "image": str(img_path)})
crops.append(crop)
if len(actors) % 200 == 0:
print(f" [align] {len(actors)}/{len(selected)} actors, "
f"{len(crops)} crops", file=sys.stderr)
if len(actors) < 2:
return err(f"only {len(actors)} actors survived detection/alignment")
print(f"[align] {len(actors)} actors, {len(crops)} crops, {n_nodetect} skipped "
f"in {time.time() - t0:.1f}s", file=sys.stderr)
actor_of = np.array([r["actor_idx"] for r in rows], dtype=int)
t0 = time.time()
native = stages.embed(crops)
print(f"[embed] {len(crops)} native crops in {time.time() - t0:.1f}s",
file=sys.stderr)
# ── drop duplicate mugshots ───────────────────────────────────────────────
if not args.keep_duplicates:
keep = np.ones(len(rows), bool)
for ai in range(len(actors)):
kept: list[int] = []
for i in np.nonzero(actor_of == ai)[0]:
if any(float(native[i] @ native[k]) > DEDUP_SIM for k in kept):
keep[i] = False
else:
kept.append(int(i))
n_dup = int((~keep).sum())
counts = np.bincount(actor_of[keep], minlength=len(actors))
drop_actor = counts < 2
keep &= ~drop_actor[actor_of]
remap = np.full(len(actors), -1, dtype=int)
remap[~drop_actor] = np.arange(int((~drop_actor).sum()))
actors = [a for a, d in zip(actors, drop_actor) if not d]
rows = [r for r, k in zip(rows, keep) if k]
crops = [c for c, k in zip(crops, keep) if k]
native = native[keep]
actor_of = remap[actor_of[keep]]
print(f"[dedup] dropped {n_dup} duplicates and {int(drop_actor.sum())} "
f"actors; {len(actors)} actors, {len(rows)} images remain",
file=sys.stderr)
# ── hold out one probe per actor ──────────────────────────────────────────
is_probe = np.zeros(len(rows), bool)
for ai in range(len(actors)):
idx = np.nonzero(actor_of == ai)[0]
r = random.Random(f"{args.seed}:{actors[ai]['dir']}")
is_probe[r.choice(list(idx))] = True
probe_rows = np.nonzero(is_probe)[0]
gal_rows = np.nonzero(~is_probe)[0]
print(f"[holdout] {len(probe_rows)} probes, {len(gal_rows)} gallery embeddings",
file=sys.stderr)
gal_emb = native[gal_rows]
gal_actor = actor_of[gal_rows]
probe_actor = actor_of[probe_rows]
actor_cols = [np.nonzero(gal_actor == ai)[0] for ai in range(len(actors))]
if not all(len(c) for c in actor_cols):
return err("an actor has no gallery references left; raise --min-images")
cal = calibrate_gallery(gal_emb, gal_actor)
if not cal["valid"]:
return err("calibration could not be fitted; this study will not fall back "
"to a raw cosine threshold (CLAUDE.md invariant)")
log_prior_odds = float(np.log(args.match_prior / (1.0 - args.match_prior)))
# ── the grid ──────────────────────────────────────────────────────────────
down, up = INTERP[args.down_interp], INTERP[args.up_interp]
probe_crops = [crops[i] for i in probe_rows]
cells, records = [], []
n = len(probe_rows)
for size in sizes:
for sigma in sigmas:
t0 = time.time()
degraded = [degrade(c, size, sigma, args.blur_kind, down, up,
args.motion_angle) for c in probe_crops]
sharp = [score_sharpness(d) for d in degraded]
q = stages.embed(degraded)
sims = q @ gal_emb.T
best_per_actor = np.stack([sims[:, c].max(axis=1) for c in actor_cols],
axis=1)
best_actor = best_per_actor.argmax(axis=1)
best_sim = best_per_actor.max(axis=1)
p_match = np.asarray(probability(best_sim, cal["a"], cal["b"],
log_prior_odds))
accept = p_match > args.prob_threshold
correct = best_actor == probe_actor
tpi = int(np.sum(accept & correct))
fpi = int(np.sum(accept & ~correct))
unid = int(np.sum(~accept))
cell = {"size_px": size, "sigma": sigma, "blur_kind": args.blur_kind,
"n_probes": n,
"tpi": tpi, "fpi": fpi, "unidentified": unid,
"tpi_rate": tpi / n, "fpi_rate": fpi / n,
"unidentified_rate": unid / n,
"rank1_rate": float(np.mean(correct)),
"mean_p_match": float(np.mean(p_match))}
for m in MEASURES:
cell[f"mean_{m}"] = float(np.mean([s[m] for s in sharp]))
cells.append(cell)
for j in range(n):
rec = {"size_px": size, "sigma": sigma,
"blur_kind": args.blur_kind,
"probe_image": rows[probe_rows[j]]["image"],
"p_match": float(p_match[j]),
"outcome": ("TPI" if accept[j] and correct[j]
else "FPI" if accept[j] else "unidentified")}
rec.update({m: sharp[j][m] for m in MEASURES})
records.append(rec)
print(f"[grid] {size:3d}px {args.blur_kind[:4]} {sigma:<4g} TPI {100*tpi/n:5.1f}% "
f"FPI {100*fpi/n:5.1f}% unid {100*unid/n:5.1f}% "
f"rank1 {100*np.mean(correct):5.1f}% [{time.time()-t0:.1f}s]",
file=sys.stderr)
# ── rank the candidates, then locate the knee on the winner ───────────────
is_tpi = np.array([r["outcome"] == "TPI" for r in records])
ranking = sorted(
({"measure": m,
"auc": auc(np.array([r[m] for r in records], dtype=np.float64), is_tpi)}
for m in MEASURES),
key=lambda d: -d["auc"])
knees = [knee_from_measure(records, r["measure"], args.tpi_retention)
for r in ranking]
# ── outputs ───────────────────────────────────────────────────────────────
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
csv_path = out.with_name(out.name + ".csv")
with open(csv_path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(cells[0].keys()))
w.writeheader()
w.writerows(cells)
rec_path = out.with_name(out.name + ".records.csv")
with open(rec_path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(records[0].keys()))
w.writeheader()
w.writerows(records)
meta = {
"requirement": "VR-012",
"model": arcface.stem, "detector": detector.stem,
"n_actors": len(actors), "n_probes": len(probe_rows),
"n_gallery_embeddings": len(gal_rows),
"min_images": args.min_images, "seed": args.seed,
"sizes": sizes, "sigmas": sigmas, "blur_kind": args.blur_kind,
"motion_angle": args.motion_angle,
"prob_threshold": args.prob_threshold, "match_prior": args.match_prior,
"calibration": cal,
"measure_ranking": ranking,
"knees": knees,
"sharpness_window": list(sae_embed.sharpness_window()),
"caveat": (f"FPI grows with gallery size; this ran against {len(actors)} "
f"actors and still understates a production library."),
"grid": cells,
}
json_path = out.with_name(out.name + ".json")
json_path.write_text(json.dumps(meta, indent=2) + "\n")
png_path = out.with_name(out.name + ".png")
write_plots(cells, records, ranking, png_path, meta)
# ── stdout report ─────────────────────────────────────────────────────────
print(f"\nVR-012 — quality knee, {arcface.stem}")
print(f"{len(actors)} actors, {len(probe_rows)} probes x {len(cells)} cells\n")
print(f"{'size':>5} {'sigma':>6} {'TPI':>8} {'FPI':>8} {'unid':>8} {'rank1':>8}")
for c in cells:
print(f"{c['size_px']:>5} {c['sigma']:>6g} {100*c['tpi_rate']:>7.1f}% "
f"{100*c['fpi_rate']:>7.1f}% {100*c['unidentified_rate']:>7.1f}% "
f"{100*c['rank1_rate']:>7.1f}%")
print("\nAR-029 candidate ranking — AUC for predicting correct identification:")
for r in ranking:
print(f" {r['measure']:>20} {r['auc']:.4f}")
print(f"\n[out] {csv_path}\n[out] {rec_path}\n[out] {json_path}\n[out] {png_path}")
print(f"\n{meta['caveat']}")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/validation/test_audio_offset.py
+1 -1
View File
@@ -51,7 +51,7 @@ sys.path.insert(0, str(BUILD))
import sae_audio # noqa: E402 import sae_audio # noqa: E402
FIXTURE = REPO / "tests" / "fixtures" / "audio" / "superhero_offset_200s.flac" FIXTURE = REPO / "tests" / "fixtures" / "audio" / "bali_offset_200s.flac"
TONE = REPO / "tests" / "fixtures" / "audio" / "jray_audio_v1_tone.flac" TONE = REPO / "tests" / "fixtures" / "audio" / "jray_audio_v1_tone.flac"
# Server SPEC.md section 3, "Matching and offset recovery". The cap is the # Server SPEC.md section 3, "Matching and offset recovery". The cap is the
src/backends/ort_provider.hpp
+12 -1
View File
@@ -23,12 +23,23 @@
enum class OrtProvider { CPU, CUDA, ROCm, TensorRT }; enum class OrtProvider { CPU, CUDA, ROCm, TensorRT };
inline OrtProvider detect_ort_provider() { inline OrtProvider detect_ort_provider() {
// ORT returns these in its own preference order (TensorRT, CUDA, ..., CPU
// last), so the first recognised entry is the best available and the loop
// returns on it.
auto available = Ort::GetAvailableProviders(); auto available = Ort::GetAvailableProviders();
for (const auto& p : available) { for (const auto& p : available) {
// Only the TensorRT *EP* is a build-time opt-in — it needs the headers
// and the profile plumbing below. CUDA is not: it is a plain ORT
// provider, and gating its detection on the TRT flag (as this did) made
// the CUDA branch unreachable in every build that did not also ask for
// TensorRT. The symptom is silent rather than loud — inference simply
// runs on the CPU and everything still returns correct answers — which
// is why it survived: a 300-actor VR-012 grid cell took 76 s on the CPU
// with the GPU idle at 212 MiB.
#ifdef SAE_ORT_WITH_TRT_EP #ifdef SAE_ORT_WITH_TRT_EP
if (p == "TensorrtExecutionProvider") return OrtProvider::TensorRT; if (p == "TensorrtExecutionProvider") return OrtProvider::TensorRT;
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
#endif #endif
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
if (p == "ROCMExecutionProvider") return OrtProvider::ROCm; if (p == "ROCMExecutionProvider") return OrtProvider::ROCm;
} }
return OrtProvider::CPU; return OrtProvider::CPU;
src/build_gallery.cpp
+1 -5
View File
@@ -34,7 +34,6 @@ int main(int argc, char** argv) {
std::string arcface_model = kDefaultArcfaceModel; std::string arcface_model = kDefaultArcfaceModel;
float conf = 0.5f, nms_thr = 0.4f; float conf = 0.5f, nms_thr = 0.4f;
int max_side = 500; int max_side = 500;
float min_face_px = 0.f;
for (int i = 1; i < argc; ++i) { for (int i = 1; i < argc; ++i) {
auto arg = [&](const char* f) { return std::strcmp(argv[i], f) == 0; }; auto arg = [&](const char* f) { return std::strcmp(argv[i], f) == 0; };
@@ -51,7 +50,6 @@ int main(int argc, char** argv) {
else if (arg("--conf")) conf = std::stof(next()); else if (arg("--conf")) conf = std::stof(next());
else if (arg("--nms")) nms_thr = std::stof(next()); else if (arg("--nms")) nms_thr = std::stof(next());
else if (arg("--max-side")) max_side = std::stoi(next()); else if (arg("--max-side")) max_side = std::stoi(next());
else if (arg("--min-face-px")) min_face_px = std::stof(next());
else { std::cerr << "[warn] unknown flag: " << argv[i] << "\n"; } else { std::cerr << "[warn] unknown flag: " << argv[i] << "\n"; }
} catch (const std::exception& e) { } catch (const std::exception& e) {
std::cerr << "Error: " << e.what() << "\n"; std::cerr << "Error: " << e.what() << "\n";
@@ -61,8 +59,7 @@ int main(int argc, char** argv) {
if (root_path.empty() || output_path.empty()) { if (root_path.empty() || output_path.empty()) {
std::cerr << "Usage: build_gallery --root <dir> --output <gallery.json> " std::cerr << "Usage: build_gallery --root <dir> --output <gallery.json> "
"[--detector <path>] [--arcface <path>] [--max-side <N>]\n" "[--detector <path>] [--arcface <path>] [--max-side <N>]\n";
" [--min-face-px <px>]\n";
return 1; return 1;
} }
@@ -73,7 +70,6 @@ int main(int argc, char** argv) {
cfg.detector_conf = conf; cfg.detector_conf = conf;
cfg.detector_nms = nms_thr; cfg.detector_nms = nms_thr;
cfg.max_side = max_side; cfg.max_side = max_side;
cfg.min_face_px = min_face_px;
try { try {
ActorGallery gallery = build_gallery(cfg); ActorGallery gallery = build_gallery(cfg);
src/config.hpp
+6 -4
View File
@@ -153,14 +153,16 @@ struct Config {
// Banded admission for the per-subject store, in PROBABILITY space. An // Banded admission for the per-subject store, in PROBABILITY space. An
// embedding joins only if P(same person) against something already stored // embedding joins only if P(same person) against something already stored
// lands inside [lo, hi]: above hi it is redundant, below lo it is evidence // lands inside [lo, hi]: above hi it is redundant, below lo it is evidence
// the track is not one person. The same lo is re-applied to the whole store // the track is not one person. Replaces expand_novelty_sim, a raw cosine.
// at promotion time — see track_gallery.hpp. This is the only threshold the
// expansion path has: it replaces the raw-cosine expand_novelty_sim (0.55)
// and expand_track_spread_max (0.60), which are retired (AR-024).
// Working values pending VR-007; sweep both bounds, they fail in opposite // Working values pending VR-007; sweep both bounds, they fail in opposite
// directions. // directions.
float expand_band_lo{0.90f}; float expand_band_lo{0.90f};
float expand_band_hi{0.95f}; float expand_band_hi{0.95f};
float expand_novelty_sim{0.55f}; // promote only embeddings whose best sim to the
// actor's refs is below this (gallery-far / novel)
float expand_track_spread_max{0.60f}; // reject promotion if the retained buffer's
// internal spread (1 - min pairwise sim) exceeds
// this — guards track-ID collisions / two people
int expand_min_anchor_frames{3}; // require ≥N accepted frames naming the actor before int expand_min_anchor_frames{3}; // require ≥N accepted frames naming the actor before
// the track is confirmed and its buffer promoted // the track is confirmed and its buffer promoted
std::string expand_debug_dir; // if set, dump promoted mugshots + embeddings here std::string expand_debug_dir; // if set, dump promoted mugshots + embeddings here
src/gallery/gallery_builder.cpp
-21
View File
@@ -96,27 +96,6 @@ ActorGallery build_gallery(const BuildConfig& cfg) {
return a.confidence < b.confidence; return a.confidence < b.confidence;
}); });
// Reject faces too small to embed honestly.
//
// The reference images are crops cut from the film, not mugshots, so
// the detected face can be a small fraction of the image. Upscaling a
// 30 px face to ArcFace's 112x112 feeds the model an input it was
// never trained for, and it answers with a confident, plausible,
// wrong embedding.
//
// At inference that costs one frame. Here it is permanent: a poisoned
// reference sits in the gallery and corrupts every future match
// against that character, which is exactly the kind of error that is
// invisible without a study that should not have been needed.
if (cfg.min_face_px > 0.f) {
const float side = std::min(best.bbox.width, best.bbox.height);
if (side < cfg.min_face_px) {
std::cerr << " [skip] face " << side << "px < " << cfg.min_face_px
<< "px: " << img_file.path().filename() << "\n";
continue;
}
}
cv::Mat crop = align_face(img, best.landmarks); cv::Mat crop = align_face(img, best.landmarks);
if (crop.empty()) { if (crop.empty()) {
std::cerr << " [skip] alignment failed: " << img_file.path().filename() << "\n"; std::cerr << " [skip] alignment failed: " << img_file.path().filename() << "\n";
src/gallery/gallery_builder.hpp
-10
View File
@@ -29,16 +29,6 @@ struct BuildConfig {
float detector_conf{0.5f}; float detector_conf{0.5f};
float detector_nms{0.4f}; float detector_nms{0.4f};
int max_side{500}; // downscale source images to this max dimension int max_side{500}; // downscale source images to this max dimension
/// Minimum detected-face side, in pixels of the (possibly downscaled)
/// source image. 0 disables the check.
///
/// References below this are dropped rather than upscaled: a face smaller
/// than the embedder's input is off-distribution, and a bad reference
/// poisons every match against that identity for the life of the gallery.
/// Mirrors the inference-side --min-face-px so the gallery is built from
/// the same face scales it will be matched against.
float min_face_px{0.f};
// before detection — TMDB portraits are ~2k px, // before detection — TMDB portraits are ~2k px,
// SCRFD trains on smaller faces and detection // SCRFD trains on smaller faces and detection
// confidence drops on huge inputs. 0 = disabled. // confidence drops on huge inputs. 0 = disabled.
src/gallery/track_gallery.hpp
+49 -61
View File
@@ -36,15 +36,13 @@
// 2. Promotion (on confirmation). A track is "owned" by actor A once ≥N frames // 2. Promotion (on confirmation). A track is "owned" by actor A once ≥N frames
// have been accepted (by the matcher's calibrated posterior) as A. On // have been accepted (by the matcher's calibrated posterior) as A. On
// confirmation the retained buffer — the hard, gallery-far poses — is // confirmation the retained buffer — the hard, gallery-far poses — is
// promoted into A's per-film annex, subject to one safety gate: the band's // promoted into A's per-film annex, after two safety gates:
// lower bound, re-applied across the whole store (see `store_coherence`). // • novelty: only embeddings whose best sim to A's refs is below
// // expand_novelty_sim are added (skip poses already covered);
// There is exactly one threshold here, the AR-018 band, and it is a calibrated // • spread: if the retained buffer's internal spread (1 − min pairwise
// probability. Novelty is no longer a threshold at all — the eviction policy // cosine sim) exceeds expand_track_spread_max the whole track is
// above *orders* by gallery similarity rather than cutting at a constant, and // rejected — such spread signals a track-ID collision merging two
// the band's upper bound refuses the redundant views at the door. The raw // people, whose embeddings must never enter A's annex.
// cosines this replaces, expand_novelty_sim and expand_track_spread_max, are
// retired under AR-024.
// //
// The annex is CPU-side and in-memory: it is small (tens of embeddings) so the // The annex is CPU-side and in-memory: it is small (tens of embeddings) so the
// matcher scans it with a scalar loop, and it is discarded when the process // matcher scans it with a scalar loop, and it is discarded when the process
@@ -61,15 +59,16 @@ struct TrackGallery {
explicit TrackGallery(const Config& cfg) explicit TrackGallery(const Config& cfg)
: enabled_(cfg.expand_gallery) : enabled_(cfg.expand_gallery)
, buffer_size_(std::max(1, cfg.expand_buffer_size)) , buffer_size_(std::max(1, cfg.expand_buffer_size))
, band_lo_(cfg.expand_band_lo) , novelty_sim_(cfg.expand_novelty_sim)
, band_hi_(cfg.expand_band_hi) , spread_max_(cfg.expand_track_spread_max)
, min_anchor_frames_(std::max(1, cfg.expand_min_anchor_frames)) , min_anchor_frames_(std::max(1, cfg.expand_min_anchor_frames))
, debug_dir_(cfg.expand_debug_dir) , debug_dir_(cfg.expand_debug_dir)
{ {
if (!enabled_) return; if (!enabled_) return;
std::cerr << "[track_gallery] per-film expansion ON" std::cerr << "[track_gallery] per-film expansion ON"
<< " buffer=" << buffer_size_ << " buffer=" << buffer_size_
<< " band=[" << band_lo_ << ", " << band_hi_ << "]" << " novelty_sim<" << novelty_sim_
<< " spread_max=" << spread_max_
<< " min_anchor_frames=" << min_anchor_frames_; << " min_anchor_frames=" << min_anchor_frames_;
if (!debug_dir_.empty()) { if (!debug_dir_.empty()) {
std::filesystem::create_directories(debug_dir_); std::filesystem::create_directories(debug_dir_);
@@ -89,9 +88,7 @@ struct TrackGallery {
// track_id : face_tracker track (−1 = untracked, ignored) // track_id : face_tracker track (−1 = untracked, ignored)
// emb : this frame's raw embedding // emb : this frame's raw embedding
// best_actor : actor with the highest gallery similarity for this face // best_actor : actor with the highest gallery similarity for this face
// best_gal_sim : that similarity (best sim to best_actor's baked+annex // best_gal_sim : that similarity (best sim to best_actor's baked+annex refs)
// refs) — a raw cosine, the last one in this class: it is
// calibrated on entry and only the probability is stored
// accepted : true if the matcher accepted this face as best_actor // accepted : true if the matcher accepted this face as best_actor
// crop : aligned crop, retained only when debug dumping is on // crop : aligned crop, retained only when debug dumping is on
void observe(int track_id, const Embedding& emb, void observe(int track_id, const Embedding& emb,
@@ -137,6 +134,7 @@ struct TrackGallery {
/// band falls back to treating cosine as probability, which is wrong but /// band falls back to treating cosine as probability, which is wrong but
/// bounded — and the default is loud in the header rather than silent. /// bounded — and the default is loud in the header rather than silent.
void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); } void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }
void set_band(float lo, float hi) { band_lo_ = lo; band_hi_ = hi; }
/// Embeddings the band refused. A store that admits nothing is as wrong as /// Embeddings the band refused. A store that admits nothing is as wrong as
/// one that admits everything, and neither is visible without this. /// one that admits everything, and neither is visible without this.
@@ -148,10 +146,7 @@ struct TrackGallery {
private: private:
struct BufEntry { struct BufEntry {
Embedding emb; Embedding emb;
/// P(same person) against the owning actor's refs when observed — float gal_sim{0.f}; // best sim to owning actor's refs when observed
/// calibrated at the door (AR-024), so the eviction ordering below is a
/// comparison of probabilities and the struct holds no bare cosine.
float gal_p{0.f};
cv::Mat crop; // populated only when debug_dir_ set cv::Mat crop; // populated only when debug_dir_ set
}; };
@@ -198,8 +193,8 @@ private:
if (!admit(ts, emb)) { ++rejected_; return; } if (!admit(ts, emb)) { ++rejected_; return; }
BufEntry e; BufEntry e;
e.emb = emb; e.emb = emb;
e.gal_p = calibrate_(gal_sim); e.gal_sim = gal_sim;
if (!debug_dir_.empty() && !crop.empty()) e.crop = crop.clone(); if (!debug_dir_.empty() && !crop.empty()) e.crop = crop.clone();
if (static_cast<int>(ts.buf.size()) < buffer_size_) { if (static_cast<int>(ts.buf.size()) < buffer_size_) {
@@ -208,17 +203,13 @@ private:
} }
// Buffer full: evict the member the gallery recognises best (highest // Buffer full: evict the member the gallery recognises best (highest
// gal_p) — least informative — but only if the newcomer is at least as // gal_sim) — least informative — but only if the newcomer is at least as
// novel. Keeping the most gallery-far views is the whole point. // novel. Keeping the most gallery-far views is the whole point.
// int worst_i = -1;
// This is an *ordering*, not a threshold: there is no constant to tune, float worst_sim = e.gal_sim; // newcomer's sim is the bar to beat
// and novelty-seeking lives here rather than in a cutoff. It ranks
// probabilities, so it says the same thing across models (AR-024).
int worst_i = -1;
float worst_p = e.gal_p; // newcomer's probability is the bar to beat
for (int i = 0; i < static_cast<int>(ts.buf.size()); ++i) { for (int i = 0; i < static_cast<int>(ts.buf.size()); ++i) {
if (ts.buf[i].gal_p > worst_p) { if (ts.buf[i].gal_sim > worst_sim) {
worst_p = ts.buf[i].gal_p; worst_sim = ts.buf[i].gal_sim;
worst_i = i; worst_i = i;
} }
} }
@@ -233,18 +224,25 @@ private:
int actor = owning_actor(ts); int actor = owning_actor(ts);
if (actor < 0) return; if (actor < 0) return;
// ── Safety gate: the band's lower bound, across the whole store ────── // ── Safety gate: internal spread ─────────────────────────────────────
float worst = store_coherence(ts.buf); // A legitimate single-person track varies in pose but stays reasonably
if (worst < band_lo_) { // self-similar. Large spread signals two people merged under one track
// ID — reject the whole track rather than poison the actor's annex.
float spread = buffer_spread(ts.buf);
if (spread > spread_max_) {
std::cerr << "[track_gallery] track " << track_id std::cerr << "[track_gallery] track " << track_id
<< " → actor " << actor << " → actor " << actor
<< " REJECTED (worst pairwise P=" << worst << " REJECTED (spread " << spread
<< " < " << band_lo_ << ", likely ID collision)\n"; << " > " << spread_max_ << ", likely ID collision)\n";
return; return;
} }
int added = 0; int added = 0;
for (const auto& be : ts.buf) { for (const auto& be : ts.buf) {
// ── Safety gate: novelty ─────────────────────────────────────────
// Skip poses the gallery already covers; only gallery-far views are
// worth the annex slot (and the extra per-frame scan cost).
if (be.gal_sim >= novelty_sim_) continue;
annex_.push_back({be.emb, actor}); annex_.push_back({be.emb, actor});
if (!debug_dir_.empty() && !be.crop.empty()) if (!debug_dir_.empty() && !be.crop.empty())
dump_mugshot(track_id, actor, added, be); dump_mugshot(track_id, actor, added, be);
@@ -253,9 +251,10 @@ private:
std::cerr << "[track_gallery] track " << track_id std::cerr << "[track_gallery] track " << track_id
<< " confirmed actor " << actor << " confirmed actor " << actor
<< " (" << ts.accepted_frames << " accepted frames, worst " << " (" << ts.accepted_frames << " accepted frames, spread "
<< "pairwise P=" << worst << ") — promoted " << added << spread << ") — promoted " << added << "/"
<< " views; annex now " << annex_.size() << "\n"; << ts.buf.size() << " views; annex now "
<< annex_.size() << "\n";
} }
/// Prefer the registry's verdict; fall back to the local tally only when no /// Prefer the registry's verdict; fall back to the local tally only when no
@@ -273,34 +272,21 @@ private:
return best; return best;
} }
/// TRACES: AR-018, AR-024 | SR-005 // Spread = 1 − min pairwise cosine similarity over the buffer (0 when <2).
/// The store's weakest pairwise P(same person) — the band's lower bound static float buffer_spread(const std::vector<BufEntry>& buf) {
/// asked of every pair, not just of the best match at the door. float min_sim = std::numeric_limits<float>::max();
///
/// `admit` compares a newcomer against its *closest* existing member, so a
/// track that drifts gradually can chain A→B→C with every step inside the
/// band while A and C are strangers. That is precisely the shape a track-ID
/// collision takes when two people are merged over a slow pan, so the bound
/// is re-asked here across all pairs before anything reaches an actor's
/// annex. Same bound, same probability space — not a second constant.
///
/// A store of one has no pair to disagree; it is coherent by construction,
/// hence 1.
float store_coherence(const std::vector<BufEntry>& buf) const {
float worst = std::numeric_limits<float>::max();
for (size_t i = 0; i < buf.size(); ++i) for (size_t i = 0; i < buf.size(); ++i)
for (size_t j = i + 1; j < buf.size(); ++j) for (size_t j = i + 1; j < buf.size(); ++j)
worst = std::min(worst, min_sim = std::min(min_sim, cosine_similarity(buf[i].emb, buf[j].emb));
calibrate_(cosine_similarity(buf[i].emb, buf[j].emb))); if (min_sim == std::numeric_limits<float>::max()) return 0.f;
if (worst == std::numeric_limits<float>::max()) return 1.f; return 1.f - min_sim;
return worst;
} }
void dump_mugshot(int track_id, int actor, int idx, const BufEntry& be) { void dump_mugshot(int track_id, int actor, int idx, const BufEntry& be) {
#ifdef SAE_DEBUG #ifdef SAE_DEBUG
char name[64]; char name[64];
std::snprintf(name, sizeof(name), "trk%d_actor%d_%d_p%.3f.jpg", std::snprintf(name, sizeof(name), "trk%d_actor%d_%d_sim%.3f.jpg",
track_id, actor, idx, be.gal_p); track_id, actor, idx, be.gal_sim);
cv::imwrite((std::filesystem::path(debug_dir_) / name).string(), be.crop); cv::imwrite((std::filesystem::path(debug_dir_) / name).string(), be.crop);
#else #else
(void)track_id; (void)actor; (void)idx; (void)be; (void)track_id; (void)actor; (void)idx; (void)be;
@@ -310,12 +296,14 @@ private:
/// cosine → P(same person). The one probability space the pipeline reasons /// cosine → P(same person). The one probability space the pipeline reasons
/// in; see gallery_calibration.hpp's same_person_probability. /// in; see gallery_calibration.hpp's same_person_probability.
std::function<float(float)> calibrate_{[](float c) { return std::max(0.f, c); }}; std::function<float(float)> calibrate_{[](float c) { return std::max(0.f, c); }};
float band_lo_{0.90f};
float band_hi_{0.95f};
std::size_t rejected_{0}; ///< admissions refused by the band std::size_t rejected_{0}; ///< admissions refused by the band
bool enabled_; bool enabled_;
int buffer_size_; int buffer_size_;
float band_lo_; ///< AR-018, from cfg.expand_band_lo float novelty_sim_;
float band_hi_; ///< AR-018, from cfg.expand_band_hi float spread_max_;
int min_anchor_frames_; int min_anchor_frames_;
std::string debug_dir_; std::string debug_dir_;
src/main.cpp
+4 -24
View File
@@ -39,8 +39,8 @@
// --max-faces <N> max faces kept per frame (default: 10) // --max-faces <N> max faces kept per frame (default: 10)
// --expand-gallery enable per-film gallery expansion from track continuity // --expand-gallery enable per-film gallery expansion from track continuity
// --expand-buffer <N> per-track diversity buffer size (default: 20) // --expand-buffer <N> per-track diversity buffer size (default: 20)
// --expand-band-lo <p> store admission floor, P(same person) (default: 0.90) // --expand-novelty-sim <f> promote only views with best sim < f (default: 0.55)
// --expand-band-hi <p> store admission ceiling, P(same person) (default: 0.95) // --expand-spread-max <f> reject track if buffer spread > f (default: 0.60)
// --expand-min-anchor <N> accepted frames before a track confirms (default: 3) // --expand-min-anchor <N> accepted frames before a track confirms (default: 3)
// --expand-debug-dir <p> dump promoted mugshots + embeddings here (SAE_DEBUG) // --expand-debug-dir <p> dump promoted mugshots + embeddings here (SAE_DEBUG)
// (SAE_DEBUG only) // (SAE_DEBUG only)
@@ -147,8 +147,8 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--anneal")) cfg.anneal_sec = std::stod(next()); else if (arg("--anneal")) cfg.anneal_sec = std::stod(next());
else if (arg("--expand-gallery")) cfg.expand_gallery = true; else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next()); else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
else if (arg("--expand-band-lo")) cfg.expand_band_lo = std::stof(next()); else if (arg("--expand-novelty-sim")) cfg.expand_novelty_sim = std::stof(next());
else if (arg("--expand-band-hi")) cfg.expand_band_hi = std::stof(next()); else if (arg("--expand-spread-max")) cfg.expand_track_spread_max = std::stof(next());
else if (arg("--expand-min-anchor")) cfg.expand_min_anchor_frames = std::stoi(next()); else if (arg("--expand-min-anchor")) cfg.expand_min_anchor_frames = std::stoi(next());
else if (arg("--expand-debug-dir")) cfg.expand_debug_dir = next(); else if (arg("--expand-debug-dir")) cfg.expand_debug_dir = next();
else if (arg("--trt-cache")) cfg.trt.cache_dir = next(); else if (arg("--trt-cache")) cfg.trt.cache_dir = next();
@@ -281,26 +281,6 @@ int main(int argc, char** argv) {
} }
}); });
// Report *why* a node died. A Closed event alone says only that one
// stopped; the exception it carried is what identifies the fault, and
// without this listener it is discarded at the node boundary. Returning
// false keeps the existing semantics — the node still stops and the
// Closed handler above still aborts the run — but the run now names the
// cause instead of leaving it to be reconstructed from a debugger.
net.set_error_handler(
[&](std::string_view node_name, std::exception_ptr eptr) {
std::string what = "unknown exception";
try {
if (eptr) std::rethrow_exception(eptr);
} catch (const std::exception& e) {
what = e.what();
} catch (...) {
}
std::lock_guard<std::mutex> lk(event_mtx);
std::cerr << "[main] node '" << node_name << "' threw: " << what << "\n";
return false;
});
std::cerr << "[main] starting pipeline…\n"; std::cerr << "[main] starting pipeline…\n";
net.start(); net.start();
src/nodes/face_aligner_node.hpp
+3 -7
View File
@@ -4,16 +4,12 @@
#include <iostream> #include <iostream>
// ── FaceAlignerFunc ─────────────────────────────────────────────────────────── // ── FaceAlignerFunc ───────────────────────────────────────────────────────────
/// TRACES: AR-005, AR-030 | SR-002
///
// KPN node: applies a 5-point similarity transform to each detected face, // KPN node: applies a 5-point similarity transform to each detected face,
// producing a 112×112 BGR crop suitable for ArcFace inference. // producing a 112×112 BGR crop suitable for ArcFace inference.
// //
// Alignment is an Umeyama least-squares fit over all five landmarks (AR-005), // Alignment uses cv::estimateAffinePartial2D (RANSAC) to fit the detected
// not a robust one: a RANSAC fit discards the very landmarks AR-030 reads. // landmarks to ArcFace canonical positions. Degenerate detections (where the
// Degenerate detections (where the fit fails) are dropped from the output // affine fit fails) are silently dropped from the output vectors.
// vectors. The fit's residual is the AR-030 visibility measure and comes free,
// since the warp needs the transform anyway.
struct FaceAlignerFunc { struct FaceAlignerFunc {
static constexpr std::string_view label() { return "face_aligner"; } static constexpr std::string_view label() { return "face_aligner"; }
src/python_bindings.cpp
+50
View File
@@ -16,6 +16,7 @@
#include "face_embedder_engine.hpp" #include "face_embedder_engine.hpp"
#include "gallery/gallery_calibration.hpp" #include "gallery/gallery_calibration.hpp"
#include "gallery/gallery_store.hpp" #include "gallery/gallery_store.hpp"
#include "quality.hpp"
#include <nanobind/nanobind.h> #include <nanobind/nanobind.h>
#include <nanobind/ndarray.h> #include <nanobind/ndarray.h>
@@ -176,6 +177,55 @@ NB_MODULE(sae_embed, m) {
}, "image"_a, }, "image"_a,
"Border-replicate pad by 50% and CLAHE, for a detector second try."); "Border-replicate pad by 50% and CLAHE, for a detector second try.");
// ── Quality (AR-028 … AR-030) ────────────────────────────────────────────
// Exposed for the same reason the calibration is: VR-012 has to select
// among the AR-029 candidates, and the measure it selects must be the one
// that ships. A numpy copy scored during the study would leave the shipped
// measure unmeasured, which is precisely the failure the whole quality axis
// exists to prevent.
nb::class_<SharpnessScores>(m, "SharpnessScores")
.def_ro("var_laplacian", &SharpnessScores::var_laplacian)
.def_ro("norm_var_laplacian", &SharpnessScores::norm_var_laplacian)
.def_ro("tenengrad", &SharpnessScores::tenengrad)
.def_ro("hf_energy_ratio", &SharpnessScores::hf_energy_ratio)
.def_ro("dir_min_tenengrad", &SharpnessScores::dir_min_tenengrad)
.def_ro("ok", &SharpnessScores::ok)
.def("__repr__", [](const SharpnessScores& s) {
return "<SharpnessScores varlap=" + std::to_string(s.var_laplacian) +
" normvarlap=" + std::to_string(s.norm_var_laplacian) +
" tenengrad=" + std::to_string(s.tenengrad) +
" hf=" + std::to_string(s.hf_energy_ratio) +
(s.ok ? ">" : " NOT-OK>");
});
m.def("assess_sharpness", [](ImageArray crop) {
return ::assess_sharpness(as_mat(crop));
}, "crop"_a,
"All four AR-029 sharpness candidates for a 112x112 aligned crop "
"(quality.hpp). Higher is sharper for every measure; scales are not "
"comparable between measures. Scored over a fixed 64x64 window on the "
"face interior, so background bokeh and hairstyle do not enter.");
m.def("sharpness_window", [] {
const cv::Rect w = ::sharpness_window();
return std::vector<int>{w.x, w.y, w.width, w.height};
},
"The (x, y, w, h) canonical-pixel window every sharpness measure is "
"taken over, so a study can show the pixels a score came from.");
m.def("alignment_residual", [](nb::ndarray<const float, nb::shape<5, 2>,
nb::c_contig, nb::device::cpu> landmarks)
-> std::optional<float> {
const Alignment a = ::estimate_alignment(as_landmarks(landmarks));
if (!a.ok) return std::nullopt; // degenerate landmarks
return a.residual;
}, "landmarks"_a,
"The AR-030 visibility measure: RMS landmark error in canonical "
"112x112 px left over after the best similarity fit onto the ArcFace "
"template (face_utils.hpp). None when the landmarks are degenerate. "
"Exposed so VR-012 can check whether blur leaks into the pose axis — "
"if it does, discounting on both would double-count one cause.");
// ── Calibration ────────────────────────────────────────────────────────── // ── Calibration ──────────────────────────────────────────────────────────
// AR-024: the pipeline reasons in one probability space. Exposed so Python // AR-024: the pipeline reasons in one probability space. Exposed so Python
// scores through the same sigmoid the C++ matcher uses, rather than a numpy // scores through the same sigmoid the C++ matcher uses, rather than a numpy
src/quality.hpp
+256
View File
@@ -0,0 +1,256 @@
#pragma once
/// TRACES: AR-028, AR-029 | SR-002
///
/// Sharpness of the aligned crop — candidate measures for AR-029.
///
/// Motion blur and soft focus destroy the high-frequency detail the embedder
/// keys on, and unlike face size they leave the bounding box looking perfectly
/// healthy. An embedder handed such a face does not fail: it returns a
/// confident, plausible, wrong vector that then competes on equal terms with
/// every good one in the gallery.
///
/// **Why four measures and not one.** AR-029's threshold has to be *located*,
/// the way VR-005 located the size floor, not chosen. Locating it means letting
/// a study rank candidates by how well each predicts real identity loss, so all
/// four ship and VR-012 picks the winner. Until that study reports, none of
/// these is "the" sharpness measure.
///
/// **They are computed on the 112×112 aligned crop**, never the raw box. The
/// crop is geometrically scale-normalised, so a measure taken there cannot
/// re-express face size the way a raw-pixel one would.
///
/// That normalisation is geometric, not informational, and the distinction
/// matters: a 40 px face upscaled into the canonical frame genuinely carries
/// less high-frequency detail than a 400 px one downscaled into it, so every
/// measure here *does* respond to source face size. It reads **effective
/// resolution in canonical space**, which is the union of "was small" and "was
/// blurred", not blur alone. Whether that makes a sharpness discount a
/// double-count against AR-002's size gate is VR-012's joint size×sigma grid to
/// settle: if identity loss is a function of the measure alone, one axis
/// suffices; if a small-but-sharp and a large-but-blurred face at equal measure
/// lose different amounts, the axes are genuinely separate. The unit test
/// `sharpness falls under downscale-upscale as well as under blur` pins this as
/// known behaviour rather than leaving it to be discovered as a surprise.
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <cmath>
// ── The measurement window ────────────────────────────────────────────────────
// All four measures see the same pixels, so a comparison between them is about
// the operator and not about the window each happened to pick.
//
// A 64×64 region centred on the face interior, not the whole crop. Under the
// ArcFace template the landmarks span x ∈ [38.3, 73.5], y ∈ [51.5, 92.4]; this
// window covers that plus the surrounding cheeks, brow and chin while excluding
// the corners.
//
// The corners are excluded because they are where the background lives, and
// studio headshots — the gallery's entire population — are very often shot at a
// wide aperture with a deliberately blurred background. Measured over the full
// crop, that bokeh drags the score down on exactly the sharpest, most
// cooperative images in the set, which would put the measure's response
// backwards on the population used to calibrate it. Hair is excluded for the
// weaker version of the same reason: its high-frequency content varies with
// hairstyle rather than with capture quality.
//
// 64 is also a power of two, so the DFT below gets its natural size.
inline constexpr int kSharpWindow = 64;
inline constexpr int kSharpWindowX = 24; // (24,36) … (88,100) in canonical px
inline constexpr int kSharpWindowY = 36;
/// Every candidate, computed in one pass over the window.
///
/// Higher is sharper for all four, so a discount curve has the same orientation
/// whichever one VR-012 selects. Scales are *not* comparable between measures —
/// only within one.
struct SharpnessScores {
/// Variance of the Laplacian. The textbook measure, included as the
/// baseline every other candidate has to beat. Second derivatives amplify
/// sensor noise, and the value scales with image contrast, so a
/// low-contrast sharp face reads as blurred. Expected to lose; it should
/// lose on the record rather than by assertion.
float var_laplacian{0.f};
/// Variance of the Laplacian over the variance of the intensity. Divides
/// out the first-order contrast dependence that var_laplacian carries,
/// which is the single confound most likely to matter on a gallery drawn
/// from thousands of different cameras, lighting setups and JPEG pipelines.
float norm_var_laplacian{0.f};
/// Tenengrad: mean squared Sobel gradient magnitude. A first derivative, so
/// markedly less noise-amplifying than the Laplacian, at the cost of
/// responding to coarser structure. Still contrast-dependent.
float tenengrad{0.f};
/// Fraction of spectral energy above a quarter of Nyquist, DC excluded.
/// A ratio, so contrast divides out by construction rather than by an
/// explicit correction, and it is the most direct statement of "how much
/// fine detail is actually present". Bounded in [0,1], which makes it the
/// easiest of the four to turn into a discount.
float hf_energy_ratio{0.f};
/// The worse of the two Sobel axes, normalised by a low-frequency contrast
/// estimate. The only candidate here that satisfies both requirements at
/// once, and it exists because the other four do not.
///
/// Two independent fixes, each answering a measured failure of the four
/// above (numbers from the T1 ladders in tests/test_quality.cpp):
///
/// - **Normalise by low frequencies, not by total energy.** Dividing by
/// the whole intensity variance puts the detail being measured into the
/// denominator as well as the numerator, so a blur shrinks both and the
/// quotient barely moves. A Gaussian at sigma 4 canonical px keeps
/// illumination and coarse facial structure and discards detail, giving
/// a contrast estimate that blur leaves alone.
/// - **Take the minimum over direction, not the sum.** Motion blur is
/// directional: a horizontal smear destroys horizontal detail and
/// leaves vertical detail untouched. Summing the two axes (as
/// Tenengrad does) lets the surviving axis mask the destroyed one — the
/// reason both ratio measures are U-shaped in blur length, scoring a
/// 21 px smear about as sharp as a 3 px one. The minimum tracks the
/// axis that was ruined, which is the one the embedder suffers from.
///
/// Falls 510 → 12 monotonically across that same motion-blur ladder, stays
/// monotone under Gaussian blur and resampling, and moves 0.4% when
/// contrast is halved.
float dir_min_tenengrad{0.f};
/// False when the crop was the wrong size or degenerate (flat). Scored,
/// never silently dropped: a face whose sharpness cannot be computed is a
/// fact the dump should record, not an absence.
bool ok{false};
};
/// The window every measure is taken over. Exposed so a study can show the
/// pixels a score was computed from rather than trusting the constants.
inline cv::Rect sharpness_window() {
return {kSharpWindowX, kSharpWindowY, kSharpWindow, kSharpWindow};
}
namespace detail {
/// Fraction of spectral energy above `cutoff` × Nyquist, DC bin excluded.
///
/// A Hann window is applied first. Without it the DFT sees the region's edges
/// as a step discontinuity, and that step is broadband: it deposits energy at
/// every frequency including the high band being measured, so a uniformly
/// blurry crop still scores a substantial high-frequency fraction and the
/// measure's dynamic range collapses.
///
/// **The mean is removed before the window, not after.** Windowing a signal
/// that still carries its DC offset multiplies that constant by the Hann taper,
/// and the taper's own spectrum is not a single bin — the offset smears across
/// the low-frequency neighbourhood, where dropping bin (0,0) no longer removes
/// it. The leaked energy lands in the denominator without scaling with image
/// contrast, so the "ratio" silently becomes a function of absolute brightness:
/// on the synthetic crop, a 20/255 brightening moved it 23% and halving the
/// contrast moved it by a factor of 3.6. Subtracting the mean first restores
/// the invariance the ratio form is supposed to provide for free.
inline float hf_ratio(const cv::Mat& gray32, float cutoff = 0.25f) {
static const cv::Mat hann = [] {
cv::Mat w(kSharpWindow, kSharpWindow, CV_32F);
for (int y = 0; y < kSharpWindow; ++y) {
const float wy = 0.5f * (1.f - std::cos(2.f * float(CV_PI) * y / (kSharpWindow - 1)));
for (int x = 0; x < kSharpWindow; ++x) {
const float wx = 0.5f * (1.f - std::cos(2.f * float(CV_PI) * x / (kSharpWindow - 1)));
w.at<float>(y, x) = wx * wy;
}
}
return w;
}();
cv::Mat centred;
cv::subtract(gray32, cv::mean(gray32), centred);
cv::Mat windowed;
cv::multiply(centred, hann, windowed);
cv::Mat spectrum;
cv::dft(windowed, spectrum, cv::DFT_COMPLEX_OUTPUT);
// Quadrants are wrapped: frequency index n maps to the signed frequency
// n - N for n > N/2, so the radius has to be computed on the wrapped index.
const int N = kSharpWindow;
const float nyquist = N / 2.f;
const float r_cut = cutoff * nyquist;
double total = 0.0, high = 0.0;
for (int y = 0; y < N; ++y) {
const float fy = (y <= N / 2) ? float(y) : float(y - N);
for (int x = 0; x < N; ++x) {
if (x == 0 && y == 0) continue; // DC carries no detail
const float fx = (x <= N / 2) ? float(x) : float(x - N);
const auto& c = spectrum.at<cv::Vec2f>(y, x);
const double e = double(c[0]) * c[0] + double(c[1]) * c[1];
total += e;
if (std::sqrt(fx * fx + fy * fy) > r_cut) high += e;
}
}
if (total < 1e-12) return 0.f; // flat region
return static_cast<float>(high / total);
}
} // namespace detail
/// Score a 112×112 aligned BGR (or single-channel) crop on all four candidates.
///
/// Costs one colour conversion and three small convolutions over a 64×64 window
/// — negligible beside the embedder inference it guards.
inline SharpnessScores assess_sharpness(const cv::Mat& crop) {
SharpnessScores s;
const cv::Rect win = sharpness_window();
if (crop.empty() ||
win.x + win.width > crop.cols || win.y + win.height > crop.rows)
return s;
cv::Mat gray;
if (crop.channels() == 3) cv::cvtColor(crop(win), gray, cv::COLOR_BGR2GRAY);
else gray = crop(win).clone();
// Scale into [0,1] so a score does not depend on the 8-bit convention, and
// so the two contrast-normalised measures are comparable across builds.
cv::Mat g32;
gray.convertTo(g32, CV_32F, 1.0 / 255.0);
cv::Scalar mu, sigma;
cv::meanStdDev(g32, mu, sigma);
const double var_img = sigma[0] * sigma[0];
cv::Mat lap;
cv::Laplacian(g32, lap, CV_32F, 3);
cv::Scalar lmu, lsigma;
cv::meanStdDev(lap, lmu, lsigma);
const double var_lap = lsigma[0] * lsigma[0];
cv::Mat gx, gy;
cv::Sobel(g32, gx, CV_32F, 1, 0, 3);
cv::Sobel(g32, gy, CV_32F, 0, 1, 3);
cv::Mat gx2, gy2;
cv::multiply(gx, gx, gx2);
cv::multiply(gy, gy, gy2);
cv::Mat mag2 = gx2 + gy2;
// Contrast from low frequencies only — see dir_min_tenengrad. Blur leaves
// this denominator alone, which is exactly what the other two normalised
// measures lack.
cv::Mat lf;
cv::GaussianBlur(g32, lf, cv::Size(0, 0), 4.0);
cv::Scalar lfmu, lfsigma;
cv::meanStdDev(lf, lfmu, lfsigma);
const double var_lf = lfsigma[0] * lfsigma[0];
s.var_laplacian = static_cast<float>(var_lap);
// A flat window has no contrast to normalise by. Reporting 0 (rather than a
// huge quotient) keeps "less sharp" pointing the same way for a degenerate
// input as for a blurred one.
s.norm_var_laplacian = var_img > 1e-9 ? static_cast<float>(var_lap / var_img) : 0.f;
s.tenengrad = static_cast<float>(cv::mean(mag2)[0]);
s.hf_energy_ratio = detail::hf_ratio(g32);
s.dir_min_tenengrad = var_lf > 1e-9
? static_cast<float>(std::min(cv::mean(gx2)[0], cv::mean(gy2)[0]) / var_lf)
: 0.f;
s.ok = var_img > 1e-9;
return s;
}
src/scene_preview.cpp
+2 -2
View File
@@ -96,8 +96,8 @@ static Config parse_args(int argc, char** argv) {
// Per-film gallery expansion — preview supports it (same cfg fields). // Per-film gallery expansion — preview supports it (same cfg fields).
else if (arg("--expand-gallery")) cfg.expand_gallery = true; else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next()); else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
else if (arg("--expand-band-lo")) cfg.expand_band_lo = std::stof(next()); else if (arg("--expand-novelty-sim")) cfg.expand_novelty_sim = std::stof(next());
else if (arg("--expand-band-hi")) cfg.expand_band_hi = std::stof(next()); else if (arg("--expand-spread-max")) cfg.expand_track_spread_max = std::stof(next());
else if (arg("--expand-min-anchor")) cfg.expand_min_anchor_frames = std::stoi(next()); else if (arg("--expand-min-anchor")) cfg.expand_min_anchor_frames = std::stoi(next());
// Scene detection is scene_analyze-only (needs the dense TransNetV2 branch). // Scene detection is scene_analyze-only (needs the dense TransNetV2 branch).
// Accept the flags so a shared command line runs, but note they're inert // Accept the flags so a shared command line runs, but note they're inert
tests/CMakeLists.txt
+1
View File
@@ -19,6 +19,7 @@ add_executable(sae_tests
test_calibration.cpp test_calibration.cpp
test_gallery_store.cpp test_gallery_store.cpp
test_face_utils.cpp test_face_utils.cpp
test_quality.cpp
test_track_gallery.cpp test_track_gallery.cpp
test_face_tracker.cpp test_face_tracker.cpp
test_track_registry.cpp test_track_registry.cpp
tests/fixtures/audio/make_offset_fixture.sh
Vendored
+6 -6
View File
@@ -1,6 +1,6 @@
#!/bin/sh #!/bin/sh
# #
# Regenerate superhero_offset_200s.flac — the real-audio fixture behind VR-014, the # Regenerate bali_offset_200s.flac — the real-audio fixture behind VR-014, the
# audio-signature offset-recovery validation. # audio-signature offset-recovery validation.
# #
# sh make_offset_fixture.sh /path/to/clips # sh make_offset_fixture.sh /path/to/clips
@@ -12,8 +12,8 @@
# one signature against another finds the true alignment and only the true # one signature against another finds the true alignment and only the true
# alignment. Tones are pathologically easy for that; dialogue and score are not. # alignment. Tones are pathologically easy for that; dialogue and score are not.
# #
# Source: scene clips from SuperHero (TRECVID DVU development set), the corpus # Source: five scene clips from "Road to Bali" (1952), the public-domain corpus
# this repo already uses for the replay fixtures — tests/fixtures/dumps/superhero.h5 # this repo already uses for the replay fixtures — tests/fixtures/dumps/bali_*.h5
# are dumps of these same clips. Each is under the 120 s window on its own # are dumps of these same clips. Each is under the 120 s window on its own
# (29-77 s), so they are concatenated in scene order to make a source long # (29-77 s), so they are concatenated in scene order to make a source long
# enough that a 120 s window can slide inside it. # enough that a 120 s window can slide inside it.
@@ -41,13 +41,13 @@
set -eu set -eu
CLIPS="${1:-../../../../hero}" CLIPS="${1:-../../../../bali}"
OUT="$(dirname "$0")/superhero_offset_200s.flac" OUT="$(dirname "$0")/bali_offset_200s.flac"
LIST="$(mktemp)" LIST="$(mktemp)"
trap 'rm -f "$LIST"' EXIT trap 'rm -f "$LIST"' EXIT
for scene in 13 27 28 31 46; do for scene in 13 27 28 31 46; do
clip="$CLIPS/SuperHero-$scene.webm" clip="$CLIPS/Road_To_Bali-$scene.webm"
[ -f "$clip" ] || { echo "missing clip: $clip" >&2; exit 1; } [ -f "$clip" ] || { echo "missing clip: $clip" >&2; exit 1; }
echo "file '$(cd "$(dirname "$clip")" && pwd)/$(basename "$clip")'" >> "$LIST" echo "file '$(cd "$(dirname "$clip")" && pwd)/$(basename "$clip")'" >> "$LIST"
done done
tests/fixtures/dumps/.gitignore
-17
View File
@@ -1,17 +0,0 @@
# Replay fixtures are distributed as artifacts, not through git.
#
# They are large (superhero.h5 is ~9 MB) and regenerating one needs the film,
# the models and a GPU — none of which CI has. So they live in the Gitea
# generic package registry and are fetched on demand:
#
# scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
# scripts/artifacts/push_artifacts.sh replay-fixtures
#
# The gallery ships alongside the dumps deliberately: a dump only replays
# meaningfully against the gallery it was produced with.
#
# bali_*.h5 predate this and remain tracked; do not add more to git.
superhero.h5
hero66.h5
gt.json
scene_bounds.json
tests/test_calibration.cpp
-9
View File
@@ -1,12 +1,6 @@
// TRACES: AR-023 | SR-002
//
// Unit tests for gallery calibration: the sigmoid math, the pairwise fit on // Unit tests for gallery calibration: the sigmoid math, the pairwise fit on
// separable data, and the in-memory hash-keyed cache (hit / stale / cold). // separable data, and the in-memory hash-keyed cache (hit / stale / cold).
// All pure, GPU-free, model-free. // All pure, GPU-free, model-free.
//
// The three `[report]` cases at the bottom carry their own GR-003 tags: they
// verify the build report, which is fitted from the same distributions but is a
// separate requirement.
#include <catch2/catch_test_macros.hpp> #include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp> #include <catch2/matchers/catch_matchers_floating_point.hpp>
@@ -189,7 +183,6 @@ Embedding unit_axis(int slot) {
} }
} // namespace } // namespace
// TRACES: GR-003 | SR-001
TEST_CASE("report surfaces actors that can never be recognised", "[report][GR-003]") { TEST_CASE("report surfaces actors that can never be recognised", "[report][GR-003]") {
// An actor with no usable image is a silent recall ceiling: the pipeline // An actor with no usable image is a silent recall ceiling: the pipeline
// will never name them, and nothing in the gallery says why. This is the // will never name them, and nothing in the gallery says why. This is the
@@ -219,7 +212,6 @@ TEST_CASE("report surfaces actors that can never be recognised", "[report][GR-00
CHECK(r.actors[1].references == 0); CHECK(r.actors[1].references == 0);
} }
// TRACES: GR-003 | SR-001
TEST_CASE("report surfaces actors too thin to calibrate on", "[report][GR-003]") { TEST_CASE("report surfaces actors too thin to calibrate on", "[report][GR-003]") {
// Below the positive-pair threshold an actor contributes nothing to the // Below the positive-pair threshold an actor contributes nothing to the
// intra-class side of the fit. They are not broken, so nothing complains — // intra-class side of the fit. They are not broken, so nothing complains —
@@ -245,7 +237,6 @@ TEST_CASE("report surfaces actors too thin to calibrate on", "[report][GR-003]")
CHECK(r.actors_below_positive_threshold >= 1); CHECK(r.actors_below_positive_threshold >= 1);
} }
// TRACES: GR-003 | SR-001
TEST_CASE("report round-trips", "[report][GR-003]") { TEST_CASE("report round-trips", "[report][GR-003]") {
ActorGallery g; ActorGallery g;
ActorGallery::Actor act; ActorGallery::Actor act;
tests/test_face_tracker.cpp
+5 -9
View File
@@ -1,17 +1,13 @@
// TRACES: AR-007, AR-008 | SR-002
//
// Unit tests for FaceTrackerFunc (nodes/face_tracker_node.hpp): frame-to-frame // Unit tests for FaceTrackerFunc (nodes/face_tracker_node.hpp): frame-to-frame
// track linking and, crucially, cross-cut re-association. Pure, GPU-free, // track linking and, crucially, cross-cut re-association. Pure, GPU-free,
// model-free — drives the node's operator() with hand-built EmbeddedSceneFrames // model-free — drives the node's operator() with hand-built EmbeddedSceneFrames
// and inspects the emitted track_ids. // and inspects the emitted track_ids.
// //
// The behaviour under test: there is one track pool keyed on `last_seen` // The behaviour under test: on a camera-angle change (Frame::is_cut) the tracker
// (AR-008), so a face lost across a camera-angle change (Frame::is_cut) is an // parks its tracks instead of destroying them, and revives a parked track_id
// ordinary association candidate rather than a parked track needing a revival // when a post-cut detection's raw last-frame-embedding cosine similarity clears
// path — the raw-cosine `cut_revive_sim` that guarded that path is retired // cut_revive_sim. IoU is deliberately driven to 0 across the cut (boxes moved) so
// (AR-024). On a cut the association weight drops to embedding-only (AR-007), // only the embedding path can re-link — exactly the scenario a cut creates.
// and IoU is deliberately driven to 0 across the cut (boxes moved) so only the
// embedding path can re-link — exactly the scenario a cut creates.
#include <catch2/catch_test_macros.hpp> #include <catch2/catch_test_macros.hpp>
#include "config.hpp" #include "config.hpp"
tests/test_quality.cpp
+302
View File
@@ -0,0 +1,302 @@
// TRACES: AR-029 | SR-002
//
// T1 for the AR-029 sharpness candidates: the properties that have to hold
// before a study is allowed to pick between them. GPU-free, model-free.
//
// The register's acceptance criterion is "synthetic blur ladder ->
// monotonically falling sharpness; Gaussian vs motion blur; small sharp face vs
// large soft one — size must not leak into this axis". The last clause needs
// care, and the tests below split it in two:
//
// - What must NOT leak is *geometric* scale. The measure is taken in the
// canonical frame, so changing how big the face was in the source while
// preserving its detail must not move the score. That is structural: the
// window is fixed at 64x64 canonical px.
// - What DOES legitimately move the score is lost *detail*. A face that was
// 40 px before being warped up to 112 really does carry less
// high-frequency content than one that was 400 px, and a measure blind to
// that would be blind to the thing it exists to catch.
//
// So "size must not leak" cannot mean "invariant to the source face size", and
// the ladder test below asserts the opposite on purpose. What it buys is that
// the overlap with AR-002 is a recorded property with a test naming it, rather
// than a surprise VR-012 discovers when the two axes turn out to be correlated.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "quality.hpp"
#include "types.hpp" // kArcFaceRef, for the window-placement test
#include <algorithm>
#include <cmath>
#include <utility>
#include <vector>
using Catch::Matchers::WithinAbs;
using Catch::Matchers::WithinRel;
namespace {
// A deterministic 112x112 stand-in for a face crop.
//
// **Broadband, not a sum of a few sinusoids.** An earlier version of this
// fixture used three discrete spatial frequencies, and the resampling ladder
// below was non-monotone for hf_energy_ratio because of it: a period-7
// component downsampled to 32 px lands exactly at Nyquist and aliases, so the
// ratio rose at one rung instead of falling. That is a property of a
// three-tone test pattern meeting a resampler, not of the measure or of any
// face — a real crop has energy spread across the band, where such a
// resonance averages out. Deterministic value noise, smoothed to give the
// roughly 1/f falloff of a photograph, exercises the whole band at once.
//
// Mid-grey base with bounded amplitude, so scaling the contrast in the tests
// below does not clip.
cv::Mat synthetic_crop() {
// Fixed LCG rather than cv::randu: the suite must not depend on OpenCV's
// RNG state, which other tests share.
uint32_t seed = 0x5eed1234u;
auto next = [&seed] {
seed = seed * 1664525u + 1013904223u;
return (seed >> 16) & 0xffffu;
};
cv::Mat noise(112, 112, CV_32F);
for (int y = 0; y < 112; ++y)
for (int x = 0; x < 112; ++x)
noise.at<float>(y, x) = float(next()) / 65535.f - 0.5f;
// Mild smoothing: white noise is flat to Nyquist, which no lens produces
// and which would make the sharpest rung of every ladder unrealistic.
cv::Mat smooth;
cv::GaussianBlur(noise, smooth, cv::Size(0, 0), 0.8);
cv::normalize(smooth, smooth, -1.0, 1.0, cv::NORM_MINMAX);
cv::Mat img(112, 112, CV_8UC3);
for (int y = 0; y < 112; ++y) {
for (int x = 0; x < 112; ++x) {
double v = 128.0 + 70.0 * smooth.at<float>(y, x);
const auto b = static_cast<uchar>(std::clamp(v, 0.0, 255.0));
img.at<cv::Vec3b>(y, x) = {b, b, b};
}
}
return img;
}
cv::Mat gaussian(const cv::Mat& src, double sigma) {
cv::Mat out;
cv::GaussianBlur(src, out, cv::Size(0, 0), sigma, sigma);
return out;
}
// Horizontal box blur — the camera-pan case, and the one an isotropic measure
// could in principle miss.
cv::Mat motion(const cv::Mat& src, int len) {
cv::Mat kernel = cv::Mat::zeros(1, len, CV_32F);
kernel.setTo(1.0f / len);
cv::Mat out;
cv::filter2D(src, out, -1, kernel);
return out;
}
// Throw away detail a face detected at size x size never had, then warp back up
// to the 112x112 the embedder is fed — the VR-005 degradation.
cv::Mat rescale(const cv::Mat& src, int size) {
if (size == 112) return src.clone();
cv::Mat small, out;
cv::resize(src, small, {size, size}, 0, 0, cv::INTER_AREA);
cv::resize(small, out, {112, 112}, 0, 0, cv::INTER_LINEAR);
return out;
}
std::vector<float> field(const std::vector<SharpnessScores>& s,
float SharpnessScores::* m) {
std::vector<float> v;
v.reserve(s.size());
for (const auto& x : s) v.push_back(x.*m);
return v;
}
void check_strictly_falling(const std::vector<float>& v, const char* what) {
INFO(what);
for (size_t i = 1; i < v.size(); ++i) {
INFO("step " << i << ": " << v[i - 1] << " -> " << v[i]);
CHECK(v[i] < v[i - 1]);
}
}
const std::vector<std::pair<const char*, float SharpnessScores::*>> kMeasures{
{"var_laplacian", &SharpnessScores::var_laplacian},
{"norm_var_laplacian", &SharpnessScores::norm_var_laplacian},
{"tenengrad", &SharpnessScores::tenengrad},
{"hf_energy_ratio", &SharpnessScores::hf_energy_ratio},
{"dir_min_tenengrad", &SharpnessScores::dir_min_tenengrad},
};
} // namespace
TEST_CASE("every candidate falls monotonically along a Gaussian blur ladder",
"[quality][AR-029]") {
const cv::Mat base = synthetic_crop();
std::vector<SharpnessScores> ladder;
for (double sigma : {0.0, 0.5, 1.0, 1.5, 2.0, 3.0})
ladder.push_back(assess_sharpness(sigma == 0.0 ? base : gaussian(base, sigma)));
for (const auto& [name, m] : kMeasures) {
REQUIRE(ladder.front().ok);
check_strictly_falling(field(ladder, m), name);
}
}
TEST_CASE("only the absolute and directional measures survive motion blur",
"[quality][AR-029]") {
// Motion blur is the commonest way a film frame is unusable, and it is
// where the candidates separate. A horizontal smear destroys horizontal
// detail and leaves vertical detail untouched, so what a measure does here
// depends on whether it can be fooled by the surviving axis.
const cv::Mat base = synthetic_crop();
std::vector<SharpnessScores> ladder{assess_sharpness(base)};
for (int len : {3, 5, 9, 15, 21})
ladder.push_back(assess_sharpness(motion(base, len)));
// Total gradient/Laplacian energy keeps falling: nothing replaces what the
// smear removed.
check_strictly_falling(field(ladder, &SharpnessScores::var_laplacian),
"var_laplacian");
check_strictly_falling(field(ladder, &SharpnessScores::tenengrad),
"tenengrad");
// The fix for the two below: low-frequency denominator, and the worse of
// the two axes rather than their sum.
check_strictly_falling(field(ladder, &SharpnessScores::dir_min_tenengrad),
"dir_min_tenengrad");
// The disqualifying behaviour, pinned rather than hidden. Both measures
// normalise by a quantity that contains the detail they are measuring, so
// once the horizontal band is gone the quotient climbs back toward its
// unblurred value: each is U-shaped in blur length, and a single score
// maps to two very different amounts of blur. A 21 px smear scores about
// as sharp as a 3 px one.
for (const auto& [name, m] : {
std::pair{"norm_var_laplacian", &SharpnessScores::norm_var_laplacian},
std::pair{"hf_energy_ratio", &SharpnessScores::hf_energy_ratio}}) {
const std::vector<float> v = field(ladder, m);
INFO(name);
const auto trough = std::min_element(v.begin(), v.end());
CHECK(trough != v.begin()); // it does fall at first …
CHECK(trough != v.end() - 1); // … then turns back up
CHECK(v.back() > 0.8f * v[1]); // recovering most of one rung
}
}
TEST_CASE("sharpness falls under downscale-upscale as well as under blur",
"[quality][AR-029]") {
// The overlap with AR-002, asserted rather than assumed. Losing resolution
// and losing focus are the same loss of high-frequency content, so every
// candidate reads a small upscaled face as less sharp. VR-012's joint
// size x sigma grid decides whether that makes a sharpness discount a
// double-count against the size gate, or whether the two axes carry
// separable information.
const cv::Mat base = synthetic_crop();
std::vector<SharpnessScores> ladder;
for (int size : {112, 64, 48, 32, 24, 16})
ladder.push_back(assess_sharpness(rescale(base, size)));
for (const auto& [name, m] : kMeasures)
check_strictly_falling(field(ladder, m), name);
}
TEST_CASE("the ratio measures are contrast-free and the raw ones are not",
"[quality][AR-029]") {
// The confound that decides the bake-off. A gallery drawn from thousands of
// cameras, lighting setups and JPEG pipelines varies enormously in
// contrast, and a measure that reads a low-contrast sharp face as blurred
// would discount it for the photographer's choices rather than for anything
// the embedder cares about.
const cv::Mat base = synthetic_crop();
// Halve the contrast about mid-grey, leaving spatial structure untouched.
cv::Mat low;
base.convertTo(low, CV_8UC3, 0.5, 64.0);
const auto s_hi = assess_sharpness(base);
const auto s_lo = assess_sharpness(low);
REQUIRE(s_hi.ok);
REQUIRE(s_lo.ok);
// Invariant by construction: both are ratios in which the contrast factor
// cancels.
CHECK_THAT(s_lo.norm_var_laplacian,
WithinRel(s_hi.norm_var_laplacian, 0.02f));
CHECK_THAT(s_lo.hf_energy_ratio, WithinRel(s_hi.hf_energy_ratio, 0.02f));
// Not invariant: both scale with the square of the contrast factor, so
// halving the contrast quarters them. This is the disqualifying behaviour,
// pinned so that a change making them contrast-free is a deliberate one.
CHECK_THAT(s_lo.var_laplacian, WithinRel(0.25f * s_hi.var_laplacian, 0.05f));
CHECK_THAT(s_lo.tenengrad, WithinRel(0.25f * s_hi.tenengrad, 0.05f));
}
TEST_CASE("brightness alone moves nothing", "[quality][AR-029]") {
const cv::Mat base = synthetic_crop();
cv::Mat bright;
base.convertTo(bright, CV_8UC3, 1.0, 20.0);
const auto a = assess_sharpness(base);
const auto b = assess_sharpness(bright);
for (const auto& [name, m] : kMeasures) {
INFO(name);
CHECK_THAT(b.*m, WithinRel(a.*m, 0.02f));
}
}
TEST_CASE("a flat crop is scored not-ok rather than given a number",
"[quality][AR-029]") {
// A face whose sharpness cannot be computed is a fact to record, not an
// absence — the same rule AR-030 follows for degenerate landmarks.
const cv::Mat flat(112, 112, CV_8UC3, cv::Scalar(128, 128, 128));
const auto s = assess_sharpness(flat);
CHECK_FALSE(s.ok);
for (const auto& [name, m] : kMeasures) {
INFO(name);
CHECK_THAT(s.*m, WithinAbs(0.0f, 1e-6f));
CHECK_FALSE(std::isnan(s.*m));
}
}
TEST_CASE("a crop smaller than the measurement window is scored not-ok",
"[quality][AR-029]") {
const cv::Mat small(64, 64, CV_8UC3, cv::Scalar(40, 90, 160));
CHECK_FALSE(assess_sharpness(small).ok);
CHECK_FALSE(assess_sharpness(cv::Mat()).ok);
}
TEST_CASE("the measurement window covers the face interior of the crop",
"[quality][AR-029]") {
// The landmarks the ArcFace template pins must all fall inside the window,
// or the measure is scoring background and hair rather than the face.
const cv::Rect w = sharpness_window();
CHECK(w.x >= 0);
CHECK(w.y >= 0);
CHECK(w.x + w.width <= 112);
CHECK(w.y + w.height <= 112);
for (int i = 0; i < 5; ++i) {
INFO("landmark " << i);
CHECK(w.contains(cv::Point(static_cast<int>(kArcFaceRef[i][0]),
static_cast<int>(kArcFaceRef[i][1]))));
}
}
TEST_CASE("a single-channel crop scores the same as its BGR equivalent",
"[quality][AR-029]") {
// The dump replays crops; nothing should depend on whether they arrived as
// three identical channels or one.
const cv::Mat base = synthetic_crop();
cv::Mat gray;
cv::cvtColor(base, gray, cv::COLOR_BGR2GRAY);
const auto a = assess_sharpness(base);
const auto b = assess_sharpness(gray);
for (const auto& [name, m] : kMeasures) {
INFO(name);
CHECK_THAT(b.*m, WithinRel(a.*m, 1e-3f));
}
}
tests/test_replay_fixtures.cpp
+65 -17
View File
@@ -153,34 +153,60 @@ Replay run(const Dump& d, double extinction = 10.0) {
} // namespace } // namespace
// ── AR-004 / VR-001 — the fixtures are intact and self-describing ──────────── // ── AR-004 / VR-001 — the fixtures are intact and self-describing ────────────
TEST_CASE("superhero fixture is complete", "[replay][VR-001]") { TEST_CASE("fixtures are complete and carry their embedder identity",
Dump d = load(fixture("superhero.h5")); "[replay][AR-004][VR-001]") {
CHECK(d.frames() == 5128); // Frame counts are exact rather than approximate. Before node outputs
CHECK(d.faces() == 4307); // blocked on a full channel, generation lost most of a clip and what it
CHECK(d.embedder == "LVFace-B_Glint360K.onnx"); // lost depended on timing — these numbers could not have been asserted.
struct Expect { const char* file; std::size_t frames, faces; };
const Expect all[] = {
{"bali_13.h5", 385, 693},
{"bali_27.h5", 335, 335},
{"bali_28.h5", 345, 368},
{"bali_31.h5", 145, 203},
{"bali_46.h5", 385, 140},
};
int64_t running = 0; for (const auto& x : all) {
for (std::size_t i = 0; i < d.frames(); ++i) { INFO(x.file);
REQUIRE(d.face_offset[i] == running); Dump d = load(fixture(x.file));
running += d.face_count[i]; CHECK(d.frames() == x.frames);
CHECK(d.faces() == x.faces);
CHECK(d.embedder == "LVFace-B_Glint360K.onnx");
// face_offset must be contiguous: a gap means faces went missing
// between frames, which no consumer could detect.
int64_t running = 0;
for (std::size_t i = 0; i < d.frames(); ++i) {
REQUIRE(d.face_offset[i] == running);
running += d.face_count[i];
}
CHECK(static_cast<std::size_t>(running) == d.faces());
} }
CHECK(static_cast<std::size_t>(running) == d.faces());
} }
TEST_CASE("replaying the superhero fixture twice gives identical tracks", // ── VR-002 — replay is deterministic ─────────────────────────────────────────
"[replay][VR-002]") { TEST_CASE("replaying a fixture twice gives identical tracks", "[replay][VR-002]") {
Dump d = load(fixture("superhero.h5")); // The property the whole fixture strategy rests on. If this fails, every
// golden output derived from a fixture is unreliable and the CI replay
// tier is worthless.
Dump d = load(fixture("bali_28.h5"));
Replay a = run(d); Replay a = run(d);
Replay b = run(d); Replay b = run(d);
REQUIRE(a.track_ids.size() == b.track_ids.size()); REQUIRE(a.track_ids.size() == b.track_ids.size());
CHECK(a.track_ids == b.track_ids); CHECK(a.track_ids == b.track_ids);
REQUIRE(a.claims.size() == b.claims.size()); REQUIRE(a.claims.size() == b.claims.size());
for (std::size_t i = 0; i < a.claims.size(); ++i) {
CHECK(a.claims[i].first_seen == b.claims[i].first_seen);
CHECK(a.claims[i].last_seen == b.claims[i].last_seen);
}
} }
// ── AR-012 / AR-013 — window invariants on real footage ──────────────────────
TEST_CASE("every face is assigned a track and every track closes", TEST_CASE("every face is assigned a track and every track closes",
"[replay][AR-012]") { "[replay][AR-012]") {
Dump d = load(fixture("superhero.h5")); Dump d = load(fixture("bali_13.h5"));
Replay r = run(d); Replay r = run(d);
CHECK(r.track_ids.size() == r.faces_seen); CHECK(r.track_ids.size() == r.faces_seen);
@@ -191,9 +217,9 @@ TEST_CASE("every face is assigned a track and every track closes",
CHECK(r.claims.size() > 0); CHECK(r.claims.size() > 0);
} }
TEST_CASE("windows are well-formed and inside the film", "[replay][AR-013]") { TEST_CASE("windows are well-formed and inside the clip", "[replay][AR-013]") {
for (const char* f : {"superhero.h5", "superhero.h5", "superhero.h5", for (const char* f : {"bali_13.h5", "bali_27.h5", "bali_28.h5",
"superhero.h5", "superhero.h5"}) { "bali_31.h5", "bali_46.h5"}) {
INFO(f); INFO(f);
Dump d = load(fixture(f)); Dump d = load(fixture(f));
Replay r = run(d); Replay r = run(d);
@@ -208,3 +234,25 @@ TEST_CASE("windows are well-formed and inside the film", "[replay][AR-013]") {
} }
} }
} }
TEST_CASE("a longer extinction window yields fewer, longer tracks",
"[replay][AR-013]") {
// The timeout decides whether a gap is absorbed into one window or splits
// it in two, so lengthening it must merge tracks rather than multiply them.
// On sparse footage this is the difference the constant actually makes.
Dump d = load(fixture("bali_46.h5")); // 140 faces over 385 frames
Replay tight = run(d, /*extinction=*/1.0);
Replay loose = run(d, /*extinction=*/30.0);
CHECK(loose.claims.size() <= tight.claims.size());
}
// ── AR-007 — cuts are exercised by the corpus, not just by construction ──────
TEST_CASE("the cut-heavy fixture actually contains cuts", "[replay][AR-007]") {
// Guards the corpus rather than the code: if a regeneration produced a
// fixture with no cuts, the association tests above would still pass while
// silently testing nothing about viewpoint changes.
Dump d = load(fixture("bali_28.h5"));
const int cuts = std::count(d.is_cut.begin(), d.is_cut.end(), uint8_t{1});
CHECK(cuts >= 5);
}
tests/test_similarity.cpp
-6
View File
@@ -1,11 +1,5 @@
// TRACES: AR-026 | SR-001
//
// Unit tests for the CPU reference similarity engine (backends/gemm_backend.cpp, // Unit tests for the CPU reference similarity engine (backends/gemm_backend.cpp,
// SAE_GEMM_CPU) and the l2_normalise helper. All pure, GPU-free, model-free. // SAE_GEMM_CPU) and the l2_normalise helper. All pure, GPU-free, model-free.
//
// This is the CI half of AR-026: equivalence between the GEMM path and
// hand-computed dot products on small input. Throughput at scale (AR-027) is T4
// and cannot run here.
#include <catch2/catch_test_macros.hpp> #include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp> #include <catch2/matchers/catch_matchers_floating_point.hpp>
tests/test_track_gallery.cpp
+62 -237
View File
@@ -1,16 +1,7 @@
// TRACES: AR-018, AR-019, AR-024 | SR-005
//
// Unit tests for TrackGallery (gallery/track_gallery.hpp): per-film gallery // Unit tests for TrackGallery (gallery/track_gallery.hpp): per-film gallery
// expansion driven by track continuity. Pure, GPU-free, model-free — exercises // expansion driven by track continuity. Pure, GPU-free, model-free — exercises
// the AR-018 banded admission at both bounds, the promotion-time coherence // the diversity-buffer eviction policy, the novelty/spread safety gates,
// gate, the diversity-buffer eviction policy, plurality ownership, and // plurality ownership, and idempotent promotion via the public interface.
// idempotent promotion, all through the public interface.
//
// The band is defined in PROBABILITY space (AR-024), so every case below states
// its own cosine → probability map instead of inheriting the header's fallback.
// A test that never names the mapping is not testing the band, it is testing a
// coincidence: with the fallback the two spaces happen to coincide, and a gate
// that silently reverted to raw cosine would still pass.
#include <catch2/catch_test_macros.hpp> #include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp> #include <catch2/matchers/catch_matchers_floating_point.hpp>
@@ -23,60 +14,30 @@
namespace { namespace {
constexpr float kPi = 3.14159265358979323846f; // Unit-norm embedding pointing along one axis (cosine sim to another one-hot is
// 0, to itself 1) — lets tests dial gallery similarity precisely.
// The band as the tests drive it. Kept in one place so a change to the shipped
// defaults does not silently invalidate the arithmetic in each case.
constexpr float kBandLo = 0.90f;
constexpr float kBandHi = 0.95f;
// Identity map: probability == cosine, so a case can place an embedding at an
// exact probability. cosine_similarity is a bare dot product over unit vectors
// (types.hpp), so the placements below are bit-exact, not approximate.
float identity_cal(float c) { return c; }
// Unit-norm embedding pointing along one axis. Cosine sim to another one-hot is
// 0, to itself 1.
Embedding one_hot(int slot) { Embedding one_hot(int slot) {
Embedding e{}; Embedding e{};
e[slot] = 1.0f; e[slot] = 1.0f;
return e; return e;
} }
// Unit-norm embedding in the plane of axes i,j at cosine `cos_t` from axis i. // Unit-norm embedding in the plane of axes i,j at angle t from i. Cosine sim to
// Cosine sim to one_hot(i) is exactly cos_t. // one_hot(i) is cos(t) — used to place a view at a chosen gallery similarity.
Embedding at_sim(int i, int j, float cos_t) { Embedding at_sim(int i, int j, float cos_t) {
Embedding e{}; Embedding e{};
float s = std::sqrt(std::max(0.f, 1.f - cos_t * cos_t));
e[i] = cos_t; e[i] = cos_t;
e[j] = std::sqrt(std::max(0.f, 1.f - cos_t * cos_t)); e[j] = s;
return e;
}
// A spoke: shares axis 0 with every other spoke, and is otherwise unique. Any
// two DISTINCT spokes have cosine similarity exactly cos_t², so one constant
// places a whole mutually-in-band store. A spoke against itself is 1.0 — above
// the band's ceiling, i.e. redundant, which is the intended reading.
Embedding spoke(int k, float cos_t) { return at_sim(0, k, cos_t); }
// cos_t chosen so pairwise similarity between distinct spokes is 0.9197 —
// comfortably inside [0.90, 0.95], clear of both bounds.
constexpr float kSpokeCos = 0.959f;
// Two embeddings `deg` apart in the plane of axes 0,1. Cosine is cos(deg), so a
// chain of these can step through the band while its endpoints fall outside it.
Embedding on_circle(float deg) {
Embedding e{};
e[0] = std::cos(deg * kPi / 180.f);
e[1] = std::sin(deg * kPi / 180.f);
return e; return e;
} }
Config expand_cfg() { Config expand_cfg() {
Config cfg; Config cfg;
cfg.expand_gallery = true; cfg.expand_gallery = true;
cfg.expand_buffer_size = 3; cfg.expand_buffer_size = 3;
cfg.expand_band_lo = kBandLo; cfg.expand_novelty_sim = 0.55f;
cfg.expand_band_hi = kBandHi; cfg.expand_track_spread_max = 0.60f;
cfg.expand_min_anchor_frames = 3; cfg.expand_min_anchor_frames = 3;
return cfg; return cfg;
} }
@@ -98,228 +59,92 @@ TEST_CASE("disabled: no annex growth when expand_gallery is off", "[track_galler
CHECK(tg.annex().empty()); CHECK(tg.annex().empty());
} }
// ── AR-018: the band ───────────────────────────────────────────────────────── TEST_CASE("confirmed track promotes gallery-far views", "[track_gallery]") {
TEST_CASE("band bounds come from config, not a hardcoded default", "[track_gallery][AR-018]") {
// The bounds were declared in Config and read nowhere, so the gate ran at
// whatever the header happened to initialise. Drive them somewhere the
// defaults are not and require the gate to follow.
Config cfg = expand_cfg();
cfg.expand_band_lo = 0.40f;
cfg.expand_band_hi = 0.60f;
TrackGallery tg(cfg);
tg.set_calibration(identity_cal);
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
// P = 0.50: inside the configured band, far below the shipped default lo.
tg.observe(1, at_sim(0, 1, 0.50f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 0);
// P = 0.92: inside the shipped default band, above the configured ceiling.
tg.observe(1, at_sim(0, 2, 0.92f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 1);
}
TEST_CASE("band admits at each bound exactly", "[track_gallery][AR-018]") {
// The verification plan asks for the bounds themselves, not a point safely
// inside them: an off-by-one in the comparison is invisible anywhere else.
// Both bounds are inclusive.
SECTION("lower bound exactly") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
tg.observe(1, at_sim(0, 1, kBandLo), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 0);
}
SECTION("upper bound exactly") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
tg.observe(1, at_sim(0, 1, kBandHi), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 0);
}
}
TEST_CASE("store never admits below the lower bound", "[track_gallery][AR-018]") {
// The lower bound is the poisoning guard: an embedding unlike everything
// already on the track is evidence the track is not one person.
TrackGallery tg(expand_cfg()); TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal); REQUIRE(tg.enabled());
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
tg.observe(1, at_sim(0, 1, kBandLo - 0.01f), 0, 0.30f, true, kNoCrop); // A track owned by actor 0. Every frame is accepted as actor 0, but each
CHECK(tg.band_rejected() == 1); // view is gallery-far (sim 0.30 < novelty 0.55) yet mutually self-similar
// enough to pass the spread gate.
for (int f = 0; f < 3; ++f)
tg.observe(7, at_sim(0, 1, 0.30f + 0.001f * f), 0, 0.30f + 0.001f * f, true, kNoCrop);
tg.observe(1, one_hot(400), 0, 0.30f, true, kNoCrop); // orthogonal: P = 0 // 3 accepted frames == min_anchor_frames → confirmed and promoted.
CHECK(tg.band_rejected() == 2); CHECK_FALSE(tg.annex().empty());
for (const auto& ae : tg.annex()) CHECK(ae.actor_idx == 0);
} }
TEST_CASE("store never admits above the upper bound", "[track_gallery][AR-018]") { TEST_CASE("novelty gate skips views the gallery already covers", "[track_gallery]") {
// The upper bound is the redundancy guard: another look at a pose the store
// already covers teaches the annex nothing and costs a slot.
TrackGallery tg(expand_cfg()); TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal); // All views are recognised well (sim 0.90 ≥ novelty 0.55): nothing worth
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop); // promoting even though the track is confirmed.
for (int f = 0; f < 3; ++f)
tg.observe(1, at_sim(0, 1, kBandHi + 0.01f), 0, 0.30f, true, kNoCrop); tg.observe(2, one_hot(0), 0, 0.90f, true, kNoCrop);
CHECK(tg.band_rejected() == 1); CHECK(tg.annex().empty());
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop); // identical: P = 1
CHECK(tg.band_rejected() == 2);
}
TEST_CASE("band thresholds probability, not cosine", "[track_gallery][AR-018][AR-024]") {
// The invariant's actual claim, and the one a raw-cosine gate passes by
// accident under an identity calibration. With a calibration that shifts by
// +0.10, two embeddings get the OPPOSITE verdict from the one their bare
// cosines would earn — so admission here can only come from the calibrated
// value having been used.
TrackGallery tg(expand_cfg());
tg.set_calibration([](float c) { return c + 0.10f; });
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
// cosine 0.84 (below lo, would be refused raw) → P = 0.94, inside the band.
tg.observe(1, at_sim(0, 1, 0.84f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 0);
// cosine 0.92 (inside the band, would be admitted raw) → P = 1.02, above it.
tg.observe(1, at_sim(0, 2, 0.92f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 1);
} }
TEST_CASE("a two-person track never poisons the annex", "[track_gallery][AR-018]") { TEST_CASE("a two-person track never poisons the annex", "[track_gallery][AR-018]") {
TrackGallery tg(expand_cfg()); TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// Two orthogonal identities under one track ID — a track-ID collision. // Two orthogonal identities under one track ID — a track-ID collision.
// The band refuses the outsider at the door, so the store never becomes //
// two-person in the first place. // The banded admission (AR-018) now catches this EARLIER than the spread
tg.observe(3, spoke(1, kSpokeCos), 0, 0.30f, true, kNoCrop); // gate did: an embedding unlike everything already on the track falls below
tg.observe(3, spoke(2, kSpokeCos), 0, 0.30f, true, kNoCrop); // the band's lower bound and is refused entry, so the buffer never becomes
// two-person in the first place. The spread gate remains as a second line
// for a track that drifts gradually rather than jumping.
//
// The assertion is on the outcome, not the mechanism: whichever gate fires,
// the outsider must not reach the actor's annex.
tg.observe(3, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(3, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(3, one_hot(400), 0, 0.30f, true, kNoCrop); // orthogonal outlier tg.observe(3, one_hot(400), 0, 0.30f, true, kNoCrop); // orthogonal outlier
CHECK(tg.band_rejected() == 1); // refused at the door CHECK(tg.band_rejected() > 0); // refused at the door
REQUIRE_FALSE(tg.annex().empty()); // the legitimate views still promote
for (const auto& e : tg.annex()) for (const auto& e : tg.annex())
CHECK(cosine_similarity(e.emb, one_hot(400)) < 0.5f); CHECK(cosine_similarity(e.emb, one_hot(400)) < 0.5f);
} }
TEST_CASE("a track that drifts through the band is refused at promotion", TEST_CASE("unconfirmed track (too few accepts) does not promote", "[track_gallery]") {
"[track_gallery][AR-018]") {
// `admit` compares a newcomer against its CLOSEST existing member, so a
// gradual drift chains past it: each step is in-band while the endpoints are
// strangers. This is the shape a collision takes over a slow pan, and the
// reason the lower bound is re-asked across every pair before promotion.
TrackGallery tg(expand_cfg()); TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal); // Only 2 accepted frames < min_anchor_frames 3; extra non-accepted frames
// fill the buffer but don't count toward ownership.
tg.observe(5, on_circle(0.f), 0, 0.30f, true, kNoCrop); tg.observe(4, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(5, on_circle(25.f), 0, 0.30f, true, kNoCrop); // P=0.906 vs 0° → in band tg.observe(4, at_sim(0, 1, 0.31f), 0, 0.31f, true, kNoCrop);
tg.observe(5, on_circle(50.f), 0, 0.30f, true, kNoCrop); // P=0.906 vs 25° → in band tg.observe(4, at_sim(0, 1, 0.32f), 0, 0.32f, false, kNoCrop);
CHECK(tg.band_rejected() == 0); // every step passed the door...
// ...but 0° and 50° are P=0.643 apart, below the floor: the whole track goes.
CHECK(tg.annex().empty()); CHECK(tg.annex().empty());
} }
// ── AR-019: ownership and promotion ────────────────────────────────────────── TEST_CASE("plurality actor wins a mixed-vote track", "[track_gallery]") {
Config cfg = expand_cfg();
TEST_CASE("confirmed track promotes its store", "[track_gallery][AR-019]") { cfg.expand_min_anchor_frames = 3;
TrackGallery tg(expand_cfg()); TrackGallery tg(cfg);
tg.set_calibration(identity_cal); // Actor 5 accepted twice, actor 6 once → plurality is 5. All views novel.
REQUIRE(tg.enabled()); tg.observe(8, at_sim(0, 1, 0.30f), 5, 0.30f, true, kNoCrop);
tg.observe(8, at_sim(0, 1, 0.31f), 5, 0.31f, true, kNoCrop);
// A track owned by actor 0: every frame accepted, every view mutually tg.observe(8, at_sim(0, 1, 0.32f), 6, 0.32f, true, kNoCrop);
// in-band (P = 0.9197 between distinct spokes) and gallery-far (0.30).
for (int k = 1; k <= 3; ++k)
tg.observe(7, spoke(k, kSpokeCos), 0, 0.30f, true, kNoCrop);
// 3 accepted frames == min_anchor_frames → confirmed and promoted.
CHECK(tg.annex().size() == 3);
for (const auto& ae : tg.annex()) CHECK(ae.actor_idx == 0);
}
TEST_CASE("registry ownership overrides the local tally", "[track_gallery][AR-019]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// Local accepted-frame plurality says actor 5; the registry's accumulated
// posterior says actor 9. The registry is authoritative.
tg.set_owner(11, 9);
for (int k = 1; k <= 3; ++k)
tg.observe(11, spoke(k, kSpokeCos), 5, 0.30f, true, kNoCrop);
REQUIRE_FALSE(tg.annex().empty());
for (const auto& ae : tg.annex()) CHECK(ae.actor_idx == 9);
}
TEST_CASE("unconfirmed track (too few accepts) does not promote", "[track_gallery][AR-019]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// Only 2 accepted frames < min_anchor_frames 3; the third fills the buffer
// but doesn't count toward ownership.
tg.observe(4, spoke(1, kSpokeCos), 0, 0.30f, true, kNoCrop);
tg.observe(4, spoke(2, kSpokeCos), 0, 0.30f, true, kNoCrop);
tg.observe(4, spoke(3, kSpokeCos), 0, 0.30f, false, kNoCrop);
CHECK(tg.annex().empty());
}
TEST_CASE("plurality actor wins a mixed-vote track", "[track_gallery][AR-019]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// No registry attached (unit-test path): actor 5 accepted twice, actor 6
// once → plurality is 5.
tg.observe(8, spoke(1, kSpokeCos), 5, 0.30f, true, kNoCrop);
tg.observe(8, spoke(2, kSpokeCos), 5, 0.30f, true, kNoCrop);
tg.observe(8, spoke(3, kSpokeCos), 6, 0.30f, true, kNoCrop);
REQUIRE_FALSE(tg.annex().empty()); REQUIRE_FALSE(tg.annex().empty());
for (const auto& ae : tg.annex()) CHECK(ae.actor_idx == 5); for (const auto& ae : tg.annex()) CHECK(ae.actor_idx == 5);
} }
TEST_CASE("promotion is idempotent across a long track", "[track_gallery][AR-019]") { TEST_CASE("promotion is idempotent across a long track", "[track_gallery]") {
TrackGallery tg(expand_cfg()); TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal); for (int f = 0; f < 3; ++f)
for (int k = 1; k <= 3; ++k) tg.observe(9, at_sim(0, 1, 0.30f + 0.001f * f), 0, 0.30f + 0.001f * f, true, kNoCrop);
tg.observe(9, spoke(k, kSpokeCos), 0, 0.30f, true, kNoCrop);
size_t after_confirm = tg.annex().size(); size_t after_confirm = tg.annex().size();
REQUIRE(after_confirm > 0); REQUIRE(after_confirm > 0);
// Keep feeding the confirmed track: annex must not grow again. // Keep feeding the confirmed track: annex must not grow again.
for (int f = 0; f < 10; ++f) for (int f = 0; f < 10; ++f)
tg.observe(9, spoke(4 + f, kSpokeCos), 0, 0.30f, true, kNoCrop); tg.observe(9, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
CHECK(tg.annex().size() == after_confirm); CHECK(tg.annex().size() == after_confirm);
} }
TEST_CASE("clear_tracks drops buffers before confirmation", "[track_gallery][AR-019]") { TEST_CASE("clear_tracks drops buffers before confirmation", "[track_gallery]") {
TrackGallery tg(expand_cfg()); TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// Two accepts, then a cut clears buffers; the third accept starts fresh and // Two accepts, then a cut clears buffers; the third accept starts fresh and
// can't reach the anchor threshold on its own. // can't reach the anchor threshold on its own.
tg.observe(1, spoke(1, kSpokeCos), 0, 0.30f, true, kNoCrop); tg.observe(1, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(1, spoke(2, kSpokeCos), 0, 0.30f, true, kNoCrop); tg.observe(1, at_sim(0, 1, 0.31f), 0, 0.31f, true, kNoCrop);
tg.clear_tracks(); tg.clear_tracks();
tg.observe(1, spoke(3, kSpokeCos), 0, 0.30f, true, kNoCrop); tg.observe(1, at_sim(0, 1, 0.32f), 0, 0.32f, true, kNoCrop);
CHECK(tg.annex().empty()); CHECK(tg.annex().empty());
} }
TEST_CASE("eviction keeps the gallery-far views", "[track_gallery][AR-018]") {
// Novelty is no longer a threshold — it is this ordering. With the buffer
// full, a more gallery-far newcomer must displace the best-recognised
// member, and a less novel one must be dropped rather than displace a
// better sample.
Config cfg = expand_cfg();
cfg.expand_buffer_size = 2;
cfg.expand_min_anchor_frames = 4;
TrackGallery tg(cfg);
tg.set_calibration(identity_cal);
tg.observe(6, spoke(1, kSpokeCos), 0, 0.80f, true, kNoCrop); // well recognised
tg.observe(6, spoke(2, kSpokeCos), 0, 0.40f, true, kNoCrop);
tg.observe(6, spoke(3, kSpokeCos), 0, 0.20f, true, kNoCrop); // novel: evicts the 0.80
tg.observe(6, spoke(4, kSpokeCos), 0, 0.90f, true, kNoCrop); // least novel: dropped
REQUIRE(tg.annex().size() == 2);
// Survivors are the two most gallery-far views: spokes 2 and 3.
for (const auto& ae : tg.annex()) {
const bool is_2 = cosine_similarity(ae.emb, spoke(2, kSpokeCos)) > 0.99f;
const bool is_3 = cosine_similarity(ae.emb, spoke(3, kSpokeCos)) > 0.99f;
CHECK((is_2 || is_3));
}
}