dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dtourolle/scene-actor-extraction:036f44fbddce6b1a2466e6b27c1a0bd9ac5594a7
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:ci/green-the-runner dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
..
compare: dtourolle/scene-actor-extraction:35e703350f0dbf0f864d6c30b7c58c3c383b3bb9
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:ci/green-the-runner dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
21 Commits
036f44fbdd .. 35e703350f
Author SHA1 Message Date
dtourolle 35e703350f fix(kpn): park on full outputs; surface node exceptions 
Adopts the KPN backpressure fix (28e0667) and registers the application
error listener it exposes.

`push_blocking` parked a scheduler worker inside the push. Each ObjectNode
owns a private single-thread pool, so the parked thread was the only one
that could drain that node's own input: under sustained backpressure
frame_source, camera_pos, face_detector and face_aligner all slept in
nanosleep at once and the pipeline stopped. Nodes now hold the value,
release the worker, and resume on a channel space-callback.

main.cpp registers set_error_handler so a node that throws names itself
and its exception. Previously the exception was discarded at the node
boundary and survived only as "node 'x' stopped unexpectedly", which says
that a node died but not why — the missing detail that made this slow to
diagnose.

AR-004 drops from Done to Mostly. Two gaps are recorded rather than
claimed fixed: a hang surviving at roughly 1 run in 20 against a 300 s
timeout (down from every run failing), and FanoutNode still dropping on
overflow instead of parking, which sheds frames on the AR-010 scene join
precisely when the dense branch falls behind.

TRACES: AR-004 | SR-002
 2026-07-31 22:42:19 +02:00
dtourolleandClaude Opus 5 de2eb5fa5c docs: regenerate the traceability matrix for VR-014 
The committed matrix predated the audio-signature binding, so VR-014 and
the four UT tags in `test_audio_offset.py` were absent from it while being
present in the register — the one inconsistency a generated file is
supposed to make impossible.

VR-014 also needed an explicit tier row. The blanket `VR-* | Out of CI`
line is right about every other study and wrong about this one: its
fixture is committed and its signature is CPU-only DSP, so it is a test a
CI host can run rather than a measurement someone has to remember to
repeat. Left as an exception under the blanket rather than rewriting the
rule, because the rule still describes the other thirteen.

Coverage unchanged at 38/69; the gate reports no orphan tags.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-014
 2026-07-31 17:02:45 +02:00
dtourolle 7b093eee92 Merge branch 'feature/quality-knee' into feature/opencv5 
# Conflicts:
#	docs/requirements.md
 2026-07-31 16:54:09 +02:00
dtourolle 890f1946bf Merge branch 'feature/dump-provenance' into feature/opencv5 2026-07-31 16:53:58 +02:00
dtourolleandClaude Opus 5 afca0524c9 docs: VR-013 and VR-014 results; AR-002 raised to 40px 
Records study results and the requirement change that follows from them.

VR-013 measures minimum face size end to end — gallery from one recording,
probes from another — rather than by degrading an already-aligned crop. Holding
90% of the plateau needs ~50 px that way against VR-005's ~22 px, the gap being
detection and landmark error rather than the embedder. AR-002 therefore takes
40 px, not 32: VR-005 isolates the embedder and is an upper bound, and 32 admits
faces in the falling region. FPI stayed 0.0% at every scale, and the ceiling is
cross-view rather than resolution.

VR-014 exercises audio-signature offset recovery on real film audio instead of
the synthetic golden tone. Forty random in-cap offsets, every one recovered to
the nearest frame, worst error 46 ms against a 500 ms budget — and 46 ms is the
quantisation floor rather than a result, since offsets land on whole 92.88 ms
frames. The runtime/2 anchor is confirmed through head-trimmed files.

The soft spot VR-014 found is tier labelling, not accuracy: the score drops with
sub-frame misalignment, so 27 of 40 correct alignments were demoted to `loose`.
One frame of slack in the score restores all forty to `audio` with false matches
unmoved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-002, VR-005, VR-013, VR-014 | SR-002, SR-003
 2026-07-31 16:53:58 +02:00
dtourolleandClaude Opus 5 d57e489e91 feat(audio): bind the v1 signature and validate offset recovery on real content 
sae_audio exposes the shipped signature to Python. It compiles
audio_signature.cpp directly against FFmpeg rather than linking
sae_gallery: the signature needs no model, no OpenCV and no HDF5, so a
module that dragged those in would make `import sae_audio` depend on a
GPU-capable build of a path that is pure CPU DSP.

The point of binding rather than porting is that a fingerprint is only
useful if every implementation agrees byte for byte. A numpy port would
be a third implementation, and the one nobody checks against the golden
vector.

VR-014 then recovers a known trim from real film audio rather than from
the synthetic tone: 40 random in-cap offsets, every one recovered to the
nearest frame, worst error 46 ms against a 500 ms budget — and 46 ms is
the quantisation floor, not a result, since offsets land on whole
92.88 ms frames.

The soft spot is tier labelling rather than accuracy. Sub-frame
misalignment drags the score down (0.94-0.99 near a frame boundary,
0.69-0.73 at half a frame), demoting 27 of 40 correct alignments to
`loose`. Allowing +/-1 frame of slack in the score fixes it: all 40 back
to `audio` at min 0.906, false matches unmoved at 0.12-0.16, for 81 ms
of the budget.

The module stops at the producer's edge. Sliding one signature against
another is the consumer's algorithm (server SPEC §3, and the jRay
plugin implements it), so a caller writing that slide in numpy is not
duplicating anything this repo owns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: IR-004, IR-005 | VR-014 | UT-105, UT-106, UT-107, UT-108 | SR-003
 2026-07-31 16:52:11 +02:00
dtourolleandClaude Opus 5 0dbbe5f6a3 fix: belief accumulates across frames (lazy-OR), not once 
A track recognised on 318 of 385 frames was owned on none, so the truth file
named nobody while the matcher was accepting almost continuously.

The correlation discount was an annihilator rather than an attenuator. Weight
was 1 - P(same view), so once a track had one stored view every later frame of
that same face scored ~0.01 and the belief stopped moving. One observation just
over the accept threshold is logit(0.78) ~ 1.27, under the ownership bar — hence
recognised always, owned never.

Two changes, in the order they were found.

Correlated evidence is now attenuated by effective sample size,
n_eff = n / (1 + (n-1)·rho), each frame contributing the marginal gain. That has
the right shape at both ends: uncorrelated evidence accumulates linearly, and a
held pose converges on 1/rho rather than growing without bound. A constant floor
was tried first and rejected — it grows linearly forever, so a long shot could
out-argue genuinely varied evidence purely by lasting longer.

Combination is now weighted lazy-OR: P = 1 - (1-P_old)·(1-p)^w, stored as
log(1-P) so the update is additive and precision stays where it matters as P
approaches 1. Each frame is new evidence that this track is that actor, and the
belief is the probability that at least one sighting was right. It converges
faster than summing log-odds at the same effective count — 2.98 vs 2.53 after
two observations at p=0.78 — which is what a real clip needs.

Note that summing log-odds was already a correct sequential Bayesian update:
the matcher fits with prior 0.5, so logit(p) IS the per-frame log-likelihood
ratio and the running sum carries the prior forward. It was not wrong, it was
slow. What blocked ownership was the discount, not the combination rule.

Also fixes a real correctness bug: the observation count lived on the
discounter, which is shared by every track, so tracks pooled into one effective
sample and each was discounted by how many others happened to be on screen. It
is now a per-track parameter.

The registry's frame scope holds its lock for its lifetime and the mutex is not
recursive, so calling observe() inside a scope self-deadlocks. The pipeline
never does — separate nodes — but the test did, and hung rather than failing.
Documented at the call site.

Verified end to end: the same clip that produced zero actors now identifies
Bing Crosby and Dorothy Lamour with belief 0.97.

Suite: 96 cases, 6142 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-025 | SR-002
 2026-07-31 16:51:08 +02:00
dtourolle 83f38a617e docs: the RANSAC aligner was a defect, measured 
AR-005 replaced cv::estimateAffinePartial2D(..., RANSAC, 3.0) with Umeyama
least squares over all five points — the estimator InsightFace aligns with, and
so the one the ArcFace/LVFace training crops were produced by.

The first note here assumed the two agree wherever RANSAC keeps all five points,
leaving a small divergence on non-frontal faces. Measured on 400 gallery
headshots with the model held fixed, that was wrong: the crops disagree by a
median 17 source px and 83.5% embed below cos 0.99 of their Umeyama counterpart.
A 4-DoF similarity is exactly determined by two points, so every minimal sample
fits its own pair perfectly and is scored on the other three; real landmarks sit
a median 2.74 canonical px from any similarity fit, so a landmark outside the
3 px band is the common case and RANSAC returns an under-determined transform.

How much that cost in accuracy is a separate question, and the honest answer is
less than those numbers suggest. Rebuilding the full gallery moved the
intra/inter separation the AR-023 calibration is fitted from by 0.583 to 0.590:
the old warp was wrong but self-consistent, gallery and probe both went through
it, and the embedder tolerates framing variation. The sharper evidence is
duplicate detection — the rebuild dropped 1614 near-duplicates against the
original build's ~100, because unstable two-point fits gave near-identical
images visibly different vectors. That instability, not a headline accuracy
delta, is what a tracker accumulating evidence across frames was paying for.

Also records the AR-030 residual's real-data floor: on the most cooperative
images the pipeline sees, it runs a median 2.74 px, so landmark noise occupies
the first few pixels and the synthetic foreshortening ladder is optimistic about
the low end. Any discount curve has to treat that range as uninformative rather
than as mild pose, and VR-012 must set thresholds against the measured
distribution.

Tests carry the tag they verify: the residual's roll/scale invariance and
monotonicity under foreshortening are what make it a pose measure rather than a
pose-and-everything-else measure.

TRACES: AR-005, AR-030 | SR-002
 2026-07-31 16:39:12 +02:00
dtourolleandClaude Opus 5 05f30c51fc feat(artifacts): push and pull the VR-013 corpus 
The cross-source study needs two 4K recordings and a hand-sorted set of
face crops, neither of which belongs in git. Adds an xsource target to
both artifact scripts.

Push uploads the clips as-is (already compressed) and zips labelling/.
Pull fetches both and regenerates frames with ffmpeg rather than
downloading them: ~320 MB of PNG that is deterministic from the clips.
The extraction settings are pinned in the script, not left to the
caller, because the manifests key on frame filenames and on detection
order within each frame — verify_labels.py runs afterwards and fails
loudly if they drift.

Pull refuses to overwrite an existing labelling/. It is human ground
truth: somebody looked at 167 crops and placed each one, and silently
replacing that with a remote copy would destroy the expensive half of
the study.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-013
 2026-07-31 15:58:24 +02:00
dtourolleandClaude Opus 5 66c9ca0a0c refactor(VR-005): drive the study off the sae_embed bindings 
Deletes the Python ports of SCRFDDecoder, ArcFaceEmbedder, align_face,
enhance_for_retry and calibrate_gallery, and calls the shipped C++
instead. 297 lines removed, 108 added.

The ports existed because sae_embed only exposed embed(path), so a
caller could not embed a crop it had degraded. That gap is closed:
detect(), align_face(), enhance_for_retry(), embed_crop()/embed_crops()
and GalleryCalibration are bound now, so there is no longer a reason to
keep a second implementation of any of them.

The calibration is the one that mattered. A parallel copy of the sigmoid
is precisely where "always the calibrated probability, never a raw
cosine" (AR-024) breaks without anyone noticing — the copy goes on
returning plausible numbers after the original has moved. Scoring
through the binding makes the rule structural rather than remembered.

Verified against the committed run: same shape, FPI 0.0% at every size,
same operating point of 32 px. Absolute rates differ by 1-2 points
because this check sampled 100 actors / 574 crops against the original's
258 / 999, not because anything regressed.

Also: --providers and --batch are gone, since provider selection and
batching belong to the backend; embeds are chunked at its max_batch,
because the engine does not split an oversized request and a whole
gallery in one call asks CUDA for a multi-gigabyte buffer. DEDUP_SIM and
MIN_EMB_FOR_POSITIVE stay as mirrored constants — used only to report
the population the C++ fitted on, not to refit it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-005 | AR-024
 2026-07-31 15:51:50 +02:00
dtourolle 402429dc2f Merge branch 'feature/gallery-report' into feature/opencv5 2026-07-31 15:31:07 +02:00
dtourolleandClaude Opus 5 ddb748eecb feat: gallery build report 
GR-003 — the calibration fit already computed per-actor dedup counts, how many
actors are eligible for positive pairs, and a 200-bin histogram of the intra and
inter distributions, then discarded all of it to stderr. Nothing persisted, so
nobody could audit whether a gallery was any good.

The report is written alongside the gallery at build time. That is the right
moment: the matcher fits the same sigmoid at analysis time, but by then the
answer is per-run and nobody is looking, whereas build time is when a gallery's
quality is actually decided.

What it surfaces, in order of usefulness:
- actors with no usable image — a silent recall ceiling, since the pipeline can
  never name them and nothing else says why
- actors below the positive-pair threshold — not broken, so nothing complains;
  they just quietly weaken every threshold downstream
- near-duplicate references removed, per actor and total
- the fitted calibration AND the two distributions behind it

That last one is the point. Every threshold in the pipeline is expressed in the
probability space this sigmoid defines, so if the distributions overlap heavily
the calibration is weak and every downstream decision inherits it — while the
gallery still looks fine from the outside.

The gallery-derived prior, intra/(intra+inter), is computed and reported but the
shipped default of 0.5 is deliberately left alone. The spec records these as
disagreeing; now the real value is visible, so the decision can be made on
evidence rather than argument.

Three tests: a zero-image actor is visible in the report, an under-referenced
actor is counted, and the report round-trips through JSON.

Suite: 95 cases, 6142 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: GR-003 | SR-001
 2026-07-31 15:30:46 +02:00
dtourolleandClaude Opus 5 a667caa313 feat: provenance attributes on the embedding dump 
VR-010 — a dump made with one detector/embedder pair was byte-indistinguishable
from one made with another, except for the two attributes GR-004 added. Replayed
against a gallery from a different model, cosine similarities are meaningless
but look entirely plausible. The register states the principle directly: a
fixture whose provenance is unknown is worse than no fixture, because it will be
trusted.

Sixteen attributes now record everything that determines the dump's content:
detector model and thresholds, min_face_px, max_faces, cut_threshold,
dense_scale, bbox_upscale, start/end, track_assoc_min_prob, and scene_detect.

scene_detect is the one that matters most. is_scene_boundary is all-zero both
when the detector found nothing and when it never ran, and those mean completely
different things to a consumer — without the flag they are indistinguishable.

No schema_version bump: new root attributes are additive and replay.py already
reads attributes with a default, so older dumps stay readable and the committed
fixtures — which predate this — still load.

Also corrects SCHEMA.md, which claimed bbox was already mapped to original
resolution at dump time. It is not; the upscale is applied downstream in the
matcher, after the dump tap. Harmless while dense_scale is 1 and silently wrong
otherwise, so bbox_upscale is now recorded and the doc says what the code does.

Verified end to end: all sixteen attributes present and correct on a freshly
generated dump.

Suite: 92 cases, 6136 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-010, VR-001 | PR-002
 2026-07-31 15:24:06 +02:00
dtourolleandClaude Opus 5 d81fc59824 study(VR-013): cross-source identification probe over input resolution 
Gallery from one recording, probes from another, sweeping the probe's
input resolution end to end. VR-005 asked the same question over gallery
mugshots but degraded an already-aligned 112x112 crop with alignment held
perfect, so it isolates the embedder. Here the whole frame is downscaled
before the detector, so detection and landmark regression degrade with
it — which is most of the difference.

Corpus is two 4096x2160 clips of one shoot, four people, hand-sorted.
Ground truth is sorted by hand and gated by verify_labels.py; labels
carried down the scales geometrically by box position, never by
embedding similarity, which would keep only the faces the embedder
already gets right and drop the ones the sweep exists to find.

Findings, all scored through the production gallery sigmoid at
prob_threshold 0.754 — never a raw cosine:

- Holding 90% of the plateau needs ~50 px end to end, against VR-005's
  ~22 px. min_face_px at 40 looks right; 32 would admit faces in the
  falling region.
- FPI is 0.0% at every scale. Resolution loss goes entirely to TBI.
- The ceiling is cross-view, not resolution: everyone matches themselves
  within a recording (0.55-0.85) and collapses across two (0.14-0.45,
  threshold 0.335). Only the subject with frontal *gallery* references
  identified reliably, whatever their probe pose — so the lever is
  gallery pose coverage, not a better landmark source.
- Averaging SCRFD's overlapping detections instead of discarding them at
  NMS lifts cross-recording TPI 41% -> 49%, for one forward pass and no
  extra model.

Four identities and one shoot, so the shape is the result and the
absolute rates are not. Both clips contain all four people, so there is
no out-of-gallery class and the 10x-weighted out-of-cast misID is
untested here.

Clips, frames, hand-sorted crops and results are gitignored and belong
in the artifact registry — the sorting is human ground truth and
expensive to redo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-013 | AR-002, AR-005, AR-024
 2026-07-31 15:20:18 +02:00
dtourolleandClaude Opus 5 042e424961 study(VR-005): minimum face size from downscaled gallery mugshots 
Holds out one mugshot per actor, degrades that probe to each candidate
face size and matches it against a gallery held at native resolution,
reporting TPI/FPI per size. Replaces AR-002's 66x66 px working estimate
with a measurement. Needs no video and no ground truth beyond the
mugshot cache already on disk.

LVFace-B over 258 actors, 999 gallery embeddings, threshold 0.754:

    px    12    16    20    24    32    40   48+
   TPI   6.6% 46.5% 81.8% 93.4% 98.1% 99.2% 99.2%

FPI is 0.000 at every size — a face too small to identify degrades to
unidentified, never to a wrong name. rank-1 holds at >=99.6% from 24 px
up, so what fails first is the calibrated probability crossing
threshold, not the ranking.

Two limits on reading this. FPI grows with the number of actors
competing, so 258 understates it against a production library. And
detection and alignment run on the native image with only the resulting
112x112 crop degraded, so landmark error at small face sizes is excluded
by construction and the curve is an upper bound — VR-010 measures the
same question end to end, and lands well above these numbers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-005 | AR-002
 2026-07-31 15:17:39 +02:00
dtourolleandClaude Opus 5 50649c1f87 feat: expansion promotion gated on all three discontinuity signals 
AR-019 — promotion may only borrow same-identity evidence from a span where
identity is certain, so every discontinuity signal now clears the buffers rather
than just the histogram cut.

is_scene_boundary was already named in the gate but never set by anything, so
that half of it was dead until AR-010 gave it a producer. It now does what the
spec always said. The third signal, an identity contradiction, needs no code
here: AR-015 closes a track whose belief swapped, so it can no longer promote.

Ownership now comes from the registry rather than a second tally. TrackGallery
was computing its own plurality vote over accepted frames, which meant two
different answers to "who is this track" could coexist in one run — and the
expansion one ignored the Bayesian accumulation entirely, weighting thirty
near-identical looks the same as thirty distinct ones. The local tally survives
only as a fallback for callers with no registry attached, which is the unit
tests and the replay harness.

Suite: 92 cases, 6133 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-019, AR-010, AR-015 | SR-005
 2026-07-31 15:11:56 +02:00
dtourolleandClaude Opus 5 080581c050 feat: banded admission for the per-subject embedding store 
AR-018 — an embedding joins a track's store only if its similarity to something
already there falls inside a band, rather than merely being far from the gallery.

Above the upper bound it is redundant: another look at a pose the store already
covers, teaching the annex nothing while costing a slot a novel view could have
used. Below the lower bound it is suspect: within one track every face is the
same person by construction, so an embedding unlike everything else on the track
is evidence that construction failed — a track-ID collision or a bad detection.
Admitting it is exactly how an actor's annex gets poisoned with someone else's
face.

The old gate had only the upper half of that idea, expressed as a raw cosine
against the gallery. Both bounds are now calibrated probabilities (AR-024), so
the same number means the same thing here as in association and evidence
weighting rather than three different things.

This catches track-ID collisions EARLIER than the spread gate did — at the door
rather than at promotion — so the buffer never becomes two-person in the first
place. The spread gate stays as a second line for a track that drifts gradually
instead of jumping. The existing test was asserting the mechanism rather than
the outcome, so it was rewritten to assert what actually matters: whichever gate
fires, the outsider must not reach the annex.

Rejections are counted. A store that admits nothing is as broken as one that
admits everything, and neither is visible otherwise.

Band defaults 0.90-0.95 are working values pending VR-007; the two bounds fail in
opposite directions and must be swept separately.

Suite: 92 cases, 6133 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-018, AR-024 | SR-005
 2026-07-31 15:10:31 +02:00
dtourolleandClaude Opus 5 6da8ac2bdb perf: back the CPU similarity GEMM with OpenBLAS 
The CPU path was a scalar triple loop. It is the correctness oracle for the GPU
backends, but it is also what CI runs — there is no GPU on the N100 host — and
since AR-003 removed the per-frame face cap, a crowded frame now scores many
faces against a library-scale gallery. Scoring one face against 5000 embeddings
is 2.6 MFLOP; in scalar that does not hold up (AR-027).

S(g,f) viewed as row-major [n_faces x n_gallery] is exactly query * gallery^T,
so the loop nest collapses into a single cblas_sgemm.

OpenBLAS is optional in the build: found via pkg-config, and the scalar path
remains when it is absent so no hard dependency is added and the two can be
diffed when a similarity looks wrong. The configure step warns rather than
failing, since a developer without it should still get a working tree.

The test target links it too. Without that the suite compiles the scalar
fallback while the builder image ships CBLAS, so CI would be verifying a kernel
that is not the one running in production — the same class of mistake as testing
a path the gate never executes.

Recorded as required (not optional) in the DP-007 image, for the same reason.

Suite: 92 cases, 6136 assertions, with CBLAS compiled in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-026, AR-027, DP-007 | SR-001
 2026-07-31 15:04:29 +02:00
dtourolleandClaude Opus 5 bbab5aed23 feat: join the decode butterfly so scene boundaries reach the face branch 
AR-010 — is_scene_boundary had no producer: SceneDetectorFunc was a terminal
sink writing scenes.json and never annotating the frames flowing to face
detection. The flag was permanently false, so the boundary half of AR-007's
frame-dependent association was dead code that a test could still exercise
synthetically and appear to verify.

The topology already forks after decode — dense frames to TransNetV2, sampled
frames to face detection — so this is a fork-join. SceneBoundaries is the join:
the detector publishes each window's verdict with a watermark, and an annotator
on the sampled branch stamps the flag.

The watermark is the part that matters. TransNetV2 buffers 100 frames before it
can score any of them, so at any instant it has an opinion up to some time T and
none after. Without recording T a consumer cannot tell "no boundary" from "not
scored yet", and those demand opposite behaviour — treating unscored frames as
boundary-free is exactly what makes a downstream check pass while verifying
nothing.

Buffering alone does not work, which was my first attempt. Channel depth creates
lag only when the consumer is slower, and the face branch runs four orders of
magnitude faster per frame than TransNetV2 (0.01ms vs 400ms), so its channels
drain instantly and no lag accumulates. Measured: 106 of 364 frames outran the
detector. The annotator therefore waits on the watermark explicitly. The
detector signals completion so the tail cannot deadlock, and publishes from
flush_remaining too — without that the final frames arrive with no verdict.

Boundaries are deduped on publish, matching what scenes.json does at write time.
A run of adjacent high-scoring frames is one boundary, not several; leaving them
raw made this view report 357 where the file said 13. Now the two agree exactly.

Frames past the detector's last scored window remain unverified and are counted
as such rather than silently marked boundary-free.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-007, AR-010 | SR-002
 2026-07-31 14:56:38 +02:00
dtourolleandClaude Opus 5 a2c699a844 docs: AR-010 is blocked on a design decision, not an implementation gap 
Making SceneDetectorFunc a pass-through does not work. TransNetV2 buffers 100
dense frames before it can score any of them and trusts only each window's
centre, so a boundary at time T is not known until roughly 3.3s after T at
30 fps. The face pipeline runs on a parallel branch and has long since passed T.
An association hint that arrives after the association is worthless.

Three options recorded with their costs: two-pass (correct, doubles the decode
that already dominates runtime), delaying the face branch (couples the two
branches' timing, which invites heisenbugs under backpressure), or leaving it
unwired.

Leaving it unwired costs less than it looks, which is what makes this a decision
rather than a defect. The redesign made cuts and boundaries do the same thing —
both say "spatial continuity is broken, associate on embedding" — so TransNetV2
adds nothing over the histogram except on transitions the histogram cannot see:
slow dissolves and fades. That gap is real but narrow.

Where TransNetV2 still earns its cost is AR-019, whose promotion gate wants a
span free of cuts and boundaries. A late answer is fine there, because promotion
happens on track confirmation rather than per frame — so it can be wired
offline against the collected boundary list, off the hot path entirely.

Recommendation: leave the association path on is_cut alone, wire boundaries into
AR-019, and revisit if dissolve-heavy material shows association failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-010, AR-019 | SR-002
 2026-07-31 14:27:05 +02:00
dtourolleandClaude Opus 5 dfb8f5801e feat: no fixed cap on faces per frame 
AR-003 — max_faces defaults to 0, meaning no cap. A fixed cap discards the
SMALLEST faces first, which are exactly the background cast X-Ray still credits
with scene membership, so the pipeline was systematically losing the people it
is supposed to find in crowded scenes.

This is only safe now that AR-004 landed. Previously an uncapped frame would
have pushed more work into channels that dropped on overflow, trading a visible
cap for silent loss. With backpressure the producer slows instead, so per-frame
cost is contained rather than discarded.

The matcher's kMaxFaces used to throw above 32, which made it an accidental
second cap. It sizes the similarity engine's preallocated buffer, so it bounds
memory rather than face count — the frame is now scored in batches of that size.
Memory stays bounded; faces do not.

Largest-first ordering is kept even without the cap, and the comment now says
why: the Hungarian solver tie-breaks on index order, so that ordering is
load-bearing for the replay determinism test rather than a leftover of the cap.

Verified end to end on a real clip: identical output to the capped run (385
frames, 693 faces), which is expected since that footage peaks at 4 faces per
frame — the point is the absence of a regression. The committed fixtures remain
byte-identical and valid.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-003 | SR-002
 2026-07-31 14:22:20 +02:00
59 changed files with 6136 additions and 267 deletions
Expand all files Collapse all files
CMakeLists.txt
+28
View File
@@ -152,6 +152,24 @@ if(SAE_GEMM_BACKEND STREQUAL "CPU")
set_target_properties(gemm_backend PROPERTIES POSITION_INDEPENDENT_CODE ON)
target_include_directories(gemm_backend PRIVATE src)
target_compile_definitions(gemm_backend PRIVATE SAE_GEMM_CPU)
# AR-026/AR-027: back the CPU path with OpenBLAS when present. Optional, so
# the build gains no hard dependency — but without it the fallback is a
# scalar loop, which does not hold up against a library-scale gallery, and
# the CPU path is exactly what CI (no GPU) and the cpu builder image use.
find_package(PkgConfig QUIET)
if(PkgConfig_FOUND)
pkg_check_modules(OPENBLAS QUIET openblas)
endif()
if(OPENBLAS_FOUND)
message(STATUS "GEMM backend: CPU + OpenBLAS ${OPENBLAS_VERSION}")
target_compile_definitions(gemm_backend PRIVATE SAE_GEMM_CBLAS)
target_include_directories(gemm_backend PRIVATE ${OPENBLAS_INCLUDE_DIRS})
target_link_libraries(gemm_backend PRIVATE ${OPENBLAS_LINK_LIBRARIES})
else()
message(WARNING "GEMM backend: CPU scalar fallback — OpenBLAS not found. "
"Correct, but slow on a large gallery (AR-027).")
endif()
elseif(SAE_GEMM_BACKEND STREQUAL "CUDA")
find_library(CUBLAS_LIB cublas
HINTS /opt/cuda/targets/x86_64-linux/lib /opt/cuda/lib64
@@ -290,6 +308,16 @@ target_link_libraries(sae_embed PRIVATE sae_gallery)
nanobind_add_module(sae_kpn src/kpn_bindings.cpp)
target_link_libraries(sae_kpn PRIVATE sae_gallery)
# ── sae_audio — Python module: the v1 audio signature (IR-004) ────────────────
# Compiles audio_signature.cpp directly and links only FFmpeg, rather than
# linking sae_gallery: the signature needs no model, no OpenCV and no HDF5, and
# a module that dragged all three in would make `import sae_audio` depend on a
# GPU-capable build of a repo whose audio path is pure CPU DSP. tests/ compiles
# the same source the same way, for the same reason.
nanobind_add_module(sae_audio src/audio_bindings.cpp src/audio_signature.cpp)
target_include_directories(sae_audio PRIVATE src)
target_link_libraries(sae_audio PRIVATE ffmpeg_libs)
# HDF5 already found above (before sae_gallery); vars HDF5_CXX_LIBRARIES / _INCLUDE_DIRS
# are reused by scene_analyze / dump_embeddings below.
docs/SPEC.md
+301 -19
View File
@@ -40,12 +40,28 @@ Detect faces in sampled video frames.
presence (SR-002) a lower rate still answers the question, but it lengthens the
interval between samples and so weakens IoU-based association; sweep the two
together (VR-002).
- **Minimum face size is 66×66 px**, expressed in **original video resolution**,
- **Minimum face size is 40×40 px**, expressed in **original video resolution**,
not decoded-frame pixels. Stating it in original space decouples it from
`dense_scale`: otherwise a 0.5 downscale silently doubles the effective
threshold, and dense mode is exactly what scene detection uses.
66 is a working estimate of where ArcFace embeddings stop being reliable, not a
measured value — it should be replaced by the result of VR-005.
40 is **measured, not estimated** — it replaces an earlier 66 px guess. Two
studies bracket it, and the difference between them is the whole reason the
number is 40 rather than 32:
- **VR-005** degrades an already-aligned 112×112 crop and matches it against
a native-resolution gallery. Alignment is held perfect, so it isolates the
*embedder*: the knee sits at 24–32 px, and 32 px still returns 98.1% TPI.
- **VR-013** downscales the **whole frame before the detector**, so detection
and landmark regression degrade along with it. End to end, holding 90% of
the plateau needs roughly **50 px**, against VR-005's ~22 px.
The gap is detection and landmark error, which VR-005 excludes by construction
— so VR-005 is an **upper bound on quality**, not a threshold, and reading a
floor off it would admit faces in the falling region. **AR-002 therefore takes
VR-013's number.** 40 sits below the 50 px plateau deliberately: FPI is 0.0% at
every scale in both studies, so resolution loss costs recall and never
precision, and an over-tight floor discards presence that SR-002 requires.
- Emits bounding box, detector confidence, and 5-point landmarks.
- Bounding boxes must be reported in **original video pixel space**. When
`dense_scale < 1` downscales the decoded frame, coordinates are rescaled by
@@ -59,7 +75,9 @@ Detect faces in sampled video frames.
**Current:** SCRFD-500MF via `face_detector_node.hpp`, thresholds in `config.hpp`
(`detector_conf` 0.5, `detector_nms` 0.4), `min_face_px` 40, `max_faces` 10.
**Gap:** `min_face_px` → 66 and re-expressed in original resolution; `max_faces`
**Gap:** `min_face_px` re-expressed in original resolution — the value 40 is
already correct after VR-013, so what remains is the space it is measured in, not
the number; `max_faces`
removed, gated on backpressure (AR-004).
## AR-004 — Backpressure
@@ -133,9 +151,71 @@ Produce the exact input ArcFace expects.
nose, left mouth, right mouth).
- Alignment is the *only* geometric normalisation; no additional augmentation at
inference.
- **The transform is fitted by Umeyama's closed-form least squares over all five
points**, which is what InsightFace uses (skimage's `SimilarityTransform` *is*
`_umeyama`) and therefore what produced the crops ArcFace and LVFace were
trained on. The canonical warp is part of the input distribution, not an
implementation detail (AR-011).
- **Not a robust estimator.** A RANSAC fit buys a small residual by discarding
the landmarks that disagree with the model, and on a turned face those are the
foreshortened ones — the signal AR-030 reads. With five points and a two-point
minimal sample it also cannot separate a mis-detected landmark from honest
out-of-plane rotation, so the robustness is nominal while the cost to AR-030 is
total. It is RNG-driven besides, which made replay determinism a property of
thread scheduling.
**Current:** `align_face()` in `src/face_utils.hpp:9-22`, `cv::warpAffine` to
`{112, 112}`. **Gap:** none.
**Current:** `align_face()` in `src/face_utils.hpp`, Umeyama fit via
`umeyama_similarity()`, `cv::warpAffine` to `{112, 112}`. **Gap:** none.
> **Migration note — this was a defect, not a refinement.** Until this landed the
> fit was `cv::estimateAffinePartial2D(…, cv::RANSAC, 3.0)`. The expectation was
> that the two agree wherever RANSAC keeps all five points, leaving a small
> divergence on non-frontal faces. **Measured, that is wrong.** On 400 random
> gallery headshots, one model held fixed and only the estimator varied:
>
> | | median | p90 | max |
> |---|---|---|---|
> | Crop disagreement (source px, over the crop corners) | 16.97 | 75.91 | 223.31 |
> | `cos(umeyama, ransac)` for the resulting embedding | 0.791 | — | — |
>
> 83.5 % of crops embed to a cosine below 0.99 of their Umeyama counterpart —
> they are not the same face crop. The mechanism is that a 4-DoF similarity is
> exactly determined by **two** points, so every minimal RANSAC sample fits its
> own pair perfectly and is then scored on the other three. Real landmarks sit a
> median 2.74 canonical px from any similarity fit to the template (see AR-030
> below), so images with a landmark outside the 3 px band are the common case,
> not the exception; RANSAC then keeps two or three inliers and returns a wildly
> under-determined transform.
>
> **Every gallery baked before this change must be rebuilt** — GR-004's embedder
> stamp catches a model change, not an aligner change, so nothing else would say
> so.
>
> **How much this cost in accuracy is a separate question, and the answer appears
> to be: less than the crop numbers suggest.** Rebuilding the full gallery
> (2456 actors) moved the intra/inter separation the AR-023 calibration is fitted
> from only slightly:
>
> | | intra-actor | inter-actor | separation |
> |---|---|---|---|
> | RANSAC | 0.6234 | 0.0407 | 0.5827 |
> | Umeyama | 0.6340 | 0.0440 | 0.5900 |
>
> The reconciliation is that the old warp was *wrong but self-consistent*: it
> produced a differently-framed face rather than a scrambled one, gallery and
> probe went through the same estimator, and the embedder tolerates framing
> variation. So the figures in `model-bakeoff.md`, `best-model.md` and
> `pose-expansion.md` were all produced through the broken warp on both sides and
> should be re-run, but there is no measured basis for expecting them to move far.
>
> The sharper evidence of the old instability is duplicate detection: rebuilding
> with an unchanged `dedup_tol` dropped **1614** near-duplicate images, where the
> original build dropped on the order of a hundred. Near-identical source images
> used to embed to visibly different vectors — RANSAC fitting two-point subsets is
> unstable under small landmark perturbations, and being RNG-driven it was not
> reproducible either. That instability is what a tracker accumulating evidence
> across frames pays for, and it is the strongest reason the fix is worth having
> independently of any accuracy delta.
## AR-006 — Embedding
@@ -151,6 +231,121 @@ Generate a 512-d embedding per aligned crop.
**Current:** `embedder_node.hpp` + `face_embedder_engine.hpp`; default
LVFace-B_Glint360K. **Gap:** none.
## AR-028 … AR-030 — Embedding input quality
An embedder handed a face it cannot represent does not fail. It returns a
confident, plausible, wrong vector, and that vector then competes on equal terms
with every good one in the gallery — the same failure mode AR-011 names for
whole models, occurring here at the level of a single region. Quality assessment
is how that is caught **at inference**, rather than inferred afterwards from a
study of why a film scored badly.
Three axes, assessed on every face before its embedding is used as identity
evidence. They are kept separate and **not collapsed into one scalar**: they fail
for different reasons, have different remedies, and — as below — do not even earn
the same response.
- **Size** — already AR-002, floor at 40×40 px in original resolution, measured
end to end by VR-013. It is the precedent for the other two: the
threshold was *located*, not chosen.
- **Sharpness** — motion blur and soft focus destroy the high-frequency detail
the embedder keys on, and unlike size they leave the bounding box looking
perfectly healthy. Measured on the **112×112 aligned crop**, not the raw box:
the crop is already scale-normalised, so a measure taken there cannot silently
re-measure face size and double-count it against AR-002.
- **Visibility** — extreme pose or occlusion means the face presents fewer of the
features the embedding assumes are present. The measure is the **residual of
the AR-005 alignment fit**: the RMS landmark error, in canonical 112×112
pixels, left over after the best similarity transform onto the ArcFace
template. It costs nothing — the transform is computed for the warp regardless,
and the residual is what that fit could not explain.
Two properties earn it the job over an explicit yaw estimate:
- A similarity absorbs rotation, uniform scale and translation **exactly**,
so the residual is by construction the non-similarity part of the
deformation: out-of-plane rotation and foreshortening. In-plane roll
contributes nothing, so "a tilted head reads as a turned one" is excluded
structurally rather than by tuning. The destination frame is fixed, so face
size cannot leak in either — that is AR-002's axis, and double-counting it
would make a small frontal face look occluded.
- It responds to **occlusion** and to plainly broken landmark sets, which an
angle regressor by construction does not: a hand across the face is not a
rotation, but it does displace landmarks.
Indicative magnitudes from a synthetic foreshortening sweep (`k ≈ cos yaw`):
`k=1.0 → 0.00`, `0.9 → 1.18`, `0.75 → 3.11`, `0.5 → 6.72`, `0.3 → 9.85`
canonical px. Smooth and monotone with a usable range; the mapping onto real
faces is VR-012's to establish, and no threshold is set from these numbers.
**The synthetic ladder is noise-free and therefore optimistic about the low
end.** Measured on 400 real TMDB/Jellyfin headshots — the most frontal, most
cooperative population the pipeline ever sees — the residual runs p5 1.11,
median 2.74, p90 4.82, max 6.35 canonical px. So landmark noise alone occupies
roughly the first 3 px, and the synthetic sweep's "26° yaw ≈ 1.2 px" sits
*below* the noise floor on real data. VR-012 must set any threshold against
this measured distribution, and a discount curve has to treat the first few
pixels as uninformative rather than as mild pose.
Neither a dedicated landmark model (`models/2d106det.onnx` is present but
referenced nowhere — and it emits points, not pose) nor a direct pose CNN is
adopted unless VR-012 shows the residual insufficient. If one is needed the
candidate is **6DRepNet** (MIT, RepVGG-B1g2, 3.47° MAE on AFLW2000) rather than
Hopenet, which it dominates on accuracy, licence, recency and export
friendliness. Two caveats to record before that happens: both are trained on
**300W-LP**, which inherits research-only terms from 300W's constituent sets,
and both want their own loosely-framed ROI rather than the ArcFace crop — a
second warp and a second image in flight, which lands on AR-004's byte-based
backpressure gap. It would also have to run **per track** — over the bounded
view set AR-019's diversity buffer already keeps — not per face per frame,
which is the cost rule applied as written: fewer regions, never a degraded
input.
**Failing an axis discounts the observation; it does not delete the detection.**
Only size drops the face outright, and only because VR-005 measured a knee below
which the embedding carries no signal to discount. Blur and pose are different:
- A blurred or turned face is still evidence of **presence**, which is what
SR-002 actually asks about.
- The tracker admits a link on position *or* identity precisely so that a face
"whose embedding degraded (blur, profile turn)" stays linkable. Remove the
detection and the track fragments, costing the window extent AR-012/AR-013
exist to protect.
- AR-019 harvests non-frontal views *because* TMDB headshots are frontal.
Discarding turned faces starves the mechanism built to fix the pose problem of
its raw material, and AR-020 then has nothing to resolve at EOF.
The natural home for the discount is `EvidenceDiscounter` (AR-025), which already
weights how far one observation may move a track's belief. Note that its present
weight is pure *novelty*, so a profile view — maximally distant from everything
counted so far — currently scores near 1.0 and moves the belief hardest, when
against a frontal gallery it deserves the least trust. Novelty and reliability
are orthogonal and multiply; quality supplies the second term.
**Quality is carried, not consumed.** The vector travels with the face and is
written to the VR-001 dump alongside the embedding, so a threshold can be
re-litigated against recorded data instead of by re-running video, and so
VR-010's provenance records what the run actually admitted.
**No quality threshold is hand-set.** Each axis either has a measured knee
(VR-012, as VR-005 did for size) or it discounts rather than drops — a
hand-chosen cutoff on an uncalibrated measure is the same unfalsifiable magic
number AR-024 retired for similarity, and it would fail the same way: meaning
something different for every detector, every embedder and every film.
**Current:** visibility is measured and carried — `estimate_alignment()` in
`src/face_utils.hpp` returns the residual alongside the transform, and
`FaceAlignerFunc` writes it to `DetectedFace::alignment_residual`. Size is
`min_face_px` (40, decoded-frame space — AR-002 still open). Sharpness is
unmeasured. Nothing yet *consumes* any of it: no discount is applied, and
`align_face()` still drops the degenerate-fit case without counting it.
**Gap:** AR-029 entirely. For AR-030, the measure exists but the discount does
not — it must reach `EvidenceDiscounter` as the reliability term. For AR-028, the
residual does not yet reach the VR-001 dump, which is what VR-012 needs to run
from fixtures; that is the next step, since it unblocks the study that sets
every remaining behaviour.
## AR-007, AR-008 — Tracking
Link detections across frames into tracks representing one physical person.
@@ -204,16 +399,40 @@ Two distinct signals, deliberately kept separate:
> `--dump-embeddings` branch *before* the `scene_detect` branch at `:296`, so no
> dump-producing path even instantiates the detector.
>
> This makes AR-010 **not implemented**, not "in progress" — and it means a T2
> test of the frame-dependent `track_alpha` (AR-007) would **pass vacuously**,
> which is the worst possible failure for a verification gate. The fix is in the
> producer, not the schema: make `SceneDetectorFunc` a pass-through (or add a
> boundary annotator before the decimator) and add the scene branch to
> `dump_embeddings.cpp`. **No `schema_version` bump** — the column exists and
> merely stops being constant.
>
> Fixtures generated before the fix must be marked in provenance, since `0` is
> presently indistinguishable from "no boundary here".
> **It cannot be fixed by making the node a pass-through.** TransNetV2 buffers
> `kWindow` = 100 dense frames before it can score any of them, runs inference
> every `scene_stride` (50) frames, and trusts only each window's centre. So a
> boundary at time *T* is not known until roughly 100 dense frames after *T* —
> about **3.3 s at 30 fps**. The face pipeline runs on a parallel branch and has
> long since passed *T* by then. An association hint that arrives after the
> association is worthless.
>
> Three ways out, none free:
>
> 1. **Two-pass.** Run scene detection to completion, then analyse faces with
> boundaries already known. Simple and correct; costs a second decode of the
> whole file, and dense decode is already the pipeline's dominant cost.
> 2. **Delay the face branch** by the detector's window latency. Keeps one pass;
> adds a buffering stage and couples the two branches' timing, which is the
> kind of coupling that produces heisenbugs under backpressure.
> 3. **Leave it unwired.** Accept that `is_cut` is the only association hint.
>
> **Option 3 costs less than it appears**, which is why this is a decision rather
> than a bug. Since the redesign made cuts and boundaries do the *same thing* —
> both say "spatial continuity is broken, associate on embedding" — TransNetV2
> adds nothing over the histogram except on transitions the histogram misses:
> slow dissolves and fades, where there is no frame-to-frame discontinuity to
> detect. That is a real but narrow gap.
>
> The value TransNetV2 retains is in **AR-019**, whose promotion gate requires a
> span with no cut *and* no boundary. There a late answer is still usable,
> because promotion happens when a track is confirmed rather than per frame.
> Wiring it there — offline, against the collected boundary list — is cheaper
> than any of the three options above and does not touch the hot path.
>
> **Recommendation: option 3 plus the AR-019 wiring**, and revisit if dissolve-
> heavy material shows association failures the histogram misses.
Both feed AR-007 as **association hints**: they tell the tracker that spatial
continuity is broken and that association should weight embedding over IoU.
@@ -866,8 +1085,21 @@ cannot silently change what a green build meant.
| HDF5 (C++) | Galleries are HDF5-native; also the dump format |
| FFmpeg dev libs — `libavformat`, `libavcodec`, `libavutil`, `libswscale`, **`libswresample`** | Decode. See the note below on swresample |
| Python 3 + numpy, h5py, scipy | Python-side tests, replay, traceability tooling |
| **OpenBLAS** | Backs the CPU similarity GEMM. Without it the fallback is a scalar loop, and the CPU path is exactly what this host runs — see below |
| Catch2, nlohmann/json | **Vendored into the image, not fetched.** Both are `FetchContent`-ed today (`CMakeLists.txt:220`, `tests/CMakeLists.txt:8`), which makes every CI run depend on GitHub reachability |
**OpenBLAS is not optional here, despite being optional in the build.** CI has no
GPU, so `SAE_GEMM_BACKEND=CPU` is the only path it exercises — and since AR-003
removed the per-frame face cap, a crowded frame scores many faces against a
library-scale gallery. The scalar fallback is correct but scales badly, which
would make the CPU path the bottleneck in the one place it cannot be avoided
(AR-027). The build warns when it is missing rather than failing, so a developer
without it still gets a working tree; the image must not be that case.
The test target links it too. Otherwise the suite compiles the scalar fallback
while the image ships CBLAS, and CI would verify a kernel that is not the one
running in production.
**Deliberately excluded:** CUDA, TensorRT, ROCm — no GPU to use them. Also the
ONNX Runtime *GPU* providers; only the CPU provider is relevant, and only for T3
smoke tests.
@@ -1063,7 +1295,47 @@ plugin. Consequences to carry through:
- Files never processed by this pipeline still get a signature from the plugin;
the two paths coexist deliberately.
**Gap:** entire requirement — no audio path exists in the pipeline today.
**Current:** `src/audio_signature.*` implements the construction, and
`tests/fixtures/audio/` holds the golden vector shared verbatim with the plugin
repo, which now matches it byte for byte from C# (jRay `JR-042`/`JR-043`).
`sae_audio` (nanobind, as `sae_embed` and `sae_kpn` are) exposes the same C++ to
Python so a study drives the shipped code rather than a numpy port.
**VR-014 measures what the golden vector cannot** — that the signature actually
aligns a differently trimmed release, on real film audio rather than a synthetic
tone. It does, with an order of magnitude to spare.
**The accuracy question is settled and is not close.** What the offset is *for*
is shifting scene windows, which are seconds long, so half a second of error is
invisible; the budget is 500 ms. Over 40 random offsets inside the ±600-frame cap
the recovered offset was the nearest frame every time — **worst error 46 ms**.
That figure is the quantisation floor rather than a measurement of quality: the
offset is expressed in whole 92.88 ms frames, so no correct answer can ever be
worse than half a frame. The `runtime/2` anchor behaves as specified through real
head-trimmed files (cutting `delta` from the head moves the window by
`delta/2`), and both an out-of-cap offset and unrelated content are declined
outright (0.10 and 0.07).
**Where it is soft is tier labelling, not alignment.** The *score* at the correct
offset falls with sub-frame misalignment — 0.94–0.99 when the true offset lands
within 0.1 of a frame boundary, 0.69–0.73 at half a frame — because the two
windows' frame grids no longer coincide. The offset stays right, but only 13 of
40 cleared the server's 0.85 `audio` threshold and the other 27 were demoted to
`loose`, a tier that means "possibly the same cut, degraded audio". The threshold
was calibrated on a re-encode at *zero* offset, where the score is 1.00.
The remedy is measured, not proposed (UT-108): counting a frame as agreeing if
its peak bin matches **within ±1 frame** returns all 40 to `audio` (worst 0.906)
while unrelated content and out-of-cap offsets stay at 0.12 and 0.16 — the gap
that makes the threshold mean anything is untouched. It costs 81 ms of offset
accuracy, of a 500 ms budget, because the flattened peak lets the argmax pick an
adjacent frame. ±2 frames buys nothing further. Adopting it is a
[server spec](../../JRay-public-server/SPEC.md) §3 change — the score is
normative and shared by three repos — so this repo measures it and leaves the
decision there.
**Gap:** the signature is computed but **not yet emitted** into the truth file —
that is the `IR-002` field and the coordinated `schema_version` bump.
## IR-006 — Jellyfin round-trip
@@ -1249,9 +1521,11 @@ Persist pipeline state at the point where the expensive work ends.
variable-length HDF5 types and reads straight into numpy.
- Stores per frame: `timestamp_sec`, `frame_idx`, `is_cut`, `is_scene_boundary`.
Per face: `embedding` [N,512], `bbox` [N,4], `landmarks` [N,10], `confidence`.
- Invariants: embeddings unit-norm; `face_offset` contiguous; bboxes already in
original resolution; frames with no faces still get a row so timestamps stay
dense; EOF sentinels not written.
- Invariants: embeddings unit-norm; `face_offset` contiguous; bboxes and
landmarks in **decoded-frame** pixels with `bbox_upscale` recorded alongside
(the dump is a faithful tap, so it does not transform what the tracker saw —
see VR-010); frames with no faces still get a row so timestamps stay dense;
EOF sentinels not written.
- Enabled by `--dump-embeddings out.h5`; teeing must not perturb the live result.
Schema owned by [`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md).
@@ -1310,6 +1584,14 @@ Verify `landmarks`/`bbox`/`is_cut` suffice, and bump `schema_version` if not.
Quantify where ArcFace degrades, replacing the 66×66 estimate in A1 with a
measurement.
> **Result, and its limit.** Knee at 24–32 px; 32 px returns 98.1% TPI at 0.0
> FPI. But the probe is an already-aligned 112×112 crop, so alignment is held
> perfect and this measures the **embedder alone** — an upper bound, not a
> threshold. **VR-013** re-asks the question end to end, downscaling the whole
> frame before the detector, and lands near 50 px. AR-002's floor of 40 px comes
> from VR-013; this study is what shows how much of the gap is detection and
> landmark error rather than embedding.
**Method.**
1. Select ~100 gallery actors having more than one mugshot.
docs/plan.md
+64 -3
View File
@@ -243,7 +243,9 @@ Context crops opt-in behind `--dump-unidentified-crops`.
**Depends on:** nothing (AR-002, AR-011); AR-004 blocks AR-003.
- **AR-002** — `min_face_px` → 66, expressed in original resolution.
- **AR-002** — `min_face_px` stays **40** (VR-013 measured it end to end) but must
be expressed in original resolution rather than decoded-frame space. The value
is already right in `config.hpp`; the change is the coordinate space.
- **AR-011** — feed TransNetV2 at native rate; derive the dedup window from
source fps rather than the hardcoded `0.04 s`.
- **AR-004** — backpressure. `kMaxFaces` (`identity_matcher_node.hpp:133`)
@@ -324,8 +326,67 @@ Windows carry belief and route; `extraction.*` gains `extinction_sec` and
## VR-005 — Minimum face size study
**Depends on:** nothing. Standalone Python, no C++ contact. Produces the measured
value replacing AR-002's 66 px estimate.
**Depends on:** nothing. Standalone Python, no C++ contact. **Done** — knee at
24–32 px. It measures the embedder with alignment held perfect, so it bounds the
answer from below rather than setting it; AR-002's floor comes from **VR-013**,
which sweeps input resolution end to end and lands at 40 px.
## VR-013 — Cross-source identification probe
**Depends on:** `sae_embed` exposing `detect()`, `align_face()`, `embed_crop()`
and the gallery calibration — it drives the shipped C++ rather than reimplementing
it, which is what VR-005 could not do.
Gallery from one recording, probes from another, sweeping the probe's **input
resolution before the detector**, so detection and landmark regression degrade
with the frame. `experiments/xsource/`.
**Findings.** Holding 90% of the plateau needs ~50 px end to end against VR-005's
~22 px; `min_face_px` 40 is right and 32 would admit faces in the falling region.
FPI is 0.0% at every scale — resolution loss goes entirely to TBI, never to a
wrong name. The ceiling is **cross-view, not resolution**: everyone matches
themselves within a recording (0.55–0.85) and collapses across two (0.14–0.45),
and only the subject with frontal *gallery* references identified reliably — so
the lever is gallery pose coverage (`docs/pose-expansion.md`), not a better
landmark model. Averaging SCRFD's NMS-discarded landmark estimates lifts
cross-clip TPI 41% → 49% for one forward pass.
**Open.** Four identities and one shoot, so the shape is the result and the
absolute rates are not. Both clips hold all four people, so there is no
out-of-gallery class and the 10×-weighted out-of-cast misID is untested — holding
one identity out of the gallery would fix that.
## VR-014 — Audio-signature offset recovery
**Depends on:** `sae_audio` exposing `compute_signature()` and
`signature_from_mono()` — it drives the shipped C++, as VR-013 does, so the
thing measured is the thing that ships.
`scripts/validation/test_audio_offset.py` over
`tests/fixtures/audio/bali_offset_200s.flac`: 200 s of public-domain film audio
(the same Road to Bali clips the replay fixtures use), long enough for a 120 s
window to slide past the ±600-frame search cap. The slide itself is numpy here
on purpose — matching belongs to the consumer, so writing it out keeps this a
test of the signature rather than of somebody's matcher.
**Findings.** Alignment is a solved problem here: the offset is the nearest frame
in every in-cap trial, worst error **46 ms against a 500 ms budget**, and 46 ms is
the quantisation floor — offsets are whole 92.88 ms frames, so no correct answer
can be worse. The `runtime/2` anchor's factor of two holds through real trimmed
files, and out-of-cap offsets and unrelated content are both declined.
**The score is where the slack is, and it costs a tier rather than accuracy.** It
tracks sub-frame misalignment — 0.94–0.99 near a frame boundary, 0.69–0.73 at
half a frame — so two thirds of correct alignments miss the server's 0.85 `audio`
threshold and land in `loose`. UT-108 measures the fix rather than proposing one:
±1 frame of slack in the score returns all 40 to `audio` (min 0.906) with false
matches unmoved at 0.12–0.16, costing 81 ms of the budget. See
[`SPEC.md`](SPEC.md) IR-004 — the score is normative in the server spec, so the
change is theirs to make.
**Open.** One source, one language, one era of recording. The shape (offset exact,
score set by sub-frame phase) should hold generally, but the absolute scores are
this fixture's.
## VR-001 — Dump audit
docs/requirements.md
+30 -16
View File
@@ -29,15 +29,15 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | Planned |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done |
| AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Mostly** — node outputs *park* on a full channel: the value is held, the worker released, and a channel space-callback resumes the node. Replaces `push_blocking`, which parked a scheduler worker inside the push and, with one thread per node, stopped that node draining its own input. Verified: 385/385 frames, 0 drops. **Gap:** a rare hang survives, ~1 run in 20 at a 300 s timeout (was: every run). `FanoutNode` still drops on overflow (`fanout.hpp:129`) rather than parking, so the AR-010 scene join sheds frames exactly when the dense branch falls behind |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far |
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted |
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Not started** — `is_scene_boundary` has no producer; `SceneDetectorFunc` is a terminal sink and never annotates the frame |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Done** — decode butterfly joined via `SceneBoundaries`; the sampled branch waits for the detector's watermark. Frames past its last scored window are counted as unverified, never assumed boundary-free |
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | **Done** — `src/track_registry.hpp`; window is `[first_seen, last_seen]` of an owned track |
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | **Done** — `last_seen` optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed |
@@ -45,16 +45,19 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | **Done** — reverse index detects it on the causing update; counted |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | **Done** — `flush()`, idempotent, closes at last sighting or final tick |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief and observation count |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | Planned |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | In Progress |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space, bounds from `expand_band_lo/hi`; the lower bound re-asked pairwise at promotion, since `admit` compares only against the nearest member and a drifting track can chain past it. Retires `expand_novelty_sim` and `expand_track_spread_max` — novelty is now the eviction ordering, not a threshold. Rejections counted. Bounds unswept (VR-007) |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | **Done** — all three discontinuity signals clear the buffers; ownership comes from the registry, not a second local tally |
| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned |
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association and accumulation both in probability space; `track_max_embed_dist`, `cut_revive_sim` retired |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association, accumulation and expansion all in probability space; `track_max_embed_dist`, `cut_revive_sim`, `expand_novelty_sim`, `expand_track_spread_max` retired |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | Planned |
| AR-029 | Sharpness measure on the **aligned crop** (scale-normalised, so it cannot re-measure size) | SR-002 | Medium | Planned |
| AR-030 | Visibility measure from the AR-001 5-point landmarks — extreme pose or occlusion **discounts the observation, never deletes the detection** | SR-002 | Medium | **In Progress** — measure is the AR-005 alignment residual (`estimate_alignment()`), carried on `DetectedFace`; roll/scale invariance and monotonicity under foreshortening asserted. Nothing consumes it as a discount yet |
## Deployment (DP)
@@ -88,7 +91,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|---|---|---|---|---|
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | **Done** — `gallery/gallery_report.hpp`, written next to the gallery by `build_gallery`. Zero-usable-image actors come from the build audit, which a stored gallery cannot reconstruct; also `distinct_references`, `duplicates_removed`, and the intra/inter distributions the calibration fits and would otherwise discard |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | **Done** — basename + SHA-256 + `embed_dim`; mismatch fatal with no bypass, unstamped warns unless `--require-gallery-stamp`; `scripts/stamp_gallery.py` migrates in place |
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
@@ -104,13 +107,16 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | **Done** — replay driven from committed fixtures in `tests/test_replay_fixtures.cpp`; determinism asserted |
| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done |
| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | **Done** — knee at 24–32 px; 32 px gives 98.1% TPI, 0.0 FPI at every size |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | **Done** — knee at 24–32 px; 32 px gives 98.1% TPI, 0.0 FPI at every size. Degrades an already-aligned 112×112 crop, so it isolates the embedder and is an **upper bound**; VR-013 measures the same question end to end and AR-002 takes its number, not this one |
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned |
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned |
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned |
| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | Planned |
| VR-014 | Audio-signature **offset recovery on real content** — a known trim recovered from film audio, not from the synthetic golden tone | PR-002 | Medium | **Done** — 40 random in-cap offsets, every one recovered to the nearest frame: **worst error 46 ms against a 500 ms budget**, and 46 ms is the floor rather than a result, since the offset is quantised to whole 92.88 ms frames. The `runtime/2` anchor confirmed through real head-trimmed files (a `delta` trim moves the window by `delta/2`). The one soft spot is **tier labelling, not accuracy**: the score falls with sub-frame misalignment (0.94–0.99 near a frame boundary, 0.69–0.73 at half a frame), so 27/40 correct alignments were demoted to `loose`. ±1 frame of slack in the *score* fixes it — measured, all 40 back to `audio` (min 0.906), false matches unmoved at 0.12–0.16, costing 81 ms of the budget |
| VR-013 | Cross-source identification probe — gallery from one recording, probes from another, swept over input resolution end to end | PR-002 | Medium | **In Progress** — holding 90% of the plateau needs ~50 px end to end against VR-005's ~22 px, the gap being detection and landmark error; **`min_face_px` 40, since 32 admits faces in the falling region** (AR-002). FPI 0.0% at every scale. Ceiling is cross-view, not resolution |
---
@@ -195,7 +201,7 @@ as such rather than counted as covered.
| Requirement | Tier | Note |
|---|---|---|
| AR-001, AR-005, AR-006 | T3 | Smoke only — correctness of detection/embedding is a model property, not ours |
| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | Size filtering is arithmetic on dumped bboxes |
| AR-002 | T2 | Size filtering is arithmetic on dumped bboxes |
| AR-003, AR-004 | T1 + T4 | Backpressure logic is unit-testable; saturation behaviour needs real load |
| AR-007 … AR-017 | **T2** | The core of the redesign — fully replayable |
| AR-018 … AR-022 | **T2** | Expansion, deferred pass, clustering: all post-embedding |
@@ -207,6 +213,7 @@ as such rather than counted as covered.
| GR-001 … GR-005 | T1 + T3 | Gallery assembly is I/O and bookkeeping; embedding is T3 smoke |
| GR-006 … GR-008 | T1 | Tiering and outlier detection operate on stored embeddings |
| VR-* | Out of CI | Studies are run deliberately and their results committed as documents |
| VR-014 | **T2** | The exception, and the reason the blanket row above is not the whole story: its fixture is committed and its signature is CPU-only DSP, so the study *is* a test a CI host can run — not a measurement someone has to remember to repeat |
**One consequence worth stating:** AR-027 (arbitrary gallery scale) is
structurally unverifiable on the CI host. It needs a GPU host and a synthetic
@@ -236,9 +243,10 @@ from a copyrighted title could not live in the repository at all.
Two properties to design around rather than discover:
- **480×360 means small faces.** At this resolution a face is often 40–80 px, so
the AR-002 minimum of 66 px (original resolution) rejects much of what is
there. Fixture generation must set `--min-face-px` explicitly and record it,
or the dumps will be sparse for reasons unrelated to what is being tested.
the AR-002 minimum of 40 px (original resolution) sits at the very bottom of
that range: the filter is close to binding, and anything shot wider is lost.
Fixture generation must set `--min-face-px` explicitly and record it, or the
dumps will be sparse for reasons unrelated to what is being tested.
- **77 s is short.** At 1 fps that is 77 frames — too thin to exercise an
extinction window measured in tens of seconds. Generate at 5 fps (≈385 frames,
~1 MB) and record the rate in provenance, since the behaviour under test
@@ -302,10 +310,10 @@ because it will be trusted.
| ID | Tier | Test asserts | Edge cases to cover |
|---|---|---|---|
| AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only |
| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-002 | T2 | Faces below 40 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame |
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block |
| AR-005 | T1 | Known landmarks → expected 112×112 warp | Landmarks near frame edge; degenerate/collinear points |
| AR-005 | T1 | Known landmarks → expected 112×112 warp; the fit never mirrors | Landmarks near frame edge; degenerate/collinear points; a mirrored set — SVD returns a reflection unless the determinant guard rejects it |
| AR-006 | T3 | Embeddings are unit-norm | Batch smaller than, equal to, larger than `embed_batch_size` |
| AR-007 | T2 | Association picks the right track | Two faces crossing paths; one leaving frame as another enters |
| AR-008 | T2 | One pool; dormant tracks match on embedding, not IoU | Dormant track whose old bbox overlaps a *different* new face — must not match on position |
@@ -327,9 +335,15 @@ because it will be trusted.
| AR-025 | T1 | Log-odds accumulate; correlated frames discounted | 30 identical frames must **not** reach the certainty of 30 diverse ones |
| AR-026 | T1 + T4 | GEMM path produces same result as reference loop | Equivalence on small input in CI; throughput on GPU host |
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
| AR-028 | **T2** | No embedding reaches the matcher unscored; the vector survives into the dump | Face failing exactly one axis; all three healthy; a face whose landmarks are degenerate — scored, not silently vanished |
| AR-029 | T1 | Synthetic blur ladder → monotonically falling sharpness | Gaussian vs motion blur; **small sharp face vs large soft one** — size must not leak into this axis |
| AR-030 | T1 | Alignment residual rises monotonically with foreshortening | **In-plane roll, scale and translation must leave it at zero** — the property that makes it a pose measure rather than a pose-and-everything-else measure; face size must not shift it; degenerate landmarks report not-ok rather than a number |
| VR-012 | **T4** | Knee located per axis on held-out films | Report each candidate threshold's cost in **lost true presence**, not only its gain in precision — a gate that improves misID by discarding half the cast has not helped |
| VR-013 | **T4** | Identification holds across two recordings of the same people, and degrades to TBI rather than to a wrong name as input resolution falls | Gallery and probes must come from *different* recordings — a hold-one-out over one recording measures a much easier problem and will not surface the cross-view failure. Ground truth is hand-sorted; labels propagated by embedding similarity would keep only the faces the embedder already gets right |
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos |
| VR-014 | **T2** | A known trim offset is recovered from **real film audio**, to the nearest frame | An offset past the ±600-frame cap and unrelated content must both be *declined*, never given a best-effort alignment. Fixture and signature are both CPU-only, so unlike the other VR rows this one is CI-*executable* — though the repo's only workflow today is the traceability gate, so nothing runs it there yet. The signature comes from the shipped C++ through `sae_audio`; a numpy port would be a third implementation nobody checks against the golden vector |
| IR-006 | T1 + manual | Queue pull and result push against a stubbed Jellyfin API | Partial result never pushed; push only after the deferred pass |
| IR-007 | **T1** | Media < 120 s emits no signature at all | Exactly 120 s; just under; zero-length audio. Must match the plugin's cutoff exactly — a caller-varying window length is what SR-004 forbids |
| IR-008 | T1 | `v1:` prefix emitted and honoured on read | Unknown prefix rejected, not guessed |
docs/traceability.md
+315 -114
View File
@@ -3,7 +3,7 @@
<!-- GENERATED FILE - do not edit by hand. -->
<!-- Regenerate: scripts/traceability/traceability-gate.sh -->
**Generated:** 2026-07-31T08:35:29+00:00
**Generated:** 2026-07-31T15:01:49+00:00
Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`).
@@ -11,27 +11,28 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev
| Metric | Value |
|---|---|
| Source files scanned | 95 |
| TRACES tags found | 90 |
| Source files scanned | 112 |
| TRACES tags found | 137 |
| EXCEPTION tags found | 0 |
| Requirements defined | 63 |
| Requirements covered | 26 |
| **Coverage** | **41.3%** (26/63) |
| Coverage of CI-executable scope | 50.0% (26/52) |
| Tagged but unexecuted in CI | 3 |
| Requirements defined | 69 |
| Requirements covered | 38 |
| **Coverage** | **55.1%** (38/69) |
| Coverage of CI-executable scope | 67.9% (38/56) |
| Tagged but unexecuted in CI | 7 |
| Orphan tags | 0 |
### By type
| Type | Covered | Tagged but unexecuted | Defined |
|---|---|---|---|
| AR | 13 | 0 | 27 |
| AR | 22 | 1 | 30 |
| DP | 2 | 0 | 8 |
| IR | 8 | 0 | 8 |
| GR | 3 | 0 | 9 |
| VR | 0 | 3 | 11 |
| GR | 5 | 0 | 9 |
| VR | 1 | 6 | 14 |
- **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104
- **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104, UT-105, UT-106, UT-107, UT-108
- **IT** tags present (separate taxonomy, not counted in coverage): IT-001
- **PR** tags present (separate taxonomy, not counted in coverage): PR-002, PR-004
- **SR** tags present (separate taxonomy, not counted in coverage): SR-001, SR-002, SR-003, SR-005
@@ -41,19 +42,21 @@ These requirements have no verification tier this repo's CI host can run, so a t
| ID | Tiers | Tagged in source | Requirement |
|---|---|---|---|
| AR-027 | T4 | no | Throughput acceptable for **arbitrary** gallery size |
| AR-027 | T4 | yes | Throughput acceptable for **arbitrary** gallery size |
| VR-001 | out-of-ci | yes | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | out-of-ci | yes | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | out-of-ci | yes | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | out-of-ci | no | Reproducible validation corpus with ground truth |
| VR-005 | out-of-ci | no | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-004 | out-of-ci | yes | Reproducible validation corpus with ground truth |
| VR-005 | out-of-ci | yes | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | out-of-ci | no | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | out-of-ci | no | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | out-of-ci | no | Gallery scaling benchmark — throughput vs gallery size |
| VR-010 | out-of-ci | no | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-010 | out-of-ci | yes | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | out-of-ci | no | Rewrite the replay harness for the post-AR-012 output contract |
| VR-012 | T4, out-of-ci | no | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … |
| VR-013 | T4, out-of-ci | no | Cross-source identification probe — gallery from one recording, probe… |
**Tagged but unexecuted:** VR-001, VR-002, VR-003 — a test exists and is tagged, but this CI host cannot run it. Report those runs separately.
**Tagged but unexecuted:** AR-027, VR-001, VR-002, VR-003, VR-004, VR-005, VR-010 — a test exists and is tagged, but this CI host cannot run it. Report those runs separately.
## Orphan tags
@@ -78,32 +81,35 @@ _None._
| ID | Status | Tier | Traces to | Trace state | Tagged in | Requirement |
|---|---|---|---|---|---|---|
| AR-001 | Done | T3 | SR-002 | covered | `src/nodes/face_detector_node.hpp` | Detect faces in sampled frames; emit bbox, confidence, 5-point landma… |
| AR-002 | Planned | unset | SR-002 | untagged | - | Minimum face size **32×32 px** (VR-005 measured), expressed in **orig… |
| AR-003 | Planned | T1, T2, T4 | SR-002 | untagged | - | No fixed per-frame face cap — crowd scenes must not lose background c… |
| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | untagged | - | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… |
| AR-005 | Done | T1, T3 | SR-002 | covered | `src/face_utils.hpp` | Align to 112×112 via ArcFace 5-point similarity transform |
| AR-006 | Done | T3 | SR-002 | untagged | - | 512-d L2-normalised embeddings, batched |
| AR-002 | Planned | T2 | SR-002 | untagged | - | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's … |
| AR-003 | **Done** — `max_fac… | T1, T2, T4 | SR-002 | covered | `src/config.hpp`, `src/nodes/face_detector_node.hpp`, `src/nodes/identity_matcher_node.hpp` | No fixed per-frame face cap — crowd scenes must not lose background c… |
| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_replay_fixtures.cpp` | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… |
| AR-005 | **Done** — `umeyama… | T1, T3 | SR-002 | covered | `src/face_utils.hpp`, `tests/test_face_utils.cpp` | Align to 112×112 via ArcFace 5-point similarity transform, fitted by … |
| AR-006 | Done | T3 | SR-002 | covered | `src/nodes/embedder_node.hpp` | 512-d L2-normalised embeddings, batched |
| AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… |
| AR-008 | **Done** — one pool… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | One track pool keyed on `last_seen`; no separate revival path |
| AR-009 | Done | T2 | SR-002 | untagged | - | Camera-cut detection (histogram) as an association hint |
| AR-010 | **Not started** — `… | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint |
| AR-009 | Done | T2 | SR-002 | covered | `src/nodes/camera_position_change_detector_node.hpp` | Camera-cut detection (histogram) as an association hint |
| AR-010 | **Done** — decode b… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/scene_boundary_annotator_node.hpp`, `src/nodes/scene_detector_node.hpp`, `src/scene_boundaries.hpp` | Scene-boundary detection (TransNetV2) as an association hint |
| AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… |
| AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition |
| AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… |
| AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition |
| AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… |
| AR-014 | **Done** — swap clo… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Belief swap A→B terminates the track and starts a new one |
| AR-015 | **Done** — reverse … | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Two live tracks owned by one actor ⇒ treat as a detected cut, re-asso… |
| AR-016 | **Done** — `flush()… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | All tracks closed at EOF — a film ends with faces on screen |
| AR-017 | **Done** — `DeadTra… | T1, T2 | SR-002 | covered | `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Every presence claim carries its belief and identification route |
| AR-018 | Planned | T1, T2 | SR-005 | untagged | - | Per-subject embedding store with banded admission (novel enough, safe… |
| AR-019 | In Progress | T2 | SR-005 | untagged | - | Per-film gallery annex from owned tracks; acquires the non-frontal vi… |
| AR-018 | **Done** — banded a… | T1, T2 | SR-005 | covered | `src/config.hpp`, `src/gallery/track_gallery.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-subject embedding store with banded admission (novel enough, safe… |
| AR-019 | **Done** — all thre… | T2 | SR-005 | covered | `src/gallery/track_gallery.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-film gallery annex from owned tracks; acquires the non-frontal vi… |
| AR-020 | Planned | T2 | SR-005 | untagged | - | Deferred re-identification of unknown tracks against the final expand… |
| AR-021 | Planned | T2 | SR-005 | untagged | - | Cluster unknown tracks into one entity per person, under temporal can… |
| AR-022 | Planned | T1, T2 | §4 | untagged | - | Capture still-unidentified tracks: embeddings, metadata, **context cr… |
| AR-023 | Done | T1 | SR-002 | covered | `src/gallery/gallery_calibration.hpp`, `src/nodes/identity_matcher_node.hpp` | Fit sigmoid calibration from intra/inter similarity distributions |
| AR-024 | **Done** — associat… | T1, static | SR-002 | covered | `src/config.hpp`, `src/evidence_discount.hpp`, `src/gallery/gallery_calibration.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp`, `src/nodes/identity_matcher_node.hpp` | **Always the calibrated probability, never a raw cosine** — exception… |
| AR-024 | **Done** — associat… | T1, static | SR-002 | covered | `src/config.hpp`, `src/evidence_discount.hpp`, `src/gallery/gallery_calibration.hpp`, `src/gallery/track_gallery.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp`, `src/nodes/identity_matcher_node.hpp` | **Always the calibrated probability, never a raw cosine** — exception… |
| AR-025 | **Done** — log-odds… | T1 | SR-002 | covered | `src/evidence_discount.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-track Bayesian accumulation in log-odds, with correlated-observat… |
| AR-026 | In Progress | T1, T4 | SR-001 | untagged | - | All similarity computed as GEMM, including annex and deferred pass |
| AR-027 | Planned | T4 | SR-001 | untagged | - | Throughput acceptable for **arbitrary** gallery size |
| AR-026 | In Progress | T1, T4 | SR-001 | covered | `src/backends/gemm_backend.cpp` | All similarity computed as GEMM, including annex and deferred pass |
| AR-027 | Planned | T4 | SR-001 | tagged, unexecuted | `src/backends/gemm_backend.cpp` | Throughput acceptable for **arbitrary** gallery size |
| AR-028 | Planned | T2 | SR-002 | untagged | - | **Embedding input quality assessed and carried** — every face scored … |
| AR-029 | Planned | T1 | SR-002 | untagged | - | Sharpness measure on the **aligned crop** (scale-normalised, so it ca… |
| AR-030 | **In Progress** — m… | T1 | SR-002 | covered | `src/face_utils.hpp`, `tests/test_face_utils.cpp` | Visibility measure from the AR-001 5-point landmarks — extreme pose o… |
| DP-001 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | One analysis core; modes are front-ends and must not fork pipeline lo… |
| DP-002 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | Batch CLI over one title |
| DP-003 | Planned | T1, manual | PR-004 | untagged | - | On-demand resident service with bounded, observable queue |
@@ -115,31 +121,34 @@ _None._
| IR-001 | Done | T1 | SR-003 | covered | `src/nodes/result_sink_node.hpp` | Emit the JRay truth format as sibling `.jray.json` |
| IR-002 | **Done** — `schema_… | T1 | SR-003 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/result_sink_node.hpp` | Windows carry belief + route; `extraction.*` carries `extinction_sec`… |
| IR-003 | **In Progress** — s… | T1 | SR-003 | covered | `src/main.cpp` | Output written **after** the deferred pass, not at EOF |
| IR-004 | **Done** — `src/aud… | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Compute the audio signature exactly per server spec §3 |
| IR-005 | **Done** — `tests/f… | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Golden-vector fixture shared with the plugin repo to prove bit-exactn… |
| IR-004 | **Done** — `src/aud… | T1 | SR-003 | covered | `scripts/validation/test_audio_offset.py`, `src/audio_bindings.cpp`, `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Compute the audio signature exactly per server spec §3 |
| IR-005 | **Done** — `tests/f… | T1 | SR-003 | covered | `src/audio_bindings.cpp`, `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Golden-vector fixture shared with the plugin repo to prove bit-exactn… |
| IR-006 | Done | T1, manual | SR-001 | covered | `scripts/run_from_jellyfin.py` | Jellyfin round-trip: pull pending queue, push complete results only |
| IR-007 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Media < 120 s: emit no signature, apply no sync offset — identical ru… |
| IR-008 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Emit and honour the signature's own `v1:` version prefix |
| GR-001 | Done | T1, T3 | SR-001, SR-005 | covered | `scripts/make_jellyfin_gallery.py` | Build gallery from Jellyfin library cast, TMDB profile fallback |
| GR-002 | Done | T1, T3 | PR-003 | covered | `scripts/make_jellyfin_gallery.py` | Incremental `--merge` refresh without re-embedding known actors |
| GR-003 | Planned | T1, T3 | SR-001 | untagged | - | Report coverage: zero-image actors, under-referenced actors, dedup, c… |
| GR-003 | Planned | T1, T3 | SR-001 | covered | `src/build_gallery.cpp`, `src/gallery/gallery_calibration.hpp`, `src/gallery/gallery_report.hpp` | Report coverage: zero-image actors, under-referenced actors, dedup, c… |
| GR-004 | **Done** — basename… | T1, T3 | SR-001 | covered | `scripts/filter_gallery.py`, `scripts/make_gallery.py`, `scripts/make_jellyfin_gallery.py`, `scripts/movienet_eval.py`, `scripts/optimizer/fetch_missing_actors.py`, `scripts/optimizer/optimize.py`, `scripts/optimizer/reembed_gallery.py`, `scripts/optimizer/replay.py`, `scripts/sae_embed_loader.py`, `scripts/sae_gallery.py`, `scripts/sae_stamp.py`, `scripts/stamp_gallery.py`, `src/config.hpp`, `src/gallery/embedder_stamp.cpp`, `src/gallery/embedder_stamp.hpp`, `src/gallery/gallery_builder.cpp`, `src/gallery/gallery_store.cpp`, `src/kpn_bindings.cpp`, `src/main.cpp`, `src/nodes/embedding_dump_node.hpp`, `src/scene_preview.cpp`, `src/types.hpp`, `tests/test_gallery_store.cpp` | Stamp embedder identity into the gallery; **hard startup error** on m… |
| GR-005 | Done | T1, T3 | **SR-005** | untagged | - | Gallery data never leaves the instance |
| GR-005 | Done | T1, T3 | **SR-005** | covered | `src/gallery/gallery_store.hpp` | Gallery data never leaves the instance |
| GR-006 | Planned | T1 | SR-005 | untagged | - | Provenance tiers: baked / harvested / confirmed, distinguishable per … |
| GR-007 | Planned | T1 | SR-005 | untagged | - | Persist harvested embeddings **flagged and reviewable**, never silent… |
| GR-008 | Planned | T1 | SR-005 | untagged | - | Flag distributional outliers among an actor's references (poisoning g… |
| GR-009 | TBD | T1 | §4 | untagged | - | Human-confirmed associations persist and improve future extractions |
| VR-001 | Done | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp` | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py` | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-001 | Done | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp`, `tests/test_replay_fixtures.cpp` | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | **Done** — replay d… | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py`, `tests/test_replay_fixtures.cpp` | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/second_score.py` | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | Done | out-of-ci | PR-002 | untagged | - | Reproducible validation corpus with ground truth |
| VR-005 | **Done** — knee at … | out-of-ci | PR-002 | untagged | - | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-004 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/validation/ground_truth.py` | Reproducible validation corpus with ground truth |
| VR-005 | **Done** — knee at … | out-of-ci | PR-002 | tagged, unexecuted | `scripts/validation/min_face_size.py` | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | Planned | out-of-ci | PR-002 | untagged | - | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | Planned | out-of-ci | PR-002 | untagged | - | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | Planned | out-of-ci | PR-002 | untagged | - | Gallery scaling benchmark — throughput vs gallery size |
| VR-009 | Planned | T1, out-of-ci | PR-002 | untagged | - | Verify accumulated posteriors are calibrated against held-out tracks |
| VR-010 | Planned | out-of-ci | PR-002 | untagged | - | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-010 | Planned | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp` | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | Planned | out-of-ci | PR-002 | untagged | - | Rewrite the replay harness for the post-AR-012 output contract |
| VR-012 | Planned | T4, out-of-ci | PR-002 | untagged | - | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … |
| VR-013 | **In Progress** — h… | T4, out-of-ci | PR-002 | untagged | - | Cross-source identification probe — gallery from one recording, probe… |
| VR-014 | **Done** — 40 rando… | T2, out-of-ci | PR-002 | covered | `scripts/validation/test_audio_offset.py` | Audio-signature **offset recovery on real content** — a known trim re… |
## Detailed mapping
@@ -149,46 +158,92 @@ _None._
- [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown`
### AR-003
**Locations:** 3
- [`src/config.hpp:44`](../src/config.hpp#L44) — `Unknown`
- [`src/nodes/face_detector_node.hpp:47`](../src/nodes/face_detector_node.hpp#L47) — `private:`
- [`src/nodes/identity_matcher_node.hpp:166`](../src/nodes/identity_matcher_node.hpp#L166) — `std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);`
### AR-004
**Locations:** 4
- [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;`
- [`src/main.cpp:299`](../src/main.cpp#L299) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:166`](../src/nodes/identity_matcher_node.hpp#L166) — `std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);`
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
### AR-005
**Locations:** 2
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown`
- [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown`
### AR-006
**Locations:** 1
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `inline cv::Mat align_face(const cv::Mat& img,`
- [`src/nodes/embedder_node.hpp:21`](../src/nodes/embedder_node.hpp#L21) — `struct EmbedderFunc`
### AR-007
**Locations:** 3
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
### AR-008
**Locations:** 3
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
### AR-009
**Locations:** 1
- [`src/nodes/camera_position_change_detector_node.hpp:30`](../src/nodes/camera_position_change_detector_node.hpp#L30) — `struct CameraPositionChangeDetectorFunc`
### AR-010
**Locations:** 9
- [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;`
- [`src/main.cpp:309`](../src/main.cpp#L309) — `Unknown`
- [`src/main.cpp:379`](../src/main.cpp#L379) — `return run_net(std::move(net));`
- [`src/main.cpp:411`](../src/main.cpp#L411) — `Unknown`
- [`src/nodes/scene_boundary_annotator_node.hpp:2`](../src/nodes/scene_boundary_annotator_node.hpp#L2) — `Unknown`
- [`src/nodes/scene_detector_node.hpp:36`](../src/nodes/scene_detector_node.hpp#L36) — `static constexpr std::string_view label() { return "scene_detector"; }`
- [`src/nodes/scene_detector_node.hpp:110`](../src/nodes/scene_detector_node.hpp#L110) — `void flush_remaining()`
- [`src/nodes/scene_detector_node.hpp:144`](../src/nodes/scene_detector_node.hpp#L144) — `void write_output()`
- [`src/scene_boundaries.hpp:2`](../src/scene_boundaries.hpp#L2) — `Unknown`
### AR-012
**Locations:** 8
**Locations:** 9
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/identity_matcher_node.hpp:119`](../src/nodes/identity_matcher_node.hpp#L119) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:242`](../src/nodes/identity_matcher_node.hpp#L242) — `Unknown`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/identity_matcher_node.hpp:125`](../src/nodes/identity_matcher_node.hpp#L125) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:272`](../src/nodes/identity_matcher_node.hpp#L272) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-013
**Locations:** 2
**Locations:** 3
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-014
@@ -209,7 +264,7 @@ _None._
**Locations:** 4
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/result_sink_node.hpp:63`](../src/nodes/result_sink_node.hpp#L63) — `void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
@@ -222,32 +277,71 @@ _None._
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-018
**Locations:** 3
- [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown`
- [`src/gallery/track_gallery.hpp:161`](../src/gallery/track_gallery.hpp#L161) — `struct TrackState`
- [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);`
### AR-019
**Locations:** 3
- [`src/gallery/track_gallery.hpp:122`](../src/gallery/track_gallery.hpp#L122) — `void forget(int track_id) { tracks_.erase(track_id); }`
- [`src/nodes/identity_matcher_node.hpp:147`](../src/nodes/identity_matcher_node.hpp#L147) — `MatchedSceneFrame operator()(TrackedSceneFrame tf)`
- [`src/nodes/identity_matcher_node.hpp:285`](../src/nodes/identity_matcher_node.hpp#L285) — `Unknown`
### AR-023
**Locations:** 3
- [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }`
### AR-024
**Locations:** 6
**Locations:** 10
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown`
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/gallery/track_gallery.hpp:132`](../src/gallery/track_gallery.hpp#L132) — `void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }`
- [`src/gallery/track_gallery.hpp:161`](../src/gallery/track_gallery.hpp#L161) — `struct TrackState`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);`
- [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }`
### AR-025
**Locations:** 3
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:119`](../src/nodes/identity_matcher_node.hpp#L119) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:242`](../src/nodes/identity_matcher_node.hpp#L242) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:125`](../src/nodes/identity_matcher_node.hpp#L125) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:272`](../src/nodes/identity_matcher_node.hpp#L272) — `Unknown`
### AR-026
**Locations:** 1
- [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;`
### AR-027
**Locations:** 1
- [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;`
### AR-030
**Locations:** 2
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown`
- [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown`
### DP-001
@@ -273,11 +367,29 @@ _None._
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### GR-003
**Locations:** 13
- [`src/build_gallery.cpp:83`](../src/build_gallery.cpp#L83) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:80`](../src/gallery/gallery_calibration.hpp#L80) — `struct GalleryCalibrationStats`
- [`src/gallery/gallery_calibration.hpp:145`](../src/gallery/gallery_calibration.hpp#L145) — `std::vector<bool> actor_eligible(n_actors, false);`
- [`src/gallery/gallery_calibration.hpp:294`](../src/gallery/gallery_calibration.hpp#L294) — `Unknown`
- [`src/gallery/gallery_report.hpp:2`](../src/gallery/gallery_report.hpp#L2) — `Unknown`
- [`src/gallery/gallery_report.hpp:52`](../src/gallery/gallery_report.hpp#L52) — `struct GalleryBuildAudit`
- [`src/gallery/gallery_report.hpp:73`](../src/gallery/gallery_report.hpp#L73) — `struct GalleryReport`
- [`src/gallery/gallery_report.hpp:154`](../src/gallery/gallery_report.hpp#L154) — `inline GalleryReport build_gallery_report(const ActorGallery& gallery,`
- [`src/gallery/gallery_report.hpp:295`](../src/gallery/gallery_report.hpp#L295) — `inline nlohmann::json gallery_report_to_json(const GalleryReport& r)`
- [`src/gallery/gallery_report.hpp:361`](../src/gallery/gallery_report.hpp#L361) — `inline GalleryReport gallery_report_from_json(const nlohmann::json& j)`
- [`src/gallery/gallery_report.hpp:446`](../src/gallery/gallery_report.hpp#L446) — `inline void save_gallery_report(const std::string& path, const GalleryReport& r)`
- [`src/gallery/gallery_report.hpp:454`](../src/gallery/gallery_report.hpp#L454) — `inline GalleryReport load_gallery_report(const std::string& path)`
- [`src/gallery/gallery_report.hpp:464`](../src/gallery/gallery_report.hpp#L464) — `return gallery_report_from_json(j);`
### GR-004
**Locations:** 44
- [`src/config.hpp:49`](../src/config.hpp#L49) — `Unknown`
- [`src/config.hpp:54`](../src/config.hpp#L54) — `Unknown`
- [`src/gallery/embedder_stamp.cpp:1`](../src/gallery/embedder_stamp.cpp#L1) — `Unknown`
- [`src/gallery/embedder_stamp.hpp:2`](../src/gallery/embedder_stamp.hpp#L2) — `Unknown`
- [`src/gallery/gallery_builder.cpp:45`](../src/gallery/gallery_builder.cpp#L45) — `ActorGallery build_gallery(const BuildConfig& cfg)`
@@ -286,11 +398,11 @@ _None._
- [`src/gallery/gallery_store.cpp:219`](../src/gallery/gallery_store.cpp#L219) — `Unknown`
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:174`](../src/main.cpp#L174) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/main.cpp:188`](../src/main.cpp#L188) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:127`](../src/nodes/embedding_dump_node.hpp#L127) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:238`](../src/nodes/embedding_dump_node.hpp#L238) — `H5::H5File file(path_, H5F_ACC_TRUNC);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:141`](../src/types.hpp#L141) — `struct Actor`
- [`src/types.hpp:148`](../src/types.hpp#L148) — `struct Actor`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");`
@@ -310,18 +422,24 @@ _None._
- [`scripts/movienet_eval.py:65`](../scripts/movienet_eval.py#L65) — `with open(args.gt) as f:`
- [`scripts/optimizer/fetch_missing_actors.py:62`](../scripts/optimizer/fetch_missing_actors.py#L62) — `def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,`
- [`scripts/optimizer/fetch_missing_actors.py:109`](../scripts/optimizer/fetch_missing_actors.py#L109) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:123`](../scripts/optimizer/fetch_missing_actors.py#L123) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:124`](../scripts/optimizer/fetch_missing_actors.py#L124) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/optimize.py:186`](../scripts/optimizer/optimize.py#L186) — `Unknown`
- [`scripts/optimizer/optimize.py:202`](../scripts/optimizer/optimize.py#L202) — `if not Path(f["dump"]).exists():`
- [`scripts/optimizer/reembed_gallery.py:62`](../scripts/optimizer/reembed_gallery.py#L62) — `for i, a in enumerate(ref["actors"], 1):`
- [`scripts/optimizer/replay.py:113`](../scripts/optimizer/replay.py#L113) — `Unknown`
- [`scripts/optimizer/replay.py:252`](../scripts/optimizer/replay.py#L252) — `Unknown`
- [`scripts/sae_embed_loader.py:22`](../scripts/sae_embed_loader.py#L22) — `return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)`
- [`scripts/optimizer/replay.py:253`](../scripts/optimizer/replay.py#L253) — `Unknown`
- [`scripts/sae_embed_loader.py:23`](../scripts/sae_embed_loader.py#L23) — `def resolve_arcface(models_dir: str, arcface: str \| None = None) -> str:`
- [`scripts/sae_gallery.py:171`](../scripts/sae_gallery.py#L171) — `if not _stamp_empty(embedder):`
- [`scripts/sae_gallery.py:199`](../scripts/sae_gallery.py#L199) — `for a in range(len(offset)):`
- [`scripts/sae_gallery.py:200`](../scripts/sae_gallery.py#L200) — `for a in range(len(offset)):`
- [`scripts/sae_stamp.py:3`](../scripts/sae_stamp.py#L3) — `Unknown`
- [`scripts/stamp_gallery.py:4`](../scripts/stamp_gallery.py#L4) — `Unknown`
### GR-005
**Locations:** 1
- [`src/gallery/gallery_store.hpp:15`](../src/gallery/gallery_store.hpp#L15) — `Unknown`
### IR-001
**Locations:** 1
@@ -333,7 +451,7 @@ _None._
**Locations:** 5
- [`src/config.hpp:20`](../src/config.hpp#L20) — `struct Config`
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:122`](../src/nodes/result_sink_node.hpp#L122) — `void write_output()`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
@@ -342,12 +460,13 @@ _None._
**Locations:** 1
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
### IR-004
**Locations:** 16
**Locations:** 18
- [`src/audio_bindings.cpp:3`](../src/audio_bindings.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:265`](../src/audio_signature.cpp#L265) — `std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:320`](../src/audio_signature.cpp#L320) — `std::optional<std::string> signature_from_mono(const std::vector<float>& mono)`
@@ -364,11 +483,13 @@ _None._
- [`tests/test_audio_signature.cpp:314`](../tests/test_audio_signature.cpp#L314) — `kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));`
- [`tests/test_audio_signature.cpp:328`](../tests/test_audio_signature.cpp#L328) — `std::vector<float> a(kWindowSamples / 50);`
- [`tests/test_audio_signature.cpp:344`](../tests/test_audio_signature.cpp#L344) — `return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());`
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### IR-005
**Locations:** 5
**Locations:** 6
- [`src/audio_bindings.cpp:3`](../src/audio_bindings.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
@@ -406,13 +527,24 @@ _None._
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
### IT-001
**Locations:** 1
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
### PR-002
**Locations:** 3
**Locations:** 8
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:18`](../src/nodes/embedding_dump_node.hpp#L18) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:133`](../src/nodes/embedding_dump_node.hpp#L133) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:159`](../src/nodes/embedding_dump_node.hpp#L159) — `void operator()(EmbeddedSceneFrame ef)`
- [`src/nodes/embedding_dump_node.hpp:242`](../src/nodes/embedding_dump_node.hpp#L242) — `H5::H5File file(path_, H5F_ACC_TRUNC);`
- [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an`
- [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown`
- [`scripts/validation/ground_truth.py:24`](../scripts/validation/ground_truth.py#L24) — `Unknown`
### PR-004
@@ -422,22 +554,36 @@ _None._
### SR-001
**Locations:** 46
**Locations:** 60
- [`src/config.hpp:49`](../src/config.hpp#L49) — `Unknown`
- [`src/backends/gemm_backend.cpp:44`](../src/backends/gemm_backend.cpp#L44) — `constexpr int kDim = 512;`
- [`src/build_gallery.cpp:83`](../src/build_gallery.cpp#L83) — `Unknown`
- [`src/config.hpp:54`](../src/config.hpp#L54) — `Unknown`
- [`src/gallery/embedder_stamp.cpp:1`](../src/gallery/embedder_stamp.cpp#L1) — `Unknown`
- [`src/gallery/embedder_stamp.hpp:2`](../src/gallery/embedder_stamp.hpp#L2) — `Unknown`
- [`src/gallery/gallery_builder.cpp:45`](../src/gallery/gallery_builder.cpp#L45) — `ActorGallery build_gallery(const BuildConfig& cfg)`
- [`src/gallery/gallery_calibration.hpp:80`](../src/gallery/gallery_calibration.hpp#L80) — `struct GalleryCalibrationStats`
- [`src/gallery/gallery_calibration.hpp:145`](../src/gallery/gallery_calibration.hpp#L145) — `std::vector<bool> actor_eligible(n_actors, false);`
- [`src/gallery/gallery_calibration.hpp:294`](../src/gallery/gallery_calibration.hpp#L294) — `Unknown`
- [`src/gallery/gallery_report.hpp:2`](../src/gallery/gallery_report.hpp#L2) — `Unknown`
- [`src/gallery/gallery_report.hpp:52`](../src/gallery/gallery_report.hpp#L52) — `struct GalleryBuildAudit`
- [`src/gallery/gallery_report.hpp:73`](../src/gallery/gallery_report.hpp#L73) — `struct GalleryReport`
- [`src/gallery/gallery_report.hpp:154`](../src/gallery/gallery_report.hpp#L154) — `inline GalleryReport build_gallery_report(const ActorGallery& gallery,`
- [`src/gallery/gallery_report.hpp:295`](../src/gallery/gallery_report.hpp#L295) — `inline nlohmann::json gallery_report_to_json(const GalleryReport& r)`
- [`src/gallery/gallery_report.hpp:361`](../src/gallery/gallery_report.hpp#L361) — `inline GalleryReport gallery_report_from_json(const nlohmann::json& j)`
- [`src/gallery/gallery_report.hpp:446`](../src/gallery/gallery_report.hpp#L446) — `inline void save_gallery_report(const std::string& path, const GalleryReport& r)`
- [`src/gallery/gallery_report.hpp:454`](../src/gallery/gallery_report.hpp#L454) — `inline GalleryReport load_gallery_report(const std::string& path)`
- [`src/gallery/gallery_report.hpp:464`](../src/gallery/gallery_report.hpp#L464) — `return gallery_report_from_json(j);`
- [`src/gallery/gallery_store.cpp:82`](../src/gallery/gallery_store.cpp#L82) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/gallery/gallery_store.cpp:167`](../src/gallery/gallery_store.cpp#L167) — `H5::DataSpace scalar(H5S_SCALAR);`
- [`src/gallery/gallery_store.cpp:219`](../src/gallery/gallery_store.cpp#L219) — `Unknown`
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:174`](../src/main.cpp#L174) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/main.cpp:188`](../src/main.cpp#L188) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:127`](../src/nodes/embedding_dump_node.hpp#L127) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:238`](../src/nodes/embedding_dump_node.hpp#L238) — `H5::H5File file(path_, H5F_ACC_TRUNC);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:141`](../src/types.hpp#L141) — `struct Actor`
- [`src/types.hpp:148`](../src/types.hpp#L148) — `struct Actor`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");`
@@ -458,44 +604,61 @@ _None._
- [`scripts/movienet_eval.py:65`](../scripts/movienet_eval.py#L65) — `with open(args.gt) as f:`
- [`scripts/optimizer/fetch_missing_actors.py:62`](../scripts/optimizer/fetch_missing_actors.py#L62) — `def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,`
- [`scripts/optimizer/fetch_missing_actors.py:109`](../scripts/optimizer/fetch_missing_actors.py#L109) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:123`](../scripts/optimizer/fetch_missing_actors.py#L123) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:124`](../scripts/optimizer/fetch_missing_actors.py#L124) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/optimize.py:186`](../scripts/optimizer/optimize.py#L186) — `Unknown`
- [`scripts/optimizer/optimize.py:202`](../scripts/optimizer/optimize.py#L202) — `if not Path(f["dump"]).exists():`
- [`scripts/optimizer/reembed_gallery.py:62`](../scripts/optimizer/reembed_gallery.py#L62) — `for i, a in enumerate(ref["actors"], 1):`
- [`scripts/optimizer/replay.py:113`](../scripts/optimizer/replay.py#L113) — `Unknown`
- [`scripts/optimizer/replay.py:252`](../scripts/optimizer/replay.py#L252) — `Unknown`
- [`scripts/optimizer/replay.py:253`](../scripts/optimizer/replay.py#L253) — `Unknown`
- [`scripts/run_from_jellyfin.py:4`](../scripts/run_from_jellyfin.py#L4) — `Unknown`
- [`scripts/sae_embed_loader.py:22`](../scripts/sae_embed_loader.py#L22) — `return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)`
- [`scripts/sae_embed_loader.py:23`](../scripts/sae_embed_loader.py#L23) — `def resolve_arcface(models_dir: str, arcface: str \| None = None) -> str:`
- [`scripts/sae_gallery.py:171`](../scripts/sae_gallery.py#L171) — `if not _stamp_empty(embedder):`
- [`scripts/sae_gallery.py:199`](../scripts/sae_gallery.py#L199) — `for a in range(len(offset)):`
- [`scripts/sae_gallery.py:200`](../scripts/sae_gallery.py#L200) — `for a in range(len(offset)):`
- [`scripts/sae_stamp.py:3`](../scripts/sae_stamp.py#L3) — `Unknown`
- [`scripts/stamp_gallery.py:4`](../scripts/stamp_gallery.py#L4) — `Unknown`
### SR-002
**Locations:** 16
**Locations:** 32
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/config.hpp:44`](../src/config.hpp#L44) — `Unknown`
- [`src/config.hpp:108`](../src/config.hpp#L108) — `Unknown`
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `inline cv::Mat align_face(const cv::Mat& img,`
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:86`](../src/main.cpp#L86) — `static constexpr std::size_t kSceneJoinDepth = 256;`
- [`src/main.cpp:213`](../src/main.cpp#L213) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:230`](../src/main.cpp#L230) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:299`](../src/main.cpp#L299) — `Unknown`
- [`src/main.cpp:309`](../src/main.cpp#L309) — `Unknown`
- [`src/main.cpp:379`](../src/main.cpp#L379) — `return run_net(std::move(net));`
- [`src/main.cpp:411`](../src/main.cpp#L411) — `Unknown`
- [`src/nodes/camera_position_change_detector_node.hpp:30`](../src/nodes/camera_position_change_detector_node.hpp#L30) — `struct CameraPositionChangeDetectorFunc`
- [`src/nodes/embedder_node.hpp:21`](../src/nodes/embedder_node.hpp#L21) — `struct EmbedderFunc`
- [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown`
- [`src/nodes/face_detector_node.hpp:47`](../src/nodes/face_detector_node.hpp#L47) — `private:`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:119`](../src/nodes/identity_matcher_node.hpp#L119) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:242`](../src/nodes/identity_matcher_node.hpp#L242) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:117`](../src/nodes/identity_matcher_node.hpp#L117) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:125`](../src/nodes/identity_matcher_node.hpp#L125) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:166`](../src/nodes/identity_matcher_node.hpp#L166) — `std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);`
- [`src/nodes/identity_matcher_node.hpp:272`](../src/nodes/identity_matcher_node.hpp#L272) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:63`](../src/nodes/result_sink_node.hpp#L63) — `void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
- [`src/nodes/scene_boundary_annotator_node.hpp:2`](../src/nodes/scene_boundary_annotator_node.hpp#L2) — `Unknown`
- [`src/nodes/scene_detector_node.hpp:36`](../src/nodes/scene_detector_node.hpp#L36) — `static constexpr std::string_view label() { return "scene_detector"; }`
- [`src/nodes/scene_detector_node.hpp:110`](../src/nodes/scene_detector_node.hpp#L110) — `void flush_remaining()`
- [`src/nodes/scene_detector_node.hpp:144`](../src/nodes/scene_detector_node.hpp#L144) — `void write_output()`
- [`src/scene_boundaries.hpp:2`](../src/scene_boundaries.hpp#L2) — `Unknown`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_face_utils.cpp:1`](../tests/test_face_utils.cpp#L1) — `Unknown`
### SR-003
**Locations:** 6
**Locations:** 7
- [`src/audio_bindings.cpp:3`](../src/audio_bindings.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`src/config.hpp:20`](../src/config.hpp#L20) — `struct Config`
@@ -505,8 +668,16 @@ _None._
### SR-005
**Locations:** 1
**Locations:** 9
- [`src/config.hpp:152`](../src/config.hpp#L152) — `Unknown`
- [`src/gallery/gallery_store.hpp:15`](../src/gallery/gallery_store.hpp#L15) — `Unknown`
- [`src/gallery/track_gallery.hpp:122`](../src/gallery/track_gallery.hpp#L122) — `void forget(int track_id) { tracks_.erase(track_id); }`
- [`src/gallery/track_gallery.hpp:132`](../src/gallery/track_gallery.hpp#L132) — `void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }`
- [`src/gallery/track_gallery.hpp:161`](../src/gallery/track_gallery.hpp#L161) — `struct TrackState`
- [`src/nodes/identity_matcher_node.hpp:110`](../src/nodes/identity_matcher_node.hpp#L110) — `std::vector<float> host_gallery(static_cast<size_t>(n_gallery_) * 512);`
- [`src/nodes/identity_matcher_node.hpp:147`](../src/nodes/identity_matcher_node.hpp#L147) — `MatchedSceneFrame operator()(TrackedSceneFrame tf)`
- [`src/nodes/identity_matcher_node.hpp:285`](../src/nodes/identity_matcher_node.hpp#L285) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### UT-001
@@ -552,16 +723,42 @@ _None._
- [`tests/test_audio_signature.cpp:328`](../tests/test_audio_signature.cpp#L328) — `std::vector<float> a(kWindowSamples / 50);`
- [`tests/test_audio_signature.cpp:344`](../tests/test_audio_signature.cpp#L344) — `return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());`
### VR-001
### UT-105
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### UT-106
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### UT-107
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### UT-108
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
### VR-001
**Locations:** 2
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
### VR-002
**Locations:** 1
**Locations:** 2
- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown`
- [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an`
### VR-003
@@ -570,27 +767,31 @@ _None._
- [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown`
## Tag diagnostics
### VR-004
**Malformed tags:**
**Locations:** 1
- `scripts/filter_gallery.py:80` — {'ignored': ['— a filtered gallery holds the SAME vectors as its']}
- `scripts/make_gallery.py:181` — {'ignored': ['— stamp with the model actually loaded', 'resolved']}
- `scripts/make_jellyfin_gallery.py:456` — {'ignored': ["— --merge keeps the existing actors' vectors and"]}
- `scripts/movienet_eval.py:65` — {'ignored': ['— match() below is a bare dot product against the']}
- `scripts/optimizer/fetch_missing_actors.py:109` — {'ignored': ['— the legacy JSON gallery carries the same stamp as']}
- `scripts/optimizer/fetch_missing_actors.py:123` — {'ignored': ['— merging two galleries from different models makes']}
- `scripts/optimizer/optimize.py:202` — {'ignored': ['— every (dump', 'gallery) pair is checked ONCE here']}
- `scripts/optimizer/reembed_gallery.py:62` — {'ignored': ['— this script exists to produce a gallery in a']}
- `scripts/optimizer/replay.py:113` — {'ignored': ['— checked here', 'before any network is built', 'so a']}
- `scripts/optimizer/replay.py:252` — {'ignored': ['— promote an unprovable gallery/dump binding from a']}
- `scripts/sae_embed_loader.py:22` — {'ignored': ['— single source of truth for "which model is this"']}
- `scripts/sae_gallery.py:171` — {'ignored': ['— omitted entirely when unknown', 'so "unstamped"']}
- `scripts/sae_gallery.py:199` — {'ignored': ['— carried through so a derived gallery (filter']}
- [`scripts/validation/ground_truth.py:24`](../scripts/validation/ground_truth.py#L24) — `Unknown`
**Groups mixing requirement types (pipe separates types):**
### VR-005
- `src/main.cpp:216` — {'group': ['AR-012', 'AR-016', 'IR-002', 'IR-003']}
- `src/nodes/result_sink_node.hpp:49` — {'group': ['AR-012', 'AR-017', 'IR-002']}
- `src/nodes/result_sink_node.hpp:161` — {'group': ['AR-012', 'IR-002']}
**Locations:** 1
- [`scripts/validation/min_face_size.py:5`](../scripts/validation/min_face_size.py#L5) — `Unknown`
### VR-010
**Locations:** 5
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:18`](../src/nodes/embedding_dump_node.hpp#L18) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:133`](../src/nodes/embedding_dump_node.hpp#L133) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:159`](../src/nodes/embedding_dump_node.hpp#L159) — `void operator()(EmbeddedSceneFrame ef)`
- [`src/nodes/embedding_dump_node.hpp:242`](../src/nodes/embedding_dump_node.hpp#L242) — `H5::H5File file(path_, H5F_ACC_TRUNC);`
### VR-014
**Locations:** 1
- [`scripts/validation/test_audio_offset.py:5`](../scripts/validation/test_audio_offset.py#L5) — `The golden vector (IR-005) proves the *arithmetic* is identical in both`
experiments/.gitignore
+11
View File
@@ -11,6 +11,17 @@ manifests/
trajectories/
results/
# Cross-source identification study: source clips and the hand-sorted face
# crops. The sorting is human ground truth and expensive to redo, so it goes to
# the artifact registry rather than being regenerated — push it once sorted.
xsource/clips/
xsource/labelling/
xsource/frames/
xsource/cache/
xsource/results_*.json
xsource/failure_analysis.json
xsource/*.jpg
# Raw run logs and scratch scripts (regenerated by every run).
_scratch/
experiments/xsource/README.md
+92
View File
@@ -0,0 +1,92 @@
# xsource — cross-source identification probe (VR-013)
Gallery from **one** recording, probes from **another**, swept over the probe's
input resolution. Complements VR-005, which asked the same question over gallery
mugshots: that one degrades an already-aligned 112×112 crop, holding alignment
perfect, so it isolates the embedder. This one downscales the **whole frame**
before the detector, so detection and landmark regression degrade with it.
Corpus: two Pexels clips of one shoot (4096×2160, 25 fps), four people, all four
present in both.
## Getting the data
Clips, frames and hand-sorted crops are gitignored; they live in the artifact
registry.
scripts/artifacts/pull_artifacts.sh xsource # clips + labelling, frames regenerated
scripts/artifacts/push_artifacts.sh xsource # after correcting labels
Pulling fetches the two clips and the hand-sorted crops, then regenerates the
frames with ffmpeg — ~320 MB of PNG that is deterministic from the clips, so it
is not worth shipping. Extraction settings are pinned in the pull script because
the manifests key on frame filenames *and* on detection order within each frame;
`verify_labels.py` runs at the end and will fail loudly if they drift.
Pull never overwrites an existing `labelling/`. That directory is human ground
truth — somebody looked at all 167 crops and put each one in a folder — and it
is the expensive part of this study, so push it once corrected.
Clips are Pexels-licensed: free to use, no attribution required, but not
CC or MIT. Fine as a frozen CI artifact on private infrastructure; do not
redistribute them as stock content.
## Scripts
| script | does |
|---|---|
| `dump_faces.py` | detect every face, write a context crop per detection + a manifest |
| `redraw_boxes.py` | redraw those crops with the detection boxed, in place |
| `propose_labels.py` | propose labels for one clip from another clip's hand-sorted folders |
| `make_review_site.py` | local `review.html` — current label, crop, better match, correct and export |
| `apply_corrections.py` | apply the exported `corrections.json` |
| `verify_labels.py` | integrity gate: index consistency, duplicates, separation. Exits non-zero on failure |
| `resolution_sweep.py` | the VR-013 measurement |
| `failure_analysis.py` | what explains the misses — pose, size, blur, detector confidence |
| `landmark_voting.py` | average SCRFD's overlapping detections instead of discarding them |
| `pose_label.py` | mesh-estimated head pose, for hand correction (feeds VR-012) |
Everything drives the shipped C++ through `sae_embed`; nothing reimplements
detection, alignment, the embedder or the calibration. Scoring goes through the
production gallery sigmoid — never a raw cosine (AR-024).
LD_PRELOAD=/usr/lib/libcudnn_cnn.so.9 python3 resolution_sweep.py
The preload is needed while ORT's CUDA provider looks for
`cudnnGetConvolutionBackwardDataAlgorithm_v7`, which cuDNN 9 moved into
`libcudnn_cnn.so.9` behind a dispatch stub. Without it everything silently falls
back to CPU.
## What it found
**Resolution is not the binding constraint here.** TPI holds ~41–47% from 4096×2160
down to ~45 px faces, then falls: 23 px → 26%, 18 px → 12%, 14 px → 1.5%. Holding
90% of the plateau needs roughly 50 px end to end, against VR-005's ~22 px — the
gap is detection and landmark error, which VR-005 excludes by construction.
**FPI is 0.0% at every scale.** Resolution loss goes entirely to TBI: the pipeline
stops naming people rather than naming the wrong one.
**The ceiling is cross-view, not resolution.** Every person matches themselves
strongly *within* a recording (sim 0.55–0.85) and collapses *across* the two
(0.14–0.45, threshold 0.335). Only the person with frontal **gallery** references
identified reliably, whatever their probe pose — so the lever is gallery pose
coverage (`docs/pose-expansion.md`), not a better landmark model.
**Landmark voting helps.** SCRFD predicts each face from several anchors and NMS
discards all but one, throwing away a median of 3 landmark estimates per face.
Averaging them, weighted by confidence, lifts cross-clip TPI 41% → 49% for one
forward pass and no extra model. A MediaPipe mesh as landmark source went the
other way (41% → 16%): more stable within a recording, but a ring centroid is not
the annotated landmark ArcFace was trained on, and the embedder punishes the
off-distribution crop.
## Reading these numbers
Four identities, 70 probes, one shoot. The ~47% plateau is pose, not resolution —
half these faces are turned away and never clear threshold at any scale, so the
absolute rates say little and the *shape* is the result. Both clips contain all
four people, so there is no out-of-gallery class and the 10×-weighted out-of-cast
misID is **untested** here; holding one identity out of the gallery would fix
that. And the resolution curve is dominated by the single subject whose gallery
references are frontal.
experiments/xsource/apply_corrections.py
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env python3
"""Apply corrections.json exported from review.html.
python3 apply_corrections.py ~/Downloads/corrections.json [--dry-run]
Moves each crop to the folder you chose. "discard" goes to labelling/<clip>/discard/,
which the sweep ignores — nothing is deleted, so a misclick is recoverable.
Refuses to move a file it cannot find exactly once, rather than guessing: a
half-applied correction set would put a crop in two folders and quietly
duplicate a label.
"""
import sys, json, glob, os, shutil
if len(sys.argv) < 2:
sys.exit(__doc__)
path = sys.argv[1]
DRY = "--dry-run" in sys.argv
corr = json.load(open(path))
if not corr:
sys.exit("no corrections in that file")
moved = skipped = 0
for fname, c in corr.items():
clip, to = c["clip"], c["to"]
hits = glob.glob(f"labelling/{clip}/**/{fname}", recursive=True)
if len(hits) != 1:
print(f"[skip] {fname}: found {len(hits)} copies, expected 1")
skipped += 1
continue
src = hits[0]
dst_dir = f"labelling/{clip}/{to}"
dst = f"{dst_dir}/{fname}"
if os.path.abspath(src) == os.path.abspath(dst):
continue
print(f"{'would move' if DRY else 'move'} {c['from']} -> {to}: {fname}")
if not DRY:
os.makedirs(dst_dir, exist_ok=True)
shutil.move(src, dst)
moved += 1
print(f"\n{moved} moved, {skipped} skipped{' (dry run)' if DRY else ''}")
if not DRY and moved:
print("re-run verify_labels.py to confirm the set is still consistent")
experiments/xsource/dump_faces.py
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env python3
"""Dump face crops from both clips for hand-labelling.
Writes labelling/<clip>/unsorted/<name>.jpg — a context crop around each
detection, big enough to recognise a person by eye. Move them into
labelling/<clip>/person_A/, person_B/, ... and the sweep reads those folders as
ground truth.
Filenames carry a cNN_ cluster-hint prefix so visually similar faces sort next
to each other in a file manager. The hint is only an ordering convenience —
the folder you drop a file into is what counts, and the sweep never reads the
prefix.
Detection and alignment run through the shipped C++ (sae_embed). Every crop
keeps its clip, frame and native-resolution bbox in manifest.json, so probe
detections at reduced scale can be tied back to a labelled face geometrically,
by position, rather than by embedding similarity — which would be circular.
"""
import sys, glob, json, os, shutil
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
M = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/models/"
CLIPS = ["5157339", "5157344"]
MIN_PX = 60
CTX = 256 # context-crop side, for human recognisability
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "arcface_w600k_r50.onnx",
conf=0.5, nms=0.4, max_side=0)
for clip in CLIPS:
out_dir = f"labelling/{clip}/unsorted"
if os.path.isdir(f"labelling/{clip}"):
print(f"[skip] labelling/{clip} exists — not overwriting your sorting",
file=sys.stderr)
continue
os.makedirs(out_dir, exist_ok=True)
entries = []
for p in sorted(glob.glob(f"pex/d{clip}_*.png")):
frame = p.rsplit("_", 1)[-1].split(".")[0]
img = cv2.imread(p)
for i, d in enumerate(eng.detect(img)):
x, y, w, h = d.bbox
if min(w, h) < MIN_PX:
continue
lm = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
emb = np.asarray(eng.embed_crop(crop), dtype=np.float32)
pad = int(0.5 * max(w, h))
x0, y0 = max(0, int(x) - pad), max(0, int(y) - pad)
x1, y1 = min(img.shape[1], int(x + w) + pad), min(img.shape[0], int(y + h) + pad)
ctx = cv2.resize(img[y0:y1, x0:x1], (CTX, CTX))
entries.append({"clip": clip, "frame": frame, "idx": i,
"bbox": [float(x), float(y), float(w), float(h)],
"px": float(min(w, h)), "conf": float(d.confidence),
"emb": emb, "ctx": ctx})
# cluster hint only — greedy, purely to group similar faces in the file list
E = np.stack([e["emb"] for e in entries])
hint = -np.ones(len(entries), int)
k = 0
for i in range(len(entries)):
if hint[i] >= 0:
continue
hint[i] = k
for j in range(i + 1, len(entries)):
if hint[j] < 0 and float(E[i] @ E[j]) > 0.5:
hint[j] = k
k += 1
manifest = []
for e, h in zip(entries, hint):
name = f"c{h:02d}_{e['clip']}_f{e['frame']}_i{e['idx']}_{int(e['px'])}px.jpg"
cv2.imwrite(f"{out_dir}/{name}", e["ctx"])
manifest.append({k: v for k, v in e.items() if k not in ("emb", "ctx")}
| {"file": name, "cluster_hint": int(h)})
json.dump(manifest, open(f"labelling/{clip}/manifest.json", "w"), indent=1)
print(f"[{clip}] {len(manifest)} crops in {out_dir}, {k} cluster hints, "
f"face px {min(m['px'] for m in manifest):.0f}–{max(m['px'] for m in manifest):.0f}",
file=sys.stderr)
experiments/xsource/failure_analysis.py
+174
View File
@@ -0,0 +1,174 @@
#!/usr/bin/env python3
"""What explains the misses? Head pose, face size, blur, detector confidence.
For every hand-labelled probe face, computes the calibrated probability against
its OWN gallery entry — so a low value is a false negative, not a mistake about
who it is — and pairs it with covariates that might explain the failure.
Head pose comes from solvePnP of the 5 landmarks against a canonical 3D face,
giving yaw/pitch/roll in degrees.
CAVEAT, and it matters: the pose estimate is derived from the same 5
landmarks the alignment uses. Where those landmarks are unreliable the pose
estimate is unreliable too, and both degrade for the same reason. So this
can show that failures concentrate at high yaw; it cannot cleanly separate
"the head was turned" from "the landmarks were wrong because the head was
turned". Those are the same physical cause, but not the same fix — the
first argues for gallery pose coverage, the second for a better landmark
source.
A sanity check is printed first: pose is estimated per person, and if it does
not recover what is visible in the review sheets (one subject frontal, another
in profile, another looking down) then the estimate is not worth reading.
Similarities go through the production gallery sigmoid, never compared raw.
"""
import sys, glob, json, os
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
GALLERY_CLIP, PROBE_CLIP = "5157344", "5157339"
PROB_THRESHOLD = 0.754
# Canonical 3D face, ordered as types.hpp:60 —
# [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth.
# The subject's right eye sits to the LEFT in image space, hence the negative X.
FACE_3D = np.array([
(-34.0, 35.0, -28.0),
( 34.0, 35.0, -28.0),
( 0.0, 0.0, 0.0),
(-26.0, -32.0, -25.0),
( 26.0, -32.0, -25.0),
], dtype=np.float64)
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.5, nms=0.4, max_side=0)
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
def head_pose(lm, w, h):
"""yaw, pitch, roll in degrees. Focal length assumed = image width."""
cam = np.array([[w, 0, w / 2], [0, w, h / 2], [0, 0, 1]], dtype=np.float64)
ok, rvec, _ = cv2.solvePnP(FACE_3D, lm.astype(np.float64), cam, None,
flags=cv2.SOLVEPNP_EPNP)
if not ok:
return None
R, _ = cv2.Rodrigues(rvec)
sy = np.sqrt(R[0, 0] ** 2 + R[1, 0] ** 2)
if sy > 1e-6:
pitch = np.degrees(np.arctan2(-R[2, 0], sy))
yaw = np.degrees(np.arctan2(R[1, 0], R[0, 0]))
roll = np.degrees(np.arctan2(R[2, 1], R[2, 2]))
else:
pitch = np.degrees(np.arctan2(-R[2, 0], sy)); yaw = 0.0
roll = np.degrees(np.arctan2(-R[1, 2], R[1, 1]))
# solvePnP's yaw wraps near +/-180 for a face pointing at the camera;
# fold it to a "degrees away from frontal" magnitude.
yaw = ((yaw + 180) % 360) - 180
if abs(yaw) > 90:
yaw = np.sign(yaw) * (180 - abs(yaw))
return yaw, pitch, roll
def collect(clip):
lab = {os.path.basename(p): os.path.basename(os.path.dirname(p))
for p in glob.glob(f"labelling/{clip}/*/*.jpg")
if os.path.basename(os.path.dirname(p)) not in ("discard", "unsorted")}
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
rows = []
for frame in sorted({man[f]["frame"] for f in lab}):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
dets = eng.detect(img)
H, W = img.shape[:2]
for f, person in lab.items():
m = man[f]
if m["frame"] != frame or m["idx"] >= len(dets):
continue
d = dets[m["idx"]]
lm = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
pose = head_pose(lm, W, H)
x, y, w, h = d.bbox
g = cv2.cvtColor(np.asarray(crop), cv2.COLOR_BGR2GRAY)
rows.append({
"person": person, "px": float(min(w, h)), "conf": float(d.confidence),
"yaw": pose[0] if pose else np.nan, "pitch": pose[1] if pose else np.nan,
"roll": pose[2] if pose else np.nan,
"blur": float(cv2.Laplacian(g, cv2.CV_64F).var()),
"emb": np.asarray(eng.embed_crop(crop), dtype=np.float32)})
return rows
gal_rows = collect(GALLERY_CLIP)
prb_rows = collect(PROBE_CLIP)
gal = {}
for r in gal_rows:
gal.setdefault(r["person"], []).append(r["emb"])
gal = {p: np.stack(v) for p, v in gal.items()}
for r in prb_rows:
if r["person"] in gal:
s = float((gal[r["person"]] @ r["emb"]).max()) # best-of-N, own actor
r["p"] = cal.probability(s)
r["sim"] = s
else:
r["p"] = np.nan
rows = [r for r in prb_rows if not np.isnan(r.get("p", np.nan))]
print(f"[data] {len(rows)} labelled probe faces with a gallery entry\n", file=sys.stderr)
# ── sanity check: does the pose estimate recover what the sheets show? ───────
print("pose by person (does this match the review sheets?)")
print(f"{'person':>7}{'n':>5}{'|yaw| med':>11}{'pitch med':>11}{'P med':>8}{'hit rate':>10}")
for p in sorted({r['person'] for r in rows}):
sub = [r for r in rows if r["person"] == p]
print(f"{p:>7}{len(sub):>5}"
f"{np.median([abs(r['yaw']) for r in sub]):>11.1f}"
f"{np.median([r['pitch'] for r in sub]):>11.1f}"
f"{np.median([r['p'] for r in sub]):>8.3f}"
f"{100*np.mean([r['p'] > PROB_THRESHOLD for r in sub]):>9.0f}%")
# ── P binned by each covariate ───────────────────────────────────────────────
def binned(name, key, edges, fmt="{:.0f}"):
print(f"\nP(match) by {name}")
print(f"{'bin':>16}{'n':>5}{'P med':>9}{'hit rate':>10}{'sim med':>9}")
vals = np.array([r[key] for r in rows])
for lo, hi in zip(edges[:-1], edges[1:]):
sub = [r for r, v in zip(rows, vals) if lo <= v < hi]
if not sub:
continue
lbl = f"{fmt.format(lo)}–{fmt.format(hi)}"
print(f"{lbl:>16}{len(sub):>5}"
f"{np.median([r['p'] for r in sub]):>9.3f}"
f"{100*np.mean([r['p'] > PROB_THRESHOLD for r in sub]):>9.0f}%"
f"{np.median([r['sim'] for r in sub]):>9.3f}")
for r in rows:
r["absyaw"] = abs(r["yaw"])
r["abspitch"] = abs(r["pitch"])
binned("|yaw| (deg from frontal)", "absyaw", [0, 10, 20, 30, 45, 60, 91])
binned("|pitch| (deg)", "abspitch", [0, 10, 20, 30, 45, 91])
binned("face size (px)", "px", [0, 130, 150, 175, 200, 400])
binned("blur (laplacian var)", "blur", [0, 50, 150, 400, 1000, 1e9])
binned("detector confidence", "conf", [0.5, 0.6, 0.7, 0.8, 0.9, 1.01], "{:.2f}")
# ── how much does each covariate actually explain? ───────────────────────────
print("\nSpearman rank correlation with P(match):")
def spearman(a, b):
ra = np.argsort(np.argsort(a)); rb = np.argsort(np.argsort(b))
return float(np.corrcoef(ra, rb)[0, 1])
P = np.array([r["p"] for r in rows])
for key, label in [("absyaw", "|yaw|"), ("abspitch", "|pitch|"), ("px", "face px"),
("blur", "blur"), ("conf", "detector conf")]:
v = np.array([r[key] for r in rows])
print(f" {label:>14}: {spearman(v, P):+.3f}")
json.dump([{k: v for k, v in r.items() if k != "emb"} for r in rows],
open("failure_analysis.json", "w"), indent=1, default=float)
experiments/xsource/landmark_voting.py
+165
View File
@@ -0,0 +1,165 @@
#!/usr/bin/env python3
"""Landmark voting: average SCRFD's overlapping detections instead of discarding them.
SCRFD predicts a face from many anchors; NMS keeps the single highest-scoring
box and throws the rest away. Each discarded box carries its own 5-landmark
estimate of the SAME face, so the survivors are one sample from a distribution
we could be averaging over.
baseline conf 0.50, nms 0.40 — the shipped settings, one box per face
voted conf 0.30, nms 0.90 — duplicates survive, then grouped by IoU and
the 5 landmarks averaged, weighted by detection confidence
Why this is worth trying when the mesh failed: the mesh moved the landmarks off
the definition ArcFace was trained on (a lip-ring centroid is not an annotated
mouth corner), and the embedder punished it. A confidence-weighted mean of
SCRFD's OWN landmark predictions is the same kind of point, just with less
variance — it should stay on-distribution while being steadier.
Scored on cross-clip identification through the production sigmoid, which is
the thing that actually broke. Raw similarity shown only to locate the
threshold; it decides nothing.
LD_PRELOAD=/usr/lib/libcudnn_cnn.so.9 python3 landmark_voting.py
"""
import sys, glob, json, os
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed # before cv2 — see alignment_compare.py
import numpy as np
import cv2
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
CLIPS = ["5157344", "5157339"]
PROB_THRESHOLD = 0.754
GROUP_IOU = 0.55 # detections overlapping this much are the same face
MATCH_IOU = 0.35 # tie a detection to the hand-labelled face
base_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.5, nms=0.4, max_side=0)
# Same models, looser suppression: keep the duplicates NMS would have removed.
vote_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.3, nms=0.9, max_side=0)
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
def iou(a, b):
ax, ay, aw, ah = a; bx, by, bw, bh = b
x0, y0 = max(ax, bx), max(ay, by)
x1, y1 = min(ax + aw, bx + bw), min(ay + ah, by + bh)
if x1 <= x0 or y1 <= y0:
return 0.0
i = (x1 - x0) * (y1 - y0)
return i / (aw * ah + bw * bh - i)
def vote(dets):
"""Group overlapping detections, return (bbox, landmarks, conf, n_votes)."""
items = sorted(dets, key=lambda d: -d.confidence)
used, out = [False] * len(items), []
for i, d in enumerate(items):
if used[i]:
continue
grp = [d]
used[i] = True
for j in range(i + 1, len(items)):
if not used[j] and iou(list(d.bbox), list(items[j].bbox)) >= GROUP_IOU:
used[j] = True
grp.append(items[j])
w = np.array([g.confidence for g in grp], dtype=np.float32)
w = w / w.sum()
lms = np.stack([np.array(g.landmarks, dtype=np.float32).reshape(5, 2) for g in grp])
bxs = np.stack([np.array(list(g.bbox), dtype=np.float32) for g in grp])
out.append((( w[:, None] * bxs).sum(0), (w[:, None, None] * lms).sum(0),
float(grp[0].confidence), len(grp)))
return out
def collect(clip):
lab = {os.path.basename(p): os.path.basename(os.path.dirname(p))
for p in glob.glob(f"labelling/{clip}/*/*.jpg")
if os.path.basename(os.path.dirname(p)) not in ("discard", "unsorted")}
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
rows, votes = [], []
for frame in sorted({man[f]["frame"] for f in lab}):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
base = base_eng.detect(img)
voted = vote(vote_eng.detect(img))
for fname, person in lab.items():
m = man[fname]
if m["frame"] != frame or m["idx"] >= len(base):
continue
d = base[m["idx"]]
lm5 = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
c_b = sae_embed.align_face(img, lm5)
# the voted group covering the same face
best, best_v = None, 0.0
for bbox, lms, conf, n in voted:
v = iou(list(bbox), list(d.bbox))
if v > best_v:
best_v, best = v, (lms, n)
c_v = None
if best and best_v >= MATCH_IOU:
c_v = sae_embed.align_face(img, best[0].astype(np.float32))
votes.append(best[1])
rec = {"person": person}
rec["base"] = np.asarray(base_eng.embed_crop(c_b), np.float32) if c_b is not None else None
rec["voted"] = np.asarray(base_eng.embed_crop(c_v), np.float32) if c_v is not None else None
rows.append(rec)
return rows, votes
data, allv = {}, []
for c in CLIPS:
data[c], v = collect(c)
allv += v
print(f"[{c}] {len(data[c])} crops", file=sys.stderr)
print(f"[voting] group size: median {np.median(allv):.0f}, "
f"mean {np.mean(allv):.1f}, max {max(allv)} detections averaged per face",
file=sys.stderr)
GAL, PRB = "5157344", "5157339"
print(f"\ngallery {GAL} -> probe {PRB}, P>{PROB_THRESHOLD}\n")
print(f"{'align':>8}{'person':>8}{'n_gal':>7}{'n_prb':>7}"
f"{'within-clip':>13}{'cross-clip':>12}{'hit rate':>10}")
summary = {}
for key in ("base", "voted"):
gal, prb = {}, {}
for r in data[GAL]:
if r[key] is not None:
gal.setdefault(r["person"], []).append(r[key])
for r in data[PRB]:
if r[key] is not None:
prb.setdefault(r["person"], []).append(r[key])
gal = {p: np.stack(v) for p, v in gal.items()}
prb = {p: np.stack(v) for p, v in prb.items()}
hits = tot = 0
for p in sorted(set(gal) & set(prb)):
pp = prb[p] @ prb[p].T
np.fill_diagonal(pp, -1)
within = float(np.median(pp.max(axis=1))) if len(pp) > 1 else float("nan")
cross = float(np.median((gal[p] @ prb[p].T).max(axis=0)))
h = 0
for e in prb[p]:
bp, bn = 0.0, None
for q in gal:
v = cal.probability(float((gal[q] @ e).max()))
if v > bp:
bp, bn = v, q
if bp > PROB_THRESHOLD and bn == p:
h += 1
hits += h; tot += len(prb[p])
print(f"{key:>8}{p:>8}{len(gal[p]):>7}{len(prb[p]):>7}"
f"{cal.probability(within):>6.3f}/{within:<6.3f}"
f"{cal.probability(cross):>6.3f}/{cross:<5.3f}{100*h/len(prb[p]):>9.0f}%")
summary[key] = (hits, tot)
print(f"{key:>8}{'ALL':>8}{'':>14}{'':>25}{100*hits/max(tot,1):>9.0f}%\n")
hb, tb = summary["base"]; hv, tv = summary["voted"]
print(f"voting vs baseline: {100*hv/max(tv,1) - 100*hb/max(tb,1):+.1f} points "
f"of cross-clip TPI ({hb}/{tb} -> {hv}/{tv})")
experiments/xsource/make_review_site.py
+235
View File
@@ -0,0 +1,235 @@
#!/usr/bin/env python3
"""Build labelling/review.html — a local page for correcting the labels.
One row per crop, ordered most-suspicious first:
left the person it is currently filed under (medoid of that person's
hand-sorted crops, so the reference is one you trust)
centre the crop under review — context with the detection boxed, and
beneath it the 112x112 the embedder actually receives
right the person it matches better, if any, with both probabilities
Pick a destination per row, then Export to download corrections.json and apply
it with apply_corrections.py. Nothing is moved by this script.
Self-contained: images are inlined as data URIs and the page is opened from
disk, so no server runs and no face crop leaves the machine.
Ordering is by P(other) - P(self), both from the global gallery sigmoid, so
rows where the evidence disagrees with the label float to the top and the
agreement cases sink. It is a review order, not a verdict — you are the
arbiter, which is the whole point of labelling by hand.
"""
import sys, glob, json, os, base64
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
EMBEDDER = M + "LVFace-B_Glint360K.onnx"
GALLERY = ROOT + "gallery_lvface.h5"
REF_CLIP = "5157344" # the clip sorted by hand — reference faces come from here
CLIPS = ["5157344", "5157339"]
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=EMBEDDER, conf=0.5, nms=0.4, max_side=0)
cal = sae_embed.gallery_calibration(GALLERY)
def b64(img, size, q=72):
img = cv2.resize(img, (size, size))
ok, buf = cv2.imencode(".jpg", img, [cv2.IMWRITE_JPEG_QUALITY, q])
return "data:image/jpeg;base64," + base64.b64encode(buf).decode() if ok else ""
rows = []
for clip in CLIPS:
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
placed = {os.path.basename(p): (os.path.basename(os.path.dirname(p)), p)
for p in glob.glob(f"labelling/{clip}/*/*.jpg")}
by_frame = {}
for fname, (person, path) in placed.items():
if fname in man and person != "unsorted":
by_frame.setdefault(man[fname]["frame"], []).append((fname, person, path))
for frame, items in sorted(by_frame.items()):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
if img is None:
continue
dets = eng.detect(img)
for fname, person, path in items:
i = man[fname]["idx"]
if i >= len(dets):
continue
lm = np.array(dets[i].landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
rows.append({"clip": clip, "person": person, "file": fname, "path": path,
"px": man[fname]["px"], "aligned": np.asarray(crop),
"emb": np.asarray(eng.embed_crop(crop), dtype=np.float32)})
people = sorted({r["person"] for r in rows})
E = np.stack([r["emb"] for r in rows])
lab = np.array([people.index(r["person"]) for r in rows])
S = E @ E.T
np.fill_diagonal(S, -1.0)
# reference face per person: medoid of their REF_CLIP crops
ref_img = {}
for k, p in enumerate(people):
idx = [i for i in np.where(lab == k)[0] if rows[i]["clip"] == REF_CLIP]
if not idx:
idx = list(np.where(lab == k)[0])
if not idx:
continue
sub = S[np.ix_(idx, idx)].copy()
medoid = idx[int(np.argmax(sub.mean(axis=1)))]
ref_img[p] = b64(rows[medoid]["aligned"], 112)
items = []
for i, r in enumerate(rows):
k = lab[i]
same = [j for j in np.where(lab == k)[0] if j != i]
p_self = cal.probability(float(S[i, same].max())) if same else 0.0
best_other, p_other = None, 0.0
for k2, p2 in enumerate(people):
if k2 == k:
continue
other = np.where(lab == k2)[0]
if not len(other):
continue
pv = cal.probability(float(S[i, other].max()))
if pv > p_other:
p_other, best_other = pv, p2
ctx = cv2.imread(r["path"])
items.append({
"file": r["file"], "clip": r["clip"], "person": r["person"],
"px": int(r["px"]), "p_self": round(p_self, 3), "p_other": round(p_other, 3),
"other": best_other, "delta": round(p_other - p_self, 3),
"ctx": b64(ctx, 150) if ctx is not None else "",
"ali": b64(r["aligned"], 112),
})
items.sort(key=lambda x: -x["delta"])
payload = json.dumps({"people": people, "refs": ref_img, "items": items})
HTML = """<meta charset="utf-8"><title>JRay — label review</title>
<style>
:root{color-scheme:dark;--bg:#14161a;--fg:#e6e8ea;--mut:#8b929c;--line:#262b33;--warn:#e0654a;--ok:#4a9d6a}
body{margin:0;background:var(--bg);color:var(--fg);font:14px/1.5 system-ui,sans-serif}
header{position:sticky;top:0;background:#181b20;border-bottom:1px solid var(--line);
padding:12px 18px;display:flex;gap:18px;align-items:center;flex-wrap:wrap;z-index:5}
h1{font-size:15px;margin:0;font-weight:600}
.stat{color:var(--mut);font-size:13px}
button{background:#232830;color:var(--fg);border:1px solid var(--line);border-radius:6px;
padding:7px 13px;cursor:pointer;font:inherit}
button:hover{background:#2c323c}
button.go{background:#2f5d43;border-color:#3c7555}
.row{display:grid;grid-template-columns:150px 1fr 190px;gap:20px;align-items:center;
padding:14px 18px;border-bottom:1px solid var(--line)}
.row.flag{background:#1e1719}
.row.done{opacity:.4}
.cell{display:flex;gap:10px;align-items:center}
img{border-radius:5px;display:block;background:#000}
.lab{font-weight:600;font-size:15px}
.mut{color:var(--mut);font-size:12px}
.p{font-variant-numeric:tabular-nums}
.hi{color:var(--warn);font-weight:600}
.choices{display:flex;flex-wrap:wrap;gap:6px}
.choices button{padding:5px 10px;font-size:13px}
.choices button.sel{background:#2f5d43;border-color:#3c7555}
.legend{padding:10px 18px;color:var(--mut);font-size:12px;border-bottom:1px solid var(--line)}
</style>
<header>
<h1>Label review</h1>
<span class="stat" id="stat"></span>
<button id="exp" class="go">Export corrections.json</button>
<button id="onlyflag">Show only disagreements</button>
</header>
<div class="legend">Left: the person this crop is filed under. Centre: the crop (context with the
detection boxed, and the 112&times;112 the embedder actually sees). Right: the person it matches
better, if any. Ordered by P(other) &minus; P(self) &mdash; disagreements first.</div>
<div id="list"></div>
<script>
const D = __PAYLOAD__;
const choice = {};
const list = document.getElementById('list');
function render(){
list.innerHTML = '';
const flagOnly = document.body.dataset.flag === '1';
for (const it of D.items){
if (flagOnly && it.delta <= 0) continue;
const row = document.createElement('div');
row.className = 'row' + (it.delta > 0 ? ' flag' : '') + (choice[it.file] ? ' done' : '');
const left = document.createElement('div');
left.className = 'cell';
left.innerHTML = `<img src="${D.refs[it.person]||''}" width="72" height="72">
<div><div class="lab">${it.person}</div>
<div class="mut p">P(self) ${it.p_self.toFixed(3)}</div></div>`;
const mid = document.createElement('div');
mid.className = 'cell';
mid.innerHTML = `<img src="${it.ctx}" width="120" height="120">
<img src="${it.ali}" width="90" height="90">
<div><div class="mut">${it.clip} &middot; ${it.px}px</div>
<div class="mut">${it.file}</div></div>`;
const right = document.createElement('div');
const worse = it.delta > 0;
right.innerHTML = it.other
? `<div class="cell"><img src="${D.refs[it.other]||''}" width="56" height="56">
<div><div class="lab ${worse?'hi':''}">${it.other}</div>
<div class="mut p ${worse?'hi':''}">P ${it.p_other.toFixed(3)}</div></div></div>`
: '<div class="mut">—</div>';
const ch = document.createElement('div');
ch.className = 'choices';
for (const p of D.people.concat(['discard'])){
const b = document.createElement('button');
b.textContent = p === it.person ? p + ' (keep)' : p;
if (choice[it.file] === p || (!choice[it.file] && p === it.person)) b.classList.add('sel');
b.onclick = () => { choice[it.file] = p; render(); };
ch.appendChild(b);
}
right.appendChild(ch);
row.append(left, mid, right);
list.appendChild(row);
}
const changed = Object.entries(choice).filter(([f,p]) =>
p !== (D.items.find(i=>i.file===f)||{}).person).length;
document.getElementById('stat').textContent =
`${D.items.length} crops · ${D.items.filter(i=>i.delta>0).length} disagreements · ${changed} changes staged`;
}
document.getElementById('onlyflag').onclick = () => {
document.body.dataset.flag = document.body.dataset.flag === '1' ? '0' : '1';
render();
};
document.getElementById('exp').onclick = () => {
const out = {};
for (const it of D.items){
const p = choice[it.file] || it.person;
if (p !== it.person) out[it.file] = {from: it.person, to: p, clip: it.clip};
}
const blob = new Blob([JSON.stringify(out, null, 1)], {type:'application/json'});
const a = document.createElement('a');
a.href = URL.createObjectURL(blob); a.download = 'corrections.json'; a.click();
};
render();
</script>
"""
os.makedirs("labelling", exist_ok=True)
out = "labelling/review.html"
with open(out, "w") as f:
f.write(HTML.replace("__PAYLOAD__", payload))
size = os.path.getsize(out) / 1e6
flagged = sum(1 for i in items if i["delta"] > 0)
print(f"{out} {size:.1f} MB {len(items)} crops, {flagged} disagreements", file=sys.stderr)
print(f"open file://{os.path.abspath(out)}", file=sys.stderr)
experiments/xsource/pose_label.py
+200
View File
@@ -0,0 +1,200 @@
#!/usr/bin/env python3
"""Estimate head pose per crop, and build a page to confirm or correct it.
Why not solvePnP on the 5 detector landmarks: those landmarks collapse on
turned faces, so the estimator breaks precisely on the crops whose pose we care
about. Run that way it reported the profile subject as the MOST frontal of the
four, which is how we know not to trust it.
Instead the estimate comes from the MediaPipe face mesh (468 points, run via
OpenCV DNN — the same model rPPG-kahn uses) and a symmetry measure that needs
no 3D model:
yaw_ratio = (dL - dR) / (dL + dR)
over left/right symmetric vertex pairs, where dL and dR are each side's
distance from the face midline. Frontal ~ 0, profile -> +/-1. It degrades
gracefully because it averages many pairs rather than trusting any one point,
and it is scale- and translation-free.
It is still an estimate. So this writes pose_review.html with the estimate
PRE-FILLED as a proposal, ordered by confidence, for you to correct — and the
correlation is only run against your corrected labels. If the estimate turns
out to disagree with you often, that is the finding, and the automatic number
gets dropped rather than reported.
Bins are coarse on purpose: frontal / three-quarter / profile / down-or-hidden.
Finer than that and the labelling is slower and less reliable, and the question
("does pose explain the misses") does not need degrees.
"""
import sys, glob, json, os, base64
# sae_embed MUST be imported before cv2: OpenCV's DNN module loads the system
# libonnxruntime, which then shadows the newer one this module links against and
# the import fails on a missing symbol version. Order matters, so do not tidy
# these into alphabetical order.
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
import numpy as np
import cv2
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
MESH = "/home/dtourolle/Development/rPPG-kahn/models/face_landmark.tflite"
CLIPS = ["5157344", "5157339"]
BINS = ["frontal", "three-quarter", "profile", "down-or-hidden"]
# Symmetric vertex pairs (subject-left, subject-right) on the MediaPipe mesh:
# outer eye corners, inner eye corners, cheeks, mouth corners, jaw.
PAIRS = [(33, 263), (133, 362), (130, 359), (243, 463),
(61, 291), (91, 321), (146, 375), (58, 288), (172, 397), (215, 435)]
MIDLINE = [10, 168, 1, 4, 5, 195, 197, 152] # forehead -> nose -> chin
net = cv2.dnn.readNetFromTFLite(MESH)
NAMES = net.getUnconnectedOutLayersNames()
LMI, PRI = NAMES.index("conv2d_21"), NAMES.index("conv2d_31")
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.5, nms=0.4, max_side=0)
def mesh_pose(img, bbox, expand=1.6):
"""(yaw_ratio, presence) or (nan, 0). yaw_ratio in [-1, 1], 0 = frontal."""
x, y, w, h = bbox
cx, cy, s = x + w / 2, y + h / 2, max(w, h) * expand
crop = cv2.getRectSubPix(img, (int(s), int(s)), (float(cx), float(cy)))
net.setInput(cv2.dnn.blobFromImage(crop, 1 / 255.0, (192, 192), (0, 0, 0), swapRB=True))
o = net.forward(NAMES)
pres = 1 / (1 + np.exp(-float(o[PRI].ravel()[0])))
lm = o[LMI].reshape(468, 3)[:, :2]
mid = lm[MIDLINE]
# least-squares midline direction, then signed distance of each pair member
c = mid.mean(axis=0)
u, _, _ = np.linalg.svd(mid - c)
d = (mid - c)
axis = np.linalg.svd(d.T @ d)[0][:, 0] # principal direction of the midline
normal = np.array([-axis[1], axis[0]])
ratios = []
for a, b in PAIRS:
dl = float(np.dot(lm[a] - c, normal))
dr = float(np.dot(lm[b] - c, normal))
if abs(dl) + abs(dr) < 1e-6:
continue
ratios.append((abs(dl) - abs(dr)) / (abs(dl) + abs(dr)))
return (float(np.median(ratios)) if ratios else np.nan), pres
def b64(img, size, q=72):
ok, buf = cv2.imencode(".jpg", cv2.resize(img, (size, size)),
[cv2.IMWRITE_JPEG_QUALITY, q])
return "data:image/jpeg;base64," + base64.b64encode(buf).decode() if ok else ""
items = []
for clip in CLIPS:
lab = {os.path.basename(p): (os.path.basename(os.path.dirname(p)), p)
for p in glob.glob(f"labelling/{clip}/*/*.jpg")
if os.path.basename(os.path.dirname(p)) not in ("discard", "unsorted")}
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
for frame in sorted({man[f]["frame"] for f in lab}):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
dets = eng.detect(img)
for fname, (person, path) in lab.items():
m = man[fname]
if m["frame"] != frame or m["idx"] >= len(dets):
continue
d = dets[m["idx"]]
lm5 = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm5)
if crop is None:
continue
yaw, pres = mesh_pose(img, d.bbox)
a = abs(yaw) if not np.isnan(yaw) else 1.0
guess = ("frontal" if a < 0.15 else "three-quarter" if a < 0.45
else "profile")
if pres < 0.5:
guess = "down-or-hidden" # mesh could not fit at all
ctx = cv2.imread(path)
items.append({"file": fname, "clip": clip, "person": person,
"px": int(m["px"]), "yaw": None if np.isnan(yaw) else round(yaw, 3),
"pres": round(pres, 3), "guess": guess,
"ctx": b64(ctx, 140) if ctx is not None else "",
"ali": b64(np.asarray(crop), 112)})
# least-confident first: near a bin boundary, or the mesh could not fit
def uncertainty(it):
if it["pres"] < 0.5:
return 0.0
a = abs(it["yaw"]) if it["yaw"] is not None else 1.0
return min(abs(a - 0.15), abs(a - 0.45))
items.sort(key=uncertainty)
payload = json.dumps({"bins": BINS, "items": items})
HTML = """<meta charset="utf-8"><title>JRay — head pose labelling</title>
<style>
:root{color-scheme:dark}
body{margin:0;background:#14161a;color:#e6e8ea;font:14px/1.5 system-ui,sans-serif}
header{position:sticky;top:0;background:#181b20;border-bottom:1px solid #262b33;
padding:12px 18px;display:flex;gap:16px;align-items:center;flex-wrap:wrap;z-index:5}
h1{font-size:15px;margin:0}
button{background:#232830;color:#e6e8ea;border:1px solid #262b33;border-radius:6px;
padding:7px 12px;cursor:pointer;font:inherit}
button:hover{background:#2c323c}
button.go{background:#2f5d43;border-color:#3c7555}
.g{display:grid;grid-template-columns:repeat(auto-fill,minmax(300px,1fr));gap:10px;padding:14px}
.c{border:1px solid #262b33;border-radius:8px;padding:9px;display:flex;gap:9px;align-items:center}
.c.edited{border-color:#3c7555}
img{border-radius:5px;background:#000;display:block}
.m{color:#8b929c;font-size:11px}
.b{display:flex;flex-wrap:wrap;gap:4px;margin-top:5px}
.b button{padding:3px 7px;font-size:11px}
.b button.sel{background:#2f5d43;border-color:#3c7555}
</style>
<header><h1>Head pose</h1><span class="m" id="stat"></span>
<button class="go" id="exp">Export pose_labels.json</button></header>
<div class="g" id="g"></div>
<script>
const D=__PAYLOAD__; const pick={};
function render(){
const g=document.getElementById('g'); g.innerHTML='';
for(const it of D.items){
const cur=pick[it.file]||it.guess;
const c=document.createElement('div');
c.className='c'+(pick[it.file]&&pick[it.file]!==it.guess?' edited':'');
const b=D.bins.map(x=>`<button class="${x===cur?'sel':''}" data-f="${it.file}" data-b="${x}">${x}</button>`).join('');
c.innerHTML=`<img src="${it.ctx}" width="88" height="88"><img src="${it.ali}" width="66" height="66">
<div><div class="m">${it.person} · ${it.clip.slice(-3)} · ${it.px}px</div>
<div class="m">yaw ${it.yaw===null?'—':it.yaw} · presence ${it.pres}</div>
<div class="b">${b}</div></div>`;
g.appendChild(c);
}
g.onclick=e=>{const t=e.target; if(t.dataset&&t.dataset.b){pick[t.dataset.f]=t.dataset.b; render();}};
const ed=Object.entries(pick).filter(([f,v])=>v!==(D.items.find(i=>i.file===f)||{}).guess).length;
document.getElementById('stat').textContent=`${D.items.length} crops · ${ed} corrections`;
}
document.getElementById('exp').onclick=()=>{
const out={}; for(const it of D.items) out[it.file]={pose:pick[it.file]||it.guess,
guess:it.guess, yaw:it.yaw, pres:it.pres, person:it.person, clip:it.clip};
const a=document.createElement('a');
a.href=URL.createObjectURL(new Blob([JSON.stringify(out,null,1)],{type:'application/json'}));
a.download='pose_labels.json'; a.click();
};
render();
</script>
"""
out = "labelling/pose_review.html"
open(out, "w").write(HTML.replace("__PAYLOAD__", payload))
from collections import Counter
print(f"{out} {os.path.getsize(out)/1e6:.1f} MB {len(items)} crops", file=sys.stderr)
print(f"estimate: {dict(Counter(i['guess'] for i in items))}", file=sys.stderr)
print("\nestimated pose per person (does this match what you see?):", file=sys.stderr)
for p in sorted({i["person"] for i in items}):
for clip in CLIPS:
sub = [i for i in items if i["person"] == p and i["clip"] == clip]
if sub:
print(f" {p} {clip[-3:]}: {dict(Counter(i['guess'] for i in sub))}",
file=sys.stderr)
print(f"\nopen file://{os.path.abspath(out)}", file=sys.stderr)
experiments/xsource/propose_labels.py
+242
View File
@@ -0,0 +1,242 @@
#!/usr/bin/env python3
"""Propose person labels for one clip using another clip's hand-sorted labels.
Reads the clip you have already sorted (REF_CLIP) as ground truth, then proposes
a person for every crop in the other clip (TARGET_CLIP) and writes them into
matching folders for you to correct.
python3 propose_labels.py # propose, write folders + sheets
python3 propose_labels.py --dry-run # report only, move nothing
Output:
labelling/<target>/unsorted/A|B|C|D/ proposed, same names as the ref clip
labelling/<target>/unsorted/ left in place when no person is
confident enough to name
labelling/review_<person>.jpg contact sheet spanning BOTH clips:
confirmed crops first, then
proposed ones with their P
Correcting it: open a review sheet. Every face on it should be one person. The
lower block is the proposal — move any intruder to the right folder, or back to
unsorted/. The folder a file sits in is the ground truth; nothing downstream
reads the proposed name or its probability.
The proposal is a labelling aid, never the label. Scoring the sweep against
embedding-derived labels would be circular: it keeps the faces the embedder
already gets right and drops the hard ones the sweep exists to find. Your
correction is what breaks that loop, which is why the proposal is deliberately
conservative and leaves anything doubtful unnamed.
Assignment is on the calibrated probability, per-actor best-of-N, exactly as
identity_matcher_node does — never a bare cosine (AR-024). The calibration is
fitted on your labelled reference crops, which is what calibrate_gallery is for.
"""
import sys, glob, json, os, shutil
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
# The embedder and the gallery whose calibration scores it MUST be the same
# model: a Platt fit is specific to one embedding space, so LVFace probabilities
# read through an ArcFace fit are meaningless.
EMBEDDER = M + "LVFace-B_Glint360K.onnx"
GALLERY = ROOT + "gallery_lvface.h5" # 291 actors, cached fit
REF_CLIP, TARGET_CLIP = "5157344", "5157339"
ASSIGN_P = 0.90 # propose a name only when this confident
SHEET_COLS = 8
THUMB = 150
DRY = "--dry-run" in sys.argv
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=EMBEDDER,
conf=0.5, nms=0.4, max_side=0)
def embed_manifest(clip):
"""Re-derive each dumped crop's embedding from its source frame, cached.
The dumped .jpg is a context thumbnail for human eyes; the embedding must
come from the aligned crop the pipeline would actually produce, so the
frame is re-detected and the manifest's idx picks the same face.
Detecting 24 4K frames per clip costs far more than the rest of this script
put together, and the result only changes when the manifest does — so it is
cached and keyed on the manifest's mtime. Delete cache/ to force a redo.
"""
man_path = f"labelling/{clip}/manifest.json"
cache_path = f"cache/emb_{clip}.npz"
os.makedirs("cache", exist_ok=True)
if os.path.exists(cache_path) and \
os.path.getmtime(cache_path) >= os.path.getmtime(man_path):
z = np.load(cache_path, allow_pickle=True)
print(f"[cache] {clip}: {len(z['meta'])} embeddings reused", file=sys.stderr)
return [{**m, "emb": e} for m, e in zip(z["meta"], z["emb"])]
man = json.load(open(man_path))
by_frame = {}
for m in man:
by_frame.setdefault(m["frame"], []).append(m)
out = []
for frame, ms in sorted(by_frame.items()):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
if img is None:
sys.exit(f"missing frames/d{clip}_{frame}.png — extract with\n"
f" ffmpeg -i clips/{clip}.mp4 -vf fps=2 -frames:v 24 "
f"frames/d{clip}_%03d.png")
dets = eng.detect(img)
for m in ms:
if m["idx"] >= len(dets):
continue
d = dets[m["idx"]]
lm = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
out.append({**m, "emb": np.asarray(eng.embed_crop(crop), dtype=np.float32)})
np.savez(cache_path,
meta=np.array([{k: v for k, v in o.items() if k != "emb"} for o in out],
dtype=object),
emb=np.stack([o["emb"] for o in out]))
print(f"[cache] {clip}: {len(out)} embeddings written to {cache_path}",
file=sys.stderr)
return out
def sorted_dirs(clip):
"""Person folders you created, wherever you put them under labelling/<clip>."""
found = {}
for path in glob.glob(f"labelling/{clip}/**/", recursive=True):
name = os.path.basename(path.rstrip("/"))
if name in ("unsorted", "discard") or name.startswith("5157"):
continue
files = [os.path.basename(f) for f in glob.glob(path + "*.jpg")]
if files:
found[name] = files
return found
# ── reference side: your labels ──────────────────────────────────────────────
ref_rows = embed_manifest(REF_CLIP)
ref_dirs = sorted_dirs(REF_CLIP)
if not ref_dirs:
sys.exit(f"no person folders under labelling/{REF_CLIP} — sort that clip first")
file_to_person = {f: p for p, fs in ref_dirs.items() for f in fs}
ref = [(file_to_person[r["file"]], r["emb"]) for r in ref_rows
if r["file"] in file_to_person]
people = sorted({p for p, _ in ref})
print(f"[ref] {REF_CLIP}: {len(ref)} labelled crops over {len(people)} people "
f"{ {p: sum(1 for q, _ in ref if q == p) for p in people} }", file=sys.stderr)
R = np.stack([e for _, e in ref])
r_actor = [people.index(p) for p, _ in ref]
# The global gallery's sigmoid — NOT a fit over these four people. A Platt fit
# over a handful of identities saturates: it will hand back P=0.99 for faces it
# has no basis to separate, which is exactly how a wrong label acquires a
# convincing probability. The production fit spans the whole actor population,
# so a probability means the same thing here as it does in the matcher.
cal = sae_embed.gallery_calibration(GALLERY)
print(f"[calibration] global: {cal} assign boundary = sim "
f"{cal.boundary_at(ASSIGN_P):.4f}", file=sys.stderr)
# ── target side: propose ─────────────────────────────────────────────────────
tgt_rows = embed_manifest(TARGET_CLIP)
T = np.stack([t["emb"] for t in tgt_rows])
r_actor_arr = np.asarray(r_actor)
# per-actor best-of-N for every target crop at once: (n_people, n_target)
best_sim = np.stack([(R[r_actor_arr == people.index(p)] @ T.T).max(axis=0)
for p in people])
proposals = []
for j, t in enumerate(tgt_rows):
k = int(np.argmax(best_sim[:, j]))
prob = cal.probability(float(best_sim[k, j])) # calibrated, never a bare cosine
proposals.append({**t, "person": people[k] if prob >= ASSIGN_P else None,
"p": prob, "top1": people[k]})
# At the production threshold the global fit stays silent on most of these
# faces, which is the honest answer for profile and downward-gaze shots — but a
# labelling aid wants throughput, not caution. --all proposes the top-1 person
# for every crop and orders the review sheets by descending probability, so the
# proposals degrade visibly down the sheet and you can stop correcting where
# they stop being right. The probability is shown, never hidden.
if "--all" in sys.argv:
for x in proposals:
x["person"] = x["top1"]
named = [x for x in proposals if x["person"]]
print(f"[propose] {TARGET_CLIP}: {len(named)}/{len(proposals)} named at P>={ASSIGN_P}; "
f"{len(proposals) - len(named)} left unsorted", file=sys.stderr)
for p in people:
got = [x for x in named if x["person"] == p]
if got:
ps = [x["p"] for x in got]
print(f" {p}: {len(got):>3} crops P {min(ps):.3f}–{max(ps):.3f}", file=sys.stderr)
if DRY:
sys.exit(0)
# ── write proposed folders, mirroring the ref clip's layout ──────────────────
ref_parent = os.path.dirname(next(iter(glob.glob(f"labelling/{REF_CLIP}/**/{people[0]}/",
recursive=True))).rstrip("/"))
tgt_parent = ref_parent.replace(REF_CLIP, TARGET_CLIP)
for p in people:
d = f"{tgt_parent}/{p}"
if os.path.isdir(d): # never clobber corrections already made
print(f"[skip] {d} exists — leaving your sorting alone", file=sys.stderr)
continue
os.makedirs(d, exist_ok=True)
def find_crop(clip, fname):
"""Locate a crop wherever it currently sits under labelling/<clip>."""
hits = glob.glob(f"labelling/{clip}/**/{fname}", recursive=True)
return hits[0] if hits else None
moved = 0
for x in named:
src = find_crop(TARGET_CLIP, x["file"])
dst = f"{tgt_parent}/{x['person']}/{x['file']}"
if src and os.path.abspath(src) != os.path.abspath(dst):
shutil.move(src, dst)
moved += 1
print(f"[write] moved {moved} crops into proposed folders", file=sys.stderr)
# ── review sheets: confirmed block, then proposed block ─────────────────────
def load(clip, person, fname):
for cand in glob.glob(f"labelling/{clip}/**/{person}/{fname}", recursive=True):
return cv2.imread(cand)
return None
for person in people:
conf = [(REF_CLIP, f, None) for f in ref_dirs.get(person, [])]
prop = sorted([(TARGET_CLIP, x["file"], x["p"]) for x in named
if x["person"] == person],
key=lambda t: -t[2]) # most confident first
items = conf + prop
if not items:
continue
rows_n = (len(items) + SHEET_COLS - 1) // SHEET_COLS
sheet = np.full((rows_n * (THUMB + 26), SHEET_COLS * THUMB, 3), 30, np.uint8)
for n, (clip, fname, p) in enumerate(items):
img = load(clip, person, fname)
if img is None:
continue
rr, cc = divmod(n, SHEET_COLS)
y, x = rr * (THUMB + 26), cc * THUMB
sheet[y:y + THUMB, x:x + THUMB] = cv2.resize(img, (THUMB, THUMB))
if p is None:
tag, col = f"{clip[-3:]} CONFIRMED", (170, 170, 170)
else:
tag, col = f"{clip[-3:]} P={p:.2f}", (140, 255, 140)
cv2.putText(sheet, tag, (x + 3, y + THUMB + 17),
cv2.FONT_HERSHEY_SIMPLEX, 0.42, col, 1)
cv2.imwrite(f"labelling/review_{person}.jpg", sheet)
print(f" review_{person}.jpg: {len(conf)} confirmed + {len(prop)} proposed",
file=sys.stderr)
json.dump({x["file"]: {"person": x["person"], "p": x["p"]} for x in proposals},
open(f"labelling/proposed_{TARGET_CLIP}.json", "w"), indent=1)
experiments/xsource/redraw_boxes.py
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env python3
"""Redraw every dumped crop with its detection box marked.
The original thumbnails padded by 0.5x the face on each side for
recognisability, which in a crowded frame pulls a neighbour into shot — often
more prominently than the subject. A label cannot be corrected from a picture
that does not say which face it refers to.
This rewrites each .jpg IN PLACE, wherever it currently sits, so any sorting
already done is preserved: only the pixels change, never the filename or the
folder. Re-run it after dump_faces.py, and re-check any sorting done before it.
"""
import glob, json, os, sys
import cv2
CLIPS = ["5157339", "5157344"]
OUT = 256
for clip in CLIPS:
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
n = 0
for path in glob.glob(f"labelling/{clip}/**/*.jpg", recursive=True):
fname = os.path.basename(path)
m = man.get(fname)
if m is None:
continue
img = cv2.imread(f"frames/d{clip}_{m['frame']}.png")
if img is None:
sys.exit(f"missing frames/d{clip}_{m['frame']}.png")
x, y, w, h = (int(v) for v in m["bbox"])
pad = int(0.55 * max(w, h))
x0, y0 = max(0, x - pad), max(0, y - pad)
x1, y1 = min(img.shape[1], x + w + pad), min(img.shape[0], y + h + pad)
sub = img[y0:y1, x0:x1].copy()
# Box in the sub-image's coordinates, drawn before the resize so the
# line lands exactly on the face at any output size.
cv2.rectangle(sub, (x - x0, y - y0), (x - x0 + w, y - y0 + h), (0, 0, 255), 3)
# Dim everything outside the box so the subject is unmistakable even
# when a neighbour's face is larger or better lit.
mask = sub.copy()
mask[y - y0:y - y0 + h, x - x0:x - x0 + w] = 0
sub = cv2.addWeighted(sub, 1.0, mask, -0.35, 0)
scale = OUT / max(sub.shape[:2])
sub = cv2.resize(sub, (int(sub.shape[1] * scale), int(sub.shape[0] * scale)))
canvas = cv2.copyMakeBorder(
sub, 0, max(0, OUT - sub.shape[0]), 0, max(0, OUT - sub.shape[1]),
cv2.BORDER_CONSTANT, value=(20, 20, 20))[:OUT, :OUT]
cv2.putText(canvas, f"{int(m['px'])}px", (5, OUT - 8),
cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 255), 1)
cv2.imwrite(path, canvas)
n += 1
print(f"[{clip}] redrew {n} crops in place", file=sys.stderr)
experiments/xsource/resolution_sweep.py
+182
View File
@@ -0,0 +1,182 @@
#!/usr/bin/env python3
"""Impact of input resolution on cross-source identification.
Gallery is built from one clip at NATIVE resolution. Probes come from the other
clip with the WHOLE FRAME downscaled before it reaches the detector, so
detection and landmark regression degrade together with the pixels. That is the
measurement VR-005 structurally could not make: it degraded an already-aligned
112x112 crop, holding alignment perfect, so it isolated the embedder's
resolution sensitivity and excluded everything upstream of it.
python3 resolution_sweep.py [--gallery-clip 5157339] [--detector scrfd_500m_bnkps.onnx]
Ground truth
------------
Hand-sorted person folders. Probe detections at reduced scale are tied back to
a labelled face GEOMETRICALLY — the box is mapped to native coordinates and
matched by IoU. Never by embedding similarity, which would be circular: it
would keep the faces the embedder still gets right and silently drop the ones
this sweep exists to find.
A probe whose label is only in the probe clip is OUT OF GALLERY. Naming it is a
true out-of-cast misID, the error the per-scene scorer weights 10x, so it is
counted separately from naming the wrong gallery member.
Metric
------
The calibrated probability from the PRODUCTION gallery sigmoid, never a raw
cosine (AR-024). Per-actor best-of-N similarity -> probability -> accept above
prob_threshold. This is identification, so the matcher's prior applies;
config.hpp has match_prior 0.5, i.e. log_prior_odds = 0.
Everything runs through the shipped C++ via sae_embed.
"""
import sys, glob, json, os, argparse
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
PROB_THRESHOLD = 0.754 # config.hpp:67
LOG_PRIOR_ODDS = 0.0 # config.hpp:61 match_prior=0.5
IOU_MIN = 0.3 # geometric label carry-down
SCALES = [1.0, 0.8, 0.6, 0.5, 0.4, 0.3, 0.25, 0.2, 0.15, 0.12, 0.09, 0.06]
ap = argparse.ArgumentParser()
ap.add_argument("--gallery-clip", default="5157339")
ap.add_argument("--probe-clip", default="5157344")
ap.add_argument("--detector", default="scrfd_500m_bnkps.onnx")
ap.add_argument("--embedder", default="LVFace-B_Glint360K.onnx")
ap.add_argument("--gallery-calibration", default=ROOT + "gallery_lvface.h5")
ap.add_argument("--out", default="results_resolution_sweep.json")
args = ap.parse_args()
eng = sae_embed.FaceEmbedder(detector_model=M + args.detector,
arcface_model=M + args.embedder,
conf=0.5, nms=0.4, max_side=0)
cal = sae_embed.gallery_calibration(args.gallery_calibration)
print(f"[calibration] global: {cal}", file=sys.stderr)
def labelled(clip):
"""{filename: person} from the hand-sorted folders, ignoring discard."""
out = {}
for path in glob.glob(f"labelling/{clip}/*/*.jpg"):
person = os.path.basename(os.path.dirname(path))
if person in ("discard", "unsorted"):
continue
out[os.path.basename(path)] = person
return out
def manifest(clip):
return {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
def iou(a, b):
ax, ay, aw, ah = a; bx, by, bw, bh = b
x0, y0 = max(ax, bx), max(ay, by)
x1, y1 = min(ax + aw, bx + bw), min(ay + ah, by + bh)
if x1 <= x0 or y1 <= y0:
return 0.0
inter = (x1 - x0) * (y1 - y0)
return inter / (aw * ah + bw * bh - inter)
# ── gallery: native resolution, labelled faces only ──────────────────────────
g_lab, g_man = labelled(args.gallery_clip), manifest(args.gallery_clip)
gal = {}
for frame in sorted({g_man[f]["frame"] for f in g_lab}):
img = cv2.imread(f"frames/d{args.gallery_clip}_{frame}.png")
dets = eng.detect(img)
for fname, person in g_lab.items():
m = g_man[fname]
if m["frame"] != frame or m["idx"] >= len(dets):
continue
lm = np.array(dets[m["idx"]].landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
continue
gal.setdefault(person, []).append(np.asarray(eng.embed_crop(crop), dtype=np.float32))
gal = {p: np.stack(v) for p, v in gal.items() if v}
people = sorted(gal)
print(f"[gallery] {args.gallery_clip} @native: "
f"{ {p: len(v) for p, v in gal.items()} }", file=sys.stderr)
# ── probe ground truth at native resolution ──────────────────────────────────
p_lab, p_man = labelled(args.probe_clip), manifest(args.probe_clip)
truth = {} # frame -> [(bbox_native, person)]
for fname, person in p_lab.items():
m = p_man[fname]
truth.setdefault(m["frame"], []).append((m["bbox"], person))
n_out = sum(1 for p in set(p_lab.values()) if p not in people)
print(f"[probe] {args.probe_clip}: {len(p_lab)} labelled faces, "
f"{len(set(p_lab.values()))} people, {n_out} of them out-of-gallery",
file=sys.stderr)
# ── sweep ────────────────────────────────────────────────────────────────────
print(f"\n{'scale':>6}{'frame':>11}{'face px':>9}{'found':>7}{'matched':>9}"
f"{'TPI':>8}{'FPI-in':>8}{'FPI-out':>9}{'TBI':>8}")
results = []
for s in SCALES:
tpi = fpi_in = fpi_out = tbi = 0
n_found = n_matched = 0
pxs = []
for frame, gts in sorted(truth.items()):
img = cv2.imread(f"frames/d{args.probe_clip}_{frame}.png")
if s != 1.0:
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
dets = eng.detect(img)
n_found += len(dets)
for d in dets:
x, y, w, h = d.bbox
native = (x / s, y / s, w / s, h / s) # geometric carry-down
best, best_iou = None, 0.0
for gt_box, person in gts:
v = iou(native, gt_box)
if v > best_iou:
best_iou, best = v, person
if best_iou < IOU_MIN:
continue # spurious / unlabelled
n_matched += 1
pxs.append(min(w, h))
lm = np.array(d.landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
tbi += 1 # degenerate alignment
continue
emb = np.asarray(eng.embed_crop(crop), dtype=np.float32)
best_p, best_name = 0.0, None
for p in people: # per-actor best-of-N
prob = cal.probability(float((gal[p] @ emb).max()), LOG_PRIOR_ODDS)
if prob > best_p:
best_p, best_name = prob, p
if best_p <= PROB_THRESHOLD:
tbi += 1
elif best not in people:
fpi_out += 1 # named someone absent from the gallery
elif best_name == best:
tpi += 1
else:
fpi_in += 1
n = max(1, n_matched)
med_px = float(np.median(pxs)) if pxs else 0.0
print(f"{s:>6.2f}{f'{int(4096*s)}x{int(2160*s)}':>11}{med_px:>9.0f}"
f"{n_found:>7}{n_matched:>9}"
f"{100*tpi/n:>7.1f}%{100*fpi_in/n:>7.1f}%{100*fpi_out/n:>8.1f}%{100*tbi/n:>7.1f}%")
results.append({"scale": s, "median_face_px": med_px, "detections": n_found,
"matched_to_truth": n_matched, "tpi_pct": 100*tpi/n,
"fpi_in_gallery_pct": 100*fpi_in/n, "fpi_out_of_gallery_pct": 100*fpi_out/n,
"tbi_pct": 100*tbi/n})
json.dump({"gallery_clip": args.gallery_clip, "probe_clip": args.probe_clip,
"detector": args.detector, "embedder": args.embedder,
"prob_threshold": PROB_THRESHOLD, "log_prior_odds": LOG_PRIOR_ODDS,
"calibration": {"a": cal.a, "b": cal.b},
"gallery_people": people, "results": results},
open(args.out, "w"), indent=2)
print(f"\nwrote {args.out}", file=sys.stderr)
experiments/xsource/verify_labels.py
+167
View File
@@ -0,0 +1,167 @@
#!/usr/bin/env python3
"""Integrity check on the labelled set, before it is used as ground truth.
Checks, loudest failure first:
1. INDEX INTEGRITY. Each crop's embedding is taken by re-detecting its source
frame and indexing with the manifest's `idx`. If detection order is not
reproducible, the thumbnail you sorted and the embedding that gets scored
are different faces — you would see a correct picture and score the wrong
person, with nothing to signal it. Every crop's re-detected bbox is compared
against the manifest's.
2. NO CROP IN TWO FOLDERS, and every manifest entry accounted for — so a
move that half-completed cannot silently duplicate or drop a label.
3. ALIGNMENT. The 112x112 warp is what the embedder actually sees; the
thumbnail is only context for your eyes. verify_<person>.jpg pairs them:
context-with-box on top, the real aligned crop beneath. A profile face whose
alignment has collapsed is obvious there and nowhere else.
4. SEPARATION. Per person, the calibrated P of their own crops against the
other people's, using the global gallery sigmoid. A label set where someone
matches another person better than themselves is mislabelled.
Nothing here changes a label. It reports.
"""
import sys, glob, json, os
import numpy as np
import cv2
sys.path.insert(0, "/home/dtourolle/Development/Jray-project/scene-actor-extraction/build-ort")
import sae_embed
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
EMBEDDER = M + "LVFace-B_Glint360K.onnx"
GALLERY = ROOT + "gallery_lvface.h5"
CLIPS = ["5157344", "5157339"]
THUMB = 130
COLS = 10
eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=EMBEDDER, conf=0.5, nms=0.4, max_side=0)
fail = 0
rows = []
for clip in CLIPS:
man = {m["file"]: m for m in json.load(open(f"labelling/{clip}/manifest.json"))}
# where each crop currently sits -> its label
placed = {}
for path in glob.glob(f"labelling/{clip}/*/*.jpg"):
person = os.path.basename(os.path.dirname(path))
if person in ("discard", "unsorted"):
continue # not people; scoring them would invent an extra identity
fname = os.path.basename(path)
if fname in placed:
print(f"[FAIL] {fname} appears in both {placed[fname][0]} and {person}")
fail += 1
placed[fname] = (person, path)
missing = set(man) - set(placed)
extra = set(placed) - set(man)
if missing:
print(f"[warn] {clip}: {len(missing)} manifest crops not in any folder")
if extra:
print(f"[FAIL] {clip}: {len(extra)} files with no manifest entry: "
f"{sorted(extra)[:3]}")
fail += 1
# index integrity + alignment, frame by frame
by_frame = {}
for fname, (person, path) in placed.items():
if fname in man:
by_frame.setdefault(man[fname]["frame"], []).append((fname, person, path))
bad_idx = 0
for frame, items in sorted(by_frame.items()):
img = cv2.imread(f"frames/d{clip}_{frame}.png")
if img is None:
print(f"[FAIL] missing frames/d{clip}_{frame}.png")
fail += 1
continue
dets = eng.detect(img)
for fname, person, path in items:
m = man[fname]
i = m["idx"]
if i >= len(dets):
print(f"[FAIL] {fname}: idx {i} >= {len(dets)} detections now")
bad_idx += 1
continue
got = [float(v) for v in dets[i].bbox]
want = m["bbox"]
if max(abs(a - b) for a, b in zip(got, want)) > 1.0:
print(f"[FAIL] {fname}: manifest bbox {[round(v) for v in want]} "
f"!= re-detected {[round(v) for v in got]}")
bad_idx += 1
continue
lm = np.array(dets[i].landmarks, dtype=np.float32).reshape(5, 2)
crop = sae_embed.align_face(img, lm)
if crop is None:
print(f"[warn] {fname}: alignment degenerate, no crop reaches the embedder")
continue
rows.append({"clip": clip, "person": person, "file": fname, "path": path,
"px": m["px"], "aligned": np.asarray(crop),
"emb": np.asarray(eng.embed_crop(crop), dtype=np.float32)})
fail += bad_idx
print(f"[{clip}] {len(placed)} placed, {len(by_frame)} frames, "
f"index mismatches: {bad_idx}")
if not rows:
sys.exit("nothing to verify")
# ── separation, through the global gallery sigmoid ───────────────────────────
cal = sae_embed.gallery_calibration(GALLERY)
E = np.stack([r["emb"] for r in rows])
people = sorted({r["person"] for r in rows})
lab = np.array([people.index(r["person"]) for r in rows])
S = E @ E.T
np.fill_diagonal(S, -1.0)
print(f"\n{'person':>8}{'crops':>7}{'344':>6}{'339':>6}"
f"{'P(self)':>10}{'P(other)':>10}{'worst':>8}")
for k, p in enumerate(people):
mine = np.where(lab == k)[0]
if len(mine) < 2:
continue
self_sim = S[np.ix_(mine, mine)].max(axis=1)
other_sim = S[np.ix_(mine, np.where(lab != k)[0])].max(axis=1)
p_self = np.array([cal.probability(float(s)) for s in self_sim])
p_other = np.array([cal.probability(float(s)) for s in other_sim])
n344 = sum(1 for i in mine if rows[i]["clip"] == "5157344")
n339 = len(mine) - n344
# a crop that matches someone else better than anyone of its own label
worst = int((other_sim > self_sim).sum())
print(f"{p:>8}{len(mine):>7}{n344:>6}{n339:>6}"
f"{np.median(p_self):>10.3f}{np.median(p_other):>10.3f}{worst:>8}")
if worst:
for i in mine[other_sim > self_sim]:
print(f" suspect: {rows[i]['file']} "
f"P(self)={cal.probability(float(self_sim[list(mine).index(i)])):.3f} "
f"< P(other)={cal.probability(float(other_sim[list(mine).index(i)])):.3f}")
# ── verify sheets: context+box over the actual aligned crop ──────────────────
for p in people:
items = [r for r in rows if r["person"] == p]
items.sort(key=lambda r: (r["clip"], r["file"]))
n = len(items)
sheet_rows = (n + COLS - 1) // COLS
H = THUMB * 2 + 22
sheet = np.full((sheet_rows * H, COLS * THUMB, 3), 25, np.uint8)
for j, r in enumerate(items):
rr, cc = divmod(j, COLS)
y, x = rr * H, cc * THUMB
ctx = cv2.imread(r["path"])
if ctx is not None:
sheet[y:y + THUMB, x:x + THUMB] = cv2.resize(ctx, (THUMB, THUMB))
sheet[y + THUMB:y + 2 * THUMB, x:x + THUMB] = cv2.resize(r["aligned"], (THUMB, THUMB))
cv2.putText(sheet, f"{r['clip'][-3:]} {int(r['px'])}px",
(x + 3, y + 2 * THUMB + 15),
cv2.FONT_HERSHEY_SIMPLEX, 0.38, (150, 220, 150), 1)
cv2.imwrite(f"labelling/verify_{p}.jpg", sheet)
print(f" verify_{p}.jpg: {n} crops (top row context, bottom row what the embedder sees)")
print(f"\n{'PASS' if fail == 0 else f'{fail} FAILURES'}")
sys.exit(1 if fail else 0)
external/KPN
Vendored
+1 -1
Submodule external/KPN updated: 6595e6e925...28e06675f5
scripts/artifacts/pull_artifacts.sh
+67 -1
View File
@@ -11,6 +11,7 @@
# scripts/artifacts/pull_artifacts.sh montage-frames <film-slug> [version]
# scripts/artifacts/pull_artifacts.sh experiment-data [version]
# scripts/artifacts/pull_artifacts.sh report-highlights <name> [version]
# scripts/artifacts/pull_artifacts.sh xsource [version]
# version defaults to "latest" (newest uploaded version, by created_at).
set -euo pipefail
@@ -83,11 +84,71 @@ pull_report_highlight() {
curl -sf "${DL_BASE}/generic/report-highlights/${version}/${name}" -o "${dest}/${name}"
}
pull_xsource() {
local version="$1"
local dest="${REPO_ROOT}/experiments/xsource"
echo "=== xsource (version ${version}) ==="
mkdir -p "${dest}/clips" "${dest}/frames"
for clip in 5157339 5157344; do
if [ -f "${dest}/clips/${clip}.mp4" ]; then
echo " ${clip}.mp4 already present, skipping"
else
echo " fetching ${clip}.mp4..."
curl -sf "${DL_BASE}/generic/xsource/${version}/${clip}.mp4" \
-o "${dest}/clips/${clip}.mp4" \
|| { echo " [warn] ${clip}.mp4 not found at version ${version}" >&2; continue; }
fi
done
if [ -d "${dest}/labelling" ]; then
echo " labelling/ already present — NOT overwriting (it is hand-sorted"
echo " ground truth; move it aside first if you really want the remote copy)"
else
echo " fetching labelling.zip..."
local tmp; tmp="$(mktemp)"
curl -sf "${DL_BASE}/generic/xsource/${version}/labelling.zip" -o "$tmp"
unzip -qo "$tmp" -d "$dest"
rm "$tmp"
fi
# Frames are regenerated rather than shipped: they are ~320 MB of PNG that
# ffmpeg reproduces exactly from the clips. The manifests key on these
# filenames and on detection order within each frame, so the extraction
# settings must match the ones dump_faces.py ran against — hence fps and
# frame count are pinned here rather than left to the caller.
if ! command -v ffmpeg >/dev/null; then
echo " [warn] ffmpeg not found — frames not regenerated; the study" >&2
echo " scripts will fail until you extract them" >&2
return
fi
for clip in 5157339 5157344; do
[ -f "${dest}/clips/${clip}.mp4" ] || continue
if [ -f "${dest}/frames/d${clip}_001.png" ]; then
echo " frames for ${clip} already present, skipping"
continue
fi
echo " extracting frames for ${clip}..."
ffmpeg -v error -i "${dest}/clips/${clip}.mp4" -vf fps=2 -frames:v 24 \
"${dest}/frames/d${clip}_%03d.png"
done
echo " verifying the labelled set..."
if (cd "$dest" && python3 verify_labels.py >/dev/null 2>&1); then
echo " verify_labels.py passed"
else
echo " [warn] verify_labels.py failed — run it directly to see why." >&2
echo " A frame/manifest mismatch means the extraction settings" >&2
echo " differ from the ones the crops were dumped against." >&2
fi
}
if [ $# -eq 0 ]; then
echo "usage: $0 galleries [version]" >&2
echo " $0 montage-frames <film-slug> [version]" >&2
echo " $0 experiment-data [version]" >&2
echo " $0 report-highlights <name> [version]" >&2
echo " $0 xsource [version]" >&2
exit 1
fi
@@ -115,8 +176,13 @@ case "$TARGET" in
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version report-highlights)"
pull_report_highlight "$VERSION" "$NAME"
;;
xsource)
VERSION="${2:-latest}"
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version xsource)"
pull_xsource "$VERSION"
;;
*)
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, or report-highlights)" >&2
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2
exit 1
;;
esac
scripts/artifacts/push_artifacts.sh
+31 -2
View File
@@ -12,6 +12,7 @@
# scripts/artifacts/push_artifacts.sh montage-frames
# scripts/artifacts/push_artifacts.sh experiment-data
# scripts/artifacts/push_artifacts.sh report-highlights
# scripts/artifacts/push_artifacts.sh xsource
# scripts/artifacts/push_artifacts.sh galleries montage-frames experiment-data report-highlights
#
# Package layout (owner=dtourolle, repo=scene-actor-extraction):
@@ -109,8 +110,35 @@ push_report_highlights() {
upload "report-highlights" "germar_beats_xray.jpg" "$src"
}
push_xsource() {
echo "=== xsource (version ${VERSION}) ==="
local root="${REPO_ROOT}/experiments/xsource"
if [ ! -d "$root/labelling" ]; then
echo " no experiments/xsource/labelling found, skipping" >&2
return
fi
# Source recordings. Already compressed, so uploaded as-is rather than zipped.
shopt -s nullglob
for f in "$root"/clips/*.mp4; do
upload "xsource" "$(basename "$f")" "$f"
done
shopt -u nullglob
# The hand-sorted crops and their manifests. This is human ground truth and
# the expensive part of the study — a person looked at every crop and put it
# in a folder. Frames are deliberately NOT pushed: they are deterministic
# from the clips, and pulling regenerates them.
local tmp; tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' RETURN
local zipfile="${tmp}/labelling.zip"
(cd "$root" && zip -qr "$zipfile" labelling -x 'labelling/*.html' -x 'labelling/review_*.jpg' \
-x 'labelling/verify_*.jpg')
upload "xsource" "labelling.zip" "$zipfile"
}
if [ $# -eq 0 ]; then
echo "usage: $0 <galleries|montage-frames|experiment-data|report-highlights> [...]" >&2
echo "usage: $0 <galleries|montage-frames|experiment-data|report-highlights|xsource> [...]" >&2
exit 1
fi
@@ -120,7 +148,8 @@ for target in "$@"; do
montage-frames) push_montage_frames ;;
experiment-data) push_experiment_data ;;
report-highlights) push_report_highlights ;;
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, or report-highlights)" >&2; exit 1 ;;
xsource) push_xsource ;;
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2; exit 1 ;;
esac
done
scripts/optimizer/SCHEMA.md
+88 -8
View File
@@ -19,11 +19,32 @@ variable-length HDF5 types and reads straight into numpy.
/ (root)
attrs:
schema_version : int = 1
movie : str (source video path)
sample_fps : float
embed_dim : int = 512
embedder_model : str basename of the embedding model (GR-004)
embedder_sha256: str SHA-256 of that model file (GR-004)
# ── what produced the vectors (GR-004) ──────────────────────────────────
embedder_model : str basename of the embedding model
embedder_sha256: str SHA-256 of that model file
# ── what produced the faces (VR-010) ────────────────────────────────────
detector_model : str basename of the detector .onnx
detector_conf : float score floor a detection had to clear to be dumped
detector_nms : float NMS IoU threshold
min_face_px : float minimum box side, ORIGINAL-resolution px (AR-002)
max_faces : int per-frame cap; 0 = uncapped, the default (AR-003)
# ── what produced the frames (VR-010) ───────────────────────────────────
movie : str source video path
sample_fps : float frames analysed per second of movie
start_sec : float seek point
end_sec : float stop point; -1 = end of file
cut_threshold : float histogram correlation below which is_cut fires
dense_scale : float decoded-frame downscale in dense mode; 1 = off
bbox_upscale : float multiply faces/bbox and faces/landmarks by this to
reach original video pixels; 1 when dense_scale is 1
scene_detect : uint8 0/1 — was TransNetV2 running at all (see below)
# ── downstream setting recorded for comparability (VR-010) ──────────────
track_assoc_min_prob : float the run's tracker admission probability
frames/ group — one row per sampled frame
timestamp_sec : float64 [F]
@@ -35,14 +56,47 @@ variable-length HDF5 types and reads straight into numpy.
faces/ group — one row per detected face, concatenated
embedding : float32 [N, 512] L2-normalised ArcFace embedding
bbox : float32 [N, 4] x, y, w, h in original video pixels
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order
bbox : float32 [N, 4] x, y, w, h in DECODED-frame pixels
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order,
same space as bbox
confidence : float32 [N] detector confidence
```
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.
## Provenance (VR-010)
The attributes above are not documentation; they are the only thing that makes a
dump interpretable. Two dumps of the same film at `detector_conf` 0.5 and 0.7, or
at `dense_scale` 1.0 and 0.5, or with scene detection on and off, are different
measurements of different things — and they are byte-shaped identically. Without
provenance a consumer that mixes them gets a plausible number from an incoherent
input, and nothing anywhere reports a problem.
**`scene_detect` is the one that cannot be inferred.** `is_scene_boundary` is
all-zero both when TransNetV2 found no boundaries in the clip and when it was
never enabled, and those mean opposite things: the first says *this footage has
no shot changes*, the second says *nobody looked*. A consumer that reads the
array alone must guess. The flag is what removes the guess. (`dump_embeddings`
has no `--scene-detect`, so every dump it writes records `false` — which is
exactly the fact the committed fixtures needed to state.)
**`bbox_upscale` is recorded, not applied.** See the coordinate-space note below.
Reading is by name with a default or an existence check on **both** sides —
`replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in
`src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are
additive and `schema_version` stays 1: a pre-VR-010 dump still loads, and a
post-VR-010 dump still reads on old code.
A missing attribute means **unknown**, never a default value. Substituting
`detector_conf = 0.5` for a dump that does not say so manufactures the provenance
the requirement exists to prevent — per `docs/requirements.md`, *"a fixture whose
provenance is unknown is worse than no fixture, because it will be trusted."*
The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these
attributes; re-dump to bind them, as with GR-004.
## Model binding (GR-004)
`embedder_model` / `embedder_sha256` record which embedder produced every vector
@@ -57,10 +111,36 @@ warning, or a hard error under `SAE_REQUIRE_GALLERY_STAMP=1`) rather than as a
pass. Re-dump to bind an old dump; there is no in-place migration, because unlike
a gallery nobody can assert after the fact which model produced a vector.
## Coordinate space — `bbox`, `landmarks`, `bbox_upscale`
`bbox` and `landmarks` are in **decoded-frame pixels**: exactly the numbers SCRFD
produced, untransformed. To reach original video pixels, multiply by
`bbox_upscale`. With `dense_scale == 1` (the default, and every committed
fixture) `bbox_upscale == 1` and the two spaces coincide.
> Earlier revisions of this document claimed the upscale was applied at dump time.
> It never was. `embedding_dump_node.hpp` writes `f.bbox` raw; the upscale lives
> in `identity_matcher_node.hpp`, which is *downstream* of the dump tap. The
> claim was harmless only because `dense_scale` was 1 in practice.
The fix is to record the factor rather than to apply it, because the dump's whole
contract is to be a **faithful tap** at the `EmbeddedSceneFrame` channel — VR-002
requires replay to drive the real nodes, and a replay is only equivalent to the
live run if the tracker is fed the geometry the live tracker saw. Rescaling at
the tap would break that: the replayed tracker would associate on boxes the live
one never received. Two further reasons:
- The matcher's upscale is applied to `bbox` **only**, not to `landmarks`.
Pre-multiplying at the tap would leave the two arrays in different coordinate
spaces inside one file — a worse trap than the one being fixed.
- Pre-multiplying is lossy in the sense that matters: a dump that had been
upscaled would be indistinguishable from one taken at `dense_scale == 1`, so
you would have to record `bbox_upscale` anyway to know which you were holding.
## Invariants
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
- `bbox` is already mapped to original resolution (bbox_upscale applied at dump time),
matching what the identity matcher would emit.
- `bbox` and `landmarks` share one coordinate space; `bbox_upscale` maps both to
original resolution (see above).
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
- EOF sentinel frames are NOT written.
scripts/validation/README.md
+21
View File
@@ -79,9 +79,30 @@ The table caches nulls (tmdb ids TMDB has no IMDb id for) and checkpoints, so a
re-run only resolves new ids. TMDB is authoritative for this crosswalk — there is
no clean free bulk `tmdb_person ↔ nm` file, so we query the API once and cache.
## Minimum face size (VR-005)
`min_face_size.py` is a separate, self-contained study: it needs no video and no
ground truth, only the gallery mugshot cache. It holds out one image per actor,
degrades that probe to each candidate face size and matches it against a gallery
held at **native** resolution, reporting TPI/FPI per size — the measurement that
replaces AR-002's 66×66 px estimate.
```bash
python scripts/validation/min_face_size.py \
--images images --gallery gallery_lvface.h5 \
--arcface models/LVFace-B_Glint360K.onnx \
--actors 100 --out experiments/results/vr005_min_face_size
```
FPI grows with the number of actors competing, so a 100-actor run understates it
against a library of thousands: read FPI as relative across sizes, not as an
absolute rate. Re-run per `--arcface` model to see whether `min_face_px` should be
one constant or scale with the embedder (GR-004).
## Files
- `sample_eval.py` — CLI scorer.
- `ground_truth.py` — `XRayGroundTruth`, `MovieNetGroundTruth` loaders.
- `identity.py` — provider-agnostic match keys.
- `tmdb_imdb_map.py` — build/consult the cached `tmdb→imdb` crosswalk.
- `min_face_size.py` — VR-005 probe-size sweep (see above).
- `test_sample_eval.py` — self-contained tests (`python scripts/validation/test_sample_eval.py`).
scripts/validation/min_face_size.py
+822
View File
@@ -0,0 +1,822 @@
#!/usr/bin/env python3
"""
min_face_size.py — VR-005: at what face size do embeddings stop identifying people?
TRACES: VR-005
`min_face_px` is currently a working estimate (AR-002: 66x66 px in original video
resolution). This script replaces the guess with a measurement, using only gallery
mugshots already on disk — no video, no C++ changes.
Protocol
--------
1. Select ~100 gallery actors that have more than one mugshot.
2. Per actor hold out ONE image as the *probe*; that actor's remaining images stay
in the gallery at native resolution.
3. For each target size S, take the probe's native aligned 112x112 crop, downscale
it to SxS and upscale it back to 112x112, then embed. Detail is genuinely
destroyed and then the same warp the pipeline applies is re-applied on top —
which is what a face detected at SxS in a frame actually suffers.
4. Match each degraded probe against the whole gallery.
5. Record, per size, TPI (identified as the correct actor) and FPI (identified as
someone else). Everything else is an unidentified probe (TBI).
The asymmetry is the point: **the gallery stays at native resolution and only the
probe degrades.** That is the production case — reference mugshots are clean, the
face coming out of the video is small. Degrading both sides would measure
something the pipeline never does.
What this deliberately does NOT measure
---------------------------------------
The cosine between the size-S embedding and the native embedding of the *same*
image. That is embedding *drift*, and it answers the wrong question: an embedding
can drift a long way and stay perfectly separable, or drift a little in a
direction that destroys separation. What matters is the decision the pipeline
makes — probe against a competing gallery — so that is what is recorded.
CAVEAT — FPI IS RELATIVE, NOT ABSOLUTE
--------------------------------------
False positives grow with the number of actors competing for the match. A
~100-actor gallery therefore *understates* the false-positive rate against a
production library of thousands. Read the FPI column as a relative curve across
sizes ("FPI is 4x worse at 32 px than at 64 px"), never as the rate you would see
in production. Re-run with `--actors` at production scale before setting a
threshold from an absolute FPI number.
Decision rule
-------------
Per the repo invariant (CLAUDE.md: "always use the calibrated probability, never a
raw cosine"), identification goes through the same path as `identity_matcher_node`:
per-actor best-of-N cosine -> Platt sigmoid P(match) = sigma(a*sim + b + log-prior)
-> accept if P > `prob_threshold`. The sigmoid is fitted here by the same
histogram/gradient-descent procedure as `src/gallery/gallery_calibration.hpp`,
over the native gallery embeddings only (held-out probes are excluded, so the
calibration cannot see the images it will be scored on).
How this runs
-------------
Through the `sae_embed` bindings, which expose the shipped C++ stages directly:
`detect()`, `align_face()`, `embed_crops()` and `GalleryCalibration`. Nothing
here re-implements detection, the ArcFace warp, the embedder or the Platt fit.
That matters most for the calibration. A second copy of the sigmoid is exactly
where "always the calibrated probability, never a raw cosine" (AR-024) gets
broken without anyone noticing, because the copy keeps returning plausible
numbers after the original has moved. Scoring through the binding makes the rule
structural instead of remembered.
The backend is whichever was compiled in. Under `SAE_INFERENCE_BACKEND=ORT`
that is the reference fp32 path, which loads the .onnx directly. A TensorRT fp16
build is a *different realisation* of the same model and its embeddings are
measurably not the same vectors: on LVFace-B_Glint360K the stored TRT-fp16 gallery
agrees with an fp32 recompute of the same mugshot at only ~0.85 cosine, while
same-actor/different-actor separation is essentially unchanged (d' 5.3 vs 5.7).
Nothing here is invalidated by that — gallery and probes go through one session,
so the comparison is internally consistent — but the two embedding spaces are not
interchangeable, and `--verify-against <gallery.h5>` will show ~0.85, not ~1.0,
against a TRT-built gallery. It reports the separation of both sets alongside the
agreement so the two causes are distinguishable: a broken port collapses
separation, a different backend does not.
Secondary output (VR-005): running the sweep per `--arcface` model shows whether
`min_face_px` should be one constant at all, or should scale with the embedder —
which matters because the model is a build-time choice (GR-004).
Usage
-----
python scripts/validation/min_face_size.py \
--images images \
--gallery gallery_lvface.h5 \
--arcface models/LVFace-B_Glint360K.onnx \
--actors 100 --seed 0 \
--out experiments/results/vr005_min_face_size
Writes <out>.csv, <out>.json and <out>.png (plus <out>.per_probe.csv with
--per-probe).
"""
from __future__ import annotations
import argparse
import csv
import json
import random
import re
import sys
import time
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
def _find_sae_embed() -> Path | None:
"""Locate the built sae_embed module.
A git worktree has no build tree of its own, so fall back to the main
checkout via the shared git dir — otherwise running this study from a
feature worktree cannot find the bindings it now depends on.
"""
roots = [REPO]
try:
import subprocess
common = subprocess.run(["git", "-C", str(REPO), "rev-parse",
"--path-format=absolute", "--git-common-dir"],
capture_output=True, text=True, check=True).stdout.strip()
if common:
roots.append(Path(common).parent)
except Exception:
pass
for root in roots:
for b in ("build-ort", "build"):
if list((root / b).glob("sae_embed*.so")):
return root / b
return None
_SAE_BUILD = _find_sae_embed()
if _SAE_BUILD is None:
sys.exit("cannot find the built sae_embed module — build it with\n"
" cmake --build build-ort --target sae_embed")
sys.path.insert(0, str(_SAE_BUILD))
# Before cv2: OpenCV's DNN module loads the system libonnxruntime, which then
# shadows the one sae_embed links against and the import fails on a missing
# symbol version. Order matters here.
import sae_embed
import cv2
import numpy as np
IMAGE_EXTS = (".jpg", ".jpeg", ".png", ".webp")
JELLYFIN_ID_RE = re.compile(r"^[0-9a-f]{32}$")
# Interpolation used for the two halves of the degradation. Downscaling uses
# INTER_AREA (correct low-pass for shrinking, i.e. detail that a small detection
# genuinely never had); upscaling uses INTER_LINEAR, which is what warpAffine in
# align_face() uses when it blows a small detection up to 112x112.
INTERP = {
"area": cv2.INTER_AREA,
"linear": cv2.INTER_LINEAR,
"cubic": cv2.INTER_CUBIC,
"nearest": cv2.INTER_NEAREST,
"lanczos": cv2.INTER_LANCZOS4,
}
# House chart palette, shared with scripts/docs/experiment_charts.py so figures
# across the report read as one set.
INK, MUTED, GRID, SURFACE = "#0b0b0b", "#898781", "#e1e0d9", "#fcfcfb"
BLUE, GREEN, RED, AMBER = "#2a78d6", "#008300", "#e34948", "#eda100"
# ── Production stages, via the sae_embed bindings ─────────────────────────────
# detect / align_face / embed_crops / calibrate_gallery all call the shipped C++.
# There is deliberately no Python re-implementation of any of them: a second copy
# drifts from what ships, and the calibration is the one that must not — AR-024
# requires every similarity to pass through the same sigmoid the matcher uses.
# These two mirror constants in gallery_calibration.hpp. They are NOT a second
# copy of the fit — that is the binding's job — but the script reproduces the
# same dedup and eligibility filtering so the actor counts it reports describe
# the population the C++ actually fitted on. Keep them in step with the header.
MIN_EMB_FOR_POSITIVE = 5
DEDUP_SIM = 1.0 - 1e-7
class Stages:
"""Thin holder so the rest of the script has one object to call."""
def __init__(self, detector: str, arcface: str, conf: float, nms: float):
self.engine = sae_embed.FaceEmbedder(
detector_model=detector, arcface_model=arcface,
conf=conf, nms=nms, max_side=0)
def detect(self, img):
return self.engine.detect(img)
def align(self, img, landmarks):
return sae_embed.align_face(img, np.asarray(landmarks, dtype=np.float32).reshape(5, 2))
def enhance(self, img):
return sae_embed.enhance_for_retry(img)
def embed(self, crops):
"""(N,112,112,3) uint8 BGR -> (N,512) float32.
Chunked at the backend's max_batch: the engine does not split an
oversized request, so handing it a whole gallery at once asks CUDA for
a multi-gigabyte activation buffer and the allocator refuses.
"""
if not len(crops):
return np.zeros((0, 512), dtype=np.float32)
n = max(1, int(self.engine.max_batch))
arr = np.ascontiguousarray(np.stack(crops), dtype=np.uint8)
out = [np.asarray(self.engine.embed_crops(np.ascontiguousarray(arr[i:i + n])))
for i in range(0, len(arr), n)]
return np.concatenate(out, axis=0)
def calibrate_gallery(emb: np.ndarray, actor: np.ndarray) -> dict:
"""The production Platt fit (gallery_calibration.hpp), via the binding."""
cal = sae_embed.calibrate_gallery(
np.ascontiguousarray(emb, dtype=np.float32), [int(a) for a in actor])
print(f"[calibration] a={cal.a:.4f} b={cal.b:.4f} valid={cal.valid} "
f"boundary(P=0.5)=sim{cal.boundary_at(0.5):.4f}", file=sys.stderr)
# Held module-side rather than returned: the returned dict lands in the run
# metadata, and a native object there breaks the JSON dump.
_CAL["cal"] = cal
return {"a": float(cal.a), "b": float(cal.b), "valid": bool(cal.valid)}
def probability(sim, a: float, b: float, log_prior_odds: float = 0.0):
"""P(match) through GalleryCalibration — the C++ sigmoid, not a copy of it."""
cal = _CAL.get("cal")
if cal is None:
raise RuntimeError("probability() called before calibrate_gallery()")
sim = np.asarray(sim, dtype=np.float64)
flat = np.atleast_1d(sim).ravel()
out = np.array([cal.probability(float(v), log_prior_odds) for v in flat])
return out.reshape(sim.shape) if sim.shape else float(out[0])
_CAL: dict = {}
# ── Runtime / actor discovery ─────────────────────────────────────────────────
def normalise_name(name: str) -> str:
return re.sub(r"[^a-z0-9]+", "", name.lower())
def discover_actors(images_root: Path) -> list[dict]:
"""Enumerate the gallery-build image cache: <root>/<jellyfin_id>_<Name>/NN.jpg
(the layout make_jellyfin_gallery.py / reembed_gallery.py use)."""
actors = []
for d in sorted(p for p in images_root.iterdir() if p.is_dir()):
imgs = sorted(p for p in d.iterdir()
if p.is_file() and p.suffix.lower() in IMAGE_EXTS)
if not imgs:
continue
head, _, tail = d.name.partition("_")
if JELLYFIN_ID_RE.match(head) and tail:
jellyfin_id, name = head, tail.replace("_", " ")
else:
jellyfin_id, name = "", d.name.replace("_", " ")
actors.append({"dir": d, "jellyfin_id": jellyfin_id, "name": name,
"images": imgs})
return actors
def gallery_keys(gallery_path: Path) -> tuple[set[str], set[str]]:
"""(jellyfin ids, normalised names) of the actors an existing gallery holds."""
from sae_gallery import load_gallery_hdf5
g = load_gallery_hdf5(gallery_path)
ids = {a.get("jellyfin_id", "") for a in g["actors"] if a.get("jellyfin_id")}
names = {normalise_name(a.get("name", "")) for a in g["actors"] if a.get("name")}
return ids, names
# ── Degradation ───────────────────────────────────────────────────────────────
def degrade(crop: np.ndarray, size: int, down: int, up: int) -> np.ndarray:
"""Throw away everything a face detected at size x size never had, then warp
it back up to the 112x112 the embedder is fed."""
if size == 112:
return crop
small = cv2.resize(crop, (size, size), interpolation=down)
return cv2.resize(small, (112, 112), interpolation=up)
# ── Reporting ─────────────────────────────────────────────────────────────────
CAVEAT = (
"CAVEAT: FPI grows with gallery size. This ran against {n_actors} actors, so it "
"UNDERSTATES the false-positive rate of a production library of thousands. Read "
"FPI as relative across sizes, not as an absolute rate."
)
def write_plot(rows: list[dict], out_png: Path, meta: dict) -> None:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
plt.rcParams.update({
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE,
"savefig.facecolor": SURFACE, "text.color": INK,
"axes.edgecolor": MUTED, "axes.labelcolor": INK,
"xtick.color": MUTED, "ytick.color": MUTED,
"axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.8,
"axes.spines.top": False, "axes.spines.right": False,
})
sizes = [r["size_px"] for r in rows]
fig, ax = plt.subplots(figsize=(9, 5.6))
ax.plot(sizes, [100 * r["tpi_rate"] for r in rows], "-o", color=GREEN,
lw=2, label="TPI — identified, correct actor")
ax.plot(sizes, [100 * r["fpi_rate"] for r in rows], "-s", color=RED,
lw=2, label="FPI — identified, wrong actor")
ax.plot(sizes, [100 * r["unidentified_rate"] for r in rows], color=MUTED,
marker="^", lw=1.4, ls="--", label="unidentified (below P threshold)")
ax.plot(sizes, [100 * r["rank1_rate"] for r in rows], ":", color=BLUE,
lw=1.6, label="rank-1 correct (ignoring threshold)")
op = meta.get("operating_point")
if op:
ax.axvline(op, color=AMBER, lw=1.6, ls="-.", zorder=1)
ax.annotate(f"operating point {op} px", xy=(op, 50),
xytext=(4, 0), textcoords="offset points",
color=AMBER, fontsize=9, rotation=90, va="center")
ax.set_xlabel("probe face size before upscaling (px)")
ax.set_ylabel("% of probes")
ax.set_ylim(-2, 102)
ax.set_xticks(sizes)
ax.set_title(f"VR-005 — identification vs. probe face size\n"
f"{meta['model']}, {meta['n_actors']} actors, "
f"{meta['n_probes']} probes/size, gallery at native resolution",
fontsize=11, loc="left")
ax.legend(frameon=False, fontsize=9, loc="center left")
fig.text(0.01, 0.005, CAVEAT.format(n_actors=meta["n_actors"]),
fontsize=7.5, color=MUTED, wrap=True)
fig.tight_layout(rect=(0, 0.05, 1, 1))
out_png.parent.mkdir(parents=True, exist_ok=True)
fig.savefig(out_png, dpi=150)
plt.close(fig)
def pick_operating_point(rows: list[dict], retention: float, fpi_slack: float) -> int | None:
"""Smallest size that keeps `retention` of the undegraded (112 px control)
TPI rate and does not add more than `fpi_slack` absolute FPI over it.
A stated rule, not a magic number — change the rule, not the answer."""
control = next((r for r in rows if r["size_px"] == 112), None)
if control is None or control["n_probes"] == 0:
return None
tpi_floor = retention * control["tpi_rate"]
fpi_ceil = control["fpi_rate"] + fpi_slack
ok = [r["size_px"] for r in rows
if r["tpi_rate"] >= tpi_floor and r["fpi_rate"] <= fpi_ceil]
return min(ok) if ok else None
# ── Main ──────────────────────────────────────────────────────────────────────
def main() -> int:
p = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--images", required=True,
help="gallery image cache root (<jellyfin_id>_<Name>/NN.jpg)")
p.add_argument("--gallery", default=None,
help="gallery .h5 — restricts the actor pool to its members")
p.add_argument("--out", default=str(REPO / "experiments/results/vr005_min_face_size"),
help="output path prefix (.csv/.json/.png are appended)")
p.add_argument("--per-probe", action="store_true",
help="also write <out>.per_probe.csv, one row per probe per size")
p.add_argument("--actors", type=int, default=100, help="actors to sample (default 100)")
p.add_argument("--min-images", type=int, default=2,
help="minimum mugshots for an actor to be eligible (default 2)")
p.add_argument("--probes-per-actor", type=int, default=1,
help="images held out per actor; 1 is the VR-005 protocol")
p.add_argument("--seed", type=int, default=0, help="actor/probe selection seed")
p.add_argument("--keep-duplicates", action="store_true",
help="keep mugshots that are the same photograph twice; by "
"default they are dropped, since a probe identical to a "
"gallery reference is identified for free at every size")
p.add_argument("--sizes", default="12,16,20,24,32,40,48,56,64,72,80,96,112",
help="comma-separated probe sizes; 112 is the undegraded control")
p.add_argument("--models-dir", default=str(REPO / "models"))
p.add_argument("--arcface", default=None,
help="embedder ONNX (default <models-dir>/LVFace-B_Glint360K.onnx)")
p.add_argument("--detector", default=None,
help="SCRFD ONNX (default <models-dir>/scrfd_500m_bnkps.onnx)")
p.add_argument("--conf", type=float, default=0.5, help="detector confidence")
p.add_argument("--nms", type=float, default=0.4, help="detector NMS IoU")
p.add_argument("--max-side", type=int, default=500,
help="downscale mugshots to this longest side before detection, "
"matching the gallery builders' embedder settings")
p.add_argument("--prob-threshold", type=float, default=0.754,
help="accept if P(match) exceeds this (Config::prob_threshold)")
p.add_argument("--match-prior", type=float, default=0.5,
help="base-rate prior (Config::match_prior)")
p.add_argument("--calib-a", type=float, default=None,
help="override the fitted sigmoid scale instead of fitting")
p.add_argument("--calib-b", type=float, default=None,
help="override the fitted sigmoid bias instead of fitting")
p.add_argument("--down-interp", default="area", choices=sorted(INTERP),
help="interpolation for the 112 -> S downscale (default area)")
p.add_argument("--up-interp", default="linear", choices=sorted(INTERP),
help="interpolation for the S -> 112 upscale (default linear, "
"as warpAffine uses in align_face)")
p.add_argument("--tpi-retention", type=float, default=0.95,
help="operating point keeps this fraction of the control TPI rate")
p.add_argument("--fpi-slack", type=float, default=0.01,
help="operating point may add at most this absolute FPI over control")
p.add_argument("--verify-against", default=None,
help="gallery .h5 built from --images with the same model: report "
"agreement and separation of recomputed vs stored embeddings "
"(a TensorRT-built gallery will not agree; see the docstring)")
args = p.parse_args()
if (args.calib_a is None) != (args.calib_b is None):
return err("--calib-a and --calib-b must be given together")
models_dir = Path(args.models_dir)
arcface = Path(args.arcface) if args.arcface else models_dir / "LVFace-B_Glint360K.onnx"
detector = Path(args.detector) if args.detector else models_dir / "scrfd_500m_bnkps.onnx"
for path, what in ((arcface, "embedder"), (detector, "detector")):
if not path.is_file():
return err(f"{what} model not found: {path}\n"
f"Run: bash scripts/download_models.sh")
images_root = Path(args.images)
if not images_root.is_dir():
return err(f"image cache not found: {images_root}")
sizes = sorted({int(s) for s in args.sizes.split(",") if s.strip()})
if not sizes:
return err("--sizes is empty")
if args.probes_per_actor < 1:
return err("--probes-per-actor must be >= 1")
cv2.setRNGSeed(args.seed) # estimateAffinePartial2D's RANSAC draws from this
# ── actor pool ────────────────────────────────────────────────────────────
pool = discover_actors(images_root)
print(f"[select] {len(pool)} actor dirs with images under {images_root}",
file=sys.stderr)
if args.gallery:
ids, names = gallery_keys(Path(args.gallery))
pool = [a for a in pool
if (a["jellyfin_id"] and a["jellyfin_id"] in ids)
or normalise_name(a["name"]) in names]
print(f"[select] {len(pool)} of them are in {args.gallery}", file=sys.stderr)
need = max(args.min_images, args.probes_per_actor + 1)
eligible = [a for a in pool if len(a["images"]) >= need]
print(f"[select] {len(eligible)} have >= {need} mugshots", file=sys.stderr)
if len(eligible) < 2:
return err(f"need at least 2 actors with >= {need} mugshots; found "
f"{len(eligible)}. Build the image cache first "
f"(scripts/make_jellyfin_gallery.py) or lower --min-images.")
rng = random.Random(args.seed)
selected = sorted(rng.sample(eligible, min(args.actors, len(eligible))),
key=lambda a: a["dir"].name)
if len(selected) < args.actors:
print(f"[select] WARNING: only {len(selected)} eligible actors, "
f"--actors {args.actors} requested. FPI is gallery-size dependent — "
f"see the caveat.", file=sys.stderr)
# ── detect + align every mugshot of the selected actors, once ─────────────
stages = Stages(str(detector), str(arcface), args.conf, args.nms)
print(f"[models] detector={detector.name} embedder={arcface.name} "
f"batch={stages.engine.max_batch} (provider chosen by the C++ backend: "
f"CUDA, then ROCm, then CPU)", file=sys.stderr)
t0 = time.time()
crops: list[np.ndarray] = []
rows: list[dict] = [] # parallel to crops: {actor, actor_idx, image}
actors: list[dict] = []
n_nodetect = 0
for a in selected:
actor_crops, actor_paths = [], []
for img_path in a["images"]:
img = cv2.imread(str(img_path))
if img is None:
n_nodetect += 1
continue
if args.max_side > 0 and max(img.shape[:2]) > args.max_side:
s = args.max_side / max(img.shape[:2])
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
faces = stages.detect(img)
if not faces:
enhanced = stages.enhance(img)
faces = stages.detect(enhanced)
if faces:
img = enhanced
if not faces:
n_nodetect += 1
continue
best = max(faces, key=lambda f: f.confidence)
crop = stages.align(img, best.landmarks)
if crop is None:
n_nodetect += 1
continue
actor_crops.append(crop)
actor_paths.append(img_path)
if len(actor_crops) < args.probes_per_actor + 1:
continue
ai = len(actors)
actors.append({"name": a["name"], "jellyfin_id": a["jellyfin_id"],
"dir": a["dir"].name, "n_images": len(actor_crops)})
for crop, img_path in zip(actor_crops, actor_paths):
rows.append({"actor_idx": ai, "image": str(img_path)})
crops.append(crop)
if len(actors) % 20 == 0:
print(f" [align] {len(actors)}/{len(selected)} actors, "
f"{len(crops)} crops", file=sys.stderr)
if len(actors) < 2:
return err(f"only {len(actors)} actors survived detection/alignment — "
f"nothing to match against")
print(f"[align] {len(actors)} actors, {len(crops)} aligned crops, "
f"{n_nodetect} images skipped (no face / unreadable) in "
f"{time.time() - t0:.1f}s", file=sys.stderr)
actor_of = np.array([r["actor_idx"] for r in rows], dtype=int)
# ── embed everything at native resolution ────────────────────────────────
t0 = time.time()
native = stages.embed(crops)
print(f"[embed] {len(crops)} native crops in {time.time() - t0:.1f}s",
file=sys.stderr)
if args.verify_against:
verify_embeddings(Path(args.verify_against), rows, native, actor_of)
# ── drop duplicate mugshots ───────────────────────────────────────────────
# The cache holds the same photograph twice for some actors (two provider
# URLs, one picture). A probe that is identical to a gallery reference is
# identified for free at every size, which flatters the whole curve, so
# remove duplicates the same way calibrate_gallery does.
n_dup = 0
if not args.keep_duplicates:
keep = np.ones(len(rows), bool)
for ai in range(len(actors)):
kept: list[int] = []
for i in np.nonzero(actor_of == ai)[0]:
if any(float(native[i] @ native[k]) > DEDUP_SIM for k in kept):
keep[i] = False
else:
kept.append(int(i))
n_dup = int((~keep).sum())
# An actor left with too few distinct mugshots to hold one out drops out.
counts = np.bincount(actor_of[keep], minlength=len(actors))
drop_actor = counts < args.probes_per_actor + 1
keep &= ~drop_actor[actor_of]
remap = np.full(len(actors), -1, dtype=int)
remap[~drop_actor] = np.arange(int((~drop_actor).sum()))
actors = [a for a, d in zip(actors, drop_actor) if not d]
rows = [r for r, k in zip(rows, keep) if k]
crops = [c for c, k in zip(crops, keep) if k]
native = native[keep]
actor_of = remap[actor_of[keep]]
for r, ai in zip(rows, actor_of):
r["actor_idx"] = int(ai)
for ai, a in enumerate(actors):
a["n_images"] = int(np.sum(actor_of == ai))
print(f"[dedup] dropped {n_dup} duplicate mugshots and "
f"{int(drop_actor.sum())} actors left with too few; "
f"{len(actors)} actors, {len(rows)} images remain", file=sys.stderr)
if len(actors) < 2:
return err("fewer than 2 actors survive de-duplication — the image "
"cache holds too few distinct mugshots")
# ── hold out the probes ───────────────────────────────────────────────────
is_probe = np.zeros(len(rows), bool)
for ai in range(len(actors)):
idx = np.nonzero(actor_of == ai)[0]
# Seeded per actor so the choice does not depend on iteration order.
r = random.Random(f"{args.seed}:{actors[ai]['dir']}")
for pick in r.sample(list(idx), args.probes_per_actor):
is_probe[pick] = True
probe_rows = np.nonzero(is_probe)[0]
gal_rows = np.nonzero(~is_probe)[0]
print(f"[holdout] {len(probe_rows)} probes held out, "
f"{len(gal_rows)} gallery embeddings remain", file=sys.stderr)
gal_emb = native[gal_rows]
gal_actor = actor_of[gal_rows]
probe_actor = actor_of[probe_rows]
# Per-actor column masks for the best-of-N scan (identity_matcher_node).
actor_cols = [np.nonzero(gal_actor == ai)[0] for ai in range(len(actors))]
have_refs = np.array([len(c) > 0 for c in actor_cols])
if not have_refs.all():
return err("an actor ended up with no gallery references left; "
"raise --min-images")
# ── calibration ───────────────────────────────────────────────────────────
if args.calib_a is not None:
cal = {"a": args.calib_a, "b": args.calib_b, "valid": True}
print(f"[calibration] using supplied a={cal['a']} b={cal['b']}", file=sys.stderr)
else:
cal = calibrate_gallery(gal_emb, gal_actor)
if not cal["valid"]:
return err(
"calibration could not be fitted, and this study will not fall back to a "
"raw cosine threshold (CLAUDE.md invariant). Use more actors with >= "
f"{MIN_EMB_FOR_POSITIVE} mugshots, or pass --calib-a/--calib-b from a "
"production gallery.")
log_prior_odds = float(np.log(args.match_prior / (1.0 - args.match_prior)))
# ── sweep ─────────────────────────────────────────────────────────────────
down, up = INTERP[args.down_interp], INTERP[args.up_interp]
probe_crops = [crops[i] for i in probe_rows]
results, per_probe = [], []
for size in sizes:
t0 = time.time()
degraded = [degrade(c, size, down, up) for c in probe_crops]
q = stages.embed(degraded)
sims = q @ gal_emb.T # [n_probe, n_gal]
best_per_actor = np.stack([sims[:, cols].max(axis=1) for cols in actor_cols],
axis=1) # [n_probe, n_actor]
best_actor = best_per_actor.argmax(axis=1)
best_sim = best_per_actor.max(axis=1)
p_match = np.asarray(probability(best_sim, cal["a"], cal["b"], log_prior_odds))
accept = p_match > args.prob_threshold
correct = best_actor == probe_actor
tpi = int(np.sum(accept & correct))
fpi = int(np.sum(accept & ~correct))
unid = int(np.sum(~accept))
n = len(probe_rows)
results.append({
"size_px": size,
"n_probes": n,
"tpi": tpi, "fpi": fpi, "unidentified": unid,
"tpi_rate": tpi / n, "fpi_rate": fpi / n, "unidentified_rate": unid / n,
"rank1_rate": float(np.mean(correct)),
"mean_best_sim": float(np.mean(best_sim)),
"mean_p_match": float(np.mean(p_match)),
"mean_sim_true_actor": float(np.mean(
best_per_actor[np.arange(n), probe_actor])),
})
if args.per_probe:
for j in range(n):
per_probe.append({
"size_px": size,
"probe_image": rows[probe_rows[j]]["image"],
"true_actor": actors[probe_actor[j]]["name"],
"matched_actor": actors[best_actor[j]]["name"],
"best_sim": float(best_sim[j]),
"p_match": float(p_match[j]),
"outcome": ("TPI" if accept[j] and correct[j]
else "FPI" if accept[j] else "unidentified"),
})
print(f"[sweep] {size:3d}px TPI {tpi:4d} ({100 * tpi / n:5.1f}%) "
f"FPI {fpi:4d} ({100 * fpi / n:5.1f}%) "
f"unid {unid:4d} ({100 * unid / n:5.1f}%) "
f"rank1 {100 * np.mean(correct):5.1f}% "
f"[{time.time() - t0:.1f}s]", file=sys.stderr)
op = pick_operating_point(results, args.tpi_retention, args.fpi_slack)
# ── outputs ───────────────────────────────────────────────────────────────
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
# Append rather than with_suffix() so a prefix containing a dot keeps its name.
csv_path = out.with_name(out.name + ".csv")
json_path = out.with_name(out.name + ".json")
png_path = out.with_name(out.name + ".png")
fields = list(results[0].keys())
with open(csv_path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=fields)
w.writeheader()
w.writerows(results)
meta = {
"requirement": "VR-005",
"caveat": CAVEAT.format(n_actors=len(actors)),
"model": arcface.stem,
"detector": detector.stem,
"backend": "sae_embed / the compiled-in inference backend (fp32 ONNX under "
"SAE_INFERENCE_BACKEND=ORT; a TensorRT fp16 build is a different "
"embedding space)",
"n_actors": len(actors),
"n_probes": len(probe_rows),
"n_gallery_embeddings": len(gal_rows),
"probes_per_actor": args.probes_per_actor,
"seed": args.seed,
"sizes": sizes,
"prob_threshold": args.prob_threshold,
"match_prior": args.match_prior,
"calibration": cal,
"calibration_source": "supplied" if args.calib_a is not None else "fitted",
"sim_boundary_at_threshold": float(
(np.log(args.prob_threshold / (1 - args.prob_threshold))
- cal["b"] - log_prior_odds) / cal["a"]),
"down_interp": args.down_interp,
"up_interp": args.up_interp,
"max_side": args.max_side,
"duplicate_mugshots_dropped": n_dup,
"operating_point_rule": (
f"smallest size retaining >= {args.tpi_retention:.0%} of the 112 px "
f"control TPI rate with <= +{args.fpi_slack:.1%} absolute FPI"),
"operating_point": op,
"images_skipped_no_face": n_nodetect,
"curve": results,
"actors": actors,
}
json_path.write_text(json.dumps(meta, indent=2) + "\n")
if per_probe:
pp = out.with_name(out.name + ".per_probe.csv")
with open(pp, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(per_probe[0].keys()))
w.writeheader()
w.writerows(per_probe)
print(f"[out] {pp}", file=sys.stderr)
write_plot(results, png_path, meta)
# ── stdout report ─────────────────────────────────────────────────────────
print(f"\nVR-005 — minimum face size, {arcface.stem}")
print(f"{len(actors)} actors, {len(probe_rows)} probes/size, "
f"{len(gal_rows)} gallery embeddings at native resolution")
print(f"identify when P>{args.prob_threshold}, i.e. cosine above "
f"{meta['sim_boundary_at_threshold']:.4f} under the calibration fitted "
f"on this gallery\n")
print(f"{'size':>5} {'TPI':>8} {'FPI':>8} {'unid':>8} {'rank1':>8} {'mean sim':>9}")
for r in results:
print(f"{r['size_px']:>5} {100 * r['tpi_rate']:>7.1f}% "
f"{100 * r['fpi_rate']:>7.1f}% {100 * r['unidentified_rate']:>7.1f}% "
f"{100 * r['rank1_rate']:>7.1f}% {r['mean_best_sim']:>9.4f}")
print(f"\noperating point: {op if op else 'none of the swept sizes qualifies'}"
f" ({meta['operating_point_rule']})")
print(f"\n{meta['caveat']}")
print(f"\n[out] {csv_path}\n[out] {json_path}\n[out] {png_path}")
return 0
def _separation(emb: np.ndarray, actor: np.ndarray) -> tuple[float, float, float]:
"""(mean same-actor sim, mean different-actor sim, d') — the property that has
to survive for an embedding space to be usable, whatever its coordinates."""
iu, ju = np.triu_indices(len(actor), k=1)
sims = (emb @ emb.T)[iu, ju]
same = actor[iu] == actor[ju]
pos = sims[same & (sims < 0.9999)] # drop duplicate source images
neg = sims[~same]
if pos.size < 2 or neg.size < 2:
return float("nan"), float("nan"), float("nan")
d = (pos.mean() - neg.mean()) / np.sqrt(0.5 * (pos.var() + neg.var()))
return float(pos.mean()), float(neg.mean()), float(d)
def verify_embeddings(gallery_path: Path, rows: list[dict], native: np.ndarray,
actor_of: np.ndarray) -> None:
"""Cross-check this script's ONNX port against a gallery built by the C++
pipeline from the same mugshots.
Agreement is ~1.0 only if that gallery was built with the same backend. A
TensorRT fp16 build lands around 0.85 on LVFace-B while separating just as
well, so the separation figures — not the agreement — are what says whether
the port is sound."""
from sae_gallery import load_gallery_hdf5
g = load_gallery_hdf5(gallery_path)
stored: dict[tuple[str, str], np.ndarray] = {}
for a in g["actors"]:
key = a.get("jellyfin_id") or normalise_name(a.get("name", ""))
for e, src in zip(a.get("embeddings", []), a.get("source_images", [])):
if src:
stored[(key, src)] = np.asarray(e, np.float32)
sims, paired_mine, paired_ref, paired_actor = [], [], [], []
for i, r in enumerate(rows):
path = Path(r["image"])
head, _, _ = path.parent.name.partition("_")
key = head if JELLYFIN_ID_RE.match(head) else normalise_name(
path.parent.name.replace("_", " "))
ref = stored.get((key, path.name))
if ref is None or ref.shape != native[i].shape:
continue
ref = ref / max(float(np.linalg.norm(ref)), 1e-6)
sims.append(float(native[i] @ ref))
paired_mine.append(native[i])
paired_ref.append(ref)
paired_actor.append(actor_of[i])
if not sims:
print(f"[verify] no overlap with {gallery_path} — nothing checked",
file=sys.stderr)
return
sims_arr = np.asarray(sims)
print(f"[verify] {len(sims)} embeddings vs {gallery_path.name}: "
f"mean cos={sims_arr.mean():.4f} min={sims_arr.min():.4f}",
file=sys.stderr)
act = np.asarray(paired_actor)
for label, mat in (("this script", np.asarray(paired_mine)),
("stored gallery", np.asarray(paired_ref))):
pos, neg, d = _separation(mat, act)
print(f"[verify] {label:>14s}: same-actor {pos:.3f} "
f"different-actor {neg:.3f} d'={d:.2f}", file=sys.stderr)
if sims_arr.mean() < 0.99:
print("[verify] embeddings differ from the stored gallery. If d' is "
"comparable this is a backend difference (e.g. a TensorRT fp16 "
"build), not a broken port; the study is self-consistent either "
"way. If d' collapsed, the port is wrong.", file=sys.stderr)
def err(msg: str) -> int:
print(f"error: {msg}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())
scripts/validation/test_audio_offset.py
+370
View File
@@ -0,0 +1,370 @@
#!/usr/bin/env python3
"""
VR-014 — the v1 audio signature recovers a known trim offset on real audio.
TRACES: UT-105, UT-106, UT-107, UT-108 | VR-014 | IR-004
python scripts/validation/test_audio_offset.py [build_dir]
The golden vector (IR-005) proves the *arithmetic* is identical in both
producers. It cannot prove the thing the signature exists for: that when the
same cut arrives trimmed differently, sliding one signature against the other
finds the true alignment and only the true alignment. Its fixture is a synthetic
tone sweep, which is pathologically easy to align; film dialogue and score are
not, and that is what this measures.
The signature is computed by the **shipped C++**, through the `sae_audio`
nanobind module — never a numpy port. A third implementation of a fingerprint
whose whole value rests on three implementations agreeing byte for byte would be
the one nobody checks against the golden vector.
The slide *is* written here in numpy, deliberately: matching is the consumer's
algorithm (server SPEC.md section 3), owned by the server and the jRay plugin,
not by this repo. Writing it out is what makes this a test of the signature
rather than a test of somebody's matcher.
Two independent offset mechanisms are checked, because they can fail
separately:
* a **window offset** (UT-105) — two 120 s excerpts taken from different
points, which is the alignment search itself; and
* a **head trim** (UT-106) — a real file with delta seconds removed from the
front, which additionally exercises the runtime/2 anchor: the window follows
the midpoint, so cutting delta from the head moves it by delta/2, not delta.
That factor of two is the easiest thing in the whole feature to get wrong
and nothing else checks it.
"""
import base64
import random
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
import numpy as np
REPO = Path(__file__).resolve().parent.parent.parent
BUILD = Path(sys.argv[1]) if len(sys.argv) > 1 else REPO / "build"
sys.path.insert(0, str(BUILD))
import sae_audio # noqa: E402
FIXTURE = REPO / "tests" / "fixtures" / "audio" / "bali_offset_200s.flac"
TONE = REPO / "tests" / "fixtures" / "audio" / "jray_audio_v1_tone.flac"
# Server SPEC.md section 3, "Matching and offset recovery". The cap is the
# spec's, not a convenience: +/-600 frames is ~56 s, which covers realistic trim
# differences, and an offset outside it must be declined rather than guessed at.
SEARCH_CAP_FRAMES = 600
AUDIO_TIER = 0.85
LOOSE_TIER = 0.60
TRIALS = 40
SEED = 20250731
HOP_SEC = sae_audio.hop_size / sae_audio.sample_rate
# What the offset is actually *for*: shifting scene windows, which are seconds
# long. Half a second of error is invisible against them, and that budget is
# what makes the numbers below readable — an offset is quantised to whole
# frames, so no correct answer can be worse than half a frame (46 ms) and the
# feature has an order of magnitude in hand before anything is at stake.
OFFSET_BUDGET_SEC = 0.5
def peak_bins(signature):
"""The per-frame peak band index, which is what the slide compares.
The `v1:` prefix is checked against the constant the C++ exports rather
than a literal, so a producer bump cannot be silently parsed as v1 here
(IR-008).
"""
prefix = sae_audio.version_prefix
if not signature.startswith(prefix):
raise AssertionError(f"signature is not {prefix!r}: {signature[:8]!r}")
packed = np.frombuffer(base64.b64decode(signature[len(prefix):]), dtype=np.uint8)
if np.any(packed & 0x80):
raise AssertionError("reserved bit set — not a structurally valid signature")
return packed >> 2
def best_match(reference, query, cap=SEARCH_CAP_FRAMES, slack=0):
"""Slide `query` against `reference`; return (score, offset_frames).
`offset` is how many frames later the query's window begins, so
``query[i]`` lines up with ``reference[i + offset]``. Score is the fraction
of overlapping frames whose peak bin agrees, exactly as the spec defines it.
`slack` widens what counts as agreement to a frame within +/-slack, which is
not the spec's rule — it is the candidate remedy UT-108 measures. It changes
the *score* only; the offset it reports is still a whole-frame alignment.
"""
best_score, best_offset = -1.0, 0
for offset in range(-cap, cap + 1):
if offset >= 0:
a, b = reference[offset:], query[: len(query) - offset]
else:
a, b = reference[: len(reference) + offset], query[-offset:]
n = min(len(a), len(b))
if n < 100: # too little overlap to mean anything
continue
a, b = a[:n], b[:n]
if slack == 0:
agree = a == b
else:
agree = np.zeros(n, dtype=bool)
for shift in range(-slack, slack + 1):
shifted = np.roll(a, shift)
# 255 is not a band index, so the wrapped end can never agree.
if shift > 0:
shifted[:shift] = 255
elif shift < 0:
shifted[shift:] = 255
agree |= shifted == b
score = float(np.mean(agree))
if score > best_score:
best_score, best_offset = score, offset
return best_score, best_offset
def decode_mono(path):
"""The whole fixture as float32 mono at 11025 Hz — the signature's own rate."""
raw = subprocess.run(
["ffmpeg", "-nostdin", "-v", "error", "-i", str(path),
"-ac", "1", "-ar", str(sae_audio.sample_rate), "-f", "f32le", "-"],
capture_output=True, check=True).stdout
return np.frombuffer(raw, dtype="<f4")
def trim_head(source, seconds, out):
"""`source` with `seconds` removed from the front — a differently trimmed release."""
subprocess.run(
["ffmpeg", "-nostdin", "-v", "error", "-y", "-ss", f"{seconds:.3f}", "-i", str(source),
"-ac", "1", "-ar", str(sae_audio.sample_rate), "-sample_fmt", "s16",
"-c:a", "flac", str(out)], check=True)
return out
def tier(score):
if score >= AUDIO_TIER:
return "audio"
return "loose" if score >= LOOSE_TIER else "none"
# ── The random trial set, signed once and reused ─────────────────────────────
def random_trials(pcm):
"""(reference bins, [(expected_frames, query bins)]) for TRIALS excerpts.
Signing 40 windows is the expensive part of this file, so UT-105 and UT-108
share one set — they ask different questions of the same measurements.
"""
window = sae_audio.window_samples
reference = peak_bins(sae_audio.signature_from_mono(pcm[:window]))
rng = random.Random(SEED)
queries = []
for _ in range(TRIALS):
# Within the search cap: past it, no offset is recoverable by
# construction, which UT-107 checks separately.
start = rng.randrange(0, SEARCH_CAP_FRAMES * sae_audio.hop_size)
signature = sae_audio.signature_from_mono(pcm[start:start + window])
assert signature is not None, "a full window must always sign"
queries.append((start / sae_audio.hop_size, peak_bins(signature)))
return reference, queries
# ── UT-105 — window offsets from random excerpt starts ───────────────────────
def test_random_window_offsets(reference, queries):
"""Every in-cap offset is recovered to the nearest frame, on real audio."""
rows = []
for want, query in queries:
score, offset = best_match(reference, query)
rows.append((want, offset, score, abs(want - round(want))))
expected = np.array([r[0] for r in rows])
offset = np.array([r[1] for r in rows])
score = np.array([r[2] for r in rows])
subframe = np.array([r[3] for r in rows])
error = np.abs(offset - expected)
# The offset is quantised to whole frames, so the best any correct answer
# can do is half a frame — 46 ms. What matters is the budget that half-frame
# is measured against, and it is an order of magnitude away from it.
assert error.max() <= 1.0, f"offset missed by {error.max():.2f} frames"
assert error.max() * HOP_SEC <= OFFSET_BUDGET_SEC, (
f"offset error {error.max() * HOP_SEC:.3f}s exceeds the {OFFSET_BUDGET_SEC}s budget")
# Never mistaken for different content. This is the floor that matters: the
# audio genuinely is the same cut, so a "no match" would be a false negative
# on the case the feature exists for.
assert score.min() >= LOOSE_TIER, f"same content scored {score.min():.3f}"
# An offset that lands near a frame boundary has no excuse: it should reach
# the top tier, and does.
aligned = subframe <= 0.1
assert aligned.any(), "seed no longer produces a near-aligned trial"
assert score[aligned].min() >= AUDIO_TIER, (
f"near-frame-aligned offset scored only {score[aligned].min():.3f}")
print(f"UT-105 {TRIALS} random window offsets, all within the +/-600 frame cap")
print(f" offset error : max {error.max():.2f} frames"
f" = {error.max() * HOP_SEC * 1000:.0f} ms, against a"
f" {OFFSET_BUDGET_SEC * 1000:.0f} ms budget")
print(f" score : min {score.min():.3f} median {np.median(score):.3f}"
f" max {score.max():.3f}")
print(" score by sub-frame misalignment — the offset is exact in every row:")
for lo, hi in ((0.0, 0.1), (0.1, 0.2), (0.2, 0.3), (0.3, 0.4), (0.4, 0.5)):
m = (subframe >= lo) & (subframe < hi)
if m.any():
print(f" {lo:.1f}-{hi:.1f} frame n={m.sum():2d}"
f" score {score[m].min():.3f}-{score[m].max():.3f}"
f" tier {tier(np.median(score[m]))}")
counts = {t: int(sum(1 for s in score if tier(s) == t)) for t in ("audio", "loose", "none")}
print(f" tiers : {counts}")
return counts
# ── UT-106 — head trims through real files, including the runtime/2 anchor ───
def test_head_trims():
"""A release with delta seconds of head removed aligns at delta/2 frames."""
reference = peak_bins(sae_audio.compute_signature(str(FIXTURE)))
results = []
with tempfile.TemporaryDirectory() as tmp:
for delta in (7.0, 23.5, 41.25, 60.0):
trimmed = trim_head(FIXTURE, delta, Path(tmp) / f"trim_{delta}.flac")
signature = sae_audio.compute_signature(str(trimmed))
assert signature is not None, f"trim of {delta}s should still sign"
score, offset = best_match(reference, peak_bins(signature))
# The window follows the midpoint, so removing delta from the head
# moves it by delta/2 — not by delta.
expected = (delta / 2.0) / HOP_SEC
assert abs(offset - expected) <= 1.0, (
f"head trim {delta}s: expected ~{expected:.1f} frames, got {offset}")
assert score >= LOOSE_TIER, f"head trim {delta}s scored {score:.3f}"
results.append((delta, expected, offset, score))
print("UT-106 head trims through the real decode path (compute_signature on a file)")
for delta, expected, offset, score in results:
print(f" -{delta:6.2f}s head expected {expected:7.2f} fr"
f" recovered {offset:5d} score {score:.3f} ({tier(score)})")
# ── UT-107 — what must NOT match ─────────────────────────────────────────────
def test_declines(pcm, reference):
"""Out-of-cap offsets and unrelated content are declined, not guessed at."""
window = sae_audio.window_samples
beyond = int(75.0 * sae_audio.sample_rate) # ~807 frames, past the cap
assert beyond + window <= len(pcm), "fixture too short for the out-of-cap case"
far = peak_bins(sae_audio.signature_from_mono(pcm[beyond:beyond + window]))
score_beyond, offset_beyond = best_match(reference, far)
assert score_beyond < LOOSE_TIER, (
f"an offset past the cap scored {score_beyond:.3f} at {offset_beyond} — "
"the search invented an alignment rather than declining")
tone = peak_bins(sae_audio.compute_signature(str(TONE)))
score_tone, offset_tone = best_match(reference, tone)
assert score_tone < LOOSE_TIER, f"unrelated content scored {score_tone:.3f}"
print("UT-107 declines rather than guesses")
print(f" offset past the +/-600 frame cap : best {score_beyond:.3f}"
f" at {offset_beyond} ({tier(score_beyond)})")
print(f" unrelated content (tone fixture) : best {score_tone:.3f}"
f" at {offset_tone} ({tier(score_tone)})")
return far, tone
# ── UT-108 — the sub-frame demotion, and what one frame of slack costs ───────
def test_scoring_slack(reference, queries, far, tone):
"""Measured: +/-1 frame of slack in the *score* restores the `audio` tier.
UT-105 leaves a real question open. Every offset is right, but two thirds of
them score below the server's 0.85 `audio` threshold purely because the two
windows' frame grids do not coincide — so a correctly aligned release is
demoted to `loose`, which is the tier meaning "possibly the same cut,
degraded audio". The obvious remedy is to stop demanding that frames line up
exactly, and the question is what that costs in discrimination.
Nothing is asserted about the spec's own rule here; this measures a
candidate change to it, which is the server's to make (SPEC.md section 3).
"""
print("UT-108 cost of relaxing the score's frame alignment")
print(f" {'slack':>5} {'audio':>6} {'loose':>6} {'none':>5}"
f" {'min true':>9} {'worst err':>10} {'unrelated':>10} {'out-of-cap':>11}")
measured = {}
for slack in (0, 1, 2):
score, error = [], []
for want, query in queries:
s, offset = best_match(reference, query, slack=slack)
score.append(s)
error.append(abs(offset - want))
score, error = np.array(score), np.array(error)
false_tone, _ = best_match(reference, tone, slack=slack)
false_far, _ = best_match(reference, far, slack=slack)
counts = {t: int(sum(1 for s in score if tier(s) == t)) for t in ("audio", "loose", "none")}
measured[slack] = (score, error, max(false_tone, false_far))
print(f" {slack:>5} {counts['audio']:>6} {counts['loose']:>6} {counts['none']:>5}"
f" {score.min():>9.3f} {error.max() * HOP_SEC * 1000:>7.0f} ms"
f" {false_tone:>10.3f} {false_far:>11.3f}")
score, error, worst_false = measured[1]
# One frame of slack lifts every correct alignment to the top tier...
assert score.min() >= AUDIO_TIER, (
f"one frame of slack still leaves a true match at {score.min():.3f}")
# ...without narrowing the gap that makes the threshold mean anything...
assert worst_false < LOOSE_TIER, (
f"slack lifted a false match to {worst_false:.3f}")
# ...and the offset it costs is still far inside the budget: the score's
# peak flattens slightly, so the argmax can pick an adjacent frame.
assert error.max() * HOP_SEC <= OFFSET_BUDGET_SEC, (
f"slack cost {error.max() * HOP_SEC:.3f}s of offset accuracy")
print(f" +/-1 frame: every true match reaches `audio` (min {score.min():.3f}),"
f" worst false stays at {worst_false:.3f},")
print(f" and the offset costs {error.max() * HOP_SEC * 1000:.0f} ms of a"
f" {OFFSET_BUDGET_SEC * 1000:.0f} ms budget. +/-2 buys nothing more.")
def main():
if not FIXTURE.exists():
print(f"missing fixture {FIXTURE} — regenerate with make_offset_fixture.sh", file=sys.stderr)
return 2
if shutil.which("ffmpeg") is None:
print("this validation needs the ffmpeg CLI to trim the fixture", file=sys.stderr)
return 2
print(f"VR-014 audio-signature offset recovery on {FIXTURE.name}")
print(f" {sae_audio.expected_frames} frames per signature,"
f" {HOP_SEC * 1000:.2f} ms per frame, cap +/-{SEARCH_CAP_FRAMES} frames")
pcm = decode_mono(FIXTURE)
reference, queries = random_trials(pcm)
counts = test_random_window_offsets(reference, queries)
test_head_trims()
far, tone = test_declines(pcm, reference)
test_scoring_slack(reference, queries, far, tone)
print()
print(f"PASS — every in-cap offset recovered to the nearest frame, worst"
f" {1000 * HOP_SEC / 2:.0f} ms against a {OFFSET_BUDGET_SEC * 1000:.0f} ms budget.")
if counts["audio"] < TRIALS:
# Stated rather than asserted against the spec's rule: the offset is
# right in every case, so this is the 0.85 threshold meeting a sub-frame
# shift, not a defect in the signature. The threshold was calibrated on
# a re-encode at zero offset, where the score is 1.00. UT-108 measures
# the remedy; adopting it is the server spec's call, not this repo's.
print(f"NOTE — under the spec's exact-frame score only {counts['audio']}/{TRIALS}"
f" reach `audio`; {counts['loose']} are demoted to `loose` by sub-frame"
" shift alone. See UT-108.")
return 0
if __name__ == "__main__":
sys.exit(main())
src/audio_bindings.cpp
+125
View File
@@ -0,0 +1,125 @@
// sae_audio — Python module wrapping the v1 audio signature (audio_signature.*).
//
/// TRACES: IR-004, IR-005 | SR-003
//
// Exists so a study or a test can drive the **shipped** signature code from
// Python instead of porting the DSP to numpy. A numpy port would be a third
// implementation of a fingerprint that only works if every implementation
// agrees byte for byte, and it would be the one nobody checks against the
// golden vector — so the offset-recovery validation (VR-014) calls this.
//
// Bound with nanobind, as `sae_embed` and `sae_kpn` are. Not pybind11: a second
// binding framework in one build is a second set of ABI and lifetime rules to
// get right, for a module that needs nothing nanobind lacks.
//
// The module deliberately stops at the producer's edge. Matching — sliding one
// signature against another and scoring the overlap — is the *consumer's*
// algorithm (server SPEC §3, and the jRay plugin implements it), so it is not
// bound here and a caller writing a slide in numpy is not re-implementing
// anything this repo owns.
#include "audio_signature.hpp"
#include <nanobind/nanobind.h>
#include <nanobind/ndarray.h>
#include <nanobind/stl/optional.h>
#include <nanobind/stl/pair.h>
#include <nanobind/stl/string.h>
#include <nanobind/stl/vector.h>
#include <cstdint>
#include <optional>
#include <string>
#include <utility>
#include <vector>
namespace nb = nanobind;
using namespace nb::literals;
using namespace sae::audio;
namespace {
using MonoArray = nb::ndarray<const float, nb::ndim<1>, nb::c_contig, nb::device::cpu>;
// Hand the vector's buffer to Python without copying 1.3 M samples, and let a
// capsule own it: the array outlives this call, so the storage has to as well.
nb::object own_as_ndarray(std::vector<float>&& samples) {
auto* held = new std::vector<float>(std::move(samples));
nb::capsule owner(held, [](void* p) noexcept {
delete static_cast<std::vector<float>*>(p);
});
const std::size_t n = held->size();
return nb::cast(nb::ndarray<nb::numpy, float, nb::ndim<1>>(held->data(), {n}, owner));
}
std::vector<float> to_vector(const MonoArray& a) {
return std::vector<float>(a.data(), a.data() + a.shape(0));
}
} // namespace
NB_MODULE(sae_audio, m) {
m.doc() =
"JRay v1 audio signature (JRay-public-server SPEC.md section 3), as the "
"extraction pipeline computes it. The constants below are the contract: "
"changing any of them is a v1 -> v2 change.";
m.attr("sample_rate") = kSampleRate;
m.attr("frame_size") = kFrameSize;
m.attr("hop_size") = kHopSize;
m.attr("num_bands") = kNumBands;
m.attr("band_lo_hz") = kBandLoHz;
m.attr("band_hi_hz") = kBandHiHz;
m.attr("window_sec") = kWindowSec;
m.attr("window_samples") = kWindowSamples;
m.attr("expected_frames") = kExpectedFrames;
m.attr("version_prefix") = std::string(kVersionPrefix);
m.def(
"compute_signature",
[](const std::string& path) { return compute_signature(path); },
"path"_a,
"Signature of the 120 s window centred on the media's midpoint, or None "
"for media shorter than the window (IR-007), media with no audio "
"stream, and any decode failure — degradation, never an exception.");
m.def(
"decode_centre_window",
[](const std::string& path) -> nb::object {
std::optional<std::vector<float>> mono = decode_centre_window(path);
if (!mono) {
return nb::none();
}
return own_as_ndarray(std::move(*mono));
},
"path"_a,
"The decoded centre window as float32 mono at 11025 Hz, or None. Exposed "
"so a caller can slice or perturb real audio and re-sign it without "
"going back through a container.");
m.def(
"signature_from_mono",
[](const MonoArray& mono) { return signature_from_mono(to_vector(mono)); },
"mono"_a,
"Signature of mono float32 samples already at 11025 Hz, in [-1, 1). None "
"when fewer than one whole frame is given.");
m.def(
"pack_frames",
[](const MonoArray& mono) {
std::vector<std::uint8_t> packed = pack_frames(to_vector(mono));
return nb::bytes(reinterpret_cast<const char*>(packed.data()), packed.size());
},
"mono"_a,
"One packed byte per whole STFT frame: (band << 2) | energy_class. This "
"is the payload the signature base64-encodes.");
m.def(
"band_fft_bins",
[] {
const auto& table = band_fft_bins();
return std::vector<std::pair<int, int>>(table.begin(), table.end());
},
"The half-open FFT bin range owned by each of the 32 log-spaced bands.");
}
src/backends/gemm_backend.cpp
+36 -4
View File
@@ -34,9 +34,23 @@ constexpr int kDim = 512;
#if defined(SAE_GEMM_CPU)
#if defined(SAE_GEMM_CBLAS)
#include <cblas.h>
#endif
// ── CPU reference engine ──────────────────────────────────────────────────────
// Portable, dependency-free path used for CI and as the correctness oracle for
// the GPU backends. The gallery is L2-normalised (as are the queries), so each
// Used for CI and as the correctness oracle for the GPU backends.
//
// TRACES: AR-026, AR-027 | SR-001
// Backed by CBLAS (OpenBLAS) when available, falling back to a scalar loop when
// not. The fallback is portable but scales badly: scoring one face against a
// 5000-embedding gallery is 2.6 MFLOP, and a crowded frame multiplies that by
// the face count. Since AR-003 removed the per-frame face cap and CI has no GPU,
// the CPU path is now the one that has to hold up under a library-scale gallery
// (AR-027) rather than merely be correct.
//
// The fallback is kept rather than made mandatory so the build has no hard new
// dependency, and so the two can be diffed when a similarity looks wrong. The gallery is L2-normalised (as are the queries), so each
// similarity is a plain dot product. S is stored column-major to match the GPU
// backends: the gallery similarities for face fi start at result + fi*n_gallery.
class SimilarityEngine final : public ISimilarityEngine {
@@ -47,7 +61,13 @@ public:
gallery_row_major + static_cast<size_t>(n_gallery) * kDim)
{
host_sims_.resize(static_cast<size_t>(max_faces_) * n_gallery_);
std::cerr << "[similarity] CPU reference engine: gallery resident in host RAM ("
std::cerr << "[similarity] CPU engine ("
#if defined(SAE_GEMM_CBLAS)
<< "CBLAS"
#else
<< "scalar fallback — no CBLAS; expect poor scaling on a large gallery"
#endif
<< "): gallery resident in host RAM ("
<< (gallery_.size() * sizeof(float)) / (1024 * 1024) << " MiB)\n";
}
@@ -58,7 +78,18 @@ public:
if (n_faces > max_faces_)
throw std::runtime_error("SimilarityEngine: n_faces exceeds max_faces");
// S(g, f) col-major = dot(gallery[g], query[f]).
// S(g, f) col-major = dot(gallery[g], query[f]). Viewed as row-major
// [n_faces x n_gallery] that is exactly query * gallery^T, so it is one
// GEMM rather than a loop nest.
#if defined(SAE_GEMM_CBLAS)
cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans,
/*M=*/n_faces, /*N=*/n_gallery_, /*K=*/kDim,
/*alpha=*/1.0f,
query_row_major, /*lda=*/kDim,
gallery_.data(), /*ldb=*/kDim,
/*beta=*/0.0f,
host_sims_.data(), /*ldc=*/n_gallery_);
#else
for (int f = 0; f < n_faces; ++f) {
const float* q = query_row_major + static_cast<size_t>(f) * kDim;
float* out = host_sims_.data() + static_cast<size_t>(f) * n_gallery_;
@@ -69,6 +100,7 @@ public:
out[g] = acc;
}
}
#endif
return host_sims_.data();
}
src/build_gallery.cpp
+26
View File
@@ -18,6 +18,8 @@
// --nms <f> NMS IoU threshold (default: 0.4)
#include "gallery/gallery_builder.hpp"
#include "gallery/gallery_report.hpp"
#include "gallery/gallery_calibration.hpp"
#include "gallery/gallery_store.hpp"
#include "config.hpp"
@@ -77,6 +79,30 @@ int main(int argc, char** argv) {
}
save_gallery(output_path, gallery);
std::cerr << "Gallery saved to: " << output_path << "\n";
/// TRACES: GR-003 | SR-001
// Fit the calibration here and persist what it learned. The matcher
// fits the same sigmoid at analysis time, but that is the wrong place
// to audit a gallery from: by then the answer is per-run and nobody is
// looking. Build time is when the gallery's quality is decided, and a
// gallery can be quietly bad — heavily overlapping intra/inter
// distributions, actors with no usable image — while looking fine.
std::vector<Embedding> flat;
std::vector<int> flat_actor;
for (int ai = 0; ai < static_cast<int>(gallery.actors.size()); ++ai)
for (const auto& e : gallery.actors[ai].embeddings) {
flat.push_back(e);
flat_actor.push_back(ai);
}
GalleryCalibrationStats stats;
GalleryCalibration cal = calibrate_gallery(flat, flat_actor, &stats);
const GalleryReport report =
build_gallery_report(gallery, cal, stats, nullptr, output_path);
const std::string report_path = gallery_report_path(output_path);
save_gallery_report(report_path, report);
std::cerr << "Gallery report saved to: " << report_path << "\n";
} catch (const std::exception& e) {
std::cerr << "Fatal: " << e.what() << "\n";
return 1;
src/config.hpp
+15 -1
View File
@@ -41,7 +41,12 @@ struct Config {
// ── Detection (SCRFD-500MF via cv::dnn::Net) ──────────────────────────────
std::string detector_model;
std::string detector_engine; // optional path to pre-built TRT engine; bypasses ORT
int max_faces{10}; // pipeline cap: keep only the N largest faces
// TRACES: AR-003 | SR-002
// 0 = no cap, the default. A fixed cap discards the SMALLEST faces first,
// which are exactly the background cast X-Ray still credits with scene
// membership. Per-frame cost is contained by backpressure (AR-004) rather
// than by throwing work away. Set >0 only to bound a pathological source.
int max_faces{0};
float min_face_px{40.f}; // discard detections narrower or shorter than this
float detector_conf{0.5f};
float detector_nms{0.4f};
@@ -144,6 +149,15 @@ struct Config {
// opposite of the earlier assumption that it only helps restricted galleries.
bool expand_gallery{true}; // master switch
int expand_buffer_size{20}; // per-track diversity buffer capacity
// TRACES: AR-018, AR-024 | SR-005
// Banded admission for the per-subject store, in PROBABILITY space. An
// embedding joins only if P(same person) against something already stored
// lands inside [lo, hi]: above hi it is redundant, below lo it is evidence
// the track is not one person. Replaces expand_novelty_sim, a raw cosine.
// Working values pending VR-007; sweep both bounds, they fail in opposite
// directions.
float expand_band_lo{0.90f};
float expand_band_hi{0.95f};
float expand_novelty_sim{0.55f}; // promote only embeddings whose best sim to the
// actor's refs is below this (gallery-far / novel)
float expand_track_spread_max{0.60f}; // reject promotion if the retained buffer's
src/embed_faces.cpp
+154 -17
View File
@@ -25,11 +25,25 @@
// expected to contain exactly one subject). A warning is printed to stderr
// when more than one face is found.
//
// --all-faces emits every detection instead, which is what a caller analysing
// a frame rather than a gallery portrait needs:
// [ { "image": "frame.png",
// "faces": [ {"bbox": [...], "landmarks": [[x,y] x5],
// "confidence": 0.89, "embedding": [...]} , ... ] } ]
//
// --calibration <gallery> additionally emits the gallery's fitted Platt
// sigmoid, so a non-Python client can turn a similarity into P(match) with the
// same parameters the C++ matcher uses. Output becomes
// {"calibration": {...}, "images": [...]}. Clients must score through it:
// AR-024 requires the calibrated probability, never a bare cosine — a raw
// threshold means something different for every model, gallery and face size.
//
// This binary is intentionally a thin wrapper around the same ONNX models
// used by scene_analyze, so embeddings are guaranteed compatible.
#include "config.hpp"
#include "face_utils.hpp"
#include "gallery/gallery_store.hpp"
#include "inference/face_detector.hpp"
#include "inference/face_embedder.hpp"
@@ -100,6 +114,77 @@ static void save_debug(const std::string& dir,
cv::imwrite(dir + "/" + stem + "_aligned.jpg", aligned);
}
// ── Process one image, keeping every detection ────────────────────────────────
// The --all-faces path. Same detect → align → embed chain as process() below,
// but without the highest-confidence reduction: a frame legitimately contains
// several people, and dropping all but one is a gallery-portrait assumption.
// Faces that fail alignment are reported with a null embedding rather than
// silently dropped, so a caller can count what the detector found against what
// survived the ArcFace warp.
struct MultiFaceResult {
std::string image_path;
std::string error; // set only when the image itself failed
std::vector<FaceResult> faces;
};
static MultiFaceResult process_all(
const std::string& path,
const std::function<std::vector<DetectedFace>(const cv::Mat&)>& detect,
const std::function<Embedding(const cv::Mat&)>& embed_one,
int max_side,
const std::string& debug_dir = "") {
MultiFaceResult out;
out.image_path = path;
cv::Mat img = cv::imread(path);
if (img.empty()) {
out.error = "cannot read image";
return out;
}
if (max_side > 0) {
const int big = std::max(img.cols, img.rows);
if (big > max_side) {
const double s = static_cast<double>(max_side) / big;
cv::resize(img, img, {}, s, s, cv::INTER_AREA);
}
}
std::vector<DetectedFace> faces = detect(img);
if (faces.empty()) {
cv::Mat enhanced = enhance_for_retry(img);
faces = detect(enhanced);
if (!faces.empty())
img = enhanced;
}
if (faces.empty()) {
out.error = "no face detected";
return out;
}
for (const auto& face : faces) {
FaceResult r;
r.image_path = path;
r.confidence = face.confidence;
r.bbox[0] = face.bbox.x; r.bbox[1] = face.bbox.y;
r.bbox[2] = face.bbox.width; r.bbox[3] = face.bbox.height;
r.landmarks = face.landmarks;
cv::Mat crop = align_face(img, face.landmarks);
if (crop.empty()) {
r.error = "alignment failed";
} else {
r.ok = true;
r.embedding = embed_one(crop);
if (!debug_dir.empty())
save_debug(debug_dir, path, img, face, crop);
}
out.faces.push_back(std::move(r));
}
return out;
}
// ── Process one image ─────────────────────────────────────────────────────────
static FaceResult process(const std::string& path,
@@ -177,6 +262,8 @@ int main(int argc, char** argv) {
std::string arcface_model = kDefaultArcfaceModel;
std::string arcface_engine;
std::string debug_dir;
std::string calibration_gallery;
bool all_faces = false;
float conf = 0.5f, nms = 0.4f;
int max_side = 500;
std::vector<std::string> images;
@@ -190,13 +277,16 @@ int main(int argc, char** argv) {
else if (std::strcmp(argv[i], "--nms") == 0 && i+1 < argc) { nms = std::stof(argv[++i]); }
else if (std::strcmp(argv[i], "--save-debug") == 0 && i+1 < argc) { debug_dir = argv[++i]; }
else if (std::strcmp(argv[i], "--max-side") == 0 && i+1 < argc) { max_side = std::stoi(argv[++i]); }
else if (std::strcmp(argv[i], "--calibration")== 0 && i+1 < argc) { calibration_gallery = argv[++i]; }
else if (std::strcmp(argv[i], "--all-faces") == 0) { all_faces = true; }
else if (argv[i][0] != '-') { images.push_back(argv[i]); }
else { std::cerr << "[warn] unknown flag: " << argv[i] << "\n"; }
}
if (images.empty()) {
std::cerr << "Usage: embed_faces [--detector <path>] [--arcface <path>] "
"[--save-debug <dir>] [--max-side <N>] image1.jpg ...\n";
"[--save-debug <dir>] [--max-side <N>] [--all-faces] "
"[--calibration <gallery>] image1.jpg ...\n";
return 1;
}
@@ -217,31 +307,78 @@ int main(int argc, char** argv) {
std::function<Embedding(const cv::Mat&)> embed_one =
[&](const cv::Mat& c) { return embedder->embed_one(c); };
// One face's fields, shared by both output shapes.
auto face_json = [](const FaceResult& r) {
json f;
f["confidence"] = r.confidence;
f["bbox"] = {r.bbox[0], r.bbox[1], r.bbox[2], r.bbox[3]};
json lms = json::array();
for (const auto& pt : r.landmarks) lms.push_back({pt.x, pt.y});
f["landmarks"] = std::move(lms);
if (r.ok) f["embedding"] = std::vector<float>(r.embedding.begin(),
r.embedding.end());
else { f["embedding"] = nullptr; f["error"] = r.error; }
return f;
};
// Process images and build JSON output
json output = json::array();
json images_out = json::array();
for (const auto& path : images) {
std::cerr << "[embed_faces] " << path << "\n";
FaceResult res = process(path, detect, embed_one, max_side, debug_dir);
json entry;
entry["image"] = res.image_path;
if (res.ok) {
entry["embedding"] = std::vector<float>(res.embedding.begin(),
res.embedding.end());
entry["confidence"] = res.confidence;
entry["bbox"] = {res.bbox[0], res.bbox[1], res.bbox[2], res.bbox[3]};
json lms = json::array();
for (const auto& pt : res.landmarks) lms.push_back({pt.x, pt.y});
entry["landmarks"] = std::move(lms);
entry["image"] = path;
if (all_faces) {
MultiFaceResult res = process_all(path, detect, embed_one, max_side, debug_dir);
if (!res.error.empty()) {
entry["faces"] = json::array();
entry["error"] = res.error;
std::cerr << " [skip] " << res.error << "\n";
} else {
json faces = json::array();
for (const auto& f : res.faces) faces.push_back(face_json(f));
entry["faces"] = std::move(faces);
}
} else {
entry["embedding"] = nullptr;
entry["error"] = res.error;
std::cerr << " [skip] " << res.error << "\n";
FaceResult res = process(path, detect, embed_one, max_side, debug_dir);
if (res.ok) {
entry.merge_patch(face_json(res));
} else {
entry["embedding"] = nullptr;
entry["error"] = res.error;
std::cerr << " [skip] " << res.error << "\n";
}
}
output.push_back(std::move(entry));
images_out.push_back(std::move(entry));
}
std::cout << output.dump() << "\n";
// Without --calibration the output stays a bare array, unchanged, so
// existing callers (build_gallery, fetch_missing_actors) are unaffected.
if (calibration_gallery.empty()) {
std::cout << images_out.dump() << "\n";
return 0;
}
ActorGallery gallery = load_gallery(calibration_gallery);
if (!gallery.calib_valid)
std::cerr << "[warn] " << calibration_gallery
<< " carries no valid calibration; a client cannot convert a "
"similarity to a probability from it (AR-024)\n";
json out;
out["calibration"] = {
{"a", gallery.calib_a},
{"b", gallery.calib_b},
{"valid", gallery.calib_valid},
{"form", "P(match) = 1/(1+exp(-(a*similarity + b + log_prior_odds)))"},
{"note", "Score through this. AR-024: a bare cosine threshold means "
"something different for every model, gallery and face size. "
"log_prior_odds = log(p0/(1-p0)) for a base-rate prior p0; use "
"0 for association (are these two faces one person)."},
};
out["images"] = std::move(images_out);
std::cout << out.dump() << "\n";
return 0;
}
src/evidence_discount.hpp
+55 -9
View File
@@ -41,7 +41,26 @@ public:
struct Config {
int max_views{8}; ///< distinct views remembered per track
float admit_below{0.6f}; ///< P(same view) under this ⇒ a new view
float floor{0.0f}; ///< minimum weight for a redundant observation
/// Ceiling on the correlation between two observations of one track.
///
/// This is what bounds the accumulation. `n_eff = n / (1 + (n-1)·rho)`
/// tends to `1/rho` as `n` grows, so `rho_max` sets how much a single
/// repeated view can ever be worth: 0.5 caps it at two observations,
/// no matter how long the shot runs.
///
/// 0.5 caps a repeated view at two independent observations' worth,
/// which is what lets a track the matcher accepts on frame after frame
/// actually become owned. Higher values starve ownership; the sweep
/// (VR-007) decides where it belongs.
///
/// It is capped below 1 deliberately. P(same view) near 1 says the two
/// crops look alike; it does not say the second carries no information.
/// A fresh frame is a fresh detection, a fresh alignment and a fresh
/// noise realisation, so a little independent evidence survives even a
/// perfectly held pose. Setting this to 1 recovers the original bug —
/// belief frozen after the first frame.
float rho_max{0.5f};
};
// Two constructors rather than a defaulted argument: `Config{}` as a default
@@ -53,12 +72,36 @@ public:
EvidenceDiscounter(Calibrate cal, Config cfg)
: cal_(std::move(cal)), cfg_(cfg) {}
/// Weight in [0,1] for one observation, updating `views` when the
/// observation is novel enough to count as a distinct look at the subject.
/// The marginal evidence one observation adds, in units of independent
/// observations.
///
/// The first observation on a track always counts in full: there is nothing
/// for it to be redundant with.
float weight(std::vector<Embedding>& views, const Embedding& e) const {
/// Each frame is a Bayesian update, so confidence must keep growing — but
/// correlated observations must grow it less, and must not grow it without
/// bound. The standard treatment is **effective sample size**:
///
/// n_eff(n) = n / (1 + (n-1)·rho)
///
/// and this returns `n_eff(n) - n_eff(n-1)`, the gain from *this* frame.
/// The shape is right at both ends: with rho = 0 every frame counts fully
/// and the belief accumulates linearly, while as rho rises the series
/// converges on `1/rho` and a held pose stops adding no matter how long it
/// is held.
///
/// The two failure modes it sits between are both real and both were hit:
/// a weight of 0 for repeats froze the belief after one frame, so a track
/// recognised on 318 frames was owned on none; a constant floor grew it
/// linearly forever, so a long shot could out-argue genuinely varied
/// evidence purely by lasting longer.
///
/// `rho` is estimated from P(same view) against the closest stored view,
/// capped by `rho_max`. The first observation has nothing to be redundant
/// with and counts in full.
/// `n_seen` is the count of observations already folded into THIS track.
/// It is a parameter rather than discounter state because one discounter
/// serves every track: holding the count internally would pool unrelated
/// tracks into one effective sample, so a busy film would silently discount
/// each track by how many others happened to be on screen.
float weight(std::vector<Embedding>& views, int n_seen, const Embedding& e) const {
if (views.empty()) {
views.push_back(e);
return 1.0f;
@@ -68,9 +111,12 @@ public:
for (const auto& v : views)
p_same = std::max(p_same, cal_(cosine_similarity(v, e)));
// Weight is the probability this is *not* a repeat of something already
// counted. A near-duplicate contributes ~0; an unseen pose ~1.
const float w = std::max(cfg_.floor, 1.0f - p_same);
const float rho = std::min(cfg_.rho_max, std::max(0.0f, p_same));
const float n_prev = static_cast<float>(std::max(1, n_seen));
const float n_now = n_prev + 1.0f;
auto n_eff = [rho](float n) { return n / (1.0f + (n - 1.0f) * rho); };
const float w = std::max(0.0f, n_eff(n_now) - n_eff(n_prev));
if (p_same < cfg_.admit_below &&
static_cast<int>(views.size()) < cfg_.max_views) {
src/face_embedder_engine.hpp
+19
View File
@@ -112,6 +112,25 @@ public:
return res;
}
// ── Stage accessors ──────────────────────────────────────────────────────
// embed_mat() above is the whole detect→align→embed chain, which is the
// right entry point for embedding a gallery image. Studies that need to
// intervene between the stages — swapping the landmark source, degrading a
// crop before it reaches the embedder — drive these instead, so they still
// exercise the shipped detector, alignment and embedder rather than a
// re-implementation of them.
std::vector<DetectedFace> detect(const cv::Mat& img) { return detector_->detect(img); }
Embedding embed_crop(const cv::Mat& crop) { return embedder_->embed_one(crop); }
// Batched form. A study embedding thousands of crops one at a time pays the
// per-call overhead thousands of times over; the backend already batches.
std::vector<Embedding> embed_crops(const std::vector<cv::Mat>& crops) {
return embedder_->embed(crops);
}
int max_batch() const { return embedder_->max_batch(); }
private:
std::unique_ptr<IFaceDetector> detector_;
std::unique_ptr<IFaceEmbedder> embedder_;
src/face_utils.hpp
+130 -12
View File
@@ -1,26 +1,144 @@
#pragma once
/// TRACES: AR-005 | SR-002
/// TRACES: AR-005, AR-030 | SR-002
#include "types.hpp"
#include <opencv2/calib3d.hpp>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <cmath>
// ── align_face ────────────────────────────────────────────────────────────────
// Produces a 112×112 BGR crop using the ArcFace 5-point similarity transform.
// Returns an empty Mat if the affine fit fails (degenerate detection).
inline cv::Mat align_face(const cv::Mat& img,
const std::array<cv::Point2f, 5>& landmarks) {
std::vector<cv::Point2f> src(landmarks.begin(), landmarks.end());
std::vector<cv::Point2f> dst(5);
// ── umeyama_similarity ────────────────────────────────────────────────────────
// Closed-form least-squares similarity transform (rotation + uniform scale +
// translation, 4 DoF) mapping `src` onto `dst`, by Umeyama's solution.
//
// This is the estimator InsightFace aligns with — skimage's SimilarityTransform
// is `_umeyama(..., estimate_scale=True)` — and therefore the one that produced
// the crops ArcFace and LVFace were *trained* on. The canonical warp is part of
// the input distribution, not a free implementation choice (AR-011).
//
// Deliberately **not** `cv::estimateAffinePartial2D(..., cv::RANSAC)`:
//
// - A robust estimator earns a small residual by discarding the points that
// disagree with the model. On a turned face those are precisely the
// foreshortened landmarks — the pose signal AR-030 exists to measure. RANSAC
// would suppress exactly the quantity we want to read.
// - With five points and a two-point minimal sample there is almost no
// redundancy, so it cannot distinguish a mis-detected landmark from honest
// out-of-plane rotation. The robustness is nominal.
// - It is RNG-driven (`cv::theRNG()` is thread-local); this is exact, so
// replay determinism stops depending on thread scheduling.
//
// Returns an empty Mat when the source points are degenerate (all coincident).
inline cv::Mat umeyama_similarity(const std::array<cv::Point2f, 5>& src,
const std::array<cv::Point2f, 5>& dst) {
constexpr int N = 5;
double mu_sx = 0, mu_sy = 0, mu_dx = 0, mu_dy = 0;
for (int i = 0; i < N; ++i) {
mu_sx += src[i].x; mu_sy += src[i].y;
mu_dx += dst[i].x; mu_dy += dst[i].y;
}
mu_sx /= N; mu_sy /= N; mu_dx /= N; mu_dy /= N;
// var_src and the cross-covariance Σ = (1/N) Σ (d - μ_d)(s - μ_s)ᵀ
double var_s = 0;
cv::Matx22d sigma = cv::Matx22d::zeros();
for (int i = 0; i < N; ++i) {
const double sx = src[i].x - mu_sx, sy = src[i].y - mu_sy;
const double dx = dst[i].x - mu_dx, dy = dst[i].y - mu_dy;
var_s += sx * sx + sy * sy;
sigma(0, 0) += dx * sx; sigma(0, 1) += dx * sy;
sigma(1, 0) += dy * sx; sigma(1, 1) += dy * sy;
}
var_s /= N;
sigma *= 1.0 / N;
if (var_s < 1e-12) return {}; // every source point coincides — no scale
cv::Mat w, u, vt;
cv::SVD::compute(cv::Mat(sigma), w, u, vt, cv::SVD::FULL_UV);
const cv::Matx22d U (u.at<double>(0,0), u.at<double>(0,1),
u.at<double>(1,0), u.at<double>(1,1));
const cv::Matx22d Vt(vt.at<double>(0,0), vt.at<double>(0,1),
vt.at<double>(1,0), vt.at<double>(1,1));
// A similarity may rotate but never mirror: if the fit came out
// orientation-reversing, flip the least-significant singular direction.
cv::Matx22d S = cv::Matx22d::eye();
if (cv::determinant(U) * cv::determinant(Vt) < 0) S(1, 1) = -1;
const cv::Matx22d R = U * S * Vt;
const double c = (w.at<double>(0) * S(0,0) + w.at<double>(1) * S(1,1)) / var_s;
cv::Mat M(2, 3, CV_64F);
M.at<double>(0,0) = c * R(0,0); M.at<double>(0,1) = c * R(0,1);
M.at<double>(1,0) = c * R(1,0); M.at<double>(1,1) = c * R(1,1);
M.at<double>(0,2) = mu_dx - c * (R(0,0) * mu_sx + R(0,1) * mu_sy);
M.at<double>(1,2) = mu_dy - c * (R(1,0) * mu_sx + R(1,1) * mu_sy);
return M;
}
// ── Alignment ─────────────────────────────────────────────────────────────────
// The 5-point fit, plus what it could not explain.
//
// `residual` is the RMS landmark error in **canonical 112×112 pixels** after the
// best similarity fit. Two properties make it the AR-030 visibility measure:
//
// - The similarity transform absorbs rotation, uniform scale and translation
// exactly, so the residual is by construction the part of the deformation a
// similarity *cannot* explain — out-of-plane rotation and foreshortening,
// plus landmark noise. In-plane roll contributes nothing. The "roll must not
// read as yaw" failure is excluded structurally rather than by tuning.
// - The destination frame is fixed, so a 40 px face and a 400 px face are both
// measured in the same canonical space. The measure cannot silently
// re-express face size (already AR-002's job) the way a raw-pixel one would.
//
// It also responds to occlusion and to plainly broken landmark sets, which a
// yaw-angle estimator by construction does not.
struct Alignment {
cv::Mat M; ///< 2×3 CV_64F: source pixels → canonical 112×112
float residual{0.f}; ///< RMS canonical-pixel error; 0 ⇒ a perfect fit
bool ok{false}; ///< false ⇒ degenerate landmarks, no transform
};
/// Fit the canonical ArcFace template to `landmarks` and report the misfit.
inline Alignment estimate_alignment(const std::array<cv::Point2f, 5>& landmarks) {
std::array<cv::Point2f, 5> dst;
for (int i = 0; i < 5; ++i) dst[i] = {kArcFaceRef[i][0], kArcFaceRef[i][1]};
cv::Mat M = cv::estimateAffinePartial2D(src, dst, cv::noArray(), cv::RANSAC, 3.0);
if (M.empty()) return {};
Alignment a;
a.M = umeyama_similarity(landmarks, dst);
if (a.M.empty()) return a;
double sq = 0;
for (int i = 0; i < 5; ++i) {
const double x = a.M.at<double>(0,0) * landmarks[i].x
+ a.M.at<double>(0,1) * landmarks[i].y + a.M.at<double>(0,2);
const double y = a.M.at<double>(1,0) * landmarks[i].x
+ a.M.at<double>(1,1) * landmarks[i].y + a.M.at<double>(1,2);
const double ex = x - dst[i].x, ey = y - dst[i].y;
sq += ex * ex + ey * ey;
}
a.residual = static_cast<float>(std::sqrt(sq / 5.0));
a.ok = true;
return a;
}
// ── align_face ────────────────────────────────────────────────────────────────
// Produces a 112×112 BGR crop using the ArcFace 5-point similarity transform.
// Returns an empty Mat if the fit fails (degenerate detection). When
// `residual_out` is non-null it receives the AR-030 misfit for the same fit —
// free, since the transform has already been computed.
inline cv::Mat align_face(const cv::Mat& img,
const std::array<cv::Point2f, 5>& landmarks,
float* residual_out = nullptr) {
const Alignment a = estimate_alignment(landmarks);
if (!a.ok) return {};
if (residual_out) *residual_out = a.residual;
cv::Mat crop;
cv::warpAffine(img, crop, M, {112, 112},
cv::warpAffine(img, crop, a.M, {112, 112},
cv::INTER_LINEAR, cv::BORDER_CONSTANT, {0, 0, 0});
return crop;
}
src/gallery/gallery_builder.hpp
+1
View File
@@ -1,4 +1,5 @@
#pragma once
#include "gallery/gallery_report.hpp"
#include "types.hpp"
#include <string>
src/gallery/gallery_calibration.hpp
+76 -1
View File
@@ -77,14 +77,48 @@ inline std::function<float(float)> same_person_probability(const GalleryCalibrat
return [cal](float similarity) { return cal.probability(similarity); };
}
/// TRACES: GR-003 | SR-001
///
/// Everything the fit learns about the gallery on its way to two numbers.
///
/// The fit computes per-actor dedup counts, which actors can supply positive
/// pairs at all, and the two similarity distributions the sigmoid is derived
/// from — and then returns only (a, b, valid). GR-003 exists because that is the
/// evidence for whether the calibration, and so every threshold expressed in its
/// probability space (AR-024), rests on anything. Filling this struct costs
/// nothing: the values already exist at the point they are copied out.
///
/// Per-actor vectors are indexed by the actor index used in `flat_actor`.
struct GalleryCalibrationStats {
int n_actors = 0;
int min_embeddings_for_positive = 0;
float dedup_sim_threshold = 0.f;
std::vector<int> distinct_per_actor; // after near-duplicate removal
std::vector<int> duplicates_removed_per_actor;
std::vector<char> eligible; // 1 = supplies positive pairs
int hist_bins = 0; // over sim ∈ [-1, 1]
std::vector<double> intra_hist;
std::vector<double> inter_hist;
double n_intra_pairs = 0.0;
double n_inter_pairs = 0.0;
double train_accuracy_pct = 0.0;
};
// Fit a logistic sigmoid to gallery pair similarities.
// Positive pairs: same actor, different reference images.
// Negative pairs: different actors (all cross-actor embedding pairs).
// Class weights balance the (typically skewed) pos/neg ratio.
// Requires ≥2 positive pairs and ≥1 negative pair.
//
// `stats` is optional (GR-003): pass one to receive the dedup, eligibility and
// distribution detail the fit would otherwise discard.
inline GalleryCalibration calibrate_gallery(
const std::vector<Embedding>& flat_emb,
const std::vector<int>& flat_actor)
const std::vector<int>& flat_actor,
GalleryCalibrationStats* stats = nullptr)
{
constexpr int kMinEmbeddingsForPositive = 5;
constexpr float kDedupSimThreshold = 1.f - 1e-7f; // sim above this = duplicate
@@ -108,6 +142,21 @@ inline GalleryCalibration calibrate_gallery(
std::vector<bool> actor_eligible(n_actors, false);
int n_eligible = 0;
/// TRACES: GR-003 | SR-001
// Record what the filter did, per actor, while the counts still exist.
if (stats) {
*stats = GalleryCalibrationStats{};
stats->n_actors = n_actors;
stats->min_embeddings_for_positive = kMinEmbeddingsForPositive;
stats->dedup_sim_threshold = kDedupSimThreshold;
stats->distinct_per_actor.assign(n_actors, 0);
stats->duplicates_removed_per_actor.assign(n_actors, 0);
stats->eligible.assign(n_actors, 0);
stats->hist_bins = kHistBins;
stats->intra_hist.assign(kHistBins, 0.0);
stats->inter_hist.assign(kHistBins, 0.0);
}
for (int ai = 0; ai < n_actors; ++ai) {
std::vector<Embedding> kept;
for (const auto& e : by_actor[ai]) {
@@ -121,6 +170,12 @@ inline GalleryCalibration calibrate_gallery(
actor_eligible[ai] = true;
++n_eligible;
}
if (stats) {
stats->distinct_per_actor[ai] = static_cast<int>(kept.size());
stats->duplicates_removed_per_actor[ai] =
static_cast<int>(by_actor[ai].size() - kept.size());
stats->eligible[ai] = actor_eligible[ai] ? 1 : 0;
}
for (auto& e : kept) {
flat_emb_dedup.push_back(e);
flat_actor_dedup.push_back(ai);
@@ -128,6 +183,14 @@ inline GalleryCalibration calibrate_gallery(
}
const int n = static_cast<int>(flat_emb_dedup.size());
// Nothing to fit and nothing to multiply. Returning here keeps the report
// buildable for a degenerate gallery instead of handing cv::gemm an empty
// matrix; the per-actor stats above are already filled and still useful.
if (n == 0) {
std::cerr << "[calibration] no embeddings — calibration skipped\n";
return {};
}
std::cerr << "[calibration] dedup: " << flat_emb.size() << " -> " << n
<< " embeddings (" << n_eligible << "/" << n_actors
<< " actors have >= " << kMinEmbeddingsForPositive
@@ -228,6 +291,17 @@ inline GalleryCalibration calibrate_gallery(
double n_pos = 0.0, n_neg = 0.0;
for (int b = 0; b < kHistBins; ++b) { n_pos += pos_hist[b]; n_neg += neg_hist[b]; }
/// TRACES: GR-003 | SR-001
// The two distributions the sigmoid is about to be fitted from. Emitted
// whether or not the fit succeeds — a failed fit is exactly the case where
// someone needs to see why.
if (stats) {
stats->intra_hist = pos_hist;
stats->inter_hist = neg_hist;
stats->n_intra_pairs = n_pos;
stats->n_inter_pairs = n_neg;
}
if (n_pos < 2 || n_neg < 1) {
std::cerr << "[calibration] insufficient pairs (+" << n_pos
<< "/-" << n_neg << ") — calibration skipped\n";
@@ -282,6 +356,7 @@ inline GalleryCalibration calibrate_gallery(
correct += (sig > 0.5f) ? pos_hist[b] : neg_hist[b];
}
double acc = 100.0 * correct / total;
if (stats) stats->train_accuracy_pct = acc;
GalleryCalibration cal{a, bias, true};
std::cerr << "[calibration] sigmoid fitted:"
src/gallery/gallery_report.hpp
+495
View File
@@ -0,0 +1,495 @@
#pragma once
/// TRACES: GR-003 | SR-001
///
/// The gallery build report — what the gallery *is*, written next to it.
///
/// A gallery is a silent artefact: it loads, it scores, it never complains. The
/// two ways it fails are both invisible from the outside.
///
/// 1. **An actor with zero usable images can never be recognised.** They are
/// dropped at build time (`gallery_builder.cpp` skips a directory whose
/// images all fail detection or alignment), so afterwards nothing in the
/// file records that they were ever meant to be there. Every scene they
/// appear in is a guaranteed miss, and recall is capped at a number nobody
/// computed. This is the single most useful line in the report.
/// 2. **A gallery can be quietly bad and look fine.** The Platt sigmoid
/// (AR-023) is fitted from two distributions — intra-class (same actor,
/// different reference) and inter-class (different actors) similarity — and
/// *every* threshold in the pipeline is expressed in the probability space
/// that fit defines (AR-024): identity acceptance, track association,
/// expansion admission, cluster merging. If those two distributions overlap
/// heavily the fit is weak, and every downstream decision silently inherits
/// that weakness while still reporting confident-looking probabilities. The
/// fit already computes the distributions and throws them away; emitting
/// them is what makes the quality of the whole probability space auditable
/// instead of assumed.
///
/// The report is therefore a build artefact, not a debug aid: it is the only
/// place the recall ceiling and the calibration's conditioning are written down.
///
/// **On the histograms being in cosine space.** They bin raw similarity, and
/// that is not an AR-024 violation: no decision is taken here. These two
/// distributions are the *input* the calibration is fitted from — they cannot be
/// expressed in the probability space the calibration defines, because that
/// space is their output. GR-003 asks for exactly this ("the intra/inter
/// distributions behind it"), for the same reason GR-008 characterises an
/// actor's reference spread in the metric space: shape is a property of the
/// metric, decisions are a property of the probability.
#include "gallery/gallery_calibration.hpp"
#include "gallery/embedder_stamp.hpp"
#include "types.hpp"
#include <nlohmann/json.hpp>
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <fstream>
#include <stdexcept>
#include <string>
#include <vector>
/// TRACES: GR-003 | SR-001
///
/// Per-actor image accounting from the build pass, including the actors that
/// produced nothing and were therefore dropped from the gallery.
///
/// Filled by `build_gallery()`. It has to be collected there and cannot be
/// recovered later: by the time a gallery exists, an actor with no usable image
/// is indistinguishable from an actor who was never requested.
struct GalleryBuildAudit {
struct ActorImages {
std::string imdb_id;
std::string name;
int images_seen = 0; // candidate image files in the actor's directory
int images_used = 0; // ...that yielded an embedding
int unreadable = 0; // cv::imread failed
int no_face = 0; // detector found nothing
int align_failed = 0; // 5-point warp failed
};
std::vector<ActorImages> actors; // every directory seen, in build order
};
/// TRACES: GR-003 | SR-001
struct GalleryReport {
// One row per actor the build considered. Actors with references == 0 are
// the zero-usable-image case: present in the source tree, absent from the
// gallery, unrecognisable for the life of the file.
struct Actor {
std::string imdb_id;
std::string name;
int images_seen = -1; // -1 = unknown (report built without a build audit)
int references = 0; // embeddings stored in the gallery
int distinct_references = 0; // ...after near-duplicate removal
int duplicates_removed = 0;
bool eligible_for_positive_pairs = false;
};
// The two distributions the sigmoid is fitted from, as the fit itself saw
// them: counts per similarity bin over [sim_min, sim_max].
struct Distributions {
int bins = 0;
float sim_min = -1.f;
float sim_max = 1.f;
std::vector<double> intra; // same actor, different reference image
std::vector<double> inter; // different actors
double intra_pairs = 0.0;
double inter_pairs = 0.0;
double intra_mean = 0.0;
double inter_mean = 0.0;
// Normalised histogram intersection, Σ_b min(p_intra[b], p_inter[b]).
// 0 = perfectly separated, 1 = indistinguishable. This is the number
// that says whether the calibration — and so every threshold expressed
// in its probability space — rests on anything.
double overlap = 0.0;
};
// GR-003 / AR-023 open question, reported but NOT applied. The spec asks for
// a gallery-derived prior of intra/(intra+inter); the shipped default is
// 0.5. Persisting the distributions makes the real value computable, so the
// decision can be taken on evidence rather than left implicit. Behaviour is
// unchanged: `applied` is always false here.
struct Prior {
double derived = 0.0; // intra_pairs / (intra_pairs + inter_pairs)
double derived_log_odds = 0.0; // log(p/(1-p)), the term AR-023 would add
float configured_default = 0.5f;
bool applied = false;
std::string note;
};
std::string schema{"sae.gallery_report/1"};
std::string gallery_path;
EmbedderStamp embedder;
// ── Summary ──────────────────────────────────────────────────────────────
int actors_total = 0; // considered (gallery + zero-usable)
int actors_in_gallery = 0;
int actors_zero_usable = 0;
int actors_below_positive_threshold = 0;
int64_t embeddings_total = 0;
int64_t distinct_embeddings_total = 0;
int64_t duplicates_removed_total = 0;
double mean_embeddings_per_actor = 0.0; // over actors in the gallery
int min_embeddings_for_positive_pairs = 0;
float dedup_similarity_threshold = 0.f;
// ── Calibration ──────────────────────────────────────────────────────────
float calib_a = 10.f;
float calib_b = -5.f;
bool calib_valid = false;
uint64_t calib_hash = 0;
double calib_train_accuracy_pct = 0.0;
float calib_boundary_p50 = 0.f; // similarity at which P(match) = 0.5
Distributions distributions;
Prior prior;
std::vector<Actor> actors;
// Names duplicated out of `actors` so the two failure modes are greppable
// without a JSON query. These are the lines a human reads first.
std::vector<std::string> zero_usable;
std::vector<std::string> below_positive_threshold;
};
/// TRACES: GR-003 | SR-001
///
/// Assembles the report from the three things that know a piece of the answer:
/// the gallery itself (who is in it, with how many references), the calibration
/// stats (dedup, eligibility, the two distributions), and the build audit (who
/// was considered and produced nothing). The audit is optional — a report built
/// from a stored gallery simply cannot know about the actors that never made it.
///
/// `stats` is indexed by actor index, so the flat arrays handed to
/// `calibrate_gallery()` must have used the gallery's own actor ordering.
inline GalleryReport build_gallery_report(const ActorGallery& gallery,
const GalleryCalibration& cal,
const GalleryCalibrationStats& stats,
const GalleryBuildAudit* audit = nullptr,
const std::string& gallery_path = "",
float configured_prior = 0.5f)
{
GalleryReport r;
r.gallery_path = gallery_path;
r.embedder = gallery.embedder;
r.calib_a = cal.a;
r.calib_b = cal.b;
r.calib_valid = cal.valid;
r.calib_hash = gallery.calib_hash;
r.calib_train_accuracy_pct = stats.train_accuracy_pct;
r.calib_boundary_p50 = cal.boundary_at(0.5f);
r.min_embeddings_for_positive_pairs = stats.min_embeddings_for_positive;
r.dedup_similarity_threshold = stats.dedup_sim_threshold;
auto audit_for = [&](const ActorGallery::Actor& a) -> const GalleryBuildAudit::ActorImages* {
if (!audit) return nullptr;
for (const auto& e : audit->actors) {
if (!a.imdb_id.empty() && e.imdb_id == a.imdb_id) return &e;
if (a.imdb_id.empty() && e.name == a.name) return &e;
}
return nullptr;
};
for (size_t i = 0; i < gallery.actors.size(); ++i) {
const auto& ga = gallery.actors[i];
GalleryReport::Actor row;
row.imdb_id = ga.imdb_id;
row.name = ga.name;
row.references = static_cast<int>(ga.embeddings.size());
if (const auto* au = audit_for(ga)) row.images_seen = au->images_seen;
if (i < stats.distinct_per_actor.size()) {
row.distinct_references = stats.distinct_per_actor[i];
row.duplicates_removed = stats.duplicates_removed_per_actor[i];
row.eligible_for_positive_pairs = stats.eligible[i] != 0;
} else {
// No calibration stats for this actor (the fit never saw them).
// Report the raw count rather than a fabricated distinct count.
row.distinct_references = row.references;
}
r.embeddings_total += row.references;
r.distinct_embeddings_total += row.distinct_references;
r.duplicates_removed_total += row.duplicates_removed;
if (!row.eligible_for_positive_pairs) {
++r.actors_below_positive_threshold;
r.below_positive_threshold.push_back(row.name);
}
r.actors.push_back(std::move(row));
}
r.actors_in_gallery = static_cast<int>(gallery.actors.size());
// Actors the build considered and could not use at all. They are not in the
// gallery, so this is the only record that they exist.
if (audit) {
for (const auto& e : audit->actors) {
if (e.images_used > 0) continue;
GalleryReport::Actor row;
row.imdb_id = e.imdb_id;
row.name = e.name;
row.images_seen = e.images_seen;
row.references = 0;
r.zero_usable.push_back(e.name);
r.actors.push_back(std::move(row));
}
}
r.actors_zero_usable = static_cast<int>(r.zero_usable.size());
r.actors_total = r.actors_in_gallery + r.actors_zero_usable;
r.mean_embeddings_per_actor =
r.actors_in_gallery > 0
? static_cast<double>(r.embeddings_total) / r.actors_in_gallery
: 0.0;
// ── The two distributions, straight out of the fit ───────────────────────
auto& d = r.distributions;
d.bins = stats.hist_bins;
d.sim_min = -1.f;
d.sim_max = 1.f;
d.intra = stats.intra_hist;
d.inter = stats.inter_hist;
d.intra_pairs = stats.n_intra_pairs;
d.inter_pairs = stats.n_inter_pairs;
if (d.bins > 0) {
const double bin_w = (d.sim_max - d.sim_min) / d.bins;
double si = 0.0, se = 0.0;
for (int b = 0; b < d.bins; ++b) {
const double centre = d.sim_min + (b + 0.5) * bin_w;
si += d.intra[b] * centre;
se += d.inter[b] * centre;
}
if (d.intra_pairs > 0.0) d.intra_mean = si / d.intra_pairs;
if (d.inter_pairs > 0.0) d.inter_mean = se / d.inter_pairs;
if (d.intra_pairs > 0.0 && d.inter_pairs > 0.0) {
double ov = 0.0;
for (int b = 0; b < d.bins; ++b)
ov += std::min(d.intra[b] / d.intra_pairs, d.inter[b] / d.inter_pairs);
d.overlap = ov;
}
}
// ── The prior AR-023 leaves open — computed, reported, not applied ────────
r.prior.configured_default = configured_prior;
r.prior.applied = false;
const double pair_total = d.intra_pairs + d.inter_pairs;
if (pair_total > 0.0) {
r.prior.derived = d.intra_pairs / pair_total;
const double p = std::clamp(r.prior.derived, 1e-12, 1.0 - 1e-12);
r.prior.derived_log_odds = std::log(p / (1.0 - p));
}
r.prior.note =
"AR-023 specifies a gallery-derived prior of intra/(intra+inter); the shipped "
"match_prior default is 0.5 (calibrated sigmoid used directly). The derived value "
"is the base rate of same-actor pairs among ALL enumerated gallery pairs, so it "
"falls as the cast grows (roughly (k-1)/((k-1)+(A-1)k) for A actors with k "
"references each) — it is a property of gallery size as much as of the embedder. "
"Reported here as evidence; NOT applied. Behaviour is unchanged until the choice "
"is recorded in the spec.";
return r;
}
// ── JSON ─────────────────────────────────────────────────────────────────────
/// TRACES: GR-003 | SR-001
inline nlohmann::json gallery_report_to_json(const GalleryReport& r) {
nlohmann::json j;
j["schema"] = r.schema;
j["gallery_path"] = r.gallery_path;
j["embedder"] = {{"model_name", r.embedder.model_name},
{"model_sha256", r.embedder.model_sha256},
{"embed_dim", r.embedder.embed_dim}};
j["summary"] = {
{"actors_total", r.actors_total},
{"actors_in_gallery", r.actors_in_gallery},
{"actors_zero_usable", r.actors_zero_usable},
{"actors_below_positive_threshold", r.actors_below_positive_threshold},
{"embeddings_total", r.embeddings_total},
{"distinct_embeddings_total", r.distinct_embeddings_total},
{"duplicates_removed_total", r.duplicates_removed_total},
{"mean_embeddings_per_actor", r.mean_embeddings_per_actor},
{"min_embeddings_for_positive_pairs", r.min_embeddings_for_positive_pairs},
{"dedup_similarity_threshold", r.dedup_similarity_threshold}};
j["calibration"] = {
{"a", r.calib_a},
{"b", r.calib_b},
{"valid", r.calib_valid},
{"hash", r.calib_hash},
{"train_accuracy_pct", r.calib_train_accuracy_pct},
{"boundary_p50", r.calib_boundary_p50}};
const auto& d = r.distributions;
j["distributions"] = {
{"bins", d.bins},
{"sim_min", d.sim_min},
{"sim_max", d.sim_max},
{"intra", d.intra},
{"inter", d.inter},
{"intra_pairs", d.intra_pairs},
{"inter_pairs", d.inter_pairs},
{"intra_mean", d.intra_mean},
{"inter_mean", d.inter_mean},
{"overlap", d.overlap}};
j["prior"] = {
{"derived", r.prior.derived},
{"derived_log_odds", r.prior.derived_log_odds},
{"configured_default", r.prior.configured_default},
{"applied", r.prior.applied},
{"note", r.prior.note}};
j["zero_usable"] = r.zero_usable;
j["below_positive_threshold"] = r.below_positive_threshold;
j["actors"] = nlohmann::json::array();
for (const auto& a : r.actors) {
j["actors"].push_back({
{"imdb_id", a.imdb_id},
{"name", a.name},
{"images_seen", a.images_seen},
{"references", a.references},
{"distinct_references", a.distinct_references},
{"duplicates_removed", a.duplicates_removed},
{"eligible_for_positive_pairs", a.eligible_for_positive_pairs}});
}
return j;
}
/// TRACES: GR-003 | SR-001
inline GalleryReport gallery_report_from_json(const nlohmann::json& j) {
GalleryReport r;
r.schema = j.value("schema", std::string{});
r.gallery_path = j.value("gallery_path", std::string{});
if (j.contains("embedder")) {
const auto& je = j.at("embedder");
r.embedder.model_name = je.value("model_name", "");
r.embedder.model_sha256 = je.value("model_sha256", "");
r.embedder.embed_dim = je.value("embed_dim", 512);
}
if (j.contains("summary")) {
const auto& s = j.at("summary");
r.actors_total = s.value("actors_total", 0);
r.actors_in_gallery = s.value("actors_in_gallery", 0);
r.actors_zero_usable = s.value("actors_zero_usable", 0);
r.actors_below_positive_threshold = s.value("actors_below_positive_threshold", 0);
r.embeddings_total = s.value("embeddings_total", int64_t{0});
r.distinct_embeddings_total = s.value("distinct_embeddings_total", int64_t{0});
r.duplicates_removed_total = s.value("duplicates_removed_total", int64_t{0});
r.mean_embeddings_per_actor = s.value("mean_embeddings_per_actor", 0.0);
r.min_embeddings_for_positive_pairs = s.value("min_embeddings_for_positive_pairs", 0);
r.dedup_similarity_threshold = s.value("dedup_similarity_threshold", 0.f);
}
if (j.contains("calibration")) {
const auto& c = j.at("calibration");
r.calib_a = c.value("a", 10.f);
r.calib_b = c.value("b", -5.f);
r.calib_valid = c.value("valid", false);
r.calib_hash = c.value("hash", uint64_t{0});
r.calib_train_accuracy_pct = c.value("train_accuracy_pct", 0.0);
r.calib_boundary_p50 = c.value("boundary_p50", 0.f);
}
if (j.contains("distributions")) {
const auto& d = j.at("distributions");
r.distributions.bins = d.value("bins", 0);
r.distributions.sim_min = d.value("sim_min", -1.f);
r.distributions.sim_max = d.value("sim_max", 1.f);
r.distributions.intra = d.value("intra", std::vector<double>{});
r.distributions.inter = d.value("inter", std::vector<double>{});
r.distributions.intra_pairs = d.value("intra_pairs", 0.0);
r.distributions.inter_pairs = d.value("inter_pairs", 0.0);
r.distributions.intra_mean = d.value("intra_mean", 0.0);
r.distributions.inter_mean = d.value("inter_mean", 0.0);
r.distributions.overlap = d.value("overlap", 0.0);
}
if (j.contains("prior")) {
const auto& p = j.at("prior");
r.prior.derived = p.value("derived", 0.0);
r.prior.derived_log_odds = p.value("derived_log_odds", 0.0);
r.prior.configured_default = p.value("configured_default", 0.5f);
r.prior.applied = p.value("applied", false);
r.prior.note = p.value("note", "");
}
r.zero_usable = j.value("zero_usable", std::vector<std::string>{});
r.below_positive_threshold = j.value("below_positive_threshold", std::vector<std::string>{});
if (j.contains("actors")) {
for (const auto& ja : j.at("actors")) {
GalleryReport::Actor a;
a.imdb_id = ja.value("imdb_id", "");
a.name = ja.value("name", "");
a.images_seen = ja.value("images_seen", -1);
a.references = ja.value("references", 0);
a.distinct_references = ja.value("distinct_references", 0);
a.duplicates_removed = ja.value("duplicates_removed", 0);
a.eligible_for_positive_pairs = ja.value("eligible_for_positive_pairs", false);
r.actors.push_back(std::move(a));
}
}
return r;
}
// "<dir>/cast.h5" → "<dir>/cast.report.json". A known gallery extension is
// replaced rather than appended to, so the report sits beside the gallery under
// the same stem.
inline std::string gallery_report_path(const std::string& gallery_path) {
auto slash = gallery_path.find_last_of("/\\");
auto dot = gallery_path.find_last_of('.');
std::string stem =
(dot != std::string::npos && (slash == std::string::npos || dot > slash))
? gallery_path.substr(0, dot)
: gallery_path;
return stem + ".report.json";
}
/// TRACES: GR-003 | SR-001
inline void save_gallery_report(const std::string& path, const GalleryReport& r) {
std::ofstream out(path);
if (!out.is_open())
throw std::runtime_error("save_gallery_report: cannot write " + path);
out << gallery_report_to_json(r).dump(2) << "\n";
}
/// TRACES: GR-003 | SR-001
inline GalleryReport load_gallery_report(const std::string& path) {
std::ifstream in(path);
if (!in.is_open())
throw std::runtime_error("load_gallery_report: cannot open " + path);
nlohmann::json j;
in >> j;
return gallery_report_from_json(j);
}
/// TRACES: GR-003 | SR-001
///
/// The report's headline, on stderr, at build time. The file is the audit trail;
/// this is what stops a bad gallery from being shipped without anyone noticing.
inline void log_gallery_report(const GalleryReport& r) {
std::cerr << "[gallery-report] " << r.actors_in_gallery << " actors / "
<< r.embeddings_total << " embeddings"
<< " (mean " << r.mean_embeddings_per_actor << " per actor)\n";
if (r.actors_zero_usable > 0) {
std::cerr << "[gallery-report] WARNING: " << r.actors_zero_usable
<< " actor(s) have NO usable image — they can never be recognised:\n";
for (const auto& n : r.zero_usable) std::cerr << " - " << n << "\n";
}
if (r.actors_below_positive_threshold > 0) {
std::cerr << "[gallery-report] " << r.actors_below_positive_threshold
<< " actor(s) below " << r.min_embeddings_for_positive_pairs
<< " distinct references — they contribute no positive pairs and "
"weaken the calibration\n";
}
if (r.duplicates_removed_total > 0)
std::cerr << "[gallery-report] " << r.duplicates_removed_total
<< " near-duplicate reference(s) removed\n";
std::cerr << "[gallery-report] calibration valid=" << r.calib_valid
<< " a=" << r.calib_a << " b=" << r.calib_b
<< " intra/inter overlap=" << r.distributions.overlap
<< " (intra mean=" << r.distributions.intra_mean
<< ", inter mean=" << r.distributions.inter_mean << ")\n";
std::cerr << "[gallery-report] gallery-derived prior would be "
<< r.prior.derived << " (log-odds " << r.prior.derived_log_odds
<< "); shipped default " << r.prior.configured_default
<< " is in force — reported, not applied\n";
}
src/gallery/gallery_store.hpp
+8
View File
@@ -6,6 +6,14 @@
// legacy gallery.json files are still readable for backward compatibility but
// save_gallery always writes HDF5 regardless of the requested extension.
//
// GR-005 is preserved here by absence: this is the only path that serialises a
// gallery, and it reads and writes the local filesystem only. There is no
// upload, no client, and no encoder that could put an embedding on a wire — the
// public server refuses to carry one (SR-004/UR-012), and the prohibition holds
// on this side by there being nothing that would try.
//
/// TRACES: GR-005 | SR-005
//
// HDF5 layout:
// /embeddings float32 [N, 512] all actors' refs concatenated, row-major
// /offset int64 [A] first row of actor a in /embeddings
src/gallery/track_gallery.hpp
+70 -1
View File
@@ -2,6 +2,8 @@
#include "types.hpp"
#include "config.hpp"
#include <functional>
#include <cmath>
#include <cstdio>
#include <iostream>
@@ -117,6 +119,27 @@ struct TrackGallery {
// matcher when it observes a cut or track disappearance.
void forget(int track_id) { tracks_.erase(track_id); }
/// TRACES: AR-019 | SR-005
/// The registry's verdict on who this track is. Authoritative: it comes from
/// the Bayesian accumulation (AR-025), where the local tally counted raw
/// accepted frames and so weighted thirty near-identical looks the same as
/// thirty distinct ones.
void set_owner(int track_id, int actor_idx) {
if (track_id < 0 || actor_idx < 0) return;
tracks_[track_id].registry_owner = actor_idx;
}
/// TRACES: AR-024 | SR-005
/// Supply the calibration belonging to the active embedder. Without it the
/// band falls back to treating cosine as probability, which is wrong but
/// bounded — and the default is loud in the header rather than silent.
void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }
void set_band(float lo, float hi) { band_lo_ = lo; band_hi_ = hi; }
/// Embeddings the band refused. A store that admits nothing is as wrong as
/// one that admits everything, and neither is visible without this.
std::size_t band_rejected() const { return rejected_; }
// Drop every track buffer (scene cut / EOF). Mirrors face_tracker's clear.
void clear_tracks() { tracks_.clear(); }
@@ -132,11 +155,43 @@ private:
std::map<int, int> actor_votes; // actor_idx → accepted-frame count
int accepted_frames{0};
bool promoted{false};
int registry_owner{-1}; ///< AR-019: authoritative
};
/// TRACES: AR-018, AR-024 | SR-005
/// Banded admission: an embedding joins the store only if its similarity to
/// something already there falls **inside a band**.
///
/// above the upper bound → redundant. It is another look at a pose the
/// store already covers, and adding it teaches the annex nothing while
/// costing a slot that a novel view could have used.
/// below the lower bound → suspect. Within one track every face is the
/// same person by construction, so an embedding unlike everything else
/// on the track is evidence the construction failed — a track-ID
/// collision or a bad detection. Admitting it is how an actor's annex
/// gets poisoned with someone else's face.
///
/// Both bounds are calibrated probabilities, never raw cosines (AR-024): a
/// bare similarity threshold means something different for every model and
/// every face size, and this gate has to hold across both.
///
/// The first embedding is always admitted — there is nothing for it to be
/// redundant with, and nothing to contradict it.
bool admit(const TrackState& ts, const Embedding& emb) const {
if (ts.buf.empty()) return true;
float p_max = 0.f;
for (const auto& b : ts.buf)
p_max = std::max(p_max, calibrate_(cosine_similarity(b.emb, emb)));
return p_max >= band_lo_ && p_max <= band_hi_;
}
void insert_into_buffer(TrackState& ts, const Embedding& emb,
float gal_sim, const cv::Mat& crop)
{
if (!admit(ts, emb)) { ++rejected_; return; }
BufEntry e;
e.emb = emb;
e.gal_sim = gal_sim;
@@ -166,7 +221,7 @@ private:
void promote(int track_id, TrackState& ts) {
ts.promoted = true; // idempotent: never promote a track twice
int actor = plurality_actor(ts);
int actor = owning_actor(ts);
if (actor < 0) return;
// ── Safety gate: internal spread ─────────────────────────────────────
@@ -202,6 +257,13 @@ private:
<< annex_.size() << "\n";
}
/// Prefer the registry's verdict; fall back to the local tally only when no
/// registry is attached (unit tests, replay harness).
static int owning_actor(const TrackState& ts) {
if (ts.registry_owner >= 0) return ts.registry_owner;
return plurality_actor(ts);
}
static int plurality_actor(const TrackState& ts) {
int best = -1, best_votes = 0;
for (const auto& [ai, v] : ts.actor_votes) {
@@ -231,6 +293,13 @@ private:
#endif
}
/// cosine → P(same person). The one probability space the pipeline reasons
/// in; see gallery_calibration.hpp's same_person_probability.
std::function<float(float)> calibrate_{[](float c) { return std::max(0.f, c); }};
float band_lo_{0.90f};
float band_hi_{0.95f};
std::size_t rejected_{0}; ///< admissions refused by the band
bool enabled_;
int buffer_size_;
float novelty_sim_;
src/main.cpp
+92 -7
View File
@@ -39,8 +39,8 @@
// --max-faces <N> max faces kept per frame (default: 10)
// --expand-gallery enable per-film gallery expansion from track continuity
// --expand-buffer <N> per-track diversity buffer size (default: 20)
// --expand-novelty-sim <f> promote only views with best sim < f (default: 0.55)
// --expand-spread-max <f> reject track if buffer spread > f (default: 0.60)
// --expand-band-lo <p> store admission floor, P(same person) (default: 0.90)
// --expand-band-hi <p> store admission ceiling, P(same person) (default: 0.95)
// --expand-min-anchor <N> accepted frames before a track confirms (default: 3)
// --expand-debug-dir <p> dump promoted mugshots + embeddings here (SAE_DEBUG)
// (SAE_DEBUG only)
@@ -60,6 +60,8 @@
#include "nodes/identity_matcher_node.hpp"
#include "nodes/scene_tracker_node.hpp"
#include "nodes/scene_detector_node.hpp"
#include "scene_boundaries.hpp"
#include "nodes/scene_boundary_annotator_node.hpp"
#include "nodes/result_sink_node.hpp"
#include "nodes/embedding_dump_node.hpp"
#ifdef SAE_DEBUG
@@ -81,6 +83,18 @@
// ── CLI parsing ───────────────────────────────────────────────────────────────
/// TRACES: AR-010, AR-004 | SR-002
/// How deeply the sampled branch is buffered behind the dense one. TransNetV2
/// needs kWindow (100) dense frames before it can score any of them, so the face
/// branch must lag by at least that much or it asks about frames nobody has
/// looked at yet. Backpressure turns depth into lag: the fanout blocks on the
/// slower branch rather than dropping, so the detector simply runs ahead.
static constexpr std::size_t kSceneJoinDepth = 256;
/// Set when the scene branch is built, so shutdown can report whether the join
/// actually worked.
static std::shared_ptr<SceneBoundaries> scene_stats;
static Config parse_args(int argc, char** argv) {
Config cfg;
cfg.detector_model = kDefaultDetectorModel;
@@ -133,8 +147,8 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--anneal")) cfg.anneal_sec = std::stod(next());
else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
else if (arg("--expand-novelty-sim")) cfg.expand_novelty_sim = std::stof(next());
else if (arg("--expand-spread-max")) cfg.expand_track_spread_max = std::stof(next());
else if (arg("--expand-band-lo")) cfg.expand_band_lo = std::stof(next());
else if (arg("--expand-band-hi")) cfg.expand_band_hi = std::stof(next());
else if (arg("--expand-min-anchor")) cfg.expand_min_anchor_frames = std::stoi(next());
else if (arg("--expand-debug-dir")) cfg.expand_debug_dir = next();
else if (arg("--trt-cache")) cfg.trt.cache_dir = next();
@@ -213,7 +227,7 @@ int main(int argc, char** argv) {
SceneTrackerFunc tracker_fn {cfg};
ResultSinkFunc sink_fn {cfg, done};
/// TRACES: AR-012, AR-016, IR-002, IR-003 | SR-002
/// TRACES: AR-012, AR-016 | IR-002, IR-003 | SR-002
// A reaped track goes straight to the aggregator, so the registry holds only
// live tracks and its size is bounded by concurrent on-screen faces rather
// than growing with the film.
@@ -267,6 +281,26 @@ int main(int argc, char** argv) {
}
});
// Report *why* a node died. A Closed event alone says only that one
// stopped; the exception it carried is what identifies the fault, and
// without this listener it is discarded at the node boundary. Returning
// false keeps the existing semantics — the node still stops and the
// Closed handler above still aborts the run — but the run now names the
// cause instead of leaving it to be reconstructed from a debugger.
net.set_error_handler(
[&](std::string_view node_name, std::exception_ptr eptr) {
std::string what = "unknown exception";
try {
if (eptr) std::rethrow_exception(eptr);
} catch (const std::exception& e) {
what = e.what();
} catch (...) {
}
std::lock_guard<std::mutex> lk(event_mtx);
std::cerr << "[main] node '" << node_name << "' threw: " << what << "\n";
return false;
});
std::cerr << "[main] starting pipeline…\n";
net.start();
@@ -292,6 +326,21 @@ int main(int argc, char** argv) {
//
// Reporting it in a footer and exiting 0 made both invisible: the run
// "succeeded" and the truth file looked complete. Fail instead.
/// TRACES: AR-010 | SR-002
if (scene_stats) {
std::cerr << "[scene_annotate] boundaries=" << scene_stats->count()
<< " scored_through=" << scene_stats->scored_through() << "s";
// The tail is expected: frames after the detector's last full
// window are never covered, and no amount of buffering changes
// that. They are counted rather than silently treated as
// boundary-free, which is the distinction that matters.
if (scene_stats->outran() > 0)
std::cerr << " unscored=" << scene_stats->outran()
<< " frame(s) past the detector's last window — treated as"
" boundary-free, which is unverified rather than known";
std::cerr << "\n";
}
bool dropped = false;
{
std::lock_guard<std::mutex> lk(event_mtx);
@@ -346,6 +395,21 @@ int main(int argc, char** argv) {
if (cfg.scene_detect) {
scene_done.store(false, std::memory_order_release); // now a real terminal branch
SceneDetectorFunc scene_fn{cfg, scene_done};
/// TRACES: AR-010 | SR-002
// The join of the decode butterfly. source fans out to the dense
// TransNetV2 branch and the sampled face branch; boundaries found on the
// first have to reach the second, and cannot ride the frames because the
// branches run in parallel.
//
// TransNetV2 buffers kWindow frames before it can score any of them, so
// the face branch must lag by at least that much or it will ask about
// frames nobody has looked at yet. Channel depth is what creates the lag:
// with backpressure (AR-004) the fanout blocks on the slower branch, so
// a deep face-branch channel lets the detector run ahead by its window
// rather than dropping anything.
auto boundaries = std::make_shared<SceneBoundaries>();
scene_fn.set_boundaries(boundaries);
kpn::ObjectNode<SceneDetectorFunc, kpn::in<"dense">, kpn::out<>, "scene_detector", 0>
scene_node(scene_fn, 128);
@@ -362,13 +426,34 @@ int main(int argc, char** argv) {
return true;
}
return false;
}, 32);
}, kSceneJoinDepth);
/// TRACES: AR-010 | SR-002
// Stamp is_scene_boundary from the detector's published verdict. tol is
// half a sample interval: the two branches sample at different rates, so
// a boundary found on a dense frame rarely lands exactly on a sampled
// one, and half an interval attributes it to the nearest sampled frame
// and no further.
//
// outran() counts frames that arrived before the detector had scored
// them. Nonzero means the join depth is too shallow for the window, and
// those frames were annotated from an incomplete verdict — which would
// otherwise look exactly like "no boundary here".
SceneBoundaryAnnotatorFunc annotate_fn{boundaries, 0.5 / cfg.sample_fps};
kpn::ObjectNode<SceneBoundaryAnnotatorFunc, kpn::in<"frame">, kpn::out<"frame">,
"scene_annotate", 0> annotate(annotate_fn, kSceneJoinDepth);
// Reported at shutdown: without this the join is unverifiable, and an
// annotator that never fired looks identical to footage with no
// boundaries.
scene_stats = boundaries;
auto net = kpn::make_network(
kpn::edge(source.output<"raw">(), campos.input<"raw">()),
kpn::edge(source.output<"raw">(), scene_node.input<"dense">()),
kpn::edge(campos.output<"frame">(), decimate.input<0>()),
kpn::edge(decimate.output<0>(), detector.input<"frame">()),
kpn::edge(decimate.output<0>(), annotate.input<"frame">()),
kpn::edge(annotate.output<"frame">(), detector.input<"frame">()),
kpn::edge(detector.output<"scene">(), aligner.input<"scene">()),
kpn::edge(aligner.output<"aligned">(), embedder.input<"aligned">()),
kpn::edge(embedder.output<"embedded">(), ftracker.input<"embedded">()),
src/nodes/camera_position_change_detector_node.hpp
+2 -1
View File
@@ -26,7 +26,8 @@
// The node is a pure pass-through: it forwards the Frame unchanged except for
// is_cut, so it slots between frame_source and face_detector without altering the
// downstream contract. eof frames are forwarded immediately without processing.
//
/// TRACES: AR-009 | SR-002
struct CameraPositionChangeDetectorFunc {
static constexpr std::string_view label() { return "camera_position_change_detector"; }
src/nodes/embedder_node.hpp
+2 -1
View File
@@ -17,7 +17,8 @@
// All crops in one frame are batched into a single forward pass (capped at
// embed_batch_size). The backend serialises itself; we only call it from the
// single embedder thread.
//
/// TRACES: AR-006 | SR-002
struct EmbedderFunc {
static constexpr std::string_view label() { return "embedder"; }
src/nodes/embedding_dump_node.hpp
+173 -14
View File
@@ -1,16 +1,113 @@
#pragma once
/// TRACES: VR-001 | PR-002
/// TRACES: VR-001, VR-010 | PR-002
#include "types.hpp"
#include "config.hpp"
#include "gallery/embedder_stamp.hpp"
#include <H5Cpp.h>
#include <atomic>
#include <cstdint>
#include <iostream>
#include <optional>
#include <string>
#include <type_traits>
#include <vector>
// ── DumpProvenance ────────────────────────────────────────────────────────────
/// TRACES: VR-010 | PR-002
// Everything that determined a dump's *content*, read back tolerantly.
//
// Two dumps of the same film with different detector thresholds, a different
// `dense_scale`, or scene detection on versus off are different measurements of
// different things — but they are byte-shaped identically, so a consumer that
// mixes them gets a plausible number from an incoherent input. GR-004 closed the
// worst case (a cross-model replay, where every cosine is meaningless); this
// closes the rest.
//
// Every field is optional because dumps written before VR-010 lack the
// attributes. A missing field reads as *unknown*, never as a default — a
// silently-defaulted `detector_conf` is exactly the fabricated provenance the
// requirement exists to prevent ("a fixture whose provenance is unknown is worse
// than no fixture, because it will be trusted").
struct DumpProvenance {
// Model identity
std::optional<std::string> embedder_model; // GR-004
std::optional<std::string> embedder_sha256; // GR-004
std::optional<std::string> detector_model;
// Sampling
std::optional<std::string> movie;
std::optional<float> sample_fps;
std::optional<double> start_sec;
std::optional<double> end_sec; // -1 = to end of file
// Detection — what the run admitted into the dump
std::optional<float> detector_conf;
std::optional<float> detector_nms;
std::optional<float> min_face_px;
std::optional<int> max_faces; // 0 = uncapped (AR-003)
// Frame geometry
std::optional<float> dense_scale;
std::optional<float> bbox_upscale; // faces/bbox × this = original-resolution px
std::optional<float> cut_threshold;
// Scene detection. The reason this flag exists: `is_scene_boundary` is
// all-zero both when TransNetV2 found no boundaries and when it never ran,
// and no amount of staring at the array distinguishes them.
std::optional<bool> scene_detect;
// Downstream knob that shaped nothing in the dump but everything a replay is
// compared against — recorded so a sweep can be told apart from the baseline.
std::optional<float> track_assoc_min_prob;
};
// Read whatever provenance a dump carries. Never throws on a missing attribute;
// an old dump simply yields a DumpProvenance full of empty optionals.
inline DumpProvenance read_dump_provenance(const H5::H5File& f) {
DumpProvenance p;
auto str = [&](const char* n, std::optional<std::string>& out) {
if (!f.attrExists(n)) return;
// Written as a variable-length string, so the read must name the same
// type explicitly — the default would truncate to a fixed length.
H5::StrType vlen(H5::PredType::C_S1, H5T_VARIABLE);
std::string v;
f.openAttribute(n).read(vlen, v);
out = v;
};
auto num = [&](const char* n, const H5::PredType& dt, auto& out) {
if (!f.attrExists(n)) return;
typename std::decay_t<decltype(out)>::value_type v{};
f.openAttribute(n).read(dt, &v);
out = v;
};
str("embedder_model", p.embedder_model);
str("embedder_sha256", p.embedder_sha256);
str("detector_model", p.detector_model);
str("movie", p.movie);
num("sample_fps", H5::PredType::NATIVE_FLOAT, p.sample_fps);
num("start_sec", H5::PredType::NATIVE_DOUBLE, p.start_sec);
num("end_sec", H5::PredType::NATIVE_DOUBLE, p.end_sec);
num("detector_conf", H5::PredType::NATIVE_FLOAT, p.detector_conf);
num("detector_nms", H5::PredType::NATIVE_FLOAT, p.detector_nms);
num("min_face_px", H5::PredType::NATIVE_FLOAT, p.min_face_px);
num("max_faces", H5::PredType::NATIVE_INT, p.max_faces);
num("dense_scale", H5::PredType::NATIVE_FLOAT, p.dense_scale);
num("bbox_upscale", H5::PredType::NATIVE_FLOAT, p.bbox_upscale);
num("cut_threshold", H5::PredType::NATIVE_FLOAT, p.cut_threshold);
num("track_assoc_min_prob", H5::PredType::NATIVE_FLOAT, p.track_assoc_min_prob);
if (f.attrExists("scene_detect")) {
uint8_t v = 0;
f.openAttribute("scene_detect").read(H5::PredType::NATIVE_UINT8, &v);
p.scene_detect = (v != 0);
}
return p;
}
// ── EmbeddingDumpFunc ─────────────────────────────────────────────────────────
// KPN sink that taps the EmbeddedSceneFrame channel and writes the per-frame face
// metadata + embeddings to one HDF5 file (schema: scripts/optimizer/SCHEMA.md).
@@ -32,13 +129,39 @@ struct EmbeddingDumpFunc {
// gallery hours or weeks later — the same silent cross-model hazard as the
// gallery itself, so it carries the same stamp.
stamp_ = make_embedder_stamp(cfg.arcface_model);
/// TRACES: VR-010 | PR-002
// The rest of what determined this file's content. Captured from the live
// Config at construction, so it describes the run that is being written
// rather than whatever config happens to be lying around at read time.
prov_.detector_model = basename_of(cfg.detector_model);
prov_.detector_conf = cfg.detector_conf;
prov_.detector_nms = cfg.detector_nms;
prov_.min_face_px = cfg.min_face_px;
prov_.max_faces = cfg.max_faces;
prov_.cut_threshold = cfg.cut_threshold;
prov_.dense_scale = cfg.dense_scale;
prov_.start_sec = cfg.start_sec;
prov_.end_sec = cfg.end_sec;
prov_.scene_detect = cfg.scene_detect;
prov_.track_assoc_min_prob = cfg.track_assoc_min_prob;
std::cerr << "[embedding_dump] writing " << path_
<< " embedder: " << stamp_.describe() << "\n";
<< " embedder: " << stamp_.describe()
<< " detector: " << *prov_.detector_model
<< " @conf " << cfg.detector_conf
<< " scene_detect=" << (cfg.scene_detect ? "on" : "off") << "\n";
}
void operator()(EmbeddedSceneFrame ef) {
if (ef.source.eof) { flush(); return; }
/// TRACES: VR-010 | PR-002
// Taken from the frames themselves, not recomputed from dense_scale — the
// factor the source actually stamped on them is the one that maps
// faces/bbox back to original resolution, whatever rule produced it.
if (!prov_.bbox_upscale) prov_.bbox_upscale = ef.source.bbox_upscale;
const int32_t n = static_cast<int32_t>(ef.faces.size());
ts_.push_back(ef.source.timestamp_sec);
fidx_.push_back(ef.source.frame_idx);
@@ -71,9 +194,18 @@ struct EmbeddingDumpFunc {
}
private:
// Root attributes are additive: schema_version stays 1 across VR-010, because
// every reader takes attributes by name with a default (replay.py) or an
// existence check (read_dump_provenance), so an old dump loses nothing and a
// new dump breaks nothing. A bump is for a change to the *datasets*.
static constexpr int kSchemaVersion = 1;
static constexpr int kEmbedDim = 512;
static std::string basename_of(const std::string& path) {
const auto slash = path.find_last_of("/\\");
return slash == std::string::npos ? path : path.substr(slash + 1);
}
template<typename T>
void write_vec(H5::Group& g, const char* name, const std::vector<T>& v,
const H5::PredType& dtype, hsize_t cols = 0) {
@@ -85,23 +217,49 @@ private:
if (!v.empty()) ds.write(v.data(), dtype);
}
static void attr_str(H5::H5File& f, const char* name, const std::string& v) {
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
f.createAttribute(name, str, H5::DataSpace(H5S_SCALAR)).write(str, v);
}
template<typename T>
static void attr_num(H5::H5File& f, const char* name, const H5::PredType& dt, T v) {
f.createAttribute(name, dt, H5::DataSpace(H5S_SCALAR)).write(dt, &v);
}
void write_hdf5() {
H5::H5File file(path_, H5F_ACC_TRUNC);
// root attrs
auto scalar = H5::DataSpace(H5S_SCALAR);
auto ver = file.createAttribute("schema_version", H5::PredType::NATIVE_INT, scalar);
int sv = kSchemaVersion; ver.write(H5::PredType::NATIVE_INT, &sv);
auto ed = file.createAttribute("embed_dim", H5::PredType::NATIVE_INT, scalar);
int dim = kEmbedDim; ed.write(H5::PredType::NATIVE_INT, &dim);
auto fps = file.createAttribute("sample_fps", H5::PredType::NATIVE_FLOAT, scalar);
fps.write(H5::PredType::NATIVE_FLOAT, &sample_fps_);
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
auto mv = file.createAttribute("movie", str, scalar);
mv.write(str, movie_);
attr_num(file, "schema_version", H5::PredType::NATIVE_INT, kSchemaVersion);
attr_num(file, "embed_dim", H5::PredType::NATIVE_INT, kEmbedDim);
attr_num(file, "sample_fps", H5::PredType::NATIVE_FLOAT, sample_fps_);
attr_str(file, "movie", movie_);
/// TRACES: GR-004 | SR-001
file.createAttribute("embedder_model", str, scalar).write(str, stamp_.model_name);
file.createAttribute("embedder_sha256", str, scalar).write(str, stamp_.model_sha256);
attr_str(file, "embedder_model", stamp_.model_name);
attr_str(file, "embedder_sha256", stamp_.model_sha256);
/// TRACES: VR-010 | PR-002
attr_str(file, "detector_model", prov_.detector_model.value_or(""));
attr_num(file, "detector_conf", H5::PredType::NATIVE_FLOAT, *prov_.detector_conf);
attr_num(file, "detector_nms", H5::PredType::NATIVE_FLOAT, *prov_.detector_nms);
attr_num(file, "min_face_px", H5::PredType::NATIVE_FLOAT, *prov_.min_face_px);
attr_num(file, "max_faces", H5::PredType::NATIVE_INT, *prov_.max_faces);
attr_num(file, "cut_threshold", H5::PredType::NATIVE_FLOAT, *prov_.cut_threshold);
attr_num(file, "dense_scale", H5::PredType::NATIVE_FLOAT, *prov_.dense_scale);
// Recorded, NOT applied — faces/bbox stays in the detector's own frame
// space so a replay feeds the tracker exactly what the live run fed it.
attr_num(file, "bbox_upscale", H5::PredType::NATIVE_FLOAT,
prov_.bbox_upscale.value_or(1.f));
attr_num(file, "start_sec", H5::PredType::NATIVE_DOUBLE, *prov_.start_sec);
attr_num(file, "end_sec", H5::PredType::NATIVE_DOUBLE, *prov_.end_sec);
attr_num(file, "track_assoc_min_prob", H5::PredType::NATIVE_FLOAT,
*prov_.track_assoc_min_prob);
// 0/1, matching the uint8 booleans in frames/. Tells "TransNetV2 found no
// boundaries" apart from "TransNetV2 never ran", which is/was the same
// all-zero is_scene_boundary array either way.
attr_num(file, "scene_detect", H5::PredType::NATIVE_UINT8,
static_cast<uint8_t>(*prov_.scene_detect ? 1 : 0));
H5::Group frames = file.createGroup("frames");
write_vec(frames, "timestamp_sec", ts_, H5::PredType::NATIVE_DOUBLE);
@@ -123,6 +281,7 @@ private:
std::string path_, movie_;
EmbedderStamp stamp_;
DumpProvenance prov_;
float sample_fps_;
std::atomic<bool>& done_;
std::atomic<bool> written_{false};
src/nodes/face_aligner_node.hpp
+5 -1
View File
@@ -24,11 +24,15 @@ struct FaceAlignerFunc {
crops.reserve(sf.faces.size());
for (auto& face : sf.faces) {
cv::Mat crop = align_face(sf.source.image, face.landmarks);
// The AR-030 misfit comes from the transform the warp already needs,
// so visibility costs no extra fit.
float residual = -1.f;
cv::Mat crop = align_face(sf.source.image, face.landmarks, &residual);
if (crop.empty()) {
std::cerr << "[face_aligner] degenerate detection skipped\n";
continue;
}
face.alignment_residual = residual;
good_faces.push_back(face);
crops.push_back(std::move(crop));
}
src/nodes/face_detector_node.hpp
+5 -1
View File
@@ -44,7 +44,11 @@ struct FaceDetectorFunc {
[](const DetectedFace& a, const DetectedFace& b) {
return a.bbox.area() > b.bbox.area();
});
if (static_cast<int>(faces.size()) > max_faces_)
// TRACES: AR-003 | SR-002
// Largest-first ordering is kept regardless: it is load-bearing for
// deterministic association, since the Hungarian solver tie-breaks on
// index order (see the replay determinism test).
if (max_faces_ > 0 && static_cast<int>(faces.size()) > max_faces_)
faces.resize(max_faces_);
return {std::move(f), std::move(faces)};
src/nodes/identity_matcher_node.hpp
+52 -11
View File
@@ -106,6 +106,12 @@ struct IdentityMatcherFunc {
flat_emb_[i].data(), 512 * sizeof(float));
sim_engine_ = make_similarity_engine(host_gallery.data(), n_gallery_, kMaxFaces);
/// TRACES: AR-018, AR-024 | SR-005
// The expansion store thresholds in the same probability space as
// association and evidence weighting, so a "0.9" means one thing
// pipeline-wide rather than three.
track_gallery_.set_calibration(same_person_probability(cal_));
}
/// TRACES: AR-023, AR-024 | SR-002
@@ -138,27 +144,51 @@ struct IdentityMatcherFunc {
// mix embeddings from two viewpoints under one buffer, so we still drop
// every diversity buffer here — a revived track simply re-accumulates its
// buffer from post-cut frames. Stale cross-cut embeddings are never promoted.
if (tf.source.is_cut) track_gallery_.clear_tracks();
/// TRACES: AR-019 | SR-005
// Promotion may only borrow same-identity evidence from a span where
// identity is certain, so ALL THREE discontinuity signals clear the
// buffers, not just the histogram cut:
// is_cut — camera-angle change
// is_scene_boundary — different scene (AR-010; previously never set,
// so this half of the gate was dead)
// The third, an identity contradiction (AR-015), is enforced by the
// registry: a track whose belief swapped is closed outright, so it can
// no longer promote anything.
if (tf.source.is_cut || tf.source.is_scene_boundary)
track_gallery_.clear_tracks();
const int n_faces = static_cast<int>(tf.embeddings.size());
std::vector<IdentifiedActor> actors;
actors.reserve(n_faces);
if (n_faces == 0) return {std::move(tf.source), {}};
if (n_faces > kMaxFaces)
throw std::runtime_error("identity_matcher: n_faces exceeds kMaxFaces");
std::vector<float> host_query(static_cast<size_t>(n_faces) * 512);
for (int fi = 0; fi < n_faces; ++fi) {
std::memcpy(host_query.data() + static_cast<size_t>(fi) * 512,
tf.embeddings[fi].data(), 512 * sizeof(float));
/// TRACES: AR-003, AR-004 | SR-002
// kMaxFaces sizes the similarity engine's preallocated buffer, so it
// bounds MEMORY, not how many faces a frame may contain. It used to
// throw above the bound, which made it a hard cap on crowd scenes by
// accident; now the frame is scored in batches of that size.
//
// Faces per frame are unbounded (AR-003) because X-Ray credits scene
// membership to background cast too, and a fixed cap discards exactly
// those — the smallest faces are dropped first. Cost is contained by
// backpressure (AR-004), which slows the producer, rather than by
// silently throwing work away.
std::vector<float> host_query(static_cast<size_t>(kMaxFaces) * 512);
for (int base = 0; base < n_faces; base += kMaxFaces) {
const int chunk = std::min(kMaxFaces, n_faces - base);
for (int k = 0; k < chunk; ++k) {
std::memcpy(host_query.data() + static_cast<size_t>(k) * 512,
tf.embeddings[base + k].data(), 512 * sizeof(float));
}
// S (N_gallery × n_faces) col-major: face fi's gallery sims at sims + fi*n_gallery.
const float* host_sims = sim_engine_->compute(host_query.data(), n_faces);
// S (N_gallery × chunk) col-major: face k's gallery sims at sims + k*n_gallery.
const float* host_sims = sim_engine_->compute(host_query.data(), chunk);
for (int fi = 0; fi < n_faces; ++fi) {
const float* sims = host_sims + static_cast<size_t>(fi) * n_gallery_;
for (int ci = 0; ci < chunk; ++ci) {
const int fi = base + ci;
const float* sims = host_sims + static_cast<size_t>(ci) * n_gallery_;
std::vector<float> best_sim(gallery_.actors.size(),
-std::numeric_limits<float>::max());
@@ -252,11 +282,22 @@ struct IdentityMatcherFunc {
registry_->observe(tf.track_ids[fi], best_actor, p, tf.embeddings[fi]);
}
// TRACES: AR-019 | SR-005
// Ownership is the registry's, computed once. TrackGallery used to
// tally its own plurality vote over accepted frames, which meant two
// different answers to "who is this track" could coexist — and the
// expansion one ignored the Bayesian accumulation entirely.
if (registry_ && tf.track_ids[fi] >= 0) {
if (auto owner = registry_->owner(tf.track_ids[fi]))
track_gallery_.set_owner(tf.track_ids[fi], *owner);
}
track_gallery_.observe(tf.track_ids[fi], tf.embeddings[fi],
best_actor, best_s, accept, tf.crops[fi]);
actors.push_back(std::move(ia));
}
} // chunk loop
return {std::move(tf.source), std::move(actors)};
}
src/nodes/result_sink_node.hpp
+2 -2
View File
@@ -46,7 +46,7 @@ using json = nlohmann::json;
struct ResultSinkFunc {
static constexpr std::string_view label() { return "result_sink"; }
/// TRACES: AR-012, AR-017, IR-002 | SR-002, SR-003
/// TRACES: AR-012, AR-017 | IR-002 | SR-002, SR-003
/// A finished presence claim from the registry. Called from inside the
/// registry's reap while it holds its own lock, so this must stay a cheap
/// push and must never re-enter the registry.
@@ -158,7 +158,7 @@ private:
struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };
// Core logic: merge per-frame detections into annealed [start, end] windows.
/// TRACES: AR-012, IR-002 | SR-002
/// TRACES: AR-012 | IR-002 | SR-002
/// A claim already IS a window — `[first_seen, last_seen]` of a track the
/// actor owned. There is no annealing pass: `anneal_sec` existed to bridge
/// gaps between isolated accepted frames, and a track that survives its own
src/nodes/scene_boundary_annotator_node.hpp
+67
View File
@@ -0,0 +1,67 @@
#pragma once
/// TRACES: AR-010 | SR-002
///
/// SceneBoundaryAnnotatorFunc — the join of the decode butterfly.
///
/// `source` fans out to two branches: dense frames to TransNetV2, sampled frames
/// to face detection. A boundary found on the first has to reach the second, and
/// cannot ride along in the frame because the branches run in parallel.
///
/// This node sits on the sampled branch and stamps `Frame::is_scene_boundary`
/// from the detector's published verdict.
///
/// **It only works because the sampled branch lags.** TransNetV2 buffers
/// `kWindow` frames before it can score any of them, so this node must not reach
/// a frame before the detector has an opinion about it. Channel depth creates
/// that lag: with backpressure (AR-004) the fanout blocks on the slower branch,
/// so a deep channel here lets the detector run ahead by its window instead of
/// anything being dropped.
///
/// When the lag is insufficient the node **counts it** rather than guessing.
/// Annotating an unscored frame as boundary-free is indistinguishable from a
/// genuine "no boundary here", and that is the failure that makes a downstream
/// test pass while verifying nothing.
#include "scene_boundaries.hpp"
#include "types.hpp"
#include <memory>
#include <string_view>
#include <utility>
struct SceneBoundaryAnnotatorFunc {
static constexpr std::string_view label() { return "scene_annotate"; }
/// `tol` is half a sample interval. The branches sample at different rates,
/// so a boundary found on a dense frame rarely lands exactly on a sampled
/// one; half an interval attributes it to the nearest sampled frame and no
/// further.
SceneBoundaryAnnotatorFunc(std::shared_ptr<SceneBoundaries> b, double tol)
: bounds_(std::move(b)), tol_(tol) {}
Frame operator()(Frame f) {
if (f.eof || !bounds_) return f;
// Wait for the detector's verdict to cover this frame. Channel depth
// alone cannot provide the lag: it holds frames back only when the
// consumer is slower, and this branch is orders of magnitude faster per
// frame than TransNetV2. Blocking here is what makes the join real.
//
// Safe under backpressure because the branches are independent: this
// node stalling does not stop the detector consuming dense frames, and
// the fanout keeps feeding it.
if (!bounds_->wait_until_scored(f.timestamp_sec)) {
// The detector finished without covering this frame — the tail after
// its last full window. Unknown, not negative; counted so it cannot
// pass for "no boundary here".
bounds_->note_outran();
return f;
}
f.is_scene_boundary = bounds_->is_boundary(f.timestamp_sec, tol_);
return f;
}
private:
std::shared_ptr<SceneBoundaries> bounds_;
double tol_{0.0};
};
src/nodes/scene_detector_node.hpp
+35 -2
View File
@@ -1,6 +1,9 @@
#pragma once
#include "types.hpp"
#include "config.hpp"
#include "scene_boundaries.hpp"
#include <memory>
#include "inference/scene_detector.hpp"
#include <nlohmann/json.hpp>
@@ -30,6 +33,11 @@
struct SceneDetectorFunc {
static constexpr std::string_view label() { return "scene_detector"; }
/// TRACES: AR-010 | SR-002
/// Publish each window's verdict as it is scored, so the face branch — held
/// back by channel depth — can consult it for frames it has not reached yet.
void set_boundaries(std::shared_ptr<SceneBoundaries> b) { shared_ = std::move(b); }
SceneDetectorFunc(const Config& cfg, std::atomic<bool>& done)
: detector_(make_scene_detector(cfg))
, threshold_(cfg.scene_threshold)
@@ -51,6 +59,9 @@ struct SceneDetectorFunc {
void operator()(Frame f) {
if (f.eof) {
flush_remaining();
// Release anyone waiting on the join: the tail frames after the last
// full window will never be covered, so waiting for them would hang.
if (shared_) shared_->finish();
write_output();
done_.store(true, std::memory_order_release);
return;
@@ -82,6 +93,7 @@ private:
// otherwise skip the leading guard already covered by the previous window.
const int lo = (window_base_ == 0) ? 0 : guard_;
const int hi = ISceneDetector::kWindow - guard_;
std::vector<double> fresh;
for (int i = lo; i < hi; ++i) {
if (probs[i] <= threshold_) continue;
// Local maximum → the boundary frame (avoid a run of high scores
@@ -89,9 +101,19 @@ private:
const bool peak =
(i == 0 || probs[i] >= probs[i-1]) &&
(i == kLast_() || probs[i] >= probs[i+1]);
if (peak)
if (peak) {
boundaries_.push_back({times_[i], probs[i]});
fresh.push_back(times_[i]);
}
}
/// TRACES: AR-010 | SR-002
// Publish with a watermark: everything up to times_[hi-1] now has a
// final verdict. The face branch consults this for frames it has not
// reached yet, and the watermark is what lets it tell "no boundary
// here" from "not scored yet".
if (shared_ && hi > lo)
shared_->publish(fresh, times_[hi - 1]);
}
// At EOF the tail (< kWindow frames) never formed a full window. Pad it out
@@ -107,14 +129,24 @@ private:
std::vector<float> probs = detector_->detect_window(win);
const int lo = (window_base_ == 0) ? 0 : guard_;
std::vector<double> fresh;
for (int i = lo; i < n; ++i) { // only real (non-padded) frames
if (probs[i] <= threshold_) continue;
const bool peak =
(i == 0 || probs[i] >= probs[i-1]) &&
(i == n - 1 || probs[i] >= probs[i+1]);
if (peak)
if (peak) {
boundaries_.push_back({times_[i], probs[i]});
fresh.push_back(times_[i]);
}
}
/// TRACES: AR-010 | SR-002
// Publish the tail too. Without this the final frames — everything after
// the last full window — reach the join with no verdict and are treated
// as boundary-free without evidence, which is precisely the ambiguity
// the watermark exists to prevent.
if (shared_ && n > 0) shared_->publish(fresh, times_[n - 1]);
}
void write_output() {
@@ -176,4 +208,5 @@ private:
int64_t window_base_{0}; // frame index of images_.front()
std::vector<Boundary> boundaries_;
bool written_{false};
std::shared_ptr<SceneBoundaries> shared_; ///< AR-010 join point
};
src/python_bindings.cpp
+205 -1
View File
@@ -3,17 +3,77 @@
// Loads both ONNX sessions once per FaceEmbedder instance, then embeds many
// images via repeated embed() calls — avoiding the per-process model-load
// cost of the embed_faces CLI when embedding a large gallery.
//
// Beyond whole-image embed(), the individual pipeline stages are exposed —
// detect(), align_face(), embed_crop() — plus the gallery calibration. A study
// that needs to step between stages (a different landmark source, a degraded
// crop) drives the shipped C++ from Python rather than re-implementing
// detection, alignment, the ArcFace warp or the Platt fit in numpy. Those
// re-implementations drift from what ships, and the calibration is the one
// that must not: AR-024 requires every similarity to pass through
// GalleryCalibration::probability, never a bare cosine.
#include "face_embedder_engine.hpp"
#include "gallery/gallery_calibration.hpp"
#include "gallery/gallery_store.hpp"
#include <nanobind/nanobind.h>
#include <nanobind/ndarray.h>
#include <nanobind/stl/optional.h>
#include <nanobind/stl/string.h>
#include <nanobind/stl/vector.h>
#include <array>
#include <cstring>
#include <stdexcept>
namespace nb = nanobind;
using namespace nb::literals;
namespace {
using ImageArray = nb::ndarray<const uint8_t, nb::ndim<3>, nb::c_contig, nb::device::cpu>;
// numpy HxWx3 uint8 (BGR, as cv::imread yields) → cv::Mat sharing that buffer.
// The Mat is a view: it must not outlive the caller's array, so every use here
// copies or consumes it before returning.
cv::Mat as_mat(const ImageArray& a) {
if (a.shape(2) != 3)
throw std::invalid_argument("expected an HxWx3 uint8 BGR image");
return cv::Mat(static_cast<int>(a.shape(0)), static_cast<int>(a.shape(1)),
CV_8UC3, const_cast<uint8_t*>(a.data()));
}
// cv::Mat → freshly-allocated numpy array (owns its buffer).
nb::ndarray<nb::numpy, uint8_t> mat_to_numpy(const cv::Mat& m) {
cv::Mat c = m.isContinuous() ? m : m.clone();
auto* buf = new uint8_t[c.total() * c.elemSize()];
std::memcpy(buf, c.data, c.total() * c.elemSize());
nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast<uint8_t*>(p); });
size_t shape[3] = {static_cast<size_t>(c.rows), static_cast<size_t>(c.cols),
static_cast<size_t>(c.channels())};
return nb::ndarray<nb::numpy, uint8_t>(buf, 3, shape, owner);
}
nb::ndarray<nb::numpy, float> vec_to_numpy(std::vector<float>&& v) {
auto* buf = new float[v.size()];
std::memcpy(buf, v.data(), v.size() * sizeof(float));
nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast<float*>(p); });
size_t shape[1] = {v.size()};
return nb::ndarray<nb::numpy, float>(buf, 1, shape, owner);
}
// numpy (5,2) float32 → the landmark array align_face expects. Order is
// types.hpp:60 — [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth.
std::array<cv::Point2f, 5> as_landmarks(
const nb::ndarray<const float, nb::shape<5, 2>, nb::c_contig, nb::device::cpu>& a) {
std::array<cv::Point2f, 5> lm;
for (int i = 0; i < 5; ++i) lm[i] = {a(i, 0), a(i, 1)};
return lm;
}
} // namespace
NB_MODULE(sae_embed, m) {
m.doc() = "SCRFD + ArcFace face embedding, models loaded once per FaceEmbedder";
@@ -27,6 +87,23 @@ NB_MODULE(sae_embed, m) {
})
.def_prop_ro("bbox", [](const FaceEmbedResult& r) {
return std::vector<float>{r.bbox[0], r.bbox[1], r.bbox[2], r.bbox[3]};
})
.def_prop_ro("landmarks", [](const FaceEmbedResult& r) {
std::vector<float> v;
for (const auto& p : r.landmarks) { v.push_back(p.x); v.push_back(p.y); }
return v;
});
nb::class_<DetectedFace>(m, "Detection")
.def_ro("confidence", &DetectedFace::confidence)
.def_prop_ro("bbox", [](const DetectedFace& d) {
return std::vector<float>{d.bbox.x, d.bbox.y, d.bbox.width, d.bbox.height};
})
.def_prop_ro("landmarks", [](const DetectedFace& d) {
// (5,2): [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth
std::vector<float> v;
for (const auto& p : d.landmarks) { v.push_back(p.x); v.push_back(p.y); }
return v;
});
nb::class_<FaceEmbedderEngine>(m, "FaceEmbedder")
@@ -38,5 +115,132 @@ NB_MODULE(sae_embed, m) {
.def("embed", &FaceEmbedderEngine::embed_path, "path"_a,
nb::call_guard<nb::gil_scoped_release>(),
"Detect the highest-confidence face in the image, align it, and "
"return a FaceResult with its 512-d ArcFace embedding.");
"return a FaceResult with its 512-d ArcFace embedding.")
.def("embed_mat", [](FaceEmbedderEngine& e, ImageArray img) {
return e.embed_mat(as_mat(img).clone());
}, "image"_a,
"As embed(), on an in-memory HxWx3 uint8 BGR array.")
.def("detect", [](FaceEmbedderEngine& e, ImageArray img) {
return e.detect(as_mat(img));
}, "image"_a,
"Run the configured detector. Returns every Detection, unfiltered — "
"min_face_px is applied downstream in face_detector_node.")
.def("embed_crop", [](FaceEmbedderEngine& e, ImageArray crop) {
cv::Mat c = as_mat(crop);
if (c.rows != 112 || c.cols != 112)
throw std::invalid_argument("embed_crop expects a 112x112 aligned crop");
Embedding emb = e.embed_crop(c);
return vec_to_numpy(std::vector<float>(emb.begin(), emb.end()));
}, "crop"_a,
"Embed a caller-supplied 112x112 aligned BGR crop. The stage-level "
"entry point for studies that degrade or re-align a crop themselves.")
.def("embed_crops", [](FaceEmbedderEngine& e,
nb::ndarray<const uint8_t, nb::ndim<4>, nb::c_contig,
nb::device::cpu> crops) {
if (crops.shape(1) != 112 || crops.shape(2) != 112 || crops.shape(3) != 3)
throw std::invalid_argument("embed_crops expects (N,112,112,3) uint8 BGR");
const size_t n = crops.shape(0);
std::vector<cv::Mat> mats;
mats.reserve(n);
for (size_t i = 0; i < n; ++i)
mats.emplace_back(112, 112, CV_8UC3,
const_cast<uint8_t*>(crops.data()) + i * 112 * 112 * 3);
std::vector<Embedding> out = e.embed_crops(mats);
auto* buf = new float[n * 512];
for (size_t i = 0; i < n; ++i)
std::memcpy(buf + i * 512, out[i].data(), 512 * sizeof(float));
nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast<float*>(p); });
size_t shape[2] = {n, 512};
return nb::ndarray<nb::numpy, float>(buf, 2, shape, owner);
}, "crops"_a,
"Batched embed_crop: (N,112,112,3) uint8 BGR in, (N,512) float32 out. "
"The backend batches internally, so this avoids paying per-call "
"overhead once per crop across a large study.")
.def_prop_ro("max_batch", [](FaceEmbedderEngine& e) { return e.max_batch(); });
m.def("align_face", [](ImageArray img,
nb::ndarray<const float, nb::shape<5, 2>, nb::c_contig,
nb::device::cpu> landmarks)
-> std::optional<nb::ndarray<nb::numpy, uint8_t>> {
cv::Mat crop = ::align_face(as_mat(img), as_landmarks(landmarks));
if (crop.empty()) return std::nullopt; // degenerate fit
return mat_to_numpy(crop);
}, "image"_a, "landmarks"_a,
"The ArcFace 5-point similarity transform (face_utils.hpp, AR-005). "
"Returns a 112x112 BGR crop, or None if the affine fit is degenerate. "
"Landmark order is types.hpp:60 — right-eye, left-eye, nose, "
"right-mouth, left-mouth.");
m.def("enhance_for_retry", [](ImageArray img) {
return mat_to_numpy(::enhance_for_retry(as_mat(img)));
}, "image"_a,
"Border-replicate pad by 50% and CLAHE, for a detector second try.");
// ── Calibration ──────────────────────────────────────────────────────────
// AR-024: the pipeline reasons in one probability space. Exposed so Python
// scores through the same sigmoid the C++ matcher uses, rather than a numpy
// copy of it that can silently disagree.
nb::class_<GalleryCalibration>(m, "GalleryCalibration")
.def_ro("a", &GalleryCalibration::a)
.def_ro("b", &GalleryCalibration::b)
.def_ro("valid", &GalleryCalibration::valid)
.def("probability", &GalleryCalibration::probability,
"similarity"_a, "log_prior_odds"_a = 0.f,
"P(match | sim) = sigma(a*sim + b + log_prior_odds). Pass "
"log_prior_odds = log(p0/(1-p0)) for a base-rate prior p0; leave it "
"at 0 for association (is this one person), which is what the "
"balanced fit answers — see gallery_calibration.hpp:63.")
.def("boundary_at", &GalleryCalibration::boundary_at,
"p"_a = 0.5f, "log_prior_odds"_a = 0.f,
"The similarity at which P(match) == p. Diagnostic only — decisions "
"threshold the probability, not this.")
.def("__repr__", [](const GalleryCalibration& c) {
return "<GalleryCalibration a=" + std::to_string(c.a) +
" b=" + std::to_string(c.b) +
(c.valid ? " valid>" : " INVALID>");
});
m.def("gallery_calibration", [](const std::string& gallery_path) {
ActorGallery g = load_gallery(gallery_path);
if (g.calib_valid) {
std::cerr << "[calibration] " << gallery_path << ": cached fit"
<< " over " << g.actors.size() << " actors\n";
return GalleryCalibration{g.calib_a, g.calib_b, true};
}
// Legacy JSON galleries carry no stored fit; compute it over the
// whole gallery, which is the point — the calibration must come
// from the production actor population, not a handful of people.
std::cerr << "[calibration] " << gallery_path
<< ": no cached fit, computing over " << g.actors.size()
<< " actors\n";
std::vector<Embedding> flat;
std::vector<int> actor;
for (size_t a = 0; a < g.actors.size(); ++a)
for (const auto& e : g.actors[a].embeddings) {
flat.push_back(e);
actor.push_back(static_cast<int>(a));
}
return ::calibrate_gallery(flat, actor);
}, "gallery_path"_a,
"The production gallery's calibration — the global fit over every "
"actor in it. Use this to score, not a fit over a handful of people: "
"a sigmoid fitted on a few identities saturates, so its probabilities "
"mean nothing. Reads the cached fit stored in an HDF5 gallery, or "
"computes it over the whole gallery for a legacy JSON one.");
m.def("calibrate_gallery", [](nb::ndarray<const float, nb::shape<-1, 512>, nb::c_contig,
nb::device::cpu> emb,
std::vector<int> actor) {
const size_t n = emb.shape(0);
if (actor.size() != n)
throw std::invalid_argument("embeddings and actor ids differ in length");
std::vector<Embedding> flat(n);
for (size_t i = 0; i < n; ++i)
std::memcpy(flat[i].data(), &emb(i, 0), 512 * sizeof(float));
return ::calibrate_gallery(flat, actor);
}, "embeddings"_a, "actor_ids"_a,
"Fit the Platt sigmoid from intra/inter-class pairs — the same fit the "
"gallery build performs (gallery_calibration.hpp:85). embeddings is "
"(N,512) L2-normalised float32; actor_ids is a length-N list of "
"0-based actor indices.");
}
src/scene_boundaries.hpp
+126
View File
@@ -0,0 +1,126 @@
#pragma once
/// TRACES: AR-010 | SR-002
///
/// SceneBoundaries — the join point of the decode butterfly.
///
/// The topology forks after decode: one branch runs TransNetV2 over dense
/// frames, the other runs face detection over the sampled cadence. Boundaries
/// found on the first branch have to reach the second, and they cannot be
/// carried in the frames themselves because the branches are parallel.
///
/// **Why this needs a watermark.** TransNetV2 buffers `kWindow` frames before it
/// can score any of them, so at any instant the detector has an opinion about
/// everything up to some time T and nothing after it. Without recording T, a
/// consumer asking "is there a boundary at t?" cannot distinguish *no* from
/// *not yet* — and those demand opposite behaviour. Silently treating unscored
/// frames as boundary-free is exactly the class of failure that makes a
/// verification pass vacuously.
///
/// The consumer is held back by channel depth (see main.cpp) so that by the time
/// it pulls a frame, the detector has already scored past it. `scored_through()`
/// is what lets that assumption be *checked* rather than assumed.
#include <algorithm>
#include <condition_variable>
#include <mutex>
#include <vector>
class SceneBoundaries {
public:
/// Peaks closer than this are one boundary. Matches the dedup scenes.json
/// applies, so the two views agree.
static constexpr double kMergeSec = 0.04;
/// Called by the scene detector as each window is scored. `through` is the
/// timestamp up to which its verdict is now final.
void publish(const std::vector<double>& ts, double through) {
{
std::lock_guard<std::mutex> g(mu_);
// Dedup on insert, matching what scenes.json does at write time. A run
// of adjacent high-scoring frames is one boundary, not several, and
// leaving them raw made this view report 357 where the file said 13 —
// the same event counted many times. Harmless for is_boundary(), which
// absorbs them in its tolerance, but a count nobody can reconcile with
// the output file is a bad diagnostic.
bounds_.insert(bounds_.end(), ts.begin(), ts.end());
std::sort(bounds_.begin(), bounds_.end());
bounds_.erase(std::unique(bounds_.begin(), bounds_.end(),
[](double a, double b) { return b - a < kMergeSec; }),
bounds_.end());
scored_through_ = std::max(scored_through_, through);
}
cv_.notify_all();
}
/// True if a boundary falls within `tol` of `t`.
///
/// `tol` exists because the two branches sample at different rates: a
/// boundary found on a dense frame rarely lands exactly on a sampled one.
/// Half a sample interval is the natural width — it attributes the boundary
/// to the nearest sampled frame and no further.
bool is_boundary(double t, double tol) const {
std::lock_guard<std::mutex> g(mu_);
auto it = std::lower_bound(bounds_.begin(), bounds_.end(), t - tol);
return it != bounds_.end() && *it <= t + tol;
}
/// The timestamp through which the detector's verdict is final. A consumer
/// past this point is asking about frames nobody has looked at yet.
double scored_through() const {
std::lock_guard<std::mutex> g(mu_);
return scored_through_;
}
/// Block until the detector's verdict covers `t`, or it finishes.
///
/// Channel depth alone does NOT create the required lag: it only holds
/// frames back when the consumer is slower, and the face branch is roughly
/// four orders of magnitude faster per frame than TransNetV2. So the join
/// has to wait explicitly.
///
/// Returns false if the detector finished without ever covering `t`, which
/// happens for the tail frames after its last full window. The caller must
/// distinguish that from a genuine "no boundary" rather than assuming.
bool wait_until_scored(double t) const {
std::unique_lock<std::mutex> lk(mu_);
cv_.wait(lk, [&] { return finished_ || scored_through_ >= t; });
return scored_through_ >= t;
}
/// Called when the detector will publish nothing further. Without this the
/// join would deadlock on the tail: those frames are never covered by a full
/// window, so waiting for them would wait forever.
void finish() {
{
std::lock_guard<std::mutex> g(mu_);
finished_ = true;
}
cv_.notify_all();
}
std::size_t count() const {
std::lock_guard<std::mutex> g(mu_);
return bounds_.size();
}
/// Consumers that outran the detector. Nonzero means the face branch is not
/// buffered deeply enough for the detector's window, so some frames were
/// annotated from an incomplete verdict — a real misconfiguration, and one
/// that would otherwise be invisible.
void note_outran() const {
std::lock_guard<std::mutex> g(mu_);
++outran_;
}
std::size_t outran() const {
std::lock_guard<std::mutex> g(mu_);
return outran_;
}
private:
mutable std::mutex mu_;
mutable std::condition_variable cv_;
bool finished_{false};
std::vector<double> bounds_;
double scored_through_{-1.0};
mutable std::size_t outran_{0};
};
src/track_registry.hpp
+29 -9
View File
@@ -60,7 +60,13 @@ struct Track {
double first_seen{0.0};
std::optional<double> last_seen; ///< unset ⇒ on screen
std::optional<int> actor; ///< set once a posterior crosses
std::map<int, float> belief; ///< actor_idx → accumulated log-odds
/// actor_idx → accumulated log(1 − P). Lazy-OR (noisy-OR) accumulation:
/// each frame is new evidence that this track is that actor, and the
/// combined belief is the probability that *at least one* sighting was
/// right. Stored as log(1−P) because that makes the update additive and
/// keeps precision where it matters — as P approaches 1, (1−P) is the
/// quantity with the significant digits.
std::map<int, float> belief;
Embedding mean{}; ///< running directional mean
std::vector<Embedding> views; ///< distinct looks, for AR-025 discounting
float discounted_weight{0.f}; ///< sum of applied weights
@@ -146,14 +152,20 @@ public:
if (it == tracks_.end()) { ++dropped_votes_; return; }
Track& t = it->second;
const float w = discounter_.weight(t.views, e);
t.belief[actor_idx] += w * logit(posterior);
const float w = discounter_.weight(t.views, t.n_obs, e);
// Weighted lazy-OR: P_new = 1 − (1 − P_old)·(1 − p)^w, which in log
// space is a plain sum. w is the discounted evidence (AR-025), so a
// repeated view still advances the belief but by a fraction of what a
// genuinely new look would.
const float p = std::min(1.f - 1e-6f, std::max(1e-6f, posterior));
t.belief[actor_idx] += w * std::log(1.f - p);
t.discounted_weight += w;
++t.n_obs;
const int best = argmax_belief(t);
const float best_lo = t.belief[best];
if (best_lo < cfg_.ownership_logodds) return;
const int best = argmax_belief(t);
const float best_p = 1.f - std::exp(t.belief[best]);
if (best_p < own_threshold()) return;
if (!t.actor.has_value()) {
claim_locked(t, best);
@@ -285,20 +297,28 @@ private:
d.effective_obs = t.discounted_weight;
if (t.actor) {
d.actor_idx = *t.actor;
d.belief = logistic(t.belief[*t.actor]);
d.belief = 1.f - std::exp(t.belief[*t.actor]);
auto oi = owner_index_.find(*t.actor);
if (oi != owner_index_.end() && oi->second == t.id) owner_index_.erase(oi);
}
on_dead_(d);
}
/// Most-believed actor. belief holds log(1 − P), so the strongest claim is
/// the *most negative* entry, not the largest.
static int argmax_belief(const Track& t) {
int best = -1;
float hi = -1e30f;
for (const auto& [a, lo] : t.belief) if (lo > hi) { hi = lo; best = a; }
float lo = 1e30f;
for (const auto& [a, v] : t.belief) if (v < lo) { lo = v; best = a; }
return best;
}
/// Ownership expressed as a probability. Config still carries log-odds so
/// the knob keeps its meaning across this change.
float own_threshold() const {
return 1.f / (1.f + std::exp(-cfg_.ownership_logodds));
}
static void update_mean(Track& t, const Embedding& e) {
// Directional mean: accumulate then re-normalise to the unit sphere, so
// cosine against it stays a plain dot product.
src/types.hpp
+7
View File
@@ -63,6 +63,13 @@ struct DetectedFace {
cv::Rect2f bbox;
std::array<cv::Point2f, 5> landmarks;
float confidence{0.f};
// AR-030 visibility: RMS landmark misfit, in canonical 112×112 pixels, left
// over after the best similarity fit to the ArcFace template. Rises with
// out-of-plane pose and with occlusion; blind to in-plane roll and to face
// size, both of which the fit absorbs. Set by the aligner, which is where
// the transform is computed; -1 until then.
float alignment_residual{-1.f};
};
// ── Pipeline messages ─────────────────────────────────────────────────────────
tests/CMakeLists.txt
+13
View File
@@ -34,7 +34,20 @@ target_include_directories(sae_tests PRIVATE ${CMAKE_SOURCE_DIR}/src)
# SAE_MODELS_DIR: config.hpp (pulled in by track_gallery.hpp) bakes model paths.
# SAE_TEST_FIXTURES_DIR: the audio golden vector is read from the source tree,
# not copied, so the file the plugin repo shares is the file under test.
# AR-026/AR-027: exercise the same kernel CI actually runs. Without this the
# suite compiles the scalar fallback while the CPU builder image links OpenBLAS,
# so the tested path and the shipped path would differ.
find_package(PkgConfig QUIET)
if(PkgConfig_FOUND)
pkg_check_modules(OPENBLAS_T QUIET openblas)
endif()
if(OPENBLAS_T_FOUND)
target_include_directories(sae_tests PRIVATE ${OPENBLAS_T_INCLUDE_DIRS})
target_link_libraries(sae_tests PRIVATE ${OPENBLAS_T_LINK_LIBRARIES})
endif()
target_compile_definitions(sae_tests PRIVATE
$<$<BOOL:${OPENBLAS_T_FOUND}>:SAE_GEMM_CBLAS>
SAE_GEMM_CPU
SAE_MODELS_DIR="${SAE_MODELS_DIR}"
SAE_TEST_FIXTURES_DIR="${CMAKE_CURRENT_SOURCE_DIR}/fixtures")
tests/fixtures/audio/bali_offset_200s.flac
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/audio/make_offset_fixture.sh
Vendored Executable
+60
View File
@@ -0,0 +1,60 @@
#!/bin/sh
#
# Regenerate bali_offset_200s.flac — the real-audio fixture behind VR-014, the
# audio-signature offset-recovery validation.
#
# sh make_offset_fixture.sh /path/to/clips
#
# Why real audio and not a second synthetic tone: jray_audio_v1_tone.flac pins
# the *arithmetic* (IR-005) and is deliberately built so every band and every
# energy class appears. It cannot answer the question VR-014 asks — whether the
# peak-bin sequence of ordinary film audio is distinctive enough that sliding
# one signature against another finds the true alignment and only the true
# alignment. Tones are pathologically easy for that; dialogue and score are not.
#
# Source: five scene clips from "Road to Bali" (1952), the public-domain corpus
# this repo already uses for the replay fixtures — tests/fixtures/dumps/bali_*.h5
# are dumps of these same clips. Each is under the 120 s window on its own
# (29-77 s), so they are concatenated in scene order to make a source long
# enough that a 120 s window can slide inside it.
#
# 200 s is chosen, not arbitrary: the window is 120 s and the match search is
# capped at +/-600 frames (~55.7 s), so a source of 120 + 56 s is the shortest
# one that can place two windows at the edge of the cap. The 200 s here leaves
# room to go past it as well, which is what lets the test check that an
# out-of-range offset is declined rather than guessed.
#
# Encoded mono at 11025 Hz, 16-bit, which is exactly what the signature decodes
# to anyway. That keeps a 200 s fixture at ~2.4 MB instead of ~20 MB, and makes
# every trim below sample-exact — the test measures offset recovery, not the
# resampler, which tests/test_audio_signature.cpp already covers (UT-103).
#
# FLAC because it is lossless: the decoded PCM is the same on every machine, so
# a signature computed from this file is reproducible. A lossy fixture would
# make the measurement depend on the decoder version.
#
# sha256 of the committed file:
# 4a952e46a090a9acd9eae56996250ec03e08e0d04ee139ac0a42f1690a536c83
# A regenerated file that hashes differently means the source clips or the
# encoder changed, and VR-014's recorded numbers should be re-measured — the
# offsets will still be exact, but the scores are this audio's.
set -eu
CLIPS="${1:-../../../../bali}"
OUT="$(dirname "$0")/bali_offset_200s.flac"
LIST="$(mktemp)"
trap 'rm -f "$LIST"' EXIT
for scene in 13 27 28 31 46; do
clip="$CLIPS/Road_To_Bali-$scene.webm"
[ -f "$clip" ] || { echo "missing clip: $clip" >&2; exit 1; }
echo "file '$(cd "$(dirname "$clip")" && pwd)/$(basename "$clip")'" >> "$LIST"
done
ffmpeg -nostdin -v error -y -f concat -safe 0 -i "$LIST" \
-vn -t 200 -ac 1 -ar 11025 -sample_fmt s16 \
-c:a flac -compression_level 12 "$OUT"
echo "wrote $OUT"
sha256sum "$OUT" 2>/dev/null || shasum -a 256 "$OUT"
tests/test_calibration.cpp
+93
View File
@@ -168,3 +168,96 @@ TEST_CASE("calibrate_gallery_cached treats hash=0 as always-recompute", "[calibr
CHECK(recomputed);
CHECK(cal.valid);
}
// ── GR-003 — the build report ────────────────────────────────────────────────
#include "gallery/gallery_report.hpp"
namespace {
// A unit vector on one axis. Distinct axes are orthogonal, which is unrealistic
// as a same-actor cluster but irrelevant here: these tests count actors, they do
// not assess fit quality.
Embedding unit_axis(int slot) {
Embedding e{};
e[slot % 512] = 1.0f;
return e;
}
} // namespace
TEST_CASE("report surfaces actors that can never be recognised", "[report][GR-003]") {
// An actor with no usable image is a silent recall ceiling: the pipeline
// will never name them, and nothing in the gallery says why. This is the
// single most useful number in the report.
ActorGallery g;
for (int a = 0; a < 3; ++a) {
ActorGallery::Actor act;
act.name = "actor" + std::to_string(a);
if (a != 1) // actor1 gets nothing
for (int i = 0; i < 6; ++i) act.embeddings.push_back(unit_axis(a * 10 + i));
g.actors.push_back(std::move(act));
}
std::vector<Embedding> flat;
std::vector<int> flat_actor;
for (int a = 0; a < static_cast<int>(g.actors.size()); ++a)
for (const auto& e : g.actors[a].embeddings) { flat.push_back(e); flat_actor.push_back(a); }
GalleryCalibrationStats stats;
GalleryCalibration cal = calibrate_gallery(flat, flat_actor, &stats);
GalleryReport r = build_gallery_report(g, cal, stats);
// An actor present in the gallery with no embeddings is counted as
// in-gallery but contributes nothing; the zero-usable list is populated
// from the build audit, which a stored gallery cannot supply.
CHECK(r.actors_in_gallery == 3);
CHECK(r.actors[1].references == 0);
}
TEST_CASE("report surfaces actors too thin to calibrate on", "[report][GR-003]") {
// Below the positive-pair threshold an actor contributes nothing to the
// intra-class side of the fit. They are not broken, so nothing complains —
// they just quietly weaken every threshold downstream.
ActorGallery g;
for (int a = 0; a < 2; ++a) {
ActorGallery::Actor act;
act.name = "actor" + std::to_string(a);
const int n = (a == 0) ? 6 : 2; // actor1 is under-referenced
for (int i = 0; i < n; ++i) act.embeddings.push_back(unit_axis(a * 10 + i));
g.actors.push_back(std::move(act));
}
std::vector<Embedding> flat;
std::vector<int> flat_actor;
for (int a = 0; a < static_cast<int>(g.actors.size()); ++a)
for (const auto& e : g.actors[a].embeddings) { flat.push_back(e); flat_actor.push_back(a); }
GalleryCalibrationStats stats;
GalleryCalibration cal = calibrate_gallery(flat, flat_actor, &stats);
GalleryReport r = build_gallery_report(g, cal, stats);
CHECK(r.actors_below_positive_threshold >= 1);
}
TEST_CASE("report round-trips", "[report][GR-003]") {
ActorGallery g;
ActorGallery::Actor act;
act.name = "solo";
for (int i = 0; i < 6; ++i) act.embeddings.push_back(unit_axis(i));
g.actors.push_back(std::move(act));
std::vector<Embedding> flat;
std::vector<int> flat_actor;
for (const auto& e : g.actors[0].embeddings) { flat.push_back(e); flat_actor.push_back(0); }
GalleryCalibrationStats stats;
GalleryCalibration cal = calibrate_gallery(flat, flat_actor, &stats);
GalleryReport r = build_gallery_report(g, cal, stats);
const std::string path = "/tmp/gr003_roundtrip.report.json";
save_gallery_report(path, r);
GalleryReport back = load_gallery_report(path);
CHECK(back.actors_in_gallery == r.actors_in_gallery);
CHECK(back.actors_below_positive_threshold == r.actors_below_positive_threshold);
CHECK(back.calib_a == r.calib_a);
std::remove(path.c_str());
}
tests/test_face_utils.cpp
+107 -2
View File
@@ -1,6 +1,8 @@
// TRACES: AR-005, AR-030 | SR-002
//
// Unit tests for the geometric/numeric helpers in types.hpp and face_utils.hpp:
// cosine_similarity and the ArcFace 5-point alignment transform. GPU-free,
// model-free.
// cosine_similarity, the ArcFace 5-point alignment transform, and the alignment
// residual that AR-030 reads as its visibility measure. GPU-free, model-free.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
@@ -74,3 +76,106 @@ TEST_CASE("align_face returns empty on degenerate (collinear) landmarks", "[face
cv::Mat crop = align_face(img, lm);
CHECK(crop.empty());
}
// ── AR-030: the alignment residual as a visibility measure ────────────────────
// These assert the *properties* the measure is relied on for, not a magic value.
// Each would fail under a RANSAC fit, which buys a small residual by discarding
// the very landmarks that carry the signal.
namespace {
std::array<cv::Point2f, 5> canonical() {
std::array<cv::Point2f, 5> lm;
for (int i = 0; i < 5; ++i) lm[i] = {kArcFaceRef[i][0], kArcFaceRef[i][1]};
return lm;
}
// Rotate by `deg` in-plane, scale uniformly, translate — i.e. exactly the 4 DoF
// the similarity transform models.
std::array<cv::Point2f, 5> similarity(const std::array<cv::Point2f, 5>& in,
float deg, float s, float tx, float ty) {
const float r = deg * 3.14159265358979f / 180.f;
const float c = std::cos(r), sn = std::sin(r);
std::array<cv::Point2f, 5> out;
for (int i = 0; i < 5; ++i)
out[i] = {s * (c * in[i].x - sn * in[i].y) + tx,
s * (sn * in[i].x + c * in[i].y) + ty};
return out;
}
// Squash x about the centroid by `k`: the anisotropic deformation an out-of-plane
// yaw produces, and the one a similarity provably cannot absorb.
std::array<cv::Point2f, 5> foreshorten(const std::array<cv::Point2f, 5>& in, float k) {
float cx = 0.f;
for (const auto& p : in) cx += p.x;
cx /= 5.f;
std::array<cv::Point2f, 5> out = in;
for (auto& p : out) p.x = cx + (p.x - cx) * k;
return out;
}
} // namespace
TEST_CASE("residual is zero for a face in canonical pose", "[face_utils][AR-030]") {
const Alignment a = estimate_alignment(canonical());
REQUIRE(a.ok);
CHECK_THAT(a.residual, WithinAbs(0.0f, 1e-3f));
}
TEST_CASE("residual ignores in-plane roll, scale and translation", "[face_utils][AR-030]") {
// The structural claim behind AR-030: the fit absorbs all four similarity
// DoF exactly, so what remains is only the deformation a similarity cannot
// explain. A rolled head must not read as a turned one.
for (float deg : {-40.f, -12.f, 0.f, 17.f, 65.f}) {
const Alignment a = estimate_alignment(similarity(canonical(), deg, 3.5f, 220.f, -40.f));
REQUIRE(a.ok);
CHECK_THAT(a.residual, WithinAbs(0.0f, 1e-3f));
}
}
TEST_CASE("residual rises monotonically with foreshortening", "[face_utils][AR-030]") {
float prev = -1.f;
for (float k : {1.0f, 0.9f, 0.75f, 0.5f, 0.3f}) {
const Alignment a = estimate_alignment(foreshorten(canonical(), k));
REQUIRE(a.ok);
CHECK(a.residual > prev);
prev = a.residual;
}
}
TEST_CASE("residual is independent of face size", "[face_utils][AR-030]") {
// The measure must not silently re-express face size — that is AR-002's job,
// and double-counting it would make a small frontal face look occluded.
// Same deformation, two very different face sizes, one answer.
const auto small = similarity(foreshorten(canonical(), 0.7f), 20.f, 1.0f, 0.f, 0.f);
const auto large = similarity(foreshorten(canonical(), 0.7f), 20.f, 12.0f, 500.f, 300.f);
const Alignment a = estimate_alignment(small);
const Alignment b = estimate_alignment(large);
REQUIRE(a.ok);
REQUIRE(b.ok);
CHECK_THAT(b.residual, WithinAbs(a.residual, 1e-2f));
}
TEST_CASE("the fit never mirrors the face", "[face_utils][AR-030]") {
// SVD will happily return an orientation-reversing solution; a similarity
// transform may rotate but never reflect. Without the determinant guard a
// mirrored landmark set fits "perfectly" as a reflection.
const auto mirrored = foreshorten(canonical(), -1.f);
const Alignment a = estimate_alignment(mirrored);
REQUIRE(a.ok);
const double det = a.M.at<double>(0,0) * a.M.at<double>(1,1)
- a.M.at<double>(0,1) * a.M.at<double>(1,0);
CHECK(det > 0.0);
CHECK(a.residual > 1.0f); // and the mirroring shows up as misfit
}
TEST_CASE("degenerate landmarks report not-ok rather than a residual", "[face_utils][AR-030]") {
std::array<cv::Point2f, 5> lm;
for (auto& p : lm) p = {50.f, 50.f};
const Alignment a = estimate_alignment(lm);
CHECK_FALSE(a.ok);
CHECK(a.M.empty());
}
tests/test_replay_fixtures.cpp
+1 -1
View File
@@ -1,6 +1,6 @@
// Replay tests — the real tracker and registry driven from committed fixtures.
//
// TRACES: AR-012, AR-013, AR-004, VR-001, VR-002 | IT-001
// TRACES: AR-004, AR-012, AR-013 | VR-001, VR-002 | IT-001
//
// Tier T2: composition, not units. The registry tests construct awkward states
// directly; these check that the pieces behave when wired together and fed real
tests/test_track_gallery.cpp
+15 -5
View File
@@ -83,15 +83,25 @@ TEST_CASE("novelty gate skips views the gallery already covers", "[track_gallery
CHECK(tg.annex().empty());
}
TEST_CASE("spread gate rejects a two-person track", "[track_gallery]") {
TEST_CASE("a two-person track never poisons the annex", "[track_gallery][AR-018]") {
TrackGallery tg(expand_cfg());
// Two orthogonal identities under one track ID: pairwise sim 0 → spread 1.0
// > spread_max 0.60. Whole track rejected, annex stays empty even though
// frames are accepted and gallery-far.
// Two orthogonal identities under one track ID — a track-ID collision.
//
// The banded admission (AR-018) now catches this EARLIER than the spread
// gate did: an embedding unlike everything already on the track falls below
// the band's lower bound and is refused entry, so the buffer never becomes
// two-person in the first place. The spread gate remains as a second line
// for a track that drifts gradually rather than jumping.
//
// The assertion is on the outcome, not the mechanism: whichever gate fires,
// the outsider must not reach the actor's annex.
tg.observe(3, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(3, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(3, one_hot(400), 0, 0.30f, true, kNoCrop); // orthogonal outlier
CHECK(tg.annex().empty());
CHECK(tg.band_rejected() > 0); // refused at the door
for (const auto& e : tg.annex())
CHECK(cosine_similarity(e.emb, one_hot(400)) < 0.5f);
}
TEST_CASE("unconfirmed track (too few accepts) does not promote", "[track_gallery]") {
tests/test_track_registry.cpp
+40
View File
@@ -330,3 +330,43 @@ TEST_CASE("the registry takes a probability, not a cosine", "[registry][AR-024]"
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == -1); // never owned
}
// ── AR-025 — repeated evidence must GROW confidence, not cap it ──────────────
TEST_CASE("confidence grows across frames of the same face", "[registry][AR-025]") {
// Found on a real clip: 318 frame-level identifications across 385 frames
// produced ZERO owned tracks. The truth file named nobody while the matcher
// was accepting on most frames.
//
// Cause: the correlation discount was an annihilator rather than an
// attenuator. Weight = 1 - P(same view), so once a track had one stored
// view every later frame of that same face scored ~0.01 and belief stopped
// moving. A single observation just over the accept threshold is
// logit(0.78) ~ 1.27, under the ownership bar — recognised every frame,
// owned on none.
//
// Correlated evidence should accumulate SLOWER than independent evidence,
// never stop accumulating. Each frame is a Bayesian update.
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
// A face held on screen: the same person, the same pose, frame after frame.
for (int i = 0; i < 50; ++i) {
// The frame scope must close before observe(): it holds the registry
// lock for its lifetime and the mutex is not recursive, so observing
// inside the scope self-deadlocks. In the pipeline these are separate
// nodes, so the ordering falls out naturally — but the API allows the
// mistake, and it hangs rather than failing.
{ auto f = reg.begin_frame(i * 0.2); f.mark_seen(id, i * 0.2, axis(0)); }
reg.observe(id, 5, 0.78f, axis(0));
}
reg.flush(20.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == 5);
// ...but it must still be worth far less than 50 independent looks would be.
CHECK(sink.claims[0].effective_obs < 25.0f);
}