dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dtourolle/scene-actor-extraction:feature/traceability-tooling
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:ci/green-the-runner dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
...
compare: dtourolle/scene-actor-extraction:cd62d6452d48964d0ffb33d5306f984164e7e0b8
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:ci/green-the-runner dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
25 Commits
feature/traceability-tooling ... cd62d6452d
Author SHA1 Message Date
dtourolleandClaude Opus 5 cd62d6452d test: replay the real tracker and registry from committed fixtures 
Tier T2 — composition rather than units. The registry tests construct awkward
states directly; these feed the pieces real 480x360 footage with the cuts, gaps
and crowded frames that synthetic input does not produce.

Six cases:
- fixture integrity: exact frame and face counts, contiguous face_offset, and
  the embedder identity each dump carries (GR-004). The counts are asserted
  exactly rather than approximately, which was impossible before AR-004 — what
  a lossy run dropped depended on timing.
- determinism: replaying a fixture twice gives identical track ids and windows.
  This is the property the whole fixture strategy rests on; without it every
  golden output derived from a fixture is unreliable and the CI replay tier is
  worthless.
- every face is assigned a track, and flush leaves nothing open — a track still
  live at EOF is a window that never reaches the output.
- windows are well-formed and inside the clip. A window ends at the last
  sighting, so it can never extend past the footage that produced it.
- a longer extinction window yields fewer, longer tracks. On the sparse fixture
  (140 faces over 385 frames) that is the difference the constant actually
  makes: absorbing a gap versus splitting a window.
- the cut-heavy fixture still contains cuts. This guards the corpus, not the
  code: a regeneration that produced cut-free fixtures would leave the
  association tests passing while silently testing nothing.

Driving the functors directly rather than through a KPN network is deliberate —
no threads, no channels, no scheduling, so the same input gives the same output.

Suite: 86 cases, 6106 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004, AR-012, AR-013, VR-001, VR-002 | SR-002
 2026-07-31 11:23:17 +02:00
dtourolleandClaude Opus 5 c36885de73 fix: dropped frames fail the run instead of printing a footer 
A drop was reported to stderr and the process exited 0, so a run that discarded
320 frames "succeeded" and produced a truth file that looked complete. The
output in that case is a claim about footage that was never analysed, and
nothing in the file says so.

Now exits 2 and says why. Distinct from 1 (node crash) because the failures are
different: a crash produced no output, a drop produced output that cannot be
trusted.

This is also the regression test for AR-004 that otherwise did not exist. The
backpressure fix is one line in the KPN submodule — easy to lose in an update —
and with data pushes blocking, a drop can no longer occur on the data path. So
any drop now means either that fix regressed or a channel was disabled mid-run,
and both are worth stopping for.

Verified: a clean run still exits 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004 | SR-002
 2026-07-31 10:45:39 +02:00
dtourolleandClaude Opus 5 b5c7d4f6d9 test: committed replay fixtures from the public-domain corpus 
Five HDF5 embedding dumps from bali/ — Road to Bali (1952) — 3.6 MB total,
generated at 5 fps with a 32 px minimum face. CI never calls a model, so
inference happens on a GPU host and CI replays these as data; everything
downstream of embedding is cheap CPU maths.

Public domain is the reason this corpus rather than a convenient one: derived
fixtures can be committed, where anything cut from a copyrighted title could not
live in the repository at all.

The set covers distinct behaviours rather than being five of the same thing:
bali_28 has 9 cuts, so it exercises shot/reverse-shot association (AR-007);
bali_46 is sparse at 140 faces over 385 frames, so it exercises gaps and
extinction (AR-013); bali_13 is the busiest at 4 faces per frame; bali_31 is
short at 29s. All five recorded zero drops.

Both pinned parameters are consequences of measurements, not defaults: 5 fps
because 1 fps over a 77s clip is 77 frames, too thin for an extinction window
measured in tens of seconds; 32 px because that is the VR-005 floor, and the
corpus is 480x360 so a stricter value would reject most of what is there.

make_fixtures.sh regenerates them. Reproducibility is the requirement — a
fixture whose provenance is unknown is worse than none, because it will be
trusted. These are byte-reproducible only because of AR-004: before node
outputs blocked rather than dropped, the same command produced different dumps
run to run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: VR-001 | PR-002
 2026-07-31 10:41:51 +02:00
dtourolleandClaude Opus 5 09a4650fd9 feat: backpressure — the pipeline slows instead of losing frames 
Picks up the KPN fix: node data outputs block on a full channel rather than
dropping. Sentinels stay out-of-band, so EOF can always overtake a stalled data
path and the hold-and-wait deadlock that comment warns about is not reachable.

Verified on a 77s clip at 5 fps, which should yield 385 sampled frames:

  before  65 written, 320 dropped, 29s, two runs differ
  after   385 written, 0 dropped, 17s, two runs byte-identical

The determinism is the part that matters. Golden fixtures were impossible while
what got dropped depended on timing; VR-001 fixture generation is unblocked by
this, and so is the CI replay strategy that depends on it.

Faster rather than slower, which is worth recording because the intuition runs
the other way: a dropped frame has already cost its decode, and the overflow
exception cost more still.

AR-004 is not fully closed. Channel capacity remains a count of items, while a
face carries a 112x112 crop and a 512-float embedding — so a crowded frame
occupies far more memory per slot than a sparse one. Bounding by bytes in
flight is the remaining half, and it matters once max_faces is removed (AR-003).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004, VR-001 | SR-002
 2026-07-31 10:35:29 +02:00
dtourolleandClaude Opus 5 b98372bad8 docs: AR-004 is a KPN change, with the measurement behind it 
Backpressure cannot be implemented in this repository. Every node output in KPN
uses the dropping push() (pool_node.hpp:404 and :710, plus branch, fanout and
interrupt_node). A lossless push_blocking() already exists on both Channel and
OutputPort — "wait for the consumer to drain instead of dropping; the producer
just runs slower" — and nothing calls it. The fix is a per-channel policy or a
network default in KPN, and this pipeline should select lossless: a dropped
frame here does not degrade a result, it silently changes one.

Measured rather than inferred. One 77s clip at 5 fps should yield ~385 sampled
frames. On CPU it produced 49, ending at 51s, with 285 dropped at camera_pos
and 51 at face_aligner. Rebuilt with CUDA the same clip ran in 29s and reached
EOF correctly, and still dropped 320 at camera_pos, yielding 65. Faster
hardware moves where the queue backs up; it does not change what happens when
it does — which is why this is a correctness requirement rather than a
throughput one.

Raising channel capacity is therefore a stopgap: it lowers the probability of
overflow without changing the behaviour on overflow, and the failure it hides
is silent corruption of the output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-003, AR-004, VR-001 | SR-002
 2026-07-31 10:29:53 +02:00
dtourolleandClaude Opus 5 e0f9c95689 docs: minimum face size is 32px, measured (VR-005) 
Replaces the 66px working estimate with the sweep result. 258 probes degraded
to each size and matched against a native-resolution gallery: 16px 46.5% TPI,
24px 93.4%, 32px 98.1%, 40px 99.2%, flat to 112px. The knee is 24-32 and 32
sits within about a point of the ceiling.

The estimate was roughly twice too strict. At 66px a large share of usable
faces would have been discarded, and on 480x360 sources most of them — which
is exactly the resolution of the fixture corpus.

The more useful finding: false identification was 0.0 at every size, including
12px. Small faces fail by becoming unidentified, never by being attributed to
the wrong actor. That asymmetry is what makes a low threshold safe — the cost
of admitting a marginal face is a miss, not a false claim.

Caveat recorded rather than assumed: FPI grows with gallery size, so 258 actors
understates it against a full library. Treat 0.0 as an observation at this
scale, not a property.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-002, VR-005 | SR-002, PR-002
 2026-07-31 10:22:38 +02:00
dtourolleandClaude Opus 5 9e4cdc4efc docs: bali fixture corpus, and AR-004 blocks reproducible fixtures 
Records `bali/` — five ~77s clips of Road to Bali (1952) — as the fixture
source. Public domain, which is the point rather than a convenience: derived
fixtures can be committed, where anything cut from a copyrighted title could
not live in the repository at all.

Makes explicit what the tier table only implied: CI never calls a model. Not a
preference — the embedder measures ~930 ms/frame on the CPU provider, so a 77s
clip at 5 fps is six minutes of inference. Every model invocation happens
locally and CI consumes the result as data, which is what makes the T1/T2 split
load-bearing rather than stylistic.

Two properties of the corpus to design around: 480x360 puts many faces below
the AR-002 66px minimum, so generation must set and record --min-face-px; and
77s at 1 fps is too thin to exercise an extinction window measured in tens of
seconds, so fixtures want 5 fps.

The finding that matters: a trial dump produced 49 frames of an expected ~385,
stopping at 51s of 77s, with 285 frames dropped at camera_pos and 51 at
face_aligner on channel overflow. Channels drop rather than block, and what
drops depends on timing, so the same command twice can yield different dumps.
Golden fixtures cannot be built on that — AR-004 is a prerequisite for VR-001
fixtures, not just a throughput concern for crowd scenes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-004, VR-001 | PR-002
 2026-07-31 10:19:10 +02:00
dtourolleandClaude Opus 5 08941540cb feat: presence windows come from registry claims (schema_version 2) 
The sink no longer reconstructs presence from per-frame detections. A reaped
track already IS a window — [first_seen, last_seen] of a track an actor owned —
so it is pushed straight to the aggregator when it dies and written out as-is.

AR-012 completed end to end. The annealing pass is deleted, not disabled:
anneal_sec existed only to bridge gaps between isolated accepted frames, and a
track that survives its own gaps leaves it nothing to do. The field is REMOVED
from the output rather than zeroed — a field naming a mechanism the pipeline no
longer has is actively misleading to anyone reading a manifest, and would
outlive everyone who remembers why it reads 0.

IR-002 — schema_version 2, matching jRay/SPEC.md JR-002. Windows become objects
carrying `belief` and `route` rather than bare float pairs, so a consumer can
caveat or filter instead of treating every window as equally certain. The new
`extraction` block carries `extinction_sec` (the successor to anneal_sec, and
what a consumer actually needs to interpret a window) and `gallery_scope` —
global vs limited being the strongest single quality signal when two manifests
compete for one cut, since identical gallery_size can mean very different
recall.

AR-016 wired: a pre-write hook flushes the registry with the last timestamp
seen, so tracks still live at EOF are emitted. A film ends with faces on screen
and those tracks have not timed out; without this the closing scene's cast is
silently dropped, which reads as a recognition miss rather than a bookkeeping
bug.

IR-003 stays In Progress deliberately: the sink now writes after the flush, but
the deferred re-identification pass (AR-020) does not exist yet, so output is
still final at EOF rather than after it.

This is a BREAKING format change and part of the coordinated SR-003 bump — it
must ship together with the jRay reader and the server's acceptance of the new
shape, not ahead of them.

Suite: 80 cases, 3250 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-012, AR-016, IR-002, IR-003 | SR-002, SR-003
 2026-07-31 10:10:51 +02:00
dtourolleandClaude Opus 5 fe29d014da feat: identity evidence reaches the registry 
Closes the link that made AR-012 inert: the tracker was maintaining registry
state, but nothing called observe(), so no belief accumulated, no track was ever
owned, and no presence claim could be emitted. Tracking worked and presence did
not.

The matcher now feeds every scored face to the registry as a calibrated
posterior plus its embedding. Deliberately every scored face, not only the ones
clearing prob_threshold: a run of near-misses for one actor is evidence, and
discarding it would leave ownership depending on the per-frame threshold this
redesign exists to stop relying on. The registry discounts for correlation and
decides ownership from the accumulated posterior (AR-025).

The registry is an optional dependency of the matcher. Without one it behaves
exactly as before, which keeps the replay harness and the unit tests working
unchanged rather than forcing every caller to construct a registry it does not
need.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-012, AR-025 | SR-002
 2026-07-31 10:05:59 +02:00
dtourolleandClaude Opus 5 be5f67fa96 feat: tracker on the registry — one pool, calibrated, frame-dependent 
Merges feature/tracker-registry. See e9aea3f for the detail; in summary the
tracker no longer owns track state, association weights embedding over position
whenever position is uninformative, and every similarity comparison is a
calibrated probability.

Four constants retired: track_max_embed_dist, cut_revive_sim,
cut_inactive_max_frames, track_max_frames_missing.

Suite: 80 cases, 3250 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-007, AR-008, AR-024 | SR-002
 2026-07-31 10:01:31 +02:00
dtourolleandClaude Opus 5 e9aea3fc41 feat: tracker owns no state; association is frame-dependent and calibrated 
Three requirements land together because they cannot be separated. The
cross-cut revival branch was the only user of cut_revive_sim, so retiring that
raw cosine forces the pool collapse, and collapsing the pool removes the only
caller of the constant. Splitting them would have produced an intermediate
commit whose only purpose was to be split.

AR-008 — FaceTrackerFunc no longer keeps its own tracks_/inactive_ maps; it
holds a shared_ptr<TrackRegistry> and operates on it directly. Two parallel
copies of track state could disagree, and every divergence would surface as a
wrong presence window with nothing to indicate it. There is now ONE candidate
pool: last_seen alone says whether IoU is meaningful. The park/revive path is
deleted outright — matching a dormant track is ordinary inter-frame
association, and continuity falls out of the embedding comparison the tracker
already did rather than being a mechanism of its own.

AR-007 — track_alpha becomes the base weight for ordinary frames only.
Association drops to embedding-only when position carries no information:
on is_cut or is_scene_boundary, because the viewpoint changed, and for a
dormant track, because time has passed since its box was last valid. The second
case matters as much as the first and had no equivalent before.

AR-024 — association cost is a calibrated probability, never a raw cosine. The
tracker takes the calibration belonging to the active embedder, the same
function object EvidenceDiscounter uses. track_max_embed_dist becomes
track_assoc_min_prob, which means the same thing for every model, gallery and
face size, where a bare cosine threshold did not.

Retired: track_max_embed_dist, cut_revive_sim, cut_inactive_max_frames, and
track_max_frames_missing — the last superseded by the registry's extinction
window. That one is worth naming: a frame count silently changed meaning with
sample_fps, so the same configuration behaved differently at 1 fps and 5 fps.
Extinction is in seconds and lives in one place.

Tests rewritten rather than deleted. The old cases asserted revival by raw
cosine; the same behaviours are now asserted through the registry — a face lost
across a cut and re-associated is the SAME track, one unbroken window, and a
face returning past the extinction window is not. Added the case AR-007 exists
for: two people swap screen positions across a cut while keeping their faces,
and identity must follow the embedding rather than the box.

Suite: 80 cases, 3250 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-007, AR-008, AR-024 | SR-002
 2026-07-31 09:58:27 +02:00
dtourolleandClaude Opus 5 b7c96641a9 docs: no exact tier — the file-hash tier was withdrawn 
A stale reference to 'the runtime/exact tiers' as the fallback for media too
short to carry an audio signature. The exact tier keyed on a file hash and was
withdrawn on legal grounds: it fingerprinted the individual copy a user holds
rather than the cut the timings describe.

The pipeline never emitted a video_hash, so nothing in the code changes — but a
spec that still names a withdrawn tier is what makes the withdrawal look like an
oversight to the next reader, which is exactly how it nearly got re-added.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-31 09:55:50 +02:00
dtourolleandClaude Opus 5 843852e19c feat: registry owns correlation discounting (AR-024, AR-025) 
Moves two responsibilities inside the registry that callers should never have
been trusted with.

AR-025 — per-frame evidence is discounted for correlation by the registry
itself, via EvidenceDiscounter. Log-odds accumulation is only valid for
independent observations, and consecutive frames of one track are anything but:
near-identical pose, lighting and expression. Accumulated naively, thirty frames
of the same face at the same angle drive the posterior to certainty on what is
effectively one measurement.

Each observation is weighted by how much it adds — a view already contributed
counts for ~nothing, a genuinely new pose counts in full. This reuses the
novelty judgement gallery expansion already makes rather than inventing a second
one. The discounter is a separate class the registry holds, so it stays testable
and swappable, but it is a constructor argument rather than an option: there is
no correct way to accumulate without it.

AR-024 — observe() takes a calibrated probability and converts to log-odds
internally. A caller can no longer hand it a raw cosine, which would have been
silently wrong rather than obviously so. Retiring the remaining raw-cosine
constants in the tracker is still open.

DeadTrack now reports effective_obs alongside observations: the raw count and
the evidence that actually counted. A large gap between them is a track the
camera stared at, and worth seeing.

Three tests, one of which is the point: two tracks given the same number of
observations at the same posterior, one repeating a single view and one seeing
eight distinct ones, must not end up equally confident. Without discounting they
would be identical.

Fixed a test that asserted a belief swap on tied evidence. A tie leaves
ownership where it is — a challenger must out-accumulate the incumbent, since
one contrary observation is noise. The original test passed only because it fed
raw log-odds directly.

Suite: 78 cases, 3245 assertions. Coverage 20/63 to 22/63.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-024, AR-025 | SR-002
 2026-07-31 09:15:44 +02:00
dtourolleandClaude Opus 5 f0c7126f80 feat: TrackRegistry — presence follows track extent 
The spine of the redesign. Presence is now the extent of a track an actor owns,
[first_seen, last_seen], rather than the subset of frames in which recognition
happened to succeed. An actor recognised only at the end of a long track is
present for all of it, which is what the scene-scoped ground truth actually
records.

AR-013 — `last_seen` as an optional carries the entire liveness state: unset
means on screen, set means went off at that timestamp and still revivable,
reaped means emitted and erased. No missing-frame counter, no expired flag. It
subsumes the tracker's existing two-pool split, so there is no separate revival
path — matching a dormant track is ordinary inter-frame association.

The asymmetry is the point: interior gaps are claimed, the trailing cool-down is
not. A face lost and re-associated within the timeout never closed its track, so
the gap is presence — someone briefly occluded has not left the scene. But a
track that dies ends at its last sighting, never at the death time. That is
precisely the over-claim the retired extinction_sec keep-alive produced, where
presence ran on into the closing credits.

AR-014 — a belief swap A→B closes the track and opens a successor at the swap
frame. Not a correction: two non-twins both clearing the threshold on one face
is not realistic, whereas a track_id carried across a viewpoint change onto a
different person is. Treating it as a swap-and-continue would emit one window
blending two people; treating it as a boundary yields two that are each right.

AR-015 — two live tracks owned by one actor means at least one is wrong, since a
person cannot be in two places at once. A reverse index catches it on the update
that causes it rather than by scanning. This makes identity a third cut
detector, independent of the histogram and TransNetV2 and firing where those
failed.

AR-016 — flush() closes tracks still live at EOF. Without it a film ending
mid-shot silently drops its closing cast, which presents as a recognition miss
rather than a bookkeeping bug.

Reaping hands the dead track to the aggregator and erases it, so the registry
holds only live tracks and its size is bounded by concurrent on-screen faces
rather than growing with the film.

Locking: a frame's association pass is atomic as a unit via FrameScope, since
per-call locking would let another thread observe a half-updated frame.
owner() reads tally and verdict under one lock — separately, a track could be
both unowned and owned within a single promotion decision. A vote for an
already-reaped track is dropped and counted, because a nonzero count means the
timeout is shorter than the matcher's lag.

11 unit tests, driven directly against the registry with no network and no
fixture — the awkward cases are constructed rather than hunted for. Suite: 75
cases, 3236 assertions. Coverage 14/63 to 20/63.

Not yet wired into FaceTrackerFunc; that is AR-007/AR-008.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-012, AR-013, AR-014, AR-015, AR-016, AR-017 | SR-002
 2026-07-31 09:08:39 +02:00
dtourolleandClaude Opus 5 b35d49c772 docs: tag the implemented core with its requirement IDs 
Adds TRACES tags to code that already satisfies a Done requirement, so coverage
reflects what exists rather than starting from zero:

AR-001 face detection, AR-005 ArcFace alignment, AR-023 calibration fit,
DP-001/DP-002 the single analysis core behind the CLI, IR-001 truth-file
emission, IR-006 the Jellyfin round trip, GR-001/GR-002 gallery build and
incremental merge, VR-001 the embedding dump, VR-002 replay through the real
nodes, VR-003 per-second scoring.

Only Done requirements are tagged. A tag on Planned work would inflate coverage
with fiction that looks plausible — the same failure family as a gate that
cannot fail, and harder to spot.

GR-005 (gallery never leaves the instance) stays untagged deliberately: it is a
prohibition satisfied by the absence of an egress path, so there is no unit that
decides it. Same shape as PR-005 in the system spec, which has no software row
for the same reason. A goal held only by prohibitions cannot be verified by
pointing at code.

Coverage 5/63 to 14/63. The three VR tags are reported as tagged-but-unexecuted
and excluded from the numerator, since their tier cannot run on the CI host —
tagging deliberately cannot raise the number on its own.

Suite still 64 cases, 3199 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-001, AR-005, AR-023, DP-001, DP-002, IR-001, IR-006, GR-001, GR-002, VR-001, VR-002, VR-003
 2026-07-30 21:20:27 +02:00
dtourolleandClaude Opus 5 62396fce75 docs: record why exclude_dirs stays unset, refresh matrix 
The tool's defaults already exclude `vendor`, which covers the submodule at
scripts/vendor/jray-project. Setting the key explicitly is a trap worth
documenting: it REPLACES the defaults rather than extending them, and matching
is on path components rather than prefixes — so ["scripts/vendor"] matches
nothing while silently dropping __pycache__, node_modules, build and the rest.
Verified: the submodule's source is not scanned, and the only vendored path in
the report is the system spec it reads for PR/SR orphan checking.

Coverage after the merges: 5/63, 0 orphans. Every tag names a real requirement,
and nothing claims a requirement that is still Planned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 21:16:15 +02:00
dtourolleandClaude Opus 5 908d166173 feat: bind galleries to the embedder that built them 
GR-004 — a gallery built with one embedding model is meaningless with another.
Cosine similarities across models are garbage but look entirely plausible, so
this fails silently and expensively; every measurement taken against a
mismatched pair would have been quietly wrong.

The stamp is the model basename plus a SHA-256 of its bytes, with embed_dim as
a cheap extra guard. The hash decides and the name explains, because neither
works alone: a name is a promise rather than a fact — models get re-exported in
place under an unchanged filename, which is exactly the case where the weights
differ and nothing else does — while a bare hash mismatch tells an operator
nothing actionable.

Mismatch is fatal in every mode with no bypass. Unstamped only warns, because
unstamped is unknown rather than known-bad, and an error firing on every legacy
gallery trains people to reach for the bypass reflexively. scripts/stamp_gallery.py
binds an existing gallery in place with no re-embedding, so the warning is a
migration step rather than a permanent state; --require-gallery-stamp promotes
it to an error once a site has migrated.

Two gaps found that would have defeated the requirement outright:

- Embedding dumps carried no stamp, so a replay — which has no live embedder —
  had nothing to check the gallery against. Dumps now carry embedder_model and
  embedder_sha256 as root attributes. Additive; schema_version stays 1. This is
  the same gap the dump audit identified independently.
- --merge produced one file holding two embedding spaces, which no later check
  can untangle. Merge paths now verify before writing.

The stamp also survives identity_matcher's calibration write-back, which would
otherwise have stripped it on the first analysis run — the check would have
worked exactly once.

Conflicts resolved additively: both branches appended a source to sae_gallery
and to the test target, and both edited the GR-004 register row.

Merged suite: 64 cases, 3199 assertions, passing on CPU with no GPU.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: GR-004, VR-001 | SR-001
 2026-07-30 19:04:07 +02:00
dtourolleandClaude Opus 5 662a469870 feat: v1 content-derived audio signature 
Implements the server spec §3 construction: a 120 s window centred on the media
midpoint, downmixed and resampled to mono 11025 Hz, a 4096/1024 Hann STFT, 32
log-spaced bands over 300-3000 Hz, one byte per frame carrying a 5-bit peak band
and a 2-bit energy class, base64 with a v1: prefix.

IR-004 — the signature itself, in src/audio_signature.{hpp,cpp}. Decode reuses
the already-linked FFmpeg libraries; libswresample was missing from ffmpeg_libs
and is added. The FFT is written out rather than taken from a library: the
output must be bit-identical against a separate C# implementation, so a
dependency whose version can change the numerics is a liability.

IR-005 — a golden fixture at tests/fixtures/audio/, verified against an
independent Python implementation producing identical bytes. FLAC rather than
WAV because 120 s of 11025 Hz PCM is 2.6 MB and does not compress in git; both
decode to identical samples. The PCM checksum is asserted separately from the
signature so a codec-level divergence is distinguishable from a DSP one.

IR-007 — media under 120 s emits no signature at all, since the centred window
underflows. The rule must be identical in both producers or signatures never
match on exactly the short items most likely to be misidentified.

IR-008 — the v1: prefix is emitted and honoured, so a future change to the DSP
chain is detectable rather than silently non-matching.

Six parameters the spec left undefined had to be pinned to reproduce a byte
stream at all: periodic Hann, band value as the mean of linear magnitudes, ties
to the lowest band, the energy-class definition and its thresholds, byte layout,
and the base64 alphabet. These are now normative in the server spec — left only
in a C++ header, the C# side would have guessed and diverged.

Not yet emitted into the truth file; that is the coordinated schema_version bump
under IR-002/IR-003.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: IR-004, IR-005, IR-007, IR-008 | SR-003
 2026-07-30 19:00:08 +02:00
dtourolle 28e3bd9496 docs: generated traceability matrix 
Committed rather than ignored, matching house precedent: coverage becomes
visible to anyone browsing the repo, and its movement over time is real history
worth having in the log.
 2026-07-30 18:59:06 +02:00
dtourolleandClaude Opus 5 d9aaf8fa4e build: consume the shared traceability tooling via submodule 
jray-project is added at scripts/vendor/jray-project and the extractor is used
from there. Only two files are repo-local: traceability.toml, which carries
everything repo-specific, and the CI workflow that invokes the vendored gate.

The extractor is deliberately NOT copied in. One implementation, parameterised
by config — a second copy would drift from the first, and the tool already
proves it works unchanged against all three registers.

Enables system_spec so PR/SR orphan checking runs: previously uncheckable,
because the system spec lived outside every component's checkout.

Gate is green at 0.0% of 63 requirements, which is correct — nothing is tagged
yet. Eleven are flagged unverifiable on this CI host and excluded from the
numerator rather than counted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 18:58:40 +02:00
dtourolleandClaude Opus 5 a2ebdc4cdd docs: GR-004 done — gallery/embedder binding 
Stamp is model basename + SHA-256 + embed_dim: the hash decides, the name
explains. A name alone is a promise rather than a fact — models get re-exported
in place under an unchanged filename, which is exactly the case where weights
differ and nothing else does. A hash alone is unactionable in an error message.

Mismatch is fatal in every mode with no bypass. Unstamped only warns, because
unstamped is unknown rather than known-bad, and an error that fires on every
legacy gallery trains people to reach for the bypass. A migration script binds
existing galleries in place with no re-embedding, so warn is not permanent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: GR-004 | SR-001
 2026-07-30 18:37:07 +02:00
Claude 7db40f430d GR-004: bind galleries to the embedder that built them 
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.

The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.

Mismatch is a hard error in every mode, with no bypass, naming both sides.

Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.

Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.

Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.

Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.

Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 18:35:46 +02:00
dtourolleandClaude Opus 5 020306c94f docs: builder images and per-backend release binaries (DP-008) 
Adds the build/deploy story that was missing: containerised builder images for
cpu / cuda / rocm, and release jobs producing prebuilt binaries so a first
install need not compile.

States explicitly that this does not reverse DP-005. That requirement rejects
Docker as a *runtime* — GPU passthrough is fragile and exists only because of
the container. Using it as a *build* environment is the opposite case, and lets
one machine produce binaries for backends it cannot itself run. Build in a
container, run natively.

Two things deliberately cannot ship, and the installer must not imply otherwise:
TensorRT engines are GPU-architecture and TRT-version specific, so
build_trt_engines.sh still runs on the target; and models are ~725 MB in LFS,
orthogonal to the binary.

The base image is chosen by the OLDEST glibc to be supported, not by
convenience — a binary built in a container runs against the host's glibc, and
getting this wrong fails at load with GLIBC_2.xx not found. Accelerator runtimes
have the same shape of problem, so each image documents its compatible
CUDA/ROCm range and the installer checks it rather than discovering a mismatch
at first inference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: DP-005, DP-007, DP-008 | PR-004
 2026-07-30 18:35:10 +02:00
dtourolleandClaude Opus 5 2919ed68d1 docs: audit findings, verification tiers, CI image, artifact storage 
Corrections from the dump audit and the completed agent work:

- AR-010 is not started, not in progress: is_scene_boundary has no producer
  anywhere. SceneDetectorFunc is a terminal sink writing scenes.json and never
  annotates the frame, so the field is permanently false and the dump column a
  constant 0. A replay test of the frame-dependent track_alpha would pass
  vacuously — the worst failure mode for a verification gate.
- T1 (functor-level) becomes the primary verification tier, not T2. KPN node
  functors are plain callables constructed outside the network, so a node is
  tested by calling operator() with hand-built inputs. That removes four
  hazards at once: fixture provenance, replay-from-frame-0, cross-test state
  leakage, and replay-harness nondeterminism. It also means a dead upstream
  producer no longer blocks testing its consumer.
- VR-010 (dump provenance) and VR-011 (replay harness rewrite) added. A dump
  made with LVFace is currently byte-indistinguishable from one made with
  w600k-R50 — the GR-004 problem again, in the dump.
- Four requirements had no verification tier at all; the traceability gate
  found them.
- DP-007: CI builder image, CPU-only, pinned by tag in the Gitea container
  registry. Corpus fixtures go to the package registry rather than LFS: LFS is
  pulled on clone and would tax every developer for data only CI reads.
- IR-004/005/007/008 marked done; the v1 DSP parameters they had to pin are
  now normative in the server spec.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-010, DP-007, IR-004, IR-005, IR-007, IR-008, VR-001, VR-010, VR-011 | SR-002, SR-003
 2026-07-30 18:33:32 +02:00
dtourolleandClaude Opus 5 45ef7c1916 Add the v1 audio signature to the pipeline (IR-004, IR-005, IR-007, IR-008) 
Implements the content-derived spectral-peak signature from
JRay-public-server/SPEC.md §3 so a truth file is self-identifying: 120 s
window centred on the media midpoint, mono at 11025 Hz, 4096/1024 Hann
STFT, 32 log-spaced bins over 300-3000 Hz, one byte per frame (5-bit peak
band + 2-bit energy class), base64, `v1:` prefix.

Audio decode is a second stream from the FFmpeg libraries the pipeline
already links for video; libswresample is added to the existing
ffmpeg_libs interface target. The FFT is written out rather than pulled
from a library for the same reason the plugin vendors one: the output has
to be bit-identical across two languages, so a dependency whose version
could change the numerics is a liability.

The server spec fixes the geometry but not enough to reproduce a byte
stream — Hann periodicity, band aggregation, the energy-class definition,
tie-breaking and the base64 alphabet are all unconstrained by it. Those
are pinned in audio_signature.hpp and mirrored in the golden fixture, so
the plugin can be implemented from the fixture alone.

IR-005: tests/fixtures/audio/ carries a deterministic 120 s tone (FLAC —
lossless, so identical PCM to the WAV make_fixture.py emits, and 3.5x
smaller in git) plus the signature it must produce, the decoded-PCM
checksum and the full parameter contract. That directory is the artefact
shared with the plugin repo; the PCM checksum is separate from the
signature so a codec-level difference is distinguishable from a DSP one.

IR-007: media under 120 s emits no signature. Same for a file with no
audio stream or one that will not open — UR-9 is an enhancement and must
never be able to break a fetch.

Verified against an independent Python reference implementation: same
bytes. All 32 bands and all 4 energy classes appear in the golden vector,
and the window-centring test wraps the fixture in 90 s of silence either
side and requires the golden value back.

Not wired into the truth-file output yet — that is the schema_version
bump under IR-002/IR-003 and is deliberately out of scope here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-30 18:25:37 +02:00
62 changed files with 5384 additions and 370 deletions
Expand all files Collapse all files
.gitea/workflows/traceability-check.yml
+145
View File
@@ -0,0 +1,145 @@
name: Traceability Validation
# Mirrors JellyTau's .gitea/workflows/traceability-check.yml. The extractor is
# stdlib Python, so there is no toolchain install step and no jq.
#
# This workflow is component-agnostic: every repo-specific setting - which ID
# prefixes count, which file suffixes are source, which directories to scan,
# the threshold - lives in traceability.toml at the repo root, and the same
# extractor is shared by all three JRay components. Copying this file into
# another component needs no edits.
#
# NOTE: the runner here is an Intel N100 with no discrete GPU. This job is only
# ever static analysis of source comments plus markdown parsing, so it is cheap;
# the requirements it reports as "tagged but unexecuted" are the ones that need
# a GPU host, and they are deliberately never counted as covered.
on:
push:
branches:
- main
- master
- develop
pull_request:
branches:
- main
- master
- develop
jobs:
validate-traces:
runs-on: linux/amd64
name: Check requirement traces
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive
- name: Check Python is available
run: |
set -e
command -v python3 >/dev/null 2>&1 || {
echo "python3 is missing from the runner image."
echo "The traceability tooling is stdlib-only Python;"
echo "3.9+ with CLI flags, 3.11+ to read traceability.toml."
exit 1
}
python3 --version
# The gate's own arithmetic is the thing being trusted, so its tests run
# before it does. JellyTau's gate was believed for months while it was
# dividing by frozen literals; untested gate logic is how that happens.
- name: Test the extractor
run: python3 scripts/vendor/jray-project/scripts/traceability/test_extract_traces.py
# Threshold policy and every other repo-specific setting live in
# traceability.toml, not here, so local runs and CI runs cannot disagree
# about what "passing" means. Denominators come from docs/requirements.md
# at run time and are never hardcoded -- in this file or anywhere else.
#
# A misconfigured run (zero requirements parsed, zero files scanned) is a
# hard failure rather than a plausible-looking 0%.
- name: Traceability gate
run: sh scripts/vendor/jray-project/scripts/traceability/traceability-gate.sh
- name: Check modified files for traces
if: github.event_name == 'pull_request'
run: |
set -e
echo "Checking modified sources for TRACES tags..."
# The extensions come from the report the gate just wrote, which got
# them from traceability.toml. Restating them here would be a second
# place for the source-file definition to live, and the two would
# drift the first time a language is added.
PATTERN=$(python3 -c "
import json, re, sys
suffixes = json.load(open('traces-report.json'))['config']['sourceSuffixes']
print('(' + '|'.join(re.escape(s) + '\$' for s in suffixes) + ')')
")
echo "Source suffixes from traceability.toml: $PATTERN"
CHANGED=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD" \
| grep -E "$PATTERN" || true)
if [ -z "$CHANGED" ]; then
echo "No source files changed."
exit 0
fi
echo "Changed files:"
echo "$CHANGED" | sed 's/^/ /'
echo ""
# Advisory by design: not every file implements a requirement, and a
# tag on every function is noise that rots faster than it helps
# (CLAUDE.md: tag the unit that decides). This step exists to prompt,
# not to block. The blocking checks are in the gate step above.
#
# Piped into the loop rather than a here-string, and `case` rather
# than `[[ == ]]`, so this works under dash as well as bash. The loop
# body runs in a subshell, so misses are recorded in a file.
MISSING=$(mktemp)
echo "$CHANGED" | while IFS= read -r file; do
case "$file" in
*/test_*.py|*_test.py|*Tests.cs|tests/*|*/tests/*) continue ;;
esac
[ -f "$file" ] || continue
if ! grep -q 'TRACES:' "$file"; then
echo " no TRACES tag: $file"
echo "$file" >> "$MISSING"
fi
done
COUNT=$(wc -l < "$MISSING" | tr -d ' ')
rm -f "$MISSING"
if [ "$COUNT" -gt 0 ]; then
echo ""
echo "$COUNT changed file(s) carry no requirement tag."
echo "Format: // TRACES: AR-012, AR-013 | SR-002"
echo " (pipe separates requirement types, comma separates IDs)"
echo "A deliberate invariant exception is tagged separately:"
echo " // EXCEPTION: AR-024 <reason>"
echo "See CLAUDE.md and SPEC.md section 6."
fi
- name: Report summary
if: always()
run: |
echo "Traceability matrix: docs/traceability.md"
echo ""
head -40 docs/traceability.md || true
- name: Save reports
if: always()
uses: actions/upload-artifact@v3
with:
name: traceability-reports
path: |
traces-report.json
docs/traceability.md
retention-days: 30
.gitignore
+4
View File
@@ -19,6 +19,10 @@ compile_commands.json
# coverage). Regenerate with scripts/docs/run_holdout_all_models.py and
# scripts/docs/gallery_coverage_per_film.py.
!docs_data/*.json
# Exception: test fixtures are inputs, not build output. The audio golden
# vector (IR-005) is shared verbatim with the jRay plugin repo, so it has to be
# tracked. Regenerate the media with tests/fixtures/audio/make_fixture.py.
!tests/fixtures/**
# Video files
*.mp4
*.mkv
.gitmodules
+3
View File
@@ -2,3 +2,6 @@
path = external/KPN
url = https://gitea.tourolle.paris/dtourolle/KPN.git
branch = master
[submodule "jray-project"]
path = scripts/vendor/jray-project
url = git@gitea.tourolle.paris:dtourolle/jray-project.git
CMakeLists.txt
+18 -8
View File
@@ -196,23 +196,31 @@ endif()
# FFmpeg (hwaccel video decode: CUDA/VAAPI, runtime-detected + swscale colour
# conversion). Hwaccel support is built into libavcodec/libavutil; no extra
# libraries are needed here.
# libswresample is the audio side of the same dependency — downmix + resample
# for the audio signature (IR-004, src/audio_signature.cpp). Not a new project
# dependency: it ships with the libav* set already required above.
find_package(PkgConfig REQUIRED)
pkg_check_modules(AVFORMAT REQUIRED libavformat)
pkg_check_modules(AVCODEC REQUIRED libavcodec)
pkg_check_modules(AVUTIL REQUIRED libavutil)
pkg_check_modules(SWSCALE REQUIRED libswscale)
pkg_check_modules(AVFORMAT REQUIRED libavformat)
pkg_check_modules(AVCODEC REQUIRED libavcodec)
pkg_check_modules(AVUTIL REQUIRED libavutil)
pkg_check_modules(SWSCALE REQUIRED libswscale)
pkg_check_modules(SWRESAMPLE REQUIRED libswresample)
add_library(ffmpeg_libs INTERFACE)
target_compile_options(ffmpeg_libs INTERFACE
${AVFORMAT_CFLAGS_OTHER} ${AVCODEC_CFLAGS_OTHER}
${AVUTIL_CFLAGS_OTHER} ${SWSCALE_CFLAGS_OTHER})
${AVUTIL_CFLAGS_OTHER} ${SWSCALE_CFLAGS_OTHER}
${SWRESAMPLE_CFLAGS_OTHER})
target_include_directories(ffmpeg_libs INTERFACE
${AVFORMAT_INCLUDE_DIRS} ${AVCODEC_INCLUDE_DIRS}
${AVUTIL_INCLUDE_DIRS} ${SWSCALE_INCLUDE_DIRS})
${AVUTIL_INCLUDE_DIRS} ${SWSCALE_INCLUDE_DIRS}
${SWRESAMPLE_INCLUDE_DIRS})
target_link_libraries(ffmpeg_libs INTERFACE
${AVFORMAT_LIBRARIES} ${AVCODEC_LIBRARIES}
${AVUTIL_LIBRARIES} ${SWSCALE_LIBRARIES})
message(STATUS "FFmpeg: avformat=${AVFORMAT_VERSION} avcodec=${AVCODEC_VERSION}")
${AVUTIL_LIBRARIES} ${SWSCALE_LIBRARIES}
${SWRESAMPLE_LIBRARIES})
message(STATUS "FFmpeg: avformat=${AVFORMAT_VERSION} avcodec=${AVCODEC_VERSION} "
"swresample=${SWRESAMPLE_VERSION}")
# nlohmann/json (gallery + output serialisation)
include(FetchContent)
@@ -249,6 +257,8 @@ find_package(HDF5 REQUIRED COMPONENTS CXX)
add_library(sae_gallery STATIC
src/gallery/gallery_store.cpp
src/gallery/gallery_builder.cpp
src/audio_signature.cpp # IR-004 — content-derived audio signature
src/gallery/embedder_stamp.cpp # GR-004 — gallery/embedder binding
)
set_target_properties(sae_gallery PROPERTIES POSITION_INDEPENDENT_CODE ON)
target_include_directories(sae_gallery PUBLIC src ${HDF5_INCLUDE_DIRS})
docs/SPEC.md
+239 -9
View File
@@ -80,8 +80,49 @@ by dropping work or growing without limit.
- Memory is the real limit: faces carry 112×112 crops plus 512-float embeddings.
Backpressure must engage on bytes in flight, not just item counts.
**Gap:** entire requirement. This is a prerequisite for removing `max_faces`, not
a follow-up to it.
### The fix is not in this repo
**Every node output in KPN uses the dropping `push()`** (`pool_node.hpp:404`,
`:710`; also `branch.hpp`, `fanout.hpp`, `interrupt_node.hpp`). A lossless
`push_blocking()` — "wait for the consumer to drain instead of dropping; the
producer just runs slower" — already exists on both `Channel`
(`channel.hpp:144`) and `OutputPort` (`variant_node.hpp:81`), **and nothing
calls it.**
So AR-004 is a change to the KPN repository, not to this one. It needs either a
per-channel lossless policy or a network-wide default, and this pipeline should
select lossless: a dropped frame here does not degrade a result, it silently
changes one.
**Measured, not inferred.** One 77 s clip at 5 fps should yield ~385 sampled
frames. On CPU it produced 49, ending at 51 s, with 285 frames dropped at
`camera_pos` and 51 at `face_aligner`. Rebuilt with CUDA the same clip ran in
29 s and reached EOF correctly — and still dropped **320** frames at
`camera_pos`, yielding 65. Faster hardware moves where the queue backs up; it
does not change what happens when it does.
Two consequences worth stating:
- **Raising channel capacity is a stopgap, not a fix.** It lowers the
probability of overflow without changing the behaviour on overflow, and the
failure it hides is silent corruption of the output.
- **Fixture generation is blocked on this** (VR-001), because what gets dropped
depends on timing. The same command run twice can produce different dumps, and
a golden fixture cannot be built on that.
**Current:** fixed in KPN — node data outputs use `push_blocking`, sentinels
remain out-of-band so EOF can always overtake a stalled data path. Verified on
the same clip: 385 of 385 sampled frames written, zero drops, and two
consecutive runs byte-identical where previously they were not.
It also ran *faster* (29 s → 17 s). A dropped frame has already cost its decode,
and the overflow exception cost more — so the lossy path was paying for work it
then discarded.
**Gap:** the remaining half — bounding by **bytes in flight** rather than item
count. Channel capacity is still a count of items, and a face carries a 112×112
crop plus a 512-float embedding, so a crowded frame occupies far more memory per
slot than a sparse one. That matters once `max_faces` is removed (AR-003).
## AR-005 — Face alignment and crop
@@ -154,6 +195,26 @@ Two distinct signals, deliberately kept separate:
- **`is_scene_boundary`** — opt-in (`--scene-detect`). TransNetV2 over a densely
decoded, downscaled stream flags a *true shot/scene boundary*.
> **`is_scene_boundary` currently has no producer.** `grep -rn is_scene_boundary
> src/` finds no assignment anywhere: `SceneDetectorFunc` is a *terminal sink*
> (`main.cpp:298-300`, `kpn::out<>`) that writes `scenes.json` and never
> annotates the `Frame` flowing to the face pipeline. The field is therefore
> always `false`, and the dump column (`embedding_dump_node.hpp:38`) is a
> constant 0. Compounding it, `main.cpp:280` returns from the
> `--dump-embeddings` branch *before* the `scene_detect` branch at `:296`, so no
> dump-producing path even instantiates the detector.
>
> This makes AR-010 **not implemented**, not "in progress" — and it means a T2
> test of the frame-dependent `track_alpha` (AR-007) would **pass vacuously**,
> which is the worst possible failure for a verification gate. The fix is in the
> producer, not the schema: make `SceneDetectorFunc` a pass-through (or add a
> boundary annotator before the decimator) and add the scene branch to
> `dump_embeddings.cpp`. **No `schema_version` bump** — the column exists and
> merely stops being constant.
>
> Fixtures generated before the fix must be marked in provenance, since `0` is
> presently indistinguishable from "no boundary here".
Both feed AR-007 as **association hints**: they tell the tracker that spatial
continuity is broken and that association should weight embedding over IoU.
Neither ends a presence window (AR-012).
@@ -757,9 +818,16 @@ prerequisite.
## DP-005 — Installation and provisioning
- Native install, **no Docker** — GPU passthrough is the most fragile part of a
containerised setup and exists only because of the container. Natively the GPU
works with the host drivers and media paths need no re-mounting.
- Native install, **no Docker at runtime** — GPU passthrough is the most fragile
part of a containerised setup and exists only because of the container.
Natively the GPU works with the host drivers and media paths need no
re-mounting. This constrains how the software *runs*, not how it is *built*:
DP-008 uses containers as build environments precisely because that side has
none of these problems.
- The installer may **fetch a prebuilt binary** (DP-008) instead of compiling.
Compiling stays supported, but should not be the only path — it is the slowest
and most fragile step of a first install. TRT engines are still built locally
either way (DP-008).
- An installer (`scripts/build_install.py`) consuming one `install.yaml`:
platform (nvidia/amd/cpu), embedder model, gallery scan cadence, install
prefix; runtime secrets written to a `.env`, editable without recompiling.
@@ -770,6 +838,117 @@ prerequisite.
**Gap:** installer unbuilt.
## DP-007 — CI build image
CI runs on an Intel N100 with no discrete GPU, so the test build must configure
**CPU-only** and must not require CUDA, TensorRT or ROCm:
```
-DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=CPU
```
A prebuilt container image supplies the toolchain, published to the **Gitea
container registry** and pinned by tag — matching the `jellytau-builder`
precedent. Building dependencies per CI run is untenable on an N100, and OpenCV 5
from source would dominate every run.
The same registry stores corpus dump fixtures as generic packages (see the
fixtures table in `requirements.md`). Rebuild the image when its dependency set
changes, not per run, and pin CI to a tag rather than `latest` so a rebuild
cannot silently change what a green build meant.
**Required in the image:**
| Dependency | Why |
|---|---|
| CMake, C++ toolchain, pkg-config | Build |
| **OpenCV 5** | `CMakeLists.txt:25` prefers 5, falls back to 4. The branch targets 5, so the image should carry it — it is not yet in most distro repos and building it per-run is prohibitive |
| HDF5 (C++) | Galleries are HDF5-native; also the dump format |
| FFmpeg dev libs — `libavformat`, `libavcodec`, `libavutil`, `libswscale`, **`libswresample`** | Decode. See the note below on swresample |
| Python 3 + numpy, h5py, scipy | Python-side tests, replay, traceability tooling |
| Catch2, nlohmann/json | **Vendored into the image, not fetched.** Both are `FetchContent`-ed today (`CMakeLists.txt:220`, `tests/CMakeLists.txt:8`), which makes every CI run depend on GitHub reachability |
**Deliberately excluded:** CUDA, TensorRT, ROCm — no GPU to use them. Also the
ONNX Runtime *GPU* providers; only the CPU provider is relevant, and only for T3
smoke tests.
**Models are not baked into the image.** The seven ONNX files total ~725 MB and
live in Git LFS. T1/T2 tests are model-free by design
(`tests/CMakeLists.txt:1-4`), so the default image needs none. T3 smoke tests
require a model and should pull it via LFS in a separate job rather than
inflating the image tenfold for a minority of tests.
**`libswresample` is a real gap, not a formality.** The current
`pkg_check_modules` list (`CMakeLists.txt:200-203`) covers avformat, avcodec,
avutil and swscale but **not** swresample — which IR-004 needs to downmix to mono
and resample to 11025 Hz. It must be added alongside the audio-signature work.
**Gap:** entire requirement. The image does not exist, and no CI config is
present in this repo.
## DP-008 — Builder images and release binaries
Produce prebuilt binaries per backend so deployment does not require every user
to compile the project.
**This does not contradict DP-005.** That requirement rejects Docker as a
*runtime* — GPU passthrough is the most fragile part of a containerised setup and
exists only because of the container. Using Docker as a *build* environment is
the opposite case: hermetic, reproducible, and it lets one machine produce
binaries for backends it cannot itself run. Build in a container; run natively.
### Image matrix
The build has two independent axes (`CMakeLists.txt:48-49`), so the useful
combinations are:
| Image | `SAE_INFERENCE_BACKEND` | `SAE_GEMM_BACKEND` | Target |
|---|---|---|---|
| `sae-builder-cpu` | ORT | CPU | CI (DP-007), and the smoke-test fallback |
| `sae-builder-cuda` | TRT | CUDA | NVIDIA |
| `sae-builder-rocm` | ORT | ROCM | AMD |
All three carry the DP-007 dependency set (OpenCV 5, HDF5, FFmpeg incl.
swresample, vendored Catch2/nlohmann) and differ only in the accelerator stack.
The CPU image is the CI image — one artifact, two uses.
Published to the Gitea container registry, pinned by tag, rebuilt when the
dependency set changes rather than per run.
### What ships, and what cannot
**Ships:** the `scene_analyze` binary and its companions, per backend.
**Cannot ship: TensorRT engines.** `.engine` files are specific to the GPU
architecture and TRT version they were built on — `scripts/build_trt_engines.sh`
must still run on the target machine. A prebuilt binary shortens the install; it
does not remove the local engine-build step, and the installer must not imply
otherwise.
**Cannot ship: models.** ~725 MB in LFS, and orthogonal to the binary.
### The constraint that decides the base image
**A binary built in a container runs against the host's glibc.** Build on a
newer base than the oldest supported host and it fails at load with
`GLIBC_2.xx not found` — the classic and entirely avoidable trap when shipping
binaries out of containers.
So the base is chosen for the *oldest* glibc to be supported, not for
convenience or recency. Accelerator libraries have the same shape of problem:
the binary links against a driver-provided runtime, so each image must document
the CUDA/ROCm version range its output is compatible with, and the installer
must check it rather than discovering a mismatch at first inference.
### Jobs
A release job per backend, producing a tagged artifact in the registry. These are
**not** the CI gate — the gate runs the CPU image on every push (DP-007);
release builds run on tag. Their outputs are what DP-005's installer fetches
when the user does not want to compile.
**Gap:** entire requirement. No images, no release jobs.
## DP-006 — Gallery maintenance as a background concern
- Incremental gallery refresh runs on a timer (`gallery_scan_interval`, default
@@ -859,8 +1038,10 @@ rewritten.
**Media shorter than 120 s.** The window `runtime/2 ± 60 s` underflows, so no
signature is emitted and **no sync offset is applied**. Such items fall back to
the runtime/exact tiers, which is adequate: a 90-second extra or trailer is not
the content whose cut alignment matters. Both producers must apply the identical
the runtime tier, which is adequate: a 90-second extra or trailer is not the
content whose cut alignment matters. (There is no `exact` tier: the file-hash
tier was withdrawn on legal grounds — it fingerprinted an individual copy rather
than the cut the timings describe. See the server spec §3.) Both producers must apply the identical
rule, or they diverge on exactly the short items most likely to be
mis-identified.
@@ -945,8 +1126,57 @@ surfaced as a build report.
different models are meaningless but *look* plausible — this fails silently and
expensively otherwise.
**Gap:** named as step 4 of the `service-conversion.md` implementation plan;
unbuilt. This is the highest-value small fix in the document.
### The stamp
Two fields, written together: the model file's **basename** and the **SHA-256 of
its bytes** (plus `embed_dim` as a cheap extra guard). Stored as the `/embedder`
group in the gallery HDF5, and as an optional top-level `"embedder"` object in
the legacy JSON format.
The hash *decides*; the name is what a human *reads*. Neither alone is enough. A
name is a promise rather than a fact — models get re-exported, re-quantised and
overwritten in place under an unchanged filename, which is exactly the case where
the weights differ and nothing else does, so a name-only stamp is blind to the
failure it exists to catch. A hash alone is correct but unactionable: *"expected
3f2a…, got 9c1b…"* tells an operator nothing about what to do next. SHA-256 over
the file is derived from the artefact rather than asserted about it, needs no
registry kept up to date, and costs ~0.1 s for a 250 MB ONNX once per process.
### Verdicts
| Verdict | When | Default | Under strict mode |
|---|---|---|---|
| `match` | hashes agree | proceed | proceed |
| `weak_match` | names agree, one side unhashable | **warn** | **error** |
| `unstamped` | gallery predates GR-004 | **warn** | **error** |
| `unknown_embedder` | gallery stamped, embedder unidentifiable | **warn** | **error** |
| `mismatch` | proven different models | **error** | **error** |
**A mismatch is fatal in every mode, with no bypass**, and the message names both
sides — what the gallery was built with and what is loaded.
The three "cannot prove it" verdicts warn loudly instead, because they describe an
*unknown* state rather than a *known-bad* one, and because every gallery built
before this requirement is unstamped. Hard-failing all of them would make the
check something people route around rather than trust. Strict mode
(`--require-gallery-stamp`, or `SAE_REQUIRE_GALLERY_STAMP=1`, which propagates to
subprocesses) promotes them to errors — that is the mode measurement work runs in.
`scripts/stamp_gallery.py` re-binds an existing gallery without re-embedding, so
migration costs one command; that is what makes "warn" a temporary state rather
than a permanent one.
### Scope of the check
Embedding **dumps** carry the same stamp (`embedder_model` / `embedder_sha256`
root attributes, `scripts/optimizer/SCHEMA.md`): a replay has no live embedder, so
the dump *is* the embedder as far as the gallery is concerned. Derived galleries
(filter, cast-restrict) inherit their source's stamp; `--merge` and the JSON
gallery merge check *before* writing, since a merged file holding two embedding
spaces cannot be untangled afterwards by any later check.
**Gap:** none. Stamped in `gallery_builder.cpp` and the Python builders; verified
in `scene_analyze`, `scene_preview`, the `sae_kpn` matcher binding, `replay.py`,
`optimize.py`, `movienet_eval.py` and the merge paths.
## GR-006 … GR-009 — Provenance tiers and poisoning guard
docs/plan.md
+15 -5
View File
@@ -260,13 +260,23 @@ Context crops opt-in behind `--dump-unidentified-crops`.
# Gallery
## GR-004 — Model binding
## GR-004 — Model binding — **DONE**
**Depends on:** nothing. **Startable immediately, highest value per line.**
**Depended on:** nothing. Landed before any measurement work, as intended.
Stamp embedder identity into the gallery at build; verify at load in
`scene_analyze`, `replay.py` and the optimizer. Mismatch is a hard error naming
both sides.
Stamp = model basename + SHA-256 of the ONNX, written as the `/embedder` group at
build time (`gallery_builder.cpp`, `sae_gallery.save_gallery_hdf5`) and verified
at load in `scene_analyze`, `scene_preview`, the `sae_kpn` matcher binding,
`replay.py`, `optimize.py` and `movienet_eval.py`. Mismatch is a hard error naming
both sides, with no bypass. Embedding dumps carry the same stamp, since a replay
has no live embedder to check against.
Unstamped legacy galleries **warn loudly and proceed** rather than failing:
unknown is not known-bad, and hard-failing every pre-existing gallery would turn
the check into something people disable. `--require-gallery-stamp` /
`SAE_REQUIRE_GALLERY_STAMP=1` promotes that to a hard error — measurement runs
should set it. `scripts/stamp_gallery.py` re-binds an existing gallery without
re-embedding, so the warning state is cheap to leave.
Cross-model similarities are meaningless but *look* plausible — this fails
silently and expensively, and it would corrupt every measurement taken during the
docs/requirements.md
+145 -43
View File
@@ -29,30 +29,30 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
| AR-002 | Minimum face size 66×66 px, expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | Planned |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | Planned |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done |
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | In Progress |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | Planned |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted |
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | In Progress |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Not started** — `is_scene_boundary` has no producer; `SceneDetectorFunc` is a terminal sink and never annotates the frame |
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | Planned |
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | Planned |
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | Planned |
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | Planned |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | Planned |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | Planned |
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | **Done** — `src/track_registry.hpp`; window is `[first_seen, last_seen]` of an owned track |
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | **Done** — `last_seen` optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed |
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | **Done** — swap closes at `last_seen` and opens a successor at the swap frame; counted |
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | **Done** — reverse index detects it on the causing update; counted |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | **Done** — `flush()`, idempotent, closes at last sighting or final tick |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief and observation count |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | Planned |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | In Progress |
| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned |
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | Planned |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | Planned |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association and accumulation both in probability space; `track_max_embed_dist`, `cut_revive_sim` retired |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
@@ -66,18 +66,20 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| DP-004 | Opportunistic/idle mode: external trigger, hard stop, implicit re-queue | PR-004 | Medium | Planned |
| DP-005 | Native installer, no Docker; Fedora + Arch | PR-004 | Medium | Planned |
| DP-006 | Background incremental gallery refresh on a timer | PR-003 | Medium | Planned |
| DP-007 | CI builder image, CPU-only, pinned by tag in the Gitea container registry | PR-004 | High | Planned |
| DP-008 | Builder images + release jobs per backend (cpu / cuda / rocm); ship binaries, not engines | PR-004 | Medium | Planned |
## Integration (IR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done |
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | Planned |
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | Planned |
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | Planned |
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | Planned |
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | Planned |
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | Planned |
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | **Done** — `schema_version: 2`; windows are objects with `belief` + `route`; `extraction.*` carries `extinction_sec` and `gallery_scope`; `anneal_sec` removed |
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | **In Progress** — sink builds windows from registry claims and flushes at EOF; the deferred pass (AR-020) does not exist yet, so output is still final at EOF |
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | **Done** — `src/audio_signature.*`; not yet emitted into the truth file (IR-002) |
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | **Done** — `tests/fixtures/audio/`; v1 parameters now normative in server spec §3 |
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | **Done** |
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | **Done** |
| IR-006 | Jellyfin round-trip: pull pending queue, push complete results only | SR-001 | High | Done |
## Gallery (GR)
@@ -87,7 +89,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | Planned |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | **Done** — basename + SHA-256 + `embed_dim`; mismatch fatal with no bypass, unstamped warns unless `--require-gallery-stamp`; `scripts/stamp_gallery.py` migrates in place |
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
| GR-007 | Persist harvested embeddings **flagged and reviewable**, never silently equal to baked | SR-005 | Medium | Planned |
@@ -99,14 +101,16 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| VR-001 | HDF5 post-inference dump at the embedded-frame boundary | PR-002 | High | Done |
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | Done |
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | **Done** — replay driven from committed fixtures in `tests/test_replay_fixtures.cpp`; determinism asserted |
| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done |
| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | Planned |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | **Done** — knee at 24–32 px; 32 px gives 98.1% TPI, 0.0 FPI at every size |
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned |
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned |
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned |
---
@@ -120,21 +124,62 @@ Four tiers, in decreasing order of preference:
| Tier | Runs in CI | What it covers |
|---|---|---|
| **T1 — CPU unit** | Yes | Pure logic: registry state machine, belief accumulation, clustering, band admission, calibration maths |
| **T2 — Replay** | Yes | Real pipeline nodes driven from an HDF5 fixture — no GPU, no video |
| **T1 — Functor unit** | Yes | A KPN node's `operator()` driven directly with hand-built inputs |
| **T2 — Replay** | Yes | The composed pipeline driven from an HDF5 fixture — no GPU, no video |
| **T3 — CPU inference** | Yes, slowly | ORT CPU provider over a handful of frames; smoke tests only |
| **T4 — GPU** | **No** | Throughput, TRT engines, large-gallery GEMM |
**T2 is the reason this is workable.** The HDF5 dump (VR-001) captures state
after decode → detect → align → embed and before tracking and matching, so
everything downstream — which is where nearly all of the new design lives — is
cheap CPU maths replayable from a fixture. Tracking, presence windows, belief
accumulation, expansion, deferred re-identification and clustering are all
verifiable on an N100 at full fidelity, not in miniature.
### T1 is the primary tier, and KPN is why
That was already true for the optimizer. It now doubles as the CI strategy, which
is a strong argument for keeping the dump schema honest (VR-001) and for the
replay driving the *real* nodes rather than a reimplementation (VR-002).
**Node functors are plain callable structs, constructed independently of the
network that wraps them** (`main.cpp:186-207` builds them as stack objects;
`ObjectNode` merely adapts them). So a node is testable by constructing it and
calling `operator()` — no channels, no threads, no network, no fixture.
This is already the established pattern, not a proposal:
`tests/test_face_tracker.cpp` "drives the node's `operator()` with hand-built
`EmbeddedSceneFrame`s and inspects the emitted `track_ids`", and does so
"pure, GPU-free, model-free".
The consequence is that most of the redesign is verifiable **without any
fixture at all**: construct exactly the awkward state — a belief swap, two live
tracks converging on one actor, a film ending mid-track, a gap one frame under
the timeout — rather than hunting for a clip that happens to exhibit it.
Four hazards this removes outright:
- **No fixture-provenance risk** for these tests — the inputs are synthetic and
explicit.
- **No "fixture must be replayed from frame 0"** concern — state is constructed
directly.
- **No cross-test state leakage** (e.g. a tracker's `next_id_` persisting) — each
test constructs a fresh functor.
- **No replay-harness nondeterminism** — no channels, so no EOF-tail heuristics
or silent drops.
It also means **a dead upstream producer does not block testing a downstream
consumer.** `is_scene_boundary` currently has no producer (see AR-010), which
would make a *replay* test of the frame-dependent `track_alpha` pass vacuously —
but a T1 test simply constructs a frame with `is_scene_boundary = true` and
asserts the weighting changes. The producer gap is a pipeline defect to fix, not
a verification blocker.
### T2 covers what T1 cannot
Replay remains necessary for **composition** — that the nodes wired together
behave as the sum of their parts — and for realistic data at scale, which
synthetic inputs cannot honestly imitate. It is the tier that would catch a
wiring error, a channel-capacity problem, or an ordering assumption that only
appears under concurrency.
The HDF5 dump (VR-001) captures state after decode → detect → align → embed, so
replay needs no GPU and no video. That was built for the optimizer; it doubles as
CI, which is a strong argument for keeping the schema honest and for replay
driving the *real* nodes rather than a reimplementation (VR-002).
**Fixtures and studies are generated locally**, on the development machine where
the models, galleries and media already exist. CI consumes them; it never
produces them.
**Small committed fixtures are required.** A few HDF5 dumps covering the awkward
cases — a cut, a belief swap, two live tracks converging, a film ending
@@ -150,7 +195,7 @@ as such rather than counted as covered.
| Requirement | Tier | Note |
|---|---|---|
| AR-001, AR-005, AR-006 | T3 | Smoke only — correctness of detection/embedding is a model property, not ours |
| AR-002 | T2 | Size filtering is arithmetic on dumped bboxes |
| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | Size filtering is arithmetic on dumped bboxes |
| AR-003, AR-004 | T1 + T4 | Backpressure logic is unit-testable; saturation behaviour needs real load |
| AR-007 … AR-017 | **T2** | The core of the redesign — fully replayable |
| AR-018 … AR-022 | **T2** | Expansion, deferred pass, clustering: all post-embedding |
@@ -168,6 +213,46 @@ structurally unverifiable on the CI host. It needs a GPU host and a synthetic
large gallery, so it is the requirement most likely to silently regress. Its
benchmark (VR-008) should run on a schedule rather than on demand.
### CI never calls a model
**Not "should not" — cannot.** The N100 has no GPU, and even the ONNX Runtime CPU
provider is impractical: a measured run of the embedder on this hardware sits at
~930 ms per frame, so a 77 s clip at 5 fps would take roughly six minutes of
inference alone. Every model invocation therefore happens **locally, ahead of
time**, and CI consumes the result as data.
This is what makes the T1/T2 split load-bearing rather than a preference: T1 and
T2 are the only tiers that can exist in CI at all.
### Fixture corpus — `bali/`
Five clips of **Road to Bali (1952)**, ~77 s each, 480×360, 30 fps, 42 MB total.
Public domain, and that is the reason to use it rather than a convenience:
**derived fixtures — dumps, crops, golden outputs — can be committed without the
rights question that rules out sharing gallery data (SR-005).** A fixture cut
from a copyrighted title could not live in the repository at all.
Two properties to design around rather than discover:
- **480×360 means small faces.** At this resolution a face is often 40–80 px, so
the AR-002 minimum of 66 px (original resolution) rejects much of what is
there. Fixture generation must set `--min-face-px` explicitly and record it,
or the dumps will be sparse for reasons unrelated to what is being tested.
- **77 s is short.** At 1 fps that is 77 frames — too thin to exercise an
extinction window measured in tens of seconds. Generate at 5 fps (≈385 frames,
~1 MB) and record the rate in provenance, since the behaviour under test
changes with it.
> **AR-004 blocks reproducible fixture generation.** A trial run of one clip
> produced 49 frames of an expected ~385, ending at 51 s of 77 s, with the
> diagnostics reporting 285 frames dropped at `camera_pos` and 51 at
> `face_aligner`. Channels overflow and **drop** rather than blocking, and what
> gets dropped depends on timing — so the same command run twice can produce
> different dumps. Golden fixtures cannot be built on that. AR-004 is therefore
> a prerequisite for VR-001 fixtures, not merely a throughput concern for crowd
> scenes.
### Fixtures — precomputed inference, pulled by CI
The N100 cannot run inference at any useful rate, so **inference output is
@@ -176,16 +261,29 @@ what looks like GPU work into pure CPU replay.
| Fixture | Contents | Size | Storage |
|---|---|---|---|
| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~1 MB each | **Committed in-repo** |
| **Corpus dumps** | Full-length titles from the validation corpus | ~30 MB each | Pinned artifact, fetched by checksum |
| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~0.1–1 MB each | **Committed in-repo** |
| **Corpus dumps** | Full-length titles from the validation corpus | ~21–38 MB each | **Gitea package registry**, pinned by version + checksum |
| **Synthetic gallery** | Random unit-norm embeddings, fixed seed | small | Generated at test time |
| **Golden truth files** | Expected output for each edge-case dump | KB | Committed |
| **Audio golden vectors** | Short WAV + expected signature | KB | Committed, **shared with the plugin repo** |
| **Audio golden vectors** | FLAC + expected signature + parameter contract | ~600 KB | Committed, **shared with the plugin repo** |
Edge-case dumps are small enough to commit, and being in-repo means they version
with the code that reads them. Corpus dumps are pulled by pinned checksum from
the artifact store rather than committed, since they are large and change only
when the dump schema does.
with the code that reads them.
**Corpus dumps go to the Gitea package registry, not Git LFS.** Both are
available — the models already use LFS — but their fetch semantics differ in a
way that matters here. LFS objects are pulled on clone unless a developer
explicitly skips them, so ~38 MB per title behind LFS taxes everyone who clones,
forever, for data that only CI and the optimizer ever read. Registry artifacts
are fetched on demand by the job that needs them.
Rule of thumb: **LFS for what the build needs; the package registry for what a
particular job needs.** Models are the former; corpus dumps and the CI image
(DP-007) are the latter.
Pin by version and verify by checksum on fetch. A fixture that changes silently
under CI is worse than a missing one, because the failure presents as a code
regression.
**Generation must be reproducible and versioned.** A script, run on a GPU host,
regenerates every fixture from source clips; it is re-run when the VR-001 schema
@@ -204,7 +302,7 @@ because it will be trusted.
| ID | Tier | Test asserts | Edge cases to cover |
|---|---|---|---|
| AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only |
| AR-002 | T2 | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame |
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block |
| AR-005 | T1 | Known landmarks → expected 112×112 warp | Landmarks near frame edge; degenerate/collinear points |
@@ -231,8 +329,12 @@ because it will be trusted.
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | **Media < 120 s → no signature**; identical result in both repos |
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides |
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos |
| IR-006 | T1 + manual | Queue pull and result push against a stubbed Jellyfin API | Partial result never pushed; push only after the deferred pass |
| IR-007 | **T1** | Media < 120 s emits no signature at all | Exactly 120 s; just under; zero-length audio. Must match the plugin's cutoff exactly — a caller-varying window length is what SR-004 forbids |
| IR-008 | T1 | `v1:` prefix emitted and honoured on read | Unknown prefix rejected, not guessed |
| GR-009 | T1 | Human-confirmed associations persist and are tier-tagged | Survives a gallery rebuild; distinguishable from baked and harvested |
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides; **unstamped warns, and errors under `SAE_REQUIRE_GALLERY_STAMP`**; same filename + different SHA-256 must still be a mismatch |
| GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected |
| VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks |
docs/traceability.md
+596
View File
@@ -0,0 +1,596 @@
# Requirements traceability matrix
<!-- GENERATED FILE - do not edit by hand. -->
<!-- Regenerate: scripts/traceability/traceability-gate.sh -->
**Generated:** 2026-07-31T08:35:29+00:00
Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`).
## Summary
| Metric | Value |
|---|---|
| Source files scanned | 95 |
| TRACES tags found | 90 |
| EXCEPTION tags found | 0 |
| Requirements defined | 63 |
| Requirements covered | 26 |
| **Coverage** | **41.3%** (26/63) |
| Coverage of CI-executable scope | 50.0% (26/52) |
| Tagged but unexecuted in CI | 3 |
| Orphan tags | 0 |
### By type
| Type | Covered | Tagged but unexecuted | Defined |
|---|---|---|---|
| AR | 13 | 0 | 27 |
| DP | 2 | 0 | 8 |
| IR | 8 | 0 | 8 |
| GR | 3 | 0 | 9 |
| VR | 0 | 3 | 11 |
- **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104
- **PR** tags present (separate taxonomy, not counted in coverage): PR-002, PR-004
- **SR** tags present (separate taxonomy, not counted in coverage): SR-001, SR-002, SR-003, SR-005
## Not executable in CI
These requirements have no verification tier this repo's CI host can run, so a tag on them is evidence of *intent*, not of verification. They are never counted as covered.
| ID | Tiers | Tagged in source | Requirement |
|---|---|---|---|
| AR-027 | T4 | no | Throughput acceptable for **arbitrary** gallery size |
| VR-001 | out-of-ci | yes | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | out-of-ci | yes | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | out-of-ci | yes | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | out-of-ci | no | Reproducible validation corpus with ground truth |
| VR-005 | out-of-ci | no | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | out-of-ci | no | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | out-of-ci | no | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | out-of-ci | no | Gallery scaling benchmark — throughput vs gallery size |
| VR-010 | out-of-ci | no | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | out-of-ci | no | Rewrite the replay harness for the post-AR-012 output contract |
**Tagged but unexecuted:** VR-001, VR-002, VR-003 — a test exists and is tagged, but this CI host cannot run it. Report those runs separately.
## Orphan tags
A tag naming an ID `requirements.md` does not define. This is what renumbering produces, and what a typo produces.
_None._
## Requirements tracing up to nothing
A register row whose `Traces to` cell names no parent. Work serving no stated goal is how scope creeps in, and it is invisible unless something looks.
_None._
## Recorded exceptions
Deliberate, documented departures from an invariant (`EXCEPTION: XX-nnn <reason>`). Reported separately and never counted as coverage — an exception is a decision to be reviewed, not evidence a requirement is met.
_None._
## Register
| ID | Status | Tier | Traces to | Trace state | Tagged in | Requirement |
|---|---|---|---|---|---|---|
| AR-001 | Done | T3 | SR-002 | covered | `src/nodes/face_detector_node.hpp` | Detect faces in sampled frames; emit bbox, confidence, 5-point landma… |
| AR-002 | Planned | unset | SR-002 | untagged | - | Minimum face size **32×32 px** (VR-005 measured), expressed in **orig… |
| AR-003 | Planned | T1, T2, T4 | SR-002 | untagged | - | No fixed per-frame face cap — crowd scenes must not lose background c… |
| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | untagged | - | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… |
| AR-005 | Done | T1, T3 | SR-002 | covered | `src/face_utils.hpp` | Align to 112×112 via ArcFace 5-point similarity transform |
| AR-006 | Done | T3 | SR-002 | untagged | - | 512-d L2-normalised embeddings, batched |
| AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… |
| AR-008 | **Done** — one pool… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | One track pool keyed on `last_seen`; no separate revival path |
| AR-009 | Done | T2 | SR-002 | untagged | - | Camera-cut detection (histogram) as an association hint |
| AR-010 | **Not started** — `… | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint |
| AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… |
| AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition |
| AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… |
| AR-014 | **Done** — swap clo… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Belief swap A→B terminates the track and starts a new one |
| AR-015 | **Done** — reverse … | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Two live tracks owned by one actor ⇒ treat as a detected cut, re-asso… |
| AR-016 | **Done** — `flush()… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | All tracks closed at EOF — a film ends with faces on screen |
| AR-017 | **Done** — `DeadTra… | T1, T2 | SR-002 | covered | `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Every presence claim carries its belief and identification route |
| AR-018 | Planned | T1, T2 | SR-005 | untagged | - | Per-subject embedding store with banded admission (novel enough, safe… |
| AR-019 | In Progress | T2 | SR-005 | untagged | - | Per-film gallery annex from owned tracks; acquires the non-frontal vi… |
| AR-020 | Planned | T2 | SR-005 | untagged | - | Deferred re-identification of unknown tracks against the final expand… |
| AR-021 | Planned | T2 | SR-005 | untagged | - | Cluster unknown tracks into one entity per person, under temporal can… |
| AR-022 | Planned | T1, T2 | §4 | untagged | - | Capture still-unidentified tracks: embeddings, metadata, **context cr… |
| AR-023 | Done | T1 | SR-002 | covered | `src/gallery/gallery_calibration.hpp`, `src/nodes/identity_matcher_node.hpp` | Fit sigmoid calibration from intra/inter similarity distributions |
| AR-024 | **Done** — associat… | T1, static | SR-002 | covered | `src/config.hpp`, `src/evidence_discount.hpp`, `src/gallery/gallery_calibration.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp`, `src/nodes/identity_matcher_node.hpp` | **Always the calibrated probability, never a raw cosine** — exception… |
| AR-025 | **Done** — log-odds… | T1 | SR-002 | covered | `src/evidence_discount.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-track Bayesian accumulation in log-odds, with correlated-observat… |
| AR-026 | In Progress | T1, T4 | SR-001 | untagged | - | All similarity computed as GEMM, including annex and deferred pass |
| AR-027 | Planned | T4 | SR-001 | untagged | - | Throughput acceptable for **arbitrary** gallery size |
| DP-001 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | One analysis core; modes are front-ends and must not fork pipeline lo… |
| DP-002 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | Batch CLI over one title |
| DP-003 | Planned | T1, manual | PR-004 | untagged | - | On-demand resident service with bounded, observable queue |
| DP-004 | Planned | T1, manual | PR-004 | untagged | - | Opportunistic/idle mode: external trigger, hard stop, implicit re-que… |
| DP-005 | Planned | T1, manual | PR-004 | untagged | - | Native installer, no Docker; Fedora + Arch |
| DP-006 | Planned | T1, manual | PR-003 | untagged | - | Background incremental gallery refresh on a timer |
| DP-007 | Planned | T1, manual | PR-004 | untagged | - | CI builder image, CPU-only, pinned by tag in the Gitea container regi… |
| DP-008 | Planned | T1, manual | PR-004 | untagged | - | Builder images + release jobs per backend (cpu / cuda / rocm); ship b… |
| IR-001 | Done | T1 | SR-003 | covered | `src/nodes/result_sink_node.hpp` | Emit the JRay truth format as sibling `.jray.json` |
| IR-002 | **Done** — `schema_… | T1 | SR-003 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/result_sink_node.hpp` | Windows carry belief + route; `extraction.*` carries `extinction_sec`… |
| IR-003 | **In Progress** — s… | T1 | SR-003 | covered | `src/main.cpp` | Output written **after** the deferred pass, not at EOF |
| IR-004 | **Done** — `src/aud… | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Compute the audio signature exactly per server spec §3 |
| IR-005 | **Done** — `tests/f… | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Golden-vector fixture shared with the plugin repo to prove bit-exactn… |
| IR-006 | Done | T1, manual | SR-001 | covered | `scripts/run_from_jellyfin.py` | Jellyfin round-trip: pull pending queue, push complete results only |
| IR-007 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Media < 120 s: emit no signature, apply no sync offset — identical ru… |
| IR-008 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Emit and honour the signature's own `v1:` version prefix |
| GR-001 | Done | T1, T3 | SR-001, SR-005 | covered | `scripts/make_jellyfin_gallery.py` | Build gallery from Jellyfin library cast, TMDB profile fallback |
| GR-002 | Done | T1, T3 | PR-003 | covered | `scripts/make_jellyfin_gallery.py` | Incremental `--merge` refresh without re-embedding known actors |
| GR-003 | Planned | T1, T3 | SR-001 | untagged | - | Report coverage: zero-image actors, under-referenced actors, dedup, c… |
| GR-004 | **Done** — basename… | T1, T3 | SR-001 | covered | `scripts/filter_gallery.py`, `scripts/make_gallery.py`, `scripts/make_jellyfin_gallery.py`, `scripts/movienet_eval.py`, `scripts/optimizer/fetch_missing_actors.py`, `scripts/optimizer/optimize.py`, `scripts/optimizer/reembed_gallery.py`, `scripts/optimizer/replay.py`, `scripts/sae_embed_loader.py`, `scripts/sae_gallery.py`, `scripts/sae_stamp.py`, `scripts/stamp_gallery.py`, `src/config.hpp`, `src/gallery/embedder_stamp.cpp`, `src/gallery/embedder_stamp.hpp`, `src/gallery/gallery_builder.cpp`, `src/gallery/gallery_store.cpp`, `src/kpn_bindings.cpp`, `src/main.cpp`, `src/nodes/embedding_dump_node.hpp`, `src/scene_preview.cpp`, `src/types.hpp`, `tests/test_gallery_store.cpp` | Stamp embedder identity into the gallery; **hard startup error** on m… |
| GR-005 | Done | T1, T3 | **SR-005** | untagged | - | Gallery data never leaves the instance |
| GR-006 | Planned | T1 | SR-005 | untagged | - | Provenance tiers: baked / harvested / confirmed, distinguishable per … |
| GR-007 | Planned | T1 | SR-005 | untagged | - | Persist harvested embeddings **flagged and reviewable**, never silent… |
| GR-008 | Planned | T1 | SR-005 | untagged | - | Flag distributional outliers among an actor's references (poisoning g… |
| GR-009 | TBD | T1 | §4 | untagged | - | Human-confirmed associations persist and improve future extractions |
| VR-001 | Done | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp` | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py` | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/second_score.py` | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | Done | out-of-ci | PR-002 | untagged | - | Reproducible validation corpus with ground truth |
| VR-005 | **Done** — knee at … | out-of-ci | PR-002 | untagged | - | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | Planned | out-of-ci | PR-002 | untagged | - | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | Planned | out-of-ci | PR-002 | untagged | - | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | Planned | out-of-ci | PR-002 | untagged | - | Gallery scaling benchmark — throughput vs gallery size |
| VR-009 | Planned | T1, out-of-ci | PR-002 | untagged | - | Verify accumulated posteriors are calibrated against held-out tracks |
| VR-010 | Planned | out-of-ci | PR-002 | untagged | - | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | Planned | out-of-ci | PR-002 | untagged | - | Rewrite the replay harness for the post-AR-012 output contract |
## Detailed mapping
### AR-001
**Locations:** 1
- [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown`
### AR-005
**Locations:** 1
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `inline cv::Mat align_face(const cv::Mat& img,`
### AR-007
**Locations:** 3
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
### AR-008
**Locations:** 3
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
### AR-012
**Locations:** 8
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/identity_matcher_node.hpp:119`](../src/nodes/identity_matcher_node.hpp#L119) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:242`](../src/nodes/identity_matcher_node.hpp#L242) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-013
**Locations:** 2
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-014
**Locations:** 2
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-015
**Locations:** 2
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-016
**Locations:** 4
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/result_sink_node.hpp:63`](../src/nodes/result_sink_node.hpp#L63) — `void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-017
**Locations:** 3
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-023
**Locations:** 3
- [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }`
### AR-024
**Locations:** 6
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }`
### AR-025
**Locations:** 3
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:119`](../src/nodes/identity_matcher_node.hpp#L119) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:242`](../src/nodes/identity_matcher_node.hpp#L242) — `Unknown`
### DP-001
**Locations:** 1
- [`src/main.cpp:3`](../src/main.cpp#L3) — `Unknown`
### DP-002
**Locations:** 1
- [`src/main.cpp:3`](../src/main.cpp#L3) — `Unknown`
### GR-001
**Locations:** 1
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### GR-002
**Locations:** 1
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### GR-004
**Locations:** 44
- [`src/config.hpp:49`](../src/config.hpp#L49) — `Unknown`
- [`src/gallery/embedder_stamp.cpp:1`](../src/gallery/embedder_stamp.cpp#L1) — `Unknown`
- [`src/gallery/embedder_stamp.hpp:2`](../src/gallery/embedder_stamp.hpp#L2) — `Unknown`
- [`src/gallery/gallery_builder.cpp:45`](../src/gallery/gallery_builder.cpp#L45) — `ActorGallery build_gallery(const BuildConfig& cfg)`
- [`src/gallery/gallery_store.cpp:82`](../src/gallery/gallery_store.cpp#L82) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/gallery/gallery_store.cpp:167`](../src/gallery/gallery_store.cpp#L167) — `H5::DataSpace scalar(H5S_SCALAR);`
- [`src/gallery/gallery_store.cpp:219`](../src/gallery/gallery_store.cpp#L219) — `Unknown`
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:174`](../src/main.cpp#L174) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:141`](../src/types.hpp#L141) — `struct Actor`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");`
- [`tests/test_gallery_store.cpp:240`](../tests/test_gallery_store.cpp#L240) — `TempFile tf("gallery_json_stamp.json");`
- [`tests/test_gallery_store.cpp:252`](../tests/test_gallery_store.cpp#L252) — `Unknown`
- [`tests/test_gallery_store.cpp:268`](../tests/test_gallery_store.cpp#L268) — `Unknown`
- [`tests/test_gallery_store.cpp:300`](../tests/test_gallery_store.cpp#L300) — `Unknown`
- [`tests/test_gallery_store.cpp:309`](../tests/test_gallery_store.cpp#L309) — `Unknown`
- [`tests/test_gallery_store.cpp:333`](../tests/test_gallery_store.cpp#L333) — `Unknown`
- [`tests/test_gallery_store.cpp:348`](../tests/test_gallery_store.cpp#L348) — `Unknown`
- [`tests/test_gallery_store.cpp:367`](../tests/test_gallery_store.cpp#L367) — `Unknown`
- [`tests/test_gallery_store.cpp:384`](../tests/test_gallery_store.cpp#L384) — `TempFile tf("fake_model.onnx");`
- [`scripts/filter_gallery.py:80`](../scripts/filter_gallery.py#L80) — `if actor_jellyfin_id(a) in cast_ids]`
- [`scripts/make_gallery.py:181`](../scripts/make_gallery.py#L181) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:448`](../scripts/make_jellyfin_gallery.py#L448) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:456`](../scripts/make_jellyfin_gallery.py#L456) — `Unknown`
- [`scripts/movienet_eval.py:65`](../scripts/movienet_eval.py#L65) — `with open(args.gt) as f:`
- [`scripts/optimizer/fetch_missing_actors.py:62`](../scripts/optimizer/fetch_missing_actors.py#L62) — `def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,`
- [`scripts/optimizer/fetch_missing_actors.py:109`](../scripts/optimizer/fetch_missing_actors.py#L109) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:123`](../scripts/optimizer/fetch_missing_actors.py#L123) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/optimize.py:186`](../scripts/optimizer/optimize.py#L186) — `Unknown`
- [`scripts/optimizer/optimize.py:202`](../scripts/optimizer/optimize.py#L202) — `if not Path(f["dump"]).exists():`
- [`scripts/optimizer/reembed_gallery.py:62`](../scripts/optimizer/reembed_gallery.py#L62) — `for i, a in enumerate(ref["actors"], 1):`
- [`scripts/optimizer/replay.py:113`](../scripts/optimizer/replay.py#L113) — `Unknown`
- [`scripts/optimizer/replay.py:252`](../scripts/optimizer/replay.py#L252) — `Unknown`
- [`scripts/sae_embed_loader.py:22`](../scripts/sae_embed_loader.py#L22) — `return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)`
- [`scripts/sae_gallery.py:171`](../scripts/sae_gallery.py#L171) — `if not _stamp_empty(embedder):`
- [`scripts/sae_gallery.py:199`](../scripts/sae_gallery.py#L199) — `for a in range(len(offset)):`
- [`scripts/sae_stamp.py:3`](../scripts/sae_stamp.py#L3) — `Unknown`
- [`scripts/stamp_gallery.py:4`](../scripts/stamp_gallery.py#L4) — `Unknown`
### IR-001
**Locations:** 1
- [`src/nodes/result_sink_node.hpp:2`](../src/nodes/result_sink_node.hpp#L2) — `Unknown`
### IR-002
**Locations:** 5
- [`src/config.hpp:20`](../src/config.hpp#L20) — `struct Config`
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:122`](../src/nodes/result_sink_node.hpp#L122) — `void write_output()`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
### IR-003
**Locations:** 1
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
### IR-004
**Locations:** 16
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:265`](../src/audio_signature.cpp#L265) — `std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:320`](../src/audio_signature.cpp#L320) — `std::optional<std::string> signature_from_mono(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:327`](../src/audio_signature.cpp#L327) — `std::optional<std::vector<float>> decode_centre_window(const std::string& path)`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:151`](../tests/test_audio_signature.cpp#L151) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
- [`tests/test_audio_signature.cpp:236`](../tests/test_audio_signature.cpp#L236) — `Unknown`
- [`tests/test_audio_signature.cpp:263`](../tests/test_audio_signature.cpp#L263) — `TempWav w("centred300");`
- [`tests/test_audio_signature.cpp:301`](../tests/test_audio_signature.cpp#L301) — `Unknown`
- [`tests/test_audio_signature.cpp:314`](../tests/test_audio_signature.cpp#L314) — `kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));`
- [`tests/test_audio_signature.cpp:328`](../tests/test_audio_signature.cpp#L328) — `std::vector<float> a(kWindowSamples / 50);`
- [`tests/test_audio_signature.cpp:344`](../tests/test_audio_signature.cpp#L344) — `return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());`
### IR-005
**Locations:** 5
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:138`](../tests/test_audio_signature.cpp#L138) — `Unknown`
### IR-006
**Locations:** 1
- [`scripts/run_from_jellyfin.py:4`](../scripts/run_from_jellyfin.py#L4) — `Unknown`
### IR-007
**Locations:** 8
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:327`](../src/audio_signature.cpp#L327) — `std::optional<std::vector<float>> decode_centre_window(const std::string& path)`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:201`](../tests/test_audio_signature.cpp#L201) — `TempWav w("short30");`
- [`tests/test_audio_signature.cpp:219`](../tests/test_audio_signature.cpp#L219) — `TempWav w("exact120");`
- [`tests/test_audio_signature.cpp:229`](../tests/test_audio_signature.cpp#L229) — `TempWav w("exact120");`
### IR-008
**Locations:** 7
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:320`](../src/audio_signature.cpp#L320) — `std::optional<std::string> signature_from_mono(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
### PR-002
**Locations:** 3
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
- [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an`
- [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown`
### PR-004
**Locations:** 1
- [`src/main.cpp:3`](../src/main.cpp#L3) — `Unknown`
### SR-001
**Locations:** 46
- [`src/config.hpp:49`](../src/config.hpp#L49) — `Unknown`
- [`src/gallery/embedder_stamp.cpp:1`](../src/gallery/embedder_stamp.cpp#L1) — `Unknown`
- [`src/gallery/embedder_stamp.hpp:2`](../src/gallery/embedder_stamp.hpp#L2) — `Unknown`
- [`src/gallery/gallery_builder.cpp:45`](../src/gallery/gallery_builder.cpp#L45) — `ActorGallery build_gallery(const BuildConfig& cfg)`
- [`src/gallery/gallery_store.cpp:82`](../src/gallery/gallery_store.cpp#L82) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/gallery/gallery_store.cpp:167`](../src/gallery/gallery_store.cpp#L167) — `H5::DataSpace scalar(H5S_SCALAR);`
- [`src/gallery/gallery_store.cpp:219`](../src/gallery/gallery_store.cpp#L219) — `Unknown`
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:174`](../src/main.cpp#L174) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:141`](../src/types.hpp#L141) — `struct Actor`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");`
- [`tests/test_gallery_store.cpp:240`](../tests/test_gallery_store.cpp#L240) — `TempFile tf("gallery_json_stamp.json");`
- [`tests/test_gallery_store.cpp:252`](../tests/test_gallery_store.cpp#L252) — `Unknown`
- [`tests/test_gallery_store.cpp:268`](../tests/test_gallery_store.cpp#L268) — `Unknown`
- [`tests/test_gallery_store.cpp:300`](../tests/test_gallery_store.cpp#L300) — `Unknown`
- [`tests/test_gallery_store.cpp:309`](../tests/test_gallery_store.cpp#L309) — `Unknown`
- [`tests/test_gallery_store.cpp:333`](../tests/test_gallery_store.cpp#L333) — `Unknown`
- [`tests/test_gallery_store.cpp:348`](../tests/test_gallery_store.cpp#L348) — `Unknown`
- [`tests/test_gallery_store.cpp:367`](../tests/test_gallery_store.cpp#L367) — `Unknown`
- [`tests/test_gallery_store.cpp:384`](../tests/test_gallery_store.cpp#L384) — `TempFile tf("fake_model.onnx");`
- [`scripts/filter_gallery.py:80`](../scripts/filter_gallery.py#L80) — `if actor_jellyfin_id(a) in cast_ids]`
- [`scripts/make_gallery.py:181`](../scripts/make_gallery.py#L181) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:448`](../scripts/make_jellyfin_gallery.py#L448) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:456`](../scripts/make_jellyfin_gallery.py#L456) — `Unknown`
- [`scripts/movienet_eval.py:65`](../scripts/movienet_eval.py#L65) — `with open(args.gt) as f:`
- [`scripts/optimizer/fetch_missing_actors.py:62`](../scripts/optimizer/fetch_missing_actors.py#L62) — `def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,`
- [`scripts/optimizer/fetch_missing_actors.py:109`](../scripts/optimizer/fetch_missing_actors.py#L109) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:123`](../scripts/optimizer/fetch_missing_actors.py#L123) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/optimize.py:186`](../scripts/optimizer/optimize.py#L186) — `Unknown`
- [`scripts/optimizer/optimize.py:202`](../scripts/optimizer/optimize.py#L202) — `if not Path(f["dump"]).exists():`
- [`scripts/optimizer/reembed_gallery.py:62`](../scripts/optimizer/reembed_gallery.py#L62) — `for i, a in enumerate(ref["actors"], 1):`
- [`scripts/optimizer/replay.py:113`](../scripts/optimizer/replay.py#L113) — `Unknown`
- [`scripts/optimizer/replay.py:252`](../scripts/optimizer/replay.py#L252) — `Unknown`
- [`scripts/run_from_jellyfin.py:4`](../scripts/run_from_jellyfin.py#L4) — `Unknown`
- [`scripts/sae_embed_loader.py:22`](../scripts/sae_embed_loader.py#L22) — `return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)`
- [`scripts/sae_gallery.py:171`](../scripts/sae_gallery.py#L171) — `if not _stamp_empty(embedder):`
- [`scripts/sae_gallery.py:199`](../scripts/sae_gallery.py#L199) — `for a in range(len(offset)):`
- [`scripts/sae_stamp.py:3`](../scripts/sae_stamp.py#L3) — `Unknown`
- [`scripts/stamp_gallery.py:4`](../scripts/stamp_gallery.py#L4) — `Unknown`
### SR-002
**Locations:** 16
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `inline cv::Mat align_face(const cv::Mat& img,`
- [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:119`](../src/nodes/identity_matcher_node.hpp#L119) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:242`](../src/nodes/identity_matcher_node.hpp#L242) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:63`](../src/nodes/result_sink_node.hpp#L63) — `void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
### SR-003
**Locations:** 6
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`src/config.hpp:20`](../src/config.hpp#L20) — `struct Config`
- [`src/nodes/result_sink_node.hpp:2`](../src/nodes/result_sink_node.hpp#L2) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:122`](../src/nodes/result_sink_node.hpp#L122) — `void write_output()`
### SR-005
**Locations:** 1
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### UT-001
**Locations:** 1
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### UT-101
**Locations:** 5
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:138`](../tests/test_audio_signature.cpp#L138) — `Unknown`
- [`tests/test_audio_signature.cpp:151`](../tests/test_audio_signature.cpp#L151) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
### UT-102
**Locations:** 5
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:201`](../tests/test_audio_signature.cpp#L201) — `TempWav w("short30");`
- [`tests/test_audio_signature.cpp:219`](../tests/test_audio_signature.cpp#L219) — `TempWav w("exact120");`
- [`tests/test_audio_signature.cpp:229`](../tests/test_audio_signature.cpp#L229) — `TempWav w("exact120");`
- [`tests/test_audio_signature.cpp:236`](../tests/test_audio_signature.cpp#L236) — `Unknown`
### UT-103
**Locations:** 2
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:263`](../tests/test_audio_signature.cpp#L263) — `TempWav w("centred300");`
### UT-104
**Locations:** 5
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:301`](../tests/test_audio_signature.cpp#L301) — `Unknown`
- [`tests/test_audio_signature.cpp:314`](../tests/test_audio_signature.cpp#L314) — `kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));`
- [`tests/test_audio_signature.cpp:328`](../tests/test_audio_signature.cpp#L328) — `std::vector<float> a(kWindowSamples / 50);`
- [`tests/test_audio_signature.cpp:344`](../tests/test_audio_signature.cpp#L344) — `return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());`
### VR-001
**Locations:** 1
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
### VR-002
**Locations:** 1
- [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an`
### VR-003
**Locations:** 1
- [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown`
## Tag diagnostics
**Malformed tags:**
- `scripts/filter_gallery.py:80` — {'ignored': ['— a filtered gallery holds the SAME vectors as its']}
- `scripts/make_gallery.py:181` — {'ignored': ['— stamp with the model actually loaded', 'resolved']}
- `scripts/make_jellyfin_gallery.py:456` — {'ignored': ["— --merge keeps the existing actors' vectors and"]}
- `scripts/movienet_eval.py:65` — {'ignored': ['— match() below is a bare dot product against the']}
- `scripts/optimizer/fetch_missing_actors.py:109` — {'ignored': ['— the legacy JSON gallery carries the same stamp as']}
- `scripts/optimizer/fetch_missing_actors.py:123` — {'ignored': ['— merging two galleries from different models makes']}
- `scripts/optimizer/optimize.py:202` — {'ignored': ['— every (dump', 'gallery) pair is checked ONCE here']}
- `scripts/optimizer/reembed_gallery.py:62` — {'ignored': ['— this script exists to produce a gallery in a']}
- `scripts/optimizer/replay.py:113` — {'ignored': ['— checked here', 'before any network is built', 'so a']}
- `scripts/optimizer/replay.py:252` — {'ignored': ['— promote an unprovable gallery/dump binding from a']}
- `scripts/sae_embed_loader.py:22` — {'ignored': ['— single source of truth for "which model is this"']}
- `scripts/sae_gallery.py:171` — {'ignored': ['— omitted entirely when unknown', 'so "unstamped"']}
- `scripts/sae_gallery.py:199` — {'ignored': ['— carried through so a derived gallery (filter']}
**Groups mixing requirement types (pipe separates types):**
- `src/main.cpp:216` — {'group': ['AR-012', 'AR-016', 'IR-002', 'IR-003']}
- `src/nodes/result_sink_node.hpp:49` — {'group': ['AR-012', 'AR-017', 'IR-002']}
- `src/nodes/result_sink_node.hpp:161` — {'group': ['AR-012', 'IR-002']}
external/KPN
Vendored
+1 -1
Submodule external/KPN updated: 4b6e498ba7...be6e92268c
scripts/filter_gallery.py
+5 -1
View File
@@ -77,7 +77,11 @@ def main():
if missing > 0:
print(f"[warn] {missing} cast member(s) not present in gallery (not yet embedded)", file=sys.stderr)
save_gallery_hdf5({"actors": actors}, Path(args.output))
# TRACES: GR-004 | SR-001 — a filtered gallery holds the SAME vectors as its
# source, so it inherits the source's binding. Dropping the stamp here would
# silently launder a stamped gallery into an unstamped one.
save_gallery_hdf5({"actors": actors}, Path(args.output),
gallery.get("embedder"))
print(f"Saved {len(actors)} actor(s) to {args.output}", file=sys.stderr)
scripts/make_fixtures.sh
Executable
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env bash
# make_fixtures.sh — regenerate the committed replay fixtures.
#
# TRACES: VR-001 | PR-002
#
# CI never calls a model (see docs/requirements.md, "CI never calls a model"):
# the embedder is impractical on the N100 CI host, so inference happens HERE, on
# a machine with a GPU, and CI consumes the HDF5 dumps as data. Everything
# downstream of embedding — tracking, presence windows, belief accumulation,
# expansion — is cheap CPU maths and replays from these files.
#
# Reproducibility is a requirement, not a nicety. A fixture whose provenance is
# unknown is worse than no fixture, because it will be trusted. Every parameter
# that affects the output is pinned below rather than left to a default, and the
# dumps carry the embedder identity and SHA-256 (GR-004) so a replay cannot be
# silently scored against the wrong gallery.
#
# These are byte-reproducible only because node outputs block rather than drop
# on a full channel (AR-004). Before that fix the same command produced
# different dumps run to run, since what got dropped depended on timing.
#
# Source: bali/ — Road to Bali (1952), public domain. That matters: derived
# fixtures can be committed, where anything cut from a copyrighted title could
# not live in the repository at all.
set -euo pipefail
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CLIPS="${CLIPS:-$REPO/../bali}"
GALLERY="${GALLERY:-$REPO/gallery_lvface.h5}"
BIN="${BIN:-$REPO/build/scene_analyze}"
OUT="$REPO/tests/fixtures/dumps"
# Pinned. Changing either invalidates every committed fixture.
# fps 5 — 1 fps over a 77 s clip is 77 frames, too thin to exercise an
# extinction window measured in tens of seconds.
# min-face — 32 px, the VR-005 measured floor (98.1% TPI). The corpus is
# 480x360, so a stricter value would reject most faces present.
FPS=5
MIN_FACE_PX=32
[[ -x "$BIN" ]] || { echo "no scene_analyze at $BIN (set BIN=)" >&2; exit 1; }
[[ -f "$GALLERY" ]] || { echo "no gallery at $GALLERY (set GALLERY=)" >&2; exit 1; }
[[ -d "$CLIPS" ]] || { echo "no clips at $CLIPS (set CLIPS=)" >&2; exit 1; }
mkdir -p "$OUT"
for clip in "$CLIPS"/Road_To_Bali-*.webm; do
n="$(basename "$clip" .webm)"; n="${n##*-}"
echo "── bali_$n"
"$BIN" --movie "$clip" --gallery "$GALLERY" \
--fps "$FPS" --min-face-px "$MIN_FACE_PX" \
--dump-embeddings "$OUT/bali_$n.h5" \
--output /dev/null 2>&1 | grep -E "wrote|dropped" || true
done
echo
echo "Regenerated in $OUT — verify the diff is empty if nothing upstream changed."
echo "A non-empty diff means detection, alignment or embedding moved. That is"
echo "either a regression or a deliberate change, and either way the golden"
echo "outputs derived from these fixtures need reviewing."
scripts/make_gallery.py
+8 -3
View File
@@ -36,8 +36,9 @@ import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from sae_embed_loader import load_embedder
from sae_gallery import download_images, save_gallery, wikidata_image_urls
from sae_embed_loader import load_embedder, resolve_arcface
from sae_gallery import (download_images, embedder_stamp, save_gallery,
wikidata_image_urls)
from sae_tmdb import TMDB_IMG, tmdb_get, tmdb_id_from_imdb
@@ -177,7 +178,11 @@ def main():
output = Path(args.output)
image_root = Path(args.image_dir) if args.image_dir else output.parent / "images"
# TRACES: GR-004 | SR-001 — stamp with the model actually loaded, resolved
# through the same helper load_embedder uses so the two cannot diverge.
arcface_path = resolve_arcface(args.models_dir, args.arcface)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
stamp = embedder_stamp(arcface_path)
# Resolve movie ID
movie_id = args.movie_id
@@ -203,7 +208,7 @@ def main():
if n_actors == 0:
sys.exit("No actors could be processed — check models and images.")
save_gallery(gallery, missing, output)
save_gallery(gallery, missing, output, embedder=stamp)
if __name__ == "__main__":
scripts/make_jellyfin_gallery.py
+15 -3
View File
@@ -1,6 +1,8 @@
#!/usr/bin/env python3
"""make_jellyfin_gallery.py — build a gallery.h5 spanning an entire Jellyfin library.
TRACES: GR-001, GR-002 | SR-001, SR-005
Queries the Jellyfin API for every Movie/Series, collects the unique cast
across the whole library, downloads each actor's headshot directly from
Jellyfin (no TMDB key needed), embeds them with the sae_embed module (SCRFD +
@@ -51,8 +53,9 @@ import requests
sys.path.insert(0, str(Path(__file__).resolve().parent))
import sae_env # noqa: F401 — loads .env into os.environ on import
from sae_embed_loader import load_embedder
from sae_gallery import (download_image, download_images, load_gallery_hdf5,
from sae_embed_loader import load_embedder, resolve_arcface
from sae_gallery import (download_image, download_images, embedder_stamp,
enforce_embedder_stamp, load_gallery_hdf5,
save_gallery, wikidata_image_urls)
from sae_jellyfin import actor_jellyfin_id, jf_get, normalize_jellyfin_url
from sae_tmdb import (
@@ -442,11 +445,20 @@ def main():
image_root = Path(args.image_dir) if args.image_dir else output.parent / "images"
item_types = [t.strip() for t in args.item_types.split(",") if t.strip()]
# TRACES: GR-004 | SR-001
arcface_path = resolve_arcface(args.models_dir, args.arcface)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
stamp = embedder_stamp(arcface_path)
existing_actors = {}
if args.merge and output.is_file():
existing = load_gallery_hdf5(output)
# TRACES: GR-004 | SR-001 — --merge keeps the existing actors' vectors and
# embeds the new ones with THIS model. If they disagree, the result is one
# gallery holding two incompatible embedding spaces, which is worse than a
# mismatched gallery: no later check can separate them again.
enforce_embedder_stamp(existing.get("embedder"), stamp, str(output),
arcface_path)
for actor in existing.get("actors", []):
pid = actor_jellyfin_id(actor)
if pid:
@@ -475,7 +487,7 @@ def main():
if n_actors == 0:
sys.exit("No actors could be processed — check Jellyfin URL/API key and models.")
save_gallery(gallery, missing, output)
save_gallery(gallery, missing, output, embedder=stamp)
if __name__ == "__main__":
scripts/movienet_eval.py
+7 -2
View File
@@ -22,8 +22,8 @@ from pathlib import Path
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parent))
from sae_embed_loader import load_embedder
from sae_gallery import load_gallery_hdf5
from sae_embed_loader import load_embedder, resolve_arcface
from sae_gallery import load_gallery_hdf5, verify_gallery_stamp
def load_gallery(path: str) -> dict[str, dict]:
@@ -62,6 +62,11 @@ def main():
args = p.parse_args()
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001 — match() below is a bare dot product against the
# gallery's vectors; if the gallery came from another model those numbers are
# noise wearing a similarity's clothes.
verify_gallery_stamp(args.gallery,
resolve_arcface(args.models_dir, args.arcface))
gallery = load_gallery(args.gallery)
print(f"[eval] gallery: {len(gallery)} actors", file=sys.stderr)
scripts/optimizer/SCHEMA.md
+16
View File
@@ -22,6 +22,8 @@ variable-length HDF5 types and reads straight into numpy.
movie : str (source video path)
sample_fps : float
embed_dim : int = 512
embedder_model : str basename of the embedding model (GR-004)
embedder_sha256: str SHA-256 of that model file (GR-004)
frames/ group — one row per sampled frame
timestamp_sec : float64 [F]
@@ -41,6 +43,20 @@ variable-length HDF5 types and reads straight into numpy.
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.
## Model binding (GR-004)
`embedder_model` / `embedder_sha256` record which embedder produced every vector
in `faces/embedding`. A replay has no live embedder, so **the dump is the embedder
as far as the gallery is concerned**: `replay.py` checks these two attributes
against the gallery's own `/embedder` stamp and refuses to run on a mismatch,
naming both sides. Cross-model cosines are meaningless but look plausible.
The attributes are additive, not a format break — `schema_version` stays 1. Dumps
written before GR-004 simply lack them, which reports as *unverifiable* (a loud
warning, or a hard error under `SAE_REQUIRE_GALLERY_STAMP=1`) rather than as a
pass. Re-dump to bind an old dump; there is no in-place migration, because unlike
a gallery nobody can assert after the fact which model produced a vector.
## Invariants
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
scripts/optimizer/fetch_missing_actors.py
+13 -2
View File
@@ -35,7 +35,9 @@ REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
import sae_env # noqa: E402 loads .env
from sae_tmdb import tmdb_get, tmdb_person_for_imdb, TMDB_IMG # noqa: E402
from sae_gallery import download_images, wikidata_image_urls # noqa: E402
from sae_embed_loader import resolve_arcface # noqa: E402
from sae_gallery import (download_images, embedder_stamp, # noqa: E402
enforce_embedder_stamp, wikidata_image_urls)
from sae_embed_loader import load_embedder # noqa: E402
@@ -57,6 +59,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
src = "TMDB + Wikidata fallback" if use_wikidata else "TMDB"
print(f"[fetch] {len(missing)} missing actors to resolve via {src}", file=sys.stderr)
embedder = load_embedder(build_dir, models_dir, arcface)
stamp = embedder_stamp(resolve_arcface(models_dir, arcface)) # TRACES: GR-004 | SR-001
img_root = Path(tempfile.mkdtemp(prefix="missing_gallery_"))
actors = []
@@ -103,7 +106,9 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
f"(wiki={n_via_wikidata}) no_tmdb={n_no_tmdb} no_img={n_no_img} "
f"no_face={n_no_face}", file=sys.stderr)
Path(out_path).write_text(json.dumps({"actors": actors}, indent=2))
# TRACES: GR-004 | SR-001 — the legacy JSON gallery carries the same stamp as
# the HDF5 one; src/gallery/gallery_store.cpp reads it from either.
Path(out_path).write_text(json.dumps({"embedder": stamp, "actors": actors}, indent=2))
n_emb = sum(len(a["embeddings"]) for a in actors)
print(f"\n[fetch] recovered {n_resolved}/{len(missing)} actors "
f"({n_via_wikidata} via Wikidata), {n_emb} embeddings → {out_path}",
@@ -115,6 +120,12 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
def merge(base_path, add_path, out_path):
base = json.loads(Path(base_path).read_text())
add = json.loads(Path(add_path).read_text())
# TRACES: GR-004 | SR-001 — merging two galleries from different models makes
# ONE file containing two incompatible embedding spaces. Nothing downstream can
# ever untangle that, so this is the one place the check must run before, not
# after, the write.
enforce_embedder_stamp(base.get("embedder"), add.get("embedder"),
str(base_path), str(add_path))
have = {a.get("imdb_id") for a in base["actors"] if a.get("imdb_id")}
added = [a for a in add["actors"] if a.get("imdb_id") not in have]
base["actors"].extend(added)
scripts/optimizer/optimize.py
+23
View File
@@ -34,6 +34,7 @@ from scipy.optimize import differential_evolution
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts" / "optimizer"))
sys.path.insert(0, str(REPO / "scripts" / "validation"))
sys.path.insert(0, str(REPO / "scripts"))
import json as _json
import os
@@ -56,6 +57,8 @@ DE_WORKERS = int(os.environ.get("DE_WORKERS", "1"))
from second_score import score_seconds # noqa: E402 uniform per-second TPI/FPI scoring
from sample_eval import load_gallery_keys # noqa: E402
from replay import dump_embedder_stamp # noqa: E402
from sae_stamp import EmbedderMismatch, verify_gallery_stamp # noqa: E402
_GAL_KEYS: dict = {} # gallery path → key set (fair-recall FN mask), loaded once
_REPLAY_TIMEOUT = 45 # seconds per film; a wedged replay is killed, not left to hang
@@ -180,7 +183,15 @@ def main():
p.add_argument("--seed", type=int, default=0)
p.add_argument("--trajectory", help="write every evaluation here (JSON lines)")
p.add_argument("--out", help="write best config + metrics")
# TRACES: GR-004 | SR-001
p.add_argument("--require-gallery-stamp", action="store_true",
help="unprovable gallery/dump model binding is a hard error, "
"not a warning (also via SAE_REQUIRE_GALLERY_STAMP=1)")
args = p.parse_args()
if args.require_gallery_stamp:
# Set the env var rather than threading a flag through cfg: replays run as
# subprocesses and inherit it, so strictness cannot be lost in the handoff.
os.environ["SAE_REQUIRE_GALLERY_STAMP"] = "1"
films = json.loads(Path(args.manifest).read_text())
for f in films:
@@ -188,6 +199,18 @@ def main():
if not Path(f["dump"]).exists():
sys.exit(f"[opt] missing dump for {f['name']}: {f['dump']}")
# TRACES: GR-004 | SR-001 — every (dump, gallery) pair is checked ONCE here,
# before the first evaluation. A DE sweep is thousands of replays; discovering
# a cross-model pair at the end (or never) means every number it produced was
# noise. Each replay subprocess re-checks its own pair anyway.
for f in films:
try:
verify_gallery_stamp(f["gallery"], stamp=dump_embedder_stamp(f["dump"]),
embedder_desc=f"embedding dump {Path(f['dump']).name}",
require_stamp=args.require_gallery_stamp)
except EmbedderMismatch as e:
sys.exit(f"[opt] {f['name']}: {e}")
names, bounds = [], []
int_knobs = {"track_max_frames_missing", "cut_inactive_max_frames"}
for spec in args.params:
scripts/optimizer/reembed_gallery.py
+9 -3
View File
@@ -27,8 +27,9 @@ from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
from sae_embed_loader import load_embedder # noqa: E402
from sae_gallery import load_gallery_hdf5, save_gallery_hdf5 # noqa: E402
from sae_embed_loader import load_embedder, resolve_arcface # noqa: E402
from sae_gallery import (embedder_stamp, load_gallery_hdf5, # noqa: E402
save_gallery_hdf5)
def find_dir(images_root: Path, jellyfin_id: str, name: str) -> Path | None:
@@ -58,6 +59,11 @@ def main():
ref = load_gallery_hdf5(Path(args.ref))
images_root = Path(args.images)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001 — this script exists to produce a gallery in a
# DIFFERENT model's space from the reference. The output must therefore never
# inherit the reference's stamp; it carries the stamp of --arcface, which is
# the whole point of the bake-off being safe to run.
stamp = embedder_stamp(resolve_arcface(args.models_dir, args.arcface))
out_actors = []
n_ok = n_nodir = n_noemb = 0
@@ -84,7 +90,7 @@ def main():
print(f" [{i}/{total}] ok={n_ok} no_dir={n_nodir} no_emb={n_noemb}",
file=sys.stderr)
save_gallery_hdf5({"actors": out_actors}, Path(args.out))
save_gallery_hdf5({"actors": out_actors}, Path(args.out), stamp)
n_emb = sum(len(a["embeddings"]) for a in out_actors)
print(f"[reembed] {Path(args.arcface).stem}: {n_ok}/{total} actors, {n_emb} embeddings "
f"→ {args.out}", file=sys.stderr)
scripts/optimizer/replay.py
+35 -1
View File
@@ -2,6 +2,8 @@
"""
replay.py — replay a dumped embedding HDF5 through the real KPN downstream nodes.
TRACES: VR-002 | PR-002
Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an
EmbeddedSceneFrame into a Python-assembled KPN network wiring the *real* C++
face_tracker → identity_matcher → scene_tracker, and returns the same presence-window
@@ -25,6 +27,22 @@ import h5py
import numpy as np
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
from sae_stamp import verify_gallery_stamp # noqa: E402
def dump_embedder_stamp(dump_path: str) -> dict:
"""The GR-004 embedder stamp recorded in an embedding dump.
A replay has no live embedder — the dump IS the embedder as far as the gallery
is concerned, so the dump's stamp is what the gallery must be checked against.
Dumps written before GR-004 have no attributes and yield an empty stamp, which
the check reports as unverifiable rather than silently accepting."""
with h5py.File(dump_path, "r") as f:
name = f.attrs.get("embedder_model", "")
sha = f.attrs.get("embedder_sha256", "")
dec = lambda v: v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
return {"model_name": dec(name), "model_sha256": dec(sha), "embed_dim": 512}
def load_frames(dump_path: str, min_conf: float = 0.0):
@@ -92,6 +110,15 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool =
sys.path.insert(0, build_dir)
import sae_kpn
# TRACES: GR-004 | SR-001 — checked here, before any network is built, so a
# cross-model replay dies with one readable error instead of producing a
# plausible-looking score. add_identity_matcher re-checks it C++-side below;
# that is the backstop for any other caller of the binding.
stamp = dump_embedder_stamp(dump_path)
verify_gallery_stamp(gallery, stamp=stamp,
embedder_desc=f"embedding dump {Path(dump_path).name}",
require_stamp=bool(cfg.get("require_gallery_stamp", False)))
frames, movie, fps = load_frames(dump_path, min_conf=float(cfg.get("detector_conf", 0.0)))
net = sae_kpn.Network()
@@ -122,7 +149,8 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool =
cap = len(frames) * 2 + 64
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], cap)
sae_kpn.add_face_tracker(net, "tracker", cfg, cap)
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap)
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap,
stamp["model_name"], stamp["model_sha256"])
sae_kpn.add_scene_tracker(net, "scene", cfg, cap)
net.connect("replay", 0, "tracker", 0)
net.connect("tracker", 0, "matcher", 0)
@@ -221,11 +249,17 @@ def main():
# per-film gallery expansion: promotes pose-varied views of confidently-identified
# actors into an in-memory annex, recovering ~+4 recall at no precision cost.
p.add_argument("--expand-gallery", action="store_true")
# TRACES: GR-004 | SR-001 — promote an unprovable gallery/dump binding from a
# loud warning to a hard error. Measurement sweeps should set this (or
# SAE_REQUIRE_GALLERY_STAMP=1) so no number comes from an unbound pair.
p.add_argument("--require-gallery-stamp", action="store_true")
args = p.parse_args()
cfg = {k: getattr(args, k) for k in CFG_KEYS if getattr(args, k) is not None}
if args.expand_gallery:
cfg["expand_gallery"] = True
if args.require_gallery_stamp:
cfg["require_gallery_stamp"] = True
# stop=True: PyNode::stop() sets stop_flag_ before joining, so the source
# thread's run_loop actually exits. stop=False skips that, leaving stop_flag_
# false forever — the PyNode destructor's jthread.join() then blocks forever
scripts/optimizer/second_score.py
+2
View File
@@ -2,6 +2,8 @@
"""
second_score.py — uniform per-second agreement with X-Ray.
TRACES: VR-003 | PR-002
Unlike scene_score.py (which unions our detections over a whole X-Ray scene), this
samples EVERY SECOND of the film and asks: at second t, do we name the same actors
X-Ray says are on screen?
scripts/run_from_jellyfin.py
+2
View File
@@ -1,6 +1,8 @@
#!/usr/bin/env python3
"""run_from_jellyfin.py — resolve a Jellyfin title to its media file and run scene_analyze.
TRACES: IR-006 | SR-001
Looks up a Movie/Episode in Jellyfin, reads its on-disk Path (Jellyfin and this
tool must share the same media mount), filters the gallery down to that
title's credited cast (via filter_gallery's logic, fewer look-alike
scripts/sae_embed_loader.py
+16 -1
View File
@@ -3,11 +3,26 @@
sae_embed.FaceEmbedder loads both ONNX sessions once and exposes an
embed(path) -> FaceResult method, avoiding the per-process model reload cost
of spawning the embed_faces CLI binary for every image.
resolve_arcface() exposes the same default-resolution logic load_embedder uses,
so a caller can stamp the gallery it is about to write with the model that
actually produced its embeddings (GR-004) — the resolved path, not the CLI
argument, which is often None.
"""
import sys
from pathlib import Path
DEFAULT_ARCFACE = "arcface_w600k_r50.onnx"
def resolve_arcface(models_dir: str, arcface: str | None = None) -> str:
"""The ArcFace/LVFace ONNX path load_embedder would use for these arguments.
TRACES: GR-004 | SR-001 — single source of truth for "which model is this",
so the stamp written into a gallery can never drift from the model loaded."""
return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)
def load_embedder(build_dir: str, models_dir: str, arcface: str | None = None,
conf: float = 0.5, nms: float = 0.4, max_side: int = 500):
@@ -28,7 +43,7 @@ def load_embedder(build_dir: str, models_dir: str, arcface: str | None = None,
models_path = Path(models_dir)
detector_path = str(models_path / "scrfd_500m_bnkps.onnx")
arcface_path = arcface if arcface else str(models_path / "arcface_w600k_r50.onnx")
arcface_path = resolve_arcface(models_dir, arcface)
for model, name in [(detector_path, "SCRFD"), (arcface_path, "ArcFace")]:
if not Path(model).is_file():
sys.exit(f"{name} model not found: {model}\nRun: bash scripts/download_models.sh")
scripts/sae_gallery.py
+60 -10
View File
@@ -6,9 +6,13 @@ make_jellyfin_gallery.download_urls + download_person_images) and the duplicated
Galleries are written directly as HDF5 — never JSON. Same layout the C++ side
reads/writes (src/gallery/gallery_store.cpp): flat [N,512] embeddings + per-actor
offset/count, parallel imdb_id/tmdb_id/jellyfin_id/name string arrays, and a
per-embedding-row source_images array. calibration is left absent (calib_hash=0);
the C++ identity_matcher fits and writes it back into the file on first use.
offset/count, parallel imdb_id/tmdb_id/jellyfin_id/name string arrays, a
per-embedding-row source_images array, and an /embedder group carrying the
GR-004 model binding. calibration is left absent (calib_hash=0); the C++
identity_matcher fits and writes it back into the file on first use.
The GR-004 embedder stamp written into that /embedder group lives in sae_stamp
and is re-exported below, so existing callers keep importing it from here.
"""
import io
@@ -101,11 +105,36 @@ def download_images(urls: list[str], dest_dir: Path, n: int,
return paths
def save_gallery_hdf5(gallery: dict, output: Path) -> None:
# ── GR-004: gallery ↔ embedder binding ───────────────────────────────────────
# Implemented in sae_stamp (kept dependency-light so the optimizer's replay
# subprocesses can import it without pulling requests/Pillow); re-exported here
# because the gallery writers and every existing caller reach for it via this
# module. See src/gallery/embedder_stamp.hpp for the C++ twin and the rationale.
from sae_stamp import ( # noqa: F401
EmbedderMismatch,
check_embedder_stamp,
describe_stamp,
embedder_stamp,
enforce_embedder_stamp,
read_gallery_stamp,
require_gallery_stamp_from_env,
sha256_file,
verify_gallery_stamp,
_as_str,
_stamp_empty,
)
def save_gallery_hdf5(gallery: dict, output: Path, embedder: dict | None = None) -> None:
"""Write a gallery dict ({"actors": [...]}) directly as HDF5 — same schema
src/gallery/gallery_store.cpp reads/writes. No calibration group; the
C++ identity_matcher computes and writes it back into this file on first
use against an unseen set of embeddings."""
use against an unseen set of embeddings.
`embedder` is the GR-004 stamp (see embedder_stamp()); it may also be carried
on the gallery dict under "embedder", which is how a filtered/derived gallery
keeps its binding without the caller having to re-hash anything."""
embedder = embedder if embedder is not None else gallery.get("embedder")
actors = gallery["actors"]
embs, offsets, counts = [], [], []
imdb, tmdb, jf, name, src_images = [], [], [], [], []
@@ -139,8 +168,17 @@ def save_gallery_hdf5(gallery: dict, output: Path) -> None:
f.create_dataset("jellyfin_id", data=np.asarray(jf, dtype=object), dtype=str_t)
f.create_dataset("name", data=np.asarray(name, dtype=object), dtype=str_t)
f.create_dataset("source_images", data=np.asarray(src_images, dtype=object), dtype=str_t)
print(f"Saved: {output} ({len(actors)} actors, {emb_arr.shape[0]} embeddings)",
file=sys.stderr)
# TRACES: GR-004 | SR-001 — omitted entirely when unknown, so "unstamped"
# round-trips as unstamped rather than as a stamp naming no model.
if not _stamp_empty(embedder):
g = f.create_group("embedder")
g.attrs["model_name"] = embedder.get("model_name", "")
g.attrs["model_sha256"] = embedder.get("model_sha256", "")
g.attrs["embed_dim"] = np.int32(embedder.get("embed_dim", 512))
stamp_note = (f", embedder {embedder['model_name']}" if not _stamp_empty(embedder)
else ", NO EMBEDDER STAMP (GR-004)")
print(f"Saved: {output} ({len(actors)} actors, {emb_arr.shape[0]} embeddings"
f"{stamp_note})", file=sys.stderr)
def load_gallery_hdf5(path: Path) -> dict:
@@ -158,6 +196,14 @@ def load_gallery_hdf5(path: Path) -> dict:
if "source_images" in f:
src_images = [s.decode() if isinstance(s, bytes) else s
for s in f["source_images"][:]]
# TRACES: GR-004 | SR-001 — carried through so a derived gallery (filter,
# merge, cast-restrict) keeps the binding of the gallery it came from.
stamp = None
if "embedder" in f:
a = f["embedder"].attrs
stamp = {"model_name": _as_str(a.get("model_name", "")),
"model_sha256": _as_str(a.get("model_sha256", "")),
"embed_dim": int(a.get("embed_dim", 512))}
actors = []
for a in range(len(offset)):
@@ -167,15 +213,19 @@ def load_gallery_hdf5(path: Path) -> dict:
if src_images is not None:
actor["source_images"] = [src_images[s + i] for i in range(n)]
actors.append(actor)
return {"actors": actors}
out = {"actors": actors}
if stamp is not None:
out["embedder"] = stamp
return out
def save_gallery(gallery: dict, missing: list[dict], output: Path) -> None:
def save_gallery(gallery: dict, missing: list[dict], output: Path,
embedder: dict | None = None) -> None:
"""Write the gallery as HDF5 (forcing a .h5 extension) and, if any actors
lack images, a .missing_images.json sidecar."""
if output.suffix not in (".h5", ".hdf5"):
output = output.with_suffix(".h5")
save_gallery_hdf5(gallery, output)
save_gallery_hdf5(gallery, output, embedder)
if missing:
missing_path = output.with_name(output.stem + ".missing_images.json")
scripts/sae_stamp.py
+217
View File
@@ -0,0 +1,217 @@
"""Gallery ↔ embedder model binding (GR-004).
TRACES: GR-004 | SR-001
Python twin of src/gallery/embedder_stamp.{hpp,cpp}; the two implement the same
comparison rules and must stay in agreement. Kept as its own module — rather than
folded into sae_gallery — because scripts/optimizer/replay.py imports it once per
replay subprocess, thousands of times in a DE sweep, and must not pay for
sae_gallery's requests/Pillow imports to ask "were these made by the same model?".
Dependencies here are hashlib, json and h5py, all of which a replay already loads.
A gallery is only valid for the embedder that built it: cosine similarities across
models are meaningless but look plausible, so the mistake is silent and every
measurement taken afterwards is suspect. Identity = model filename + SHA-256 of
the model file. The hash decides (a model re-exported in place keeps its name but
not its bytes); the name is what makes the error readable. See
src/gallery/embedder_stamp.hpp for the full rationale.
"""
import hashlib
import json
import os
import sys
from pathlib import Path
import h5py
def _as_str(v) -> str:
return v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
_STAMP_CACHE: dict = {}
class EmbedderMismatch(RuntimeError):
"""Gallery was built with a different embedder than the one about to be used."""
def sha256_file(path) -> str:
"""Lowercase hex SHA-256 of a file's bytes; "" if it cannot be read."""
path = Path(path)
try:
st = path.stat()
except OSError:
return ""
key = (str(path), st.st_mtime_ns, st.st_size)
if key in _STAMP_CACHE:
return _STAMP_CACHE[key]
h = hashlib.sha256()
try:
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
except OSError:
return ""
_STAMP_CACHE[key] = h.hexdigest()
return _STAMP_CACHE[key]
def embedder_stamp(model_path, embed_dim: int = 512) -> dict:
"""Identify an embedder model file → {"model_name", "model_sha256", "embed_dim"}.
A model file that is absent (e.g. a TRT deployment running from a prebuilt
.engine) yields a name-only stamp: still comparable, just not provable."""
if not model_path:
return {"model_name": "", "model_sha256": "", "embed_dim": embed_dim}
sha = sha256_file(model_path)
if not sha:
print(f"[gallery] cannot hash embedder model {model_path} — model binding "
f"falls back to filename only (GR-004)", file=sys.stderr)
return {"model_name": Path(model_path).name, "model_sha256": sha,
"embed_dim": embed_dim}
def _stamp_empty(s) -> bool:
return not s or (not s.get("model_name") and not s.get("model_sha256"))
def describe_stamp(s) -> str:
if _stamp_empty(s):
return "UNKNOWN"
name = s.get("model_name") or "<unnamed model>"
sha = s.get("model_sha256") or ""
return f"{name} (sha256 {sha[:12]}…)" if sha else f"{name} (sha256 unavailable)"
def require_gallery_stamp_from_env() -> bool:
"""SAE_REQUIRE_GALLERY_STAMP=1 → an unprovable binding is fatal, not a warning."""
return os.environ.get("SAE_REQUIRE_GALLERY_STAMP", "0") not in ("", "0")
def check_embedder_stamp(built_with: dict | None, loading_with: dict | None,
gallery_desc: str = "gallery",
embedder_desc: str = "embedder") -> tuple[str, str]:
"""Pure comparison. Returns (verdict, message); verdict is one of
match / weak_match / unstamped / unknown_embedder / mismatch.
Same rules as compare_embedder_stamps() in src/gallery/embedder_stamp.cpp."""
if _stamp_empty(built_with):
return "unstamped", (
f"gallery '{gallery_desc}' carries no embedder stamp (GR-004).\n"
f" gallery was built with : UNKNOWN — this file predates model binding\n"
f" embedder now loaded : {describe_stamp(loading_with)} [{embedder_desc}]\n"
f" If these are not the same model every similarity from this run is\n"
f" meaningless but will look plausible. Rebuild or re-stamp the gallery\n"
f" (scripts/stamp_gallery.py), or run with SAE_REQUIRE_GALLERY_STAMP=1 to\n"
f" make this a hard error.")
if _stamp_empty(loading_with):
return "unknown_embedder", (
f"cannot identify the embedder being used against gallery "
f"'{gallery_desc}' (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)}\n"
f" embedder now loaded : UNKNOWN [{embedder_desc}]\n"
f" The binding cannot be checked, so it is not being checked.")
mismatch_tail = (
" Cosine similarities between embeddings from different models are\n"
" meaningless but look plausible. Rebuild the gallery with the loaded\n"
" model, or point the embedder at the model the gallery was built with.")
if int(built_with.get("embed_dim", 512)) != int(loading_with.get("embed_dim", 512)):
return "mismatch", (
f"gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)}, "
f"dim={built_with.get('embed_dim')} [{gallery_desc}]\n"
f" embedder now loaded : {describe_stamp(loading_with)}, "
f"dim={loading_with.get('embed_dim')} [{embedder_desc}]\n"
f" Embedding dimensions differ; these are not the same space.")
a, b = built_with.get("model_sha256", ""), loading_with.get("model_sha256", "")
if a and b:
if a == b:
note = ""
if built_with.get("model_name") != loading_with.get("model_name"):
note = (f" (gallery recorded it as '{built_with.get('model_name')}', "
f"loaded from '{loading_with.get('model_name')}' — "
f"same bytes, renamed file)")
return "match", f"embedder binding verified: {describe_stamp(built_with)}{note}"
return "mismatch", (
f"gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
f" gallery was built with : {built_with.get('model_name')} sha256={a}\n"
f" [{gallery_desc}]\n"
f" embedder now loaded : {loading_with.get('model_name')} sha256={b}\n"
f" [{embedder_desc}]\n" + mismatch_tail)
if built_with.get("model_name") and \
built_with.get("model_name") == loading_with.get("model_name"):
return "weak_match", (
f"embedder binding UNPROVEN for gallery '{gallery_desc}' (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)}\n"
f" embedder now loaded : {describe_stamp(loading_with)} [{embedder_desc}]\n"
f" Filenames agree but at least one SHA-256 is unavailable, so an\n"
f" in-place re-export under the same name would not be detected.")
return "mismatch", (
f"gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)} [{gallery_desc}]\n"
f" embedder now loaded : {describe_stamp(loading_with)} [{embedder_desc}]\n"
+ mismatch_tail)
def enforce_embedder_stamp(built_with, loading_with, gallery_desc, embedder_desc,
require_stamp: bool = False) -> str:
"""Apply check_embedder_stamp: raise EmbedderMismatch when fatal, else warn.
A mismatch is fatal unconditionally — there is no bypass, because a mismatch is
a known-wrong state, not an unknown one. The three "cannot prove it" verdicts
warn loudly and become fatal under require_stamp / SAE_REQUIRE_GALLERY_STAMP."""
strict = require_stamp or require_gallery_stamp_from_env()
verdict, msg = check_embedder_stamp(built_with, loading_with,
gallery_desc, embedder_desc)
if verdict == "mismatch":
raise EmbedderMismatch(msg)
if strict and verdict != "match":
raise EmbedderMismatch(
msg + "\n (fatal because SAE_REQUIRE_GALLERY_STAMP is set)")
if verdict == "match":
print(f"[gallery] {msg}", file=sys.stderr)
else:
print(f"\n[gallery] ***** WARNING (GR-004) *****\n{msg}\n"
f"[gallery] ****************************\n", file=sys.stderr)
return verdict
def read_gallery_stamp(path) -> dict | None:
"""The embedder stamp recorded in a gallery file, or None if unstamped.
Handles both the HDF5 /embedder group and the legacy JSON "embedder" object."""
path = Path(path)
if path.suffix in (".h5", ".hdf5"):
with h5py.File(path, "r") as f:
if "embedder" not in f:
return None
a = f["embedder"].attrs
return {"model_name": _as_str(a.get("model_name", "")),
"model_sha256": _as_str(a.get("model_sha256", "")),
"embed_dim": int(a.get("embed_dim", 512))}
data = json.loads(path.read_text())
return data.get("embedder") or None
def verify_gallery_stamp(gallery_path, model_path=None, *, stamp=None,
embedder_desc: str | None = None,
require_stamp: bool = False) -> str:
"""Load a gallery's stamp and check it against a model file (or an explicit
stamp, e.g. one read off an embedding dump). Raises EmbedderMismatch."""
loading = stamp if stamp is not None else embedder_stamp(model_path)
return enforce_embedder_stamp(read_gallery_stamp(gallery_path), loading,
str(gallery_path),
embedder_desc or str(model_path or "unknown"),
require_stamp)
def _as_str(v) -> str:
return v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
scripts/stamp_gallery.py
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""stamp_gallery.py — bind an existing gallery to the embedder that built it.
TRACES: GR-004 | SR-001
Galleries built before model binding carry no embedder stamp. They still load,
but every consumer warns that it cannot tell whether the gallery and the embedder
belong together — and under SAE_REQUIRE_GALLERY_STAMP=1 they refuse to run.
This is the migration path, and the reason the unstamped case is a warning rather
than a hard failure: re-binding an existing gallery costs one command and no
re-embedding, so nobody has to choose between a bricked setup and a check they
route around.
python scripts/stamp_gallery.py --gallery gallery.h5 \\
--arcface models/LVFace-B_Glint360K.onnx
The stamp is an ASSERTION: you are stating which model produced these vectors.
Nothing can verify it from the vectors themselves, which is exactly why the stamp
has to be written at build time going forward. Stamping the wrong model is worse
than leaving it unstamped, because it converts a loud warning into a false
all-clear — so --show it first if you are not certain.
python scripts/stamp_gallery.py --gallery gallery.h5 --show
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
import h5py
sys.path.insert(0, str(Path(__file__).resolve().parent))
from sae_gallery import (describe_stamp, embedder_stamp, # noqa: E402
read_gallery_stamp)
def main() -> int:
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--gallery", required=True, help="gallery .h5 to stamp in place")
p.add_argument("--arcface", help="the ONNX that built it (hashed into the stamp)")
p.add_argument("--show", action="store_true", help="print the current stamp and exit")
p.add_argument("--force", action="store_true",
help="overwrite an existing stamp (refused otherwise)")
args = p.parse_args()
path = Path(args.gallery)
if path.suffix not in (".h5", ".hdf5"):
return err(f"{path}: only HDF5 galleries can be stamped in place")
current = read_gallery_stamp(path)
print(f"{path}: current stamp = {describe_stamp(current)}", file=sys.stderr)
if args.show:
return 0
if not args.arcface:
return err("--arcface is required (or use --show)")
if current and not args.force:
return err("gallery is already stamped — pass --force to overwrite, but be "
"sure: a wrong stamp turns a warning into a false all-clear")
stamp = embedder_stamp(args.arcface)
if not stamp["model_sha256"]:
return err(f"cannot hash {args.arcface} — refusing to write a name-only "
"stamp, which would claim more certainty than it has")
with h5py.File(path, "r+") as f:
if "embedder" in f:
del f["embedder"]
g = f.create_group("embedder")
g.attrs["model_name"] = stamp["model_name"]
g.attrs["model_sha256"] = stamp["model_sha256"]
g.attrs["embed_dim"] = stamp["embed_dim"]
print(f"{path}: stamped with {describe_stamp(stamp)}", file=sys.stderr)
return 0
def err(msg: str) -> int:
print(f"[stamp_gallery] {msg}", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())
scripts/vendor/jray-project
Vendored Submodule
+1
Submodule scripts/vendor/jray-project added at 17106f3370
src/audio_signature.cpp
+428
View File
@@ -0,0 +1,428 @@
// ── JRay audio signature, v1 — implementation ────────────────────────────────
//
/// TRACES: IR-004, IR-007, IR-008 | SR-003
//
// The contract this implements is documented in full in audio_signature.hpp;
// read that before changing anything here. Every constant is load-bearing: the
// JRay Jellyfin plugin computes the same bytes in C#, and a signature that
// differs in any parameter simply does not match.
//
// Audio decode is a *second stream from an existing dependency* — the pipeline
// already links libavformat/libavcodec/libavutil for video (ffmpeg_decoder.hpp);
// this adds libswresample for the downmix+resample, no new project dependency.
// The FFT is written out here rather than pulled from a library for the same
// reason the plugin vendors one: it is a fixed, fully specified transform, and
// a dependency whose version could change the numerics is a liability when the
// output has to be bit-identical across two languages.
#include "audio_signature.hpp"
extern "C" {
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libavutil/avutil.h>
#include <libavutil/channel_layout.h>
#include <libavutil/opt.h>
#include <libavutil/samplefmt.h>
#include <libswresample/swresample.h>
}
#include <algorithm>
#include <cmath>
#include <cstring>
#include <vector>
namespace sae::audio {
namespace {
constexpr double kPi = 3.14159265358979323846;
// ── Band table ───────────────────────────────────────────────────────────────
// edge[b] = 300 * 10^(b/32); band b owns FFT bins [k_lo[b], k_lo[b+1]).
// ceil() of the edge in bins, so membership is decided once by integers rather
// than by a float comparison per bin per frame. The bands tile [112, 1115)
// contiguously with no gap and no overlap, which is what lets the frame energy
// below be accumulated from the per-band sums.
std::array<std::pair<int, int>, kNumBands> build_band_table() {
const double hz_per_bin = static_cast<double>(kSampleRate) / kFrameSize;
std::array<int, kNumBands + 1> k{};
for (int b = 0; b <= kNumBands; ++b) {
const double edge = kBandLoHz * std::pow(kBandHiHz / kBandLoHz,
static_cast<double>(b) / kNumBands);
k[b] = static_cast<int>(std::ceil(edge / hz_per_bin));
}
std::array<std::pair<int, int>, kNumBands> tbl{};
for (int b = 0; b < kNumBands; ++b) tbl[b] = {k[b], k[b + 1]};
return tbl;
}
// Hann, periodic: w[n] = 0.5 * (1 - cos(2*pi*n/N)). Not the symmetric (N-1)
// variant — the two differ, and the difference is observable.
const std::vector<double>& hann_window() {
static const std::vector<double> w = [] {
std::vector<double> v(kFrameSize);
for (int n = 0; n < kFrameSize; ++n)
v[n] = 0.5 * (1.0 - std::cos(2.0 * kPi * n / kFrameSize));
return v;
}();
return w;
}
// ── Radix-2 decimation-in-time complex FFT, in place, no normalisation ──────
// Twiddles are precomputed per stage from cos/sin of -2*pi*j/len so the angle
// is an exactly reproducible double in any language and only the libm rounding
// of cos/sin (≤1 ulp) can differ — orders of magnitude below the decision
// margins in the golden fixture.
struct FftTables {
std::vector<int> rev; // bit-reversal permutation
std::vector<std::vector<double>> wr, wi; // per stage
};
const FftTables& fft_tables() {
static const FftTables t = [] {
FftTables f;
f.rev.resize(kFrameSize);
int bits = 0;
while ((1 << bits) < kFrameSize) ++bits;
for (int i = 0; i < kFrameSize; ++i) {
int r = 0;
for (int b = 0; b < bits; ++b)
if (i & (1 << b)) r |= 1 << (bits - 1 - b);
f.rev[i] = r;
}
for (int len = 2; len <= kFrameSize; len <<= 1) {
const int half = len / 2;
std::vector<double> cr(half), ci(half);
for (int j = 0; j < half; ++j) {
const double ang = -2.0 * kPi * j / len;
cr[j] = std::cos(ang);
ci[j] = std::sin(ang);
}
f.wr.push_back(std::move(cr));
f.wi.push_back(std::move(ci));
}
return f;
}();
return t;
}
void fft_4096(std::vector<double>& re, std::vector<double>& im) {
const FftTables& t = fft_tables();
for (int i = 0; i < kFrameSize; ++i) {
const int j = t.rev[i];
if (i < j) { std::swap(re[i], re[j]); std::swap(im[i], im[j]); }
}
int stage = 0;
for (int len = 2; len <= kFrameSize; len <<= 1, ++stage) {
const int half = len / 2;
const std::vector<double>& wr = t.wr[stage];
const std::vector<double>& wi = t.wi[stage];
for (int base = 0; base < kFrameSize; base += len) {
for (int j = 0; j < half; ++j) {
const int a = base + j;
const int b = a + half;
const double tr = re[b] * wr[j] - im[b] * wi[j];
const double ti = re[b] * wi[j] + im[b] * wr[j];
re[b] = re[a] - tr; im[b] = im[a] - ti;
re[a] = re[a] + tr; im[a] = im[a] + ti;
}
}
}
}
int energy_class(double r) {
if (r < kEnergyClassEdges[0]) return 0;
if (r < kEnergyClassEdges[1]) return 1;
if (r < kEnergyClassEdges[2]) return 2;
return 3;
}
// ── FFmpeg RAII ─────────────────────────────────────────────────────────────
struct DecodeCtx {
AVFormatContext* fmt = nullptr;
AVCodecContext* dec = nullptr;
SwrContext* swr = nullptr;
AVFrame* frm = nullptr;
AVPacket* pkt = nullptr;
~DecodeCtx() {
if (swr) swr_free(&swr);
if (frm) av_frame_free(&frm);
if (pkt) av_packet_free(&pkt);
if (dec) avcodec_free_context(&dec);
if (fmt) avformat_close_input(&fmt);
}
};
bool open_resampler(DecodeCtx& c, const AVFrame* f) {
#if LIBAVUTIL_VERSION_INT >= AV_VERSION_INT(57, 24, 100)
AVChannelLayout out_layout;
av_channel_layout_default(&out_layout, 1); // mono
AVChannelLayout in_layout;
if (av_channel_layout_copy(&in_layout, &f->ch_layout) < 0) return false;
if (in_layout.nb_channels <= 0) {
av_channel_layout_uninit(&in_layout);
av_channel_layout_default(&in_layout, 1);
}
const int rc = swr_alloc_set_opts2(
&c.swr,
&out_layout, AV_SAMPLE_FMT_FLT, kSampleRate,
&in_layout, static_cast<AVSampleFormat>(f->format),
f->sample_rate ? f->sample_rate : kSampleRate,
0, nullptr);
av_channel_layout_uninit(&in_layout);
av_channel_layout_uninit(&out_layout);
if (rc < 0 || !c.swr) return false;
#else
const int64_t in_layout = f->channel_layout
? static_cast<int64_t>(f->channel_layout)
: av_get_default_channel_layout(f->channels ? f->channels : 1);
c.swr = swr_alloc_set_opts(
nullptr,
AV_CH_LAYOUT_MONO, AV_SAMPLE_FMT_FLT, kSampleRate,
in_layout, static_cast<AVSampleFormat>(f->format),
f->sample_rate ? f->sample_rate : kSampleRate,
0, nullptr);
if (!c.swr) return false;
#endif
return swr_init(c.swr) >= 0;
}
// Push one decoded frame (or a flush) through the resampler, dropping the
// leading `to_skip` output samples, and append to `out`.
void drain(SwrContext* swr, const AVFrame* f, int in_rate,
std::size_t& to_skip, std::vector<float>& out) {
const int64_t delay = swr_get_delay(swr, in_rate ? in_rate : kSampleRate);
const int in_n = f ? f->nb_samples : 0;
const int max_out = static_cast<int>(av_rescale_rnd(
delay + in_n, kSampleRate, in_rate ? in_rate : kSampleRate, AV_ROUND_UP)) + 32;
if (max_out <= 0) return;
std::vector<float> buf(static_cast<std::size_t>(max_out));
uint8_t* dst = reinterpret_cast<uint8_t*>(buf.data());
const int n = swr_convert(swr, &dst, max_out,
f ? const_cast<const uint8_t**>(f->extended_data) : nullptr,
in_n);
if (n <= 0) return;
std::size_t produced = static_cast<std::size_t>(n);
std::size_t off = 0;
if (to_skip) {
const std::size_t drop = std::min(to_skip, produced);
to_skip -= drop;
off = drop;
produced -= drop;
}
if (produced)
out.insert(out.end(), buf.begin() + off, buf.begin() + off + produced);
}
} // namespace
// ── Public surface ──────────────────────────────────────────────────────────
const std::array<std::pair<int, int>, kNumBands>& band_fft_bins() {
static const std::array<std::pair<int, int>, kNumBands> tbl = build_band_table();
return tbl;
}
std::string base64_encode(const std::uint8_t* data, std::size_t n) {
static constexpr char kAlphabet[] =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
std::string out;
out.reserve(((n + 2) / 3) * 4);
std::size_t i = 0;
for (; i + 3 <= n; i += 3) {
const std::uint32_t v = (std::uint32_t(data[i]) << 16) |
(std::uint32_t(data[i + 1]) << 8) |
std::uint32_t(data[i + 2]);
out += kAlphabet[(v >> 18) & 0x3F];
out += kAlphabet[(v >> 12) & 0x3F];
out += kAlphabet[(v >> 6) & 0x3F];
out += kAlphabet[v & 0x3F];
}
if (i < n) {
const bool two = (n - i) == 2;
const std::uint32_t v = (std::uint32_t(data[i]) << 16) |
(two ? (std::uint32_t(data[i + 1]) << 8) : 0u);
out += kAlphabet[(v >> 18) & 0x3F];
out += kAlphabet[(v >> 12) & 0x3F];
out += two ? kAlphabet[(v >> 6) & 0x3F] : '=';
out += '=';
}
return out;
}
std::uint64_t fnv1a64(const void* data, std::size_t n) {
const auto* p = static_cast<const std::uint8_t*>(data);
std::uint64_t h = 0xcbf29ce484222325ULL;
for (std::size_t i = 0; i < n; ++i) {
h ^= p[i];
h *= 0x100000001b3ULL;
}
return h;
}
/// TRACES: IR-004
std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono) {
if (mono.size() < static_cast<std::size_t>(kFrameSize)) return {};
const std::size_t nframes = 1 + (mono.size() - kFrameSize) / kHopSize;
const auto& bands = band_fft_bins();
const auto& win = hann_window();
const int k_lo = bands.front().first;
const int k_hi = bands.back().second; // exclusive
const double bin_count = static_cast<double>(k_hi - k_lo);
std::vector<double> re(kFrameSize), im(kFrameSize);
std::vector<std::uint8_t> peak(nframes);
std::vector<double> energy(nframes);
for (std::size_t f = 0; f < nframes; ++f) {
const float* src = mono.data() + f * kHopSize;
for (int n = 0; n < kFrameSize; ++n) {
re[n] = static_cast<double>(src[n]) * win[n];
im[n] = 0.0;
}
fft_4096(re, im);
// Per-band mean magnitude; the bands tile the 300–3000 Hz range with no
// gaps, so the frame's band-limited energy is the sum of the band sums.
double best = -1.0, total = 0.0;
int best_b = 0;
for (int b = 0; b < kNumBands; ++b) {
double sum = 0.0;
for (int k = bands[b].first; k < bands[b].second; ++k)
sum += std::sqrt(re[k] * re[k] + im[k] * im[k]);
total += sum;
const double mean = sum / (bands[b].second - bands[b].first);
if (mean > best) { best = mean; best_b = b; } // ties → lowest index
}
peak[f] = static_cast<std::uint8_t>(best_b);
energy[f] = total / bin_count;
}
// Reference is the upper median of the frame energies: an actually observed
// value (no averaging of the two middle samples), so it is bit-reproducible,
// gain-invariant and barely moves when the window is trimmed.
std::vector<double> sorted = energy;
std::sort(sorted.begin(), sorted.end());
const double ref = sorted[sorted.size() / 2];
std::vector<std::uint8_t> out(nframes);
for (std::size_t f = 0; f < nframes; ++f) {
const double r = std::log10((energy[f] + kEnergyEps) / (ref + kEnergyEps));
out[f] = static_cast<std::uint8_t>(((peak[f] & 0x1F) << 2) |
(energy_class(r) & 0x03));
}
return out;
}
/// TRACES: IR-004, IR-008
std::optional<std::string> signature_from_mono(const std::vector<float>& mono) {
const std::vector<std::uint8_t> packed = pack_frames(mono);
if (packed.empty()) return std::nullopt;
return std::string(kVersionPrefix) + base64_encode(packed.data(), packed.size());
}
/// TRACES: IR-004, IR-007
std::optional<std::vector<float>> decode_centre_window(const std::string& path) {
av_log_set_level(AV_LOG_ERROR);
DecodeCtx c;
if (avformat_open_input(&c.fmt, path.c_str(), nullptr, nullptr) < 0)
return std::nullopt;
if (avformat_find_stream_info(c.fmt, nullptr) < 0) return std::nullopt;
if (c.fmt->duration == AV_NOPTS_VALUE) return std::nullopt;
const double duration = static_cast<double>(c.fmt->duration) / AV_TIME_BASE;
// IR-007 — the window underflows, so there is no signature and no sync
// offset downstream. The plugin applies the identical rule.
if (duration < kWindowSec) return std::nullopt;
const int idx = av_find_best_stream(c.fmt, AVMEDIA_TYPE_AUDIO, -1, -1, nullptr, 0);
if (idx < 0) return std::nullopt; // no audio → no signature
AVStream* st = c.fmt->streams[idx];
const AVCodec* codec = avcodec_find_decoder(st->codecpar->codec_id);
if (!codec) return std::nullopt;
c.dec = avcodec_alloc_context3(codec);
if (!c.dec) return std::nullopt;
if (avcodec_parameters_to_context(c.dec, st->codecpar) < 0) return std::nullopt;
c.dec->thread_count = 0;
if (avcodec_open2(c.dec, codec, nullptr) < 0) return std::nullopt;
const double start_sec = duration / 2.0 - kWindowSec / 2.0;
// Seek to a packet at or before the window start; the exact start is then
// reached by discarding the leading output samples, which is what
// `ffmpeg -ss <t> -i <file>` does and therefore what the plugin sees.
if (start_sec > 0.0) {
const int64_t tgt = av_rescale_q(
static_cast<int64_t>(start_sec * AV_TIME_BASE), AV_TIME_BASE_Q, st->time_base);
if (av_seek_frame(c.fmt, idx, tgt, AVSEEK_FLAG_BACKWARD) >= 0)
avcodec_flush_buffers(c.dec);
}
c.frm = av_frame_alloc();
c.pkt = av_packet_alloc();
if (!c.frm || !c.pkt) return std::nullopt;
std::vector<float> mono;
mono.reserve(kWindowSamples + kSampleRate);
std::size_t to_skip = 0;
bool have_swr = false;
int in_rate = kSampleRate;
bool eof = false;
while (mono.size() < kWindowSamples && !eof) {
const int rr = av_read_frame(c.fmt, c.pkt);
if (rr < 0) {
eof = true;
avcodec_send_packet(c.dec, nullptr); // flush the decoder
} else if (c.pkt->stream_index != idx) {
av_packet_unref(c.pkt);
continue;
} else {
avcodec_send_packet(c.dec, c.pkt);
av_packet_unref(c.pkt);
}
while (avcodec_receive_frame(c.dec, c.frm) == 0) {
if (!have_swr) {
if (!open_resampler(c, c.frm)) return std::nullopt;
have_swr = true;
in_rate = c.frm->sample_rate ? c.frm->sample_rate : kSampleRate;
int64_t pts = c.frm->best_effort_timestamp;
if (pts == AV_NOPTS_VALUE) pts = c.frm->pts;
const double t0 = (pts == AV_NOPTS_VALUE)
? start_sec : av_q2d(st->time_base) * static_cast<double>(pts);
const double lead = start_sec - t0;
to_skip = lead > 0.0
? static_cast<std::size_t>(std::llround(lead * kSampleRate)) : 0;
}
drain(c.swr, c.frm, in_rate, to_skip, mono);
av_frame_unref(c.frm);
if (mono.size() >= kWindowSamples) break;
}
}
if (have_swr && mono.size() < kWindowSamples)
drain(c.swr, nullptr, in_rate, to_skip, mono); // flush the resampler
if (mono.empty()) return std::nullopt;
// Truncate to exactly 120.000 s so the frame count is 1288 for every input
// and does not wobble with seek granularity or the resampler tail.
if (mono.size() > kWindowSamples) mono.resize(kWindowSamples);
return mono;
}
/// TRACES: IR-004, IR-005, IR-007, IR-008
std::optional<std::string> compute_signature(const std::string& path) {
const std::optional<std::vector<float>> mono = decode_centre_window(path);
if (!mono) return std::nullopt;
return signature_from_mono(*mono);
}
} // namespace sae::audio
src/audio_signature.hpp
+158
View File
@@ -0,0 +1,158 @@
#pragma once
// ── JRay audio signature, v1 ─────────────────────────────────────────────────
//
/// TRACES: IR-004, IR-005, IR-007, IR-008 | SR-003
//
// A content-derived spectral-peak signature taken from the *centre* of the
// media, so a truth file is self-identifying: a consumer can tell whether a
// local file is the same cut as the one a manifest describes, and recover the
// frame offset when it is the same cut trimmed differently.
//
// The construction is owned by `JRay-public-server/SPEC.md` §3 and is
// reproduced by the JRay Jellyfin plugin in C#. **The two implementations must
// agree byte for byte** — a signature that differs in any parameter simply does
// not match, which defeats the entire point. Every deviation is therefore a
// breaking change and must go through the `v1:` prefix (see kVersionPrefix).
//
// Server spec §3, restated:
//
// 1. Decode a 120 s window centred on the midpoint (runtime/2 ± 60 s).
// 2. Downmix to mono, resample to 11025 Hz.
// 3. STFT: 4096-sample frame, 1024-sample hop, Hann window (~1290 frames).
// 4. Per frame, log-magnitude spectrum over 300–3000 Hz.
// 5. 32 logarithmically spaced bins; peak bin index + coarse 2-bit energy
// class.
// 6. Pack one byte per frame; base64-encode.
// 7. Prefix `v1:`.
//
// ── Details the server spec leaves open, pinned here for v1 ──────────────────
//
// The prose above is not sufficient to reproduce a byte stream, so the choices
// below are the contract. They are mirrored in
// `tests/fixtures/audio/jray_audio_v1_golden.json`, which is the artefact
// shared with the plugin repo (IR-005).
//
// Arithmetic All DSP in IEEE-754 **double**. float32 is not sufficient:
// the golden fixture has frames whose two strongest bands are
// within 1.3% of each other, which double resolves identically
// everywhere and float32 does not.
// Sample scale FFmpeg's native s16→flt conversion, x * (1/32768), then
// widened to double. Values in [-1, 1).
// Framing Only whole frames: n_frames = 1 + (n_samples - 4096) / 1024,
// integer division, 0 when n_samples < 4096. A 120.000 s
// window is 1 323 000 samples → **1288 frames**.
// ("~1290" in the spec; the server accepts a tolerance.)
// Window Hann, **periodic**: w[n] = 0.5 * (1 - cos(2*pi*n/4096)).
// Not the symmetric (N-1) variant.
// Transform Plain radix-2 decimation-in-time complex FFT over 4096 real
// samples (imag = 0), no normalisation. Magnitude is
// sqrt(re² + im²). Twiddles from cos/sin of
// -2*pi*k/len computed in double.
// Band edges edge[b] = 300 * (3000/300)^(b/32), b = 0..32. Band b spans
// FFT bins [k_lo[b], k_lo[b+1]) with
// k_lo[b] = ceil(edge[b] * 4096 / 11025) — i.e. bins 112..1114
// inclusive, 8 bins in the narrowest band. Precomputed as an
// integer table so no float comparison decides membership.
// Band value **Mean** of the linear magnitudes in the band. Mean, not
// sum, so a wide high band is not favoured over a narrow low
// one; magnitude, not power, because it is an energy proxy and
// more codec-robust than a single bin's peak.
// Peak bin argmax over the 32 band values; ties resolve to the **lowest
// index**. The log of step 4 is a monotone squash and so
// cannot change an argmax — it is applied only where it is
// observable, in the energy class below.
// Energy class The spec says "coarse 2-bit energy class" and no more. v1
// defines it as the frame's band-limited energy relative to
// the window, which is invariant to gain (loudness
// normalisation must not change a signature) and robust to
// trimming (the median barely moves):
// E_f = mean magnitude over *all* FFT bins 112..1114
// Eref = median over frames of E_f, taken as the upper
// median sorted[n/2] — no averaging of the two middle
// values, so the reference is always an actual
// observed value and is bit-reproducible
// r = log10((E_f + 1e-12) / (Eref + 1e-12))
// class = 0 if r < -0.6, 1 if r < -0.2, 2 if r < 0.2, else 3
// The thresholds deliberately straddle r = 0 rather than sit
// on it, so the median frame itself is not on a boundary.
// Byte layout bit 7 = 0 (reserved), bits 6..2 = 5-bit band index,
// bits 1..0 = 2-bit energy class:
// byte = (band << 2) | class → always 0..127
// This is the structural constraint the server validates on
// upload (§3 "Validation and abuse").
// Base64 Standard alphabet A–Za–z0–9+/ with '=' padding.
//
// ── Short media (IR-007) ─────────────────────────────────────────────────────
//
// `runtime/2 ± 60 s` underflows below 120 s, so **no signature is emitted** and
// no sync offset is applied downstream. Both producers apply the identical
// rule; diverging here would break exactly the short items most likely to be
// misidentified. `compute_signature` returns `std::nullopt`.
//
// The same nullopt is returned for a file with no audio stream, an unopenable
// file, or an unknown duration. UR-9 is an enhancement and must never be able
// to break a fetch — degradation, not failure.
#include <array>
#include <cstddef>
#include <cstdint>
#include <optional>
#include <string>
#include <utility>
#include <vector>
namespace sae::audio {
// ── Contract constants — changing any of these is a `v1:` bump ───────────────
inline constexpr int kSampleRate = 11025;
inline constexpr int kFrameSize = 4096;
inline constexpr int kHopSize = 1024;
inline constexpr int kNumBands = 32;
inline constexpr double kBandLoHz = 300.0;
inline constexpr double kBandHiHz = 3000.0;
inline constexpr double kWindowSec = 120.0;
inline constexpr double kEnergyEps = 1e-12;
// Class thresholds on log10(E_frame / E_median); see the header comment.
inline constexpr double kEnergyClassEdges[3] = {-0.6, -0.2, 0.2};
// 120.000 s at 11025 Hz. The decoded window is truncated to exactly this so the
// frame count does not wobble with seek granularity or resampler tail.
inline constexpr std::size_t kWindowSamples =
static_cast<std::size_t>(kWindowSec * kSampleRate); // 1 323 000
inline constexpr std::size_t kExpectedFrames =
1 + (kWindowSamples - kFrameSize) / kHopSize; // 1288
static_assert(kWindowSamples == 1323000, "120 s at 11025 Hz");
static_assert(kExpectedFrames == 1288, "server spec's ~1290 frames");
/// The version prefix is the signature's own, separate from `schema_version`:
/// a future change to the DSP chain must be *detectable* rather than silently
/// producing non-matching signatures (IR-008).
inline constexpr const char* kVersionPrefix = "v1:";
/// FFT bin range [first, last) for each of the 32 log-spaced bands.
/// Computed once from the constants above; exposed so the golden fixture can
/// assert the table itself, not merely the signature it produces.
const std::array<std::pair<int, int>, kNumBands>& band_fft_bins();
/// Decode the centre window of `path` as mono float PCM at 11025 Hz.
/// nullopt when the media is shorter than 120 s (IR-007), has no audio stream,
/// or cannot be opened. Never throws.
std::optional<std::vector<float>> decode_centre_window(const std::string& path);
/// One packed byte per whole STFT frame. Empty when `mono` is shorter than one
/// frame. This is the payload that gets base64-encoded.
std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono);
/// `v1:` + base64(pack_frames(mono)). nullopt when no whole frame fits.
std::optional<std::string> signature_from_mono(const std::vector<float>& mono);
/// Decode + sign. The one call the pipeline makes. nullopt per IR-007 and on
/// any decode failure — degradation, not failure.
std::optional<std::string> compute_signature(const std::string& path);
// ── Small utilities, exposed for the golden-fixture test ────────────────────
std::string base64_encode(const std::uint8_t* data, std::size_t n);
/// FNV-1a 64. Used only to pin the *decoded PCM* in the golden fixture, so a
/// codec-level difference is distinguishable from a DSP-level one.
std::uint64_t fnv1a64(const void* data, std::size_t n);
} // namespace sae::audio
src/config.hpp
+33 -15
View File
@@ -16,7 +16,13 @@ enum class Verbosity {
struct Config {
// ── Input ─────────────────────────────────────────────────────────────────
std::string movie_path;
std::string gallery_path; // gallery.json produced by build_gallery
std::string gallery_path;
// TRACES: IR-002 | SR-003
// "global" (matched against the whole library) or "limited" (this title's
// credited cast only). The strongest single quality signal when two
// manifests compete for the same cut: identical gallery_size can mean very
// different recall depending on which was used.
std::string gallery_scope{"global"}; // gallery.json produced by build_gallery
// ── Output ───────────────────────────────────────────────────────────────
std::string output_path; // annotations.json
@@ -40,6 +46,14 @@ struct Config {
float detector_conf{0.5f};
float detector_nms{0.4f};
/// TRACES: GR-004 | SR-001
// Gallery ↔ embedder binding. A gallery built with a different model than the
// one loaded here is a hard error, always. This flag additionally promotes
// "cannot prove they match" (unstamped legacy gallery, or a name-only match
// because the ONNX could not be hashed) from a loud warning to a hard error.
// Also settable via SAE_REQUIRE_GALLERY_STAMP=1. Measurement runs want it on.
bool require_gallery_stamp{false}; // --require-gallery-stamp
// ── Recognition (ArcFace ONNX) ────────────────────────────────────────────
std::string arcface_model;
std::string arcface_engine; // optional path to a pre-built TRT engine; bypasses ORT
@@ -86,21 +100,25 @@ struct Config {
float dense_scale{1.0f}; // dense-mode frame downscale (1 = off)
// ── Face tracking (frame-to-frame) ───────────────────────────────────────
float track_alpha{0.4f}; // cost weight: 0=embedding only, 1=spatial only
/// TRACES: AR-007, AR-008, AR-024 | SR-002
// track_alpha is the *base* weight, used on ordinary frames. It is
// frame-dependent (AR-007): on is_cut / is_scene_boundary, and for any track
// that is no longer on screen, it drops to 0 (embedding only), because
// position carries no information across a viewpoint change or a gap.
float track_alpha{0.4f}; // base cost weight: 0=embedding only, 1=spatial only
float track_min_iou{0.1f}; // IoU below which spatial link alone is rejected
float track_max_embed_dist{0.7f}; // cosine dist above which embedding link alone is rejected
int track_max_frames_missing{5}; // expire track after N consecutive missed frames
// ── Cross-cut track re-association ────────────────────────────────────────
// A camera-angle change (Frame::is_cut) breaks spatial (IoU) continuity but
// not identity: the same people are usually still on screen from a new angle.
// Instead of destroying tracks on a cut, the tracker parks them in an
// inactive pool. A post-cut detection whose raw cosine similarity to a parked
// track's last-frame embedding is ≥ cut_revive_sim revives that track_id
// (identity continuity survives the cut); otherwise it starts a fresh track.
// Parked tracks that go unrevived for cut_inactive_max_frames are dropped.
float cut_revive_sim{0.50f}; // min raw cosine sim (last-frame emb) to revive across a cut
int cut_inactive_max_frames{5}; // drop a parked track after N frames without revival
// Minimum P(same person) for an association to be admissible on appearance
// alone. This replaces track_max_embed_dist (a raw cosine distance, AR-024).
// 0.5 is not a tuned constant: it is the decision boundary. Below it the pair
// is more likely two people than one, and no amount of IoU makes that a link
// worth asserting on identity grounds.
float track_assoc_min_prob{0.5f};
// How long a track that has gone off screen stays available for association
// before the registry reaps it and emits its presence claim (AR-013).
// Replaces track_max_frames_missing: a frame count silently changed meaning
// with sample_fps, and the same number had to be guessed twice (once for an
// ordinary miss, once for a cut). Seconds mean one thing at any sample rate.
double track_extinction_sec{5.0};
// ── Scene tracking ────────────────────────────────────────────────────────
// extinction_sec re-tuned by DE against X-Ray per-second presence, 4-film rep4
src/evidence_discount.hpp
+85
View File
@@ -0,0 +1,85 @@
#pragma once
/// TRACES: AR-024, AR-025 | SR-002
///
/// EvidenceDiscounter — how much a single observation is allowed to move a
/// track's belief.
///
/// **The independence problem.** Per-frame identity evidence is accumulated as
/// log-odds along a track (AR-025), which is only valid for *independent*
/// observations. Consecutive frames of one track are nothing of the kind: near
/// identical pose, lighting and expression. Treating them as independent drives
/// the posterior to certainty on what is effectively one measurement — thirty
/// frames of the same face at the same angle is not thirty pieces of evidence.
///
/// The mitigation is to weight each observation by how much it *adds*: a view
/// the track has already contributed is discounted toward zero, a genuinely new
/// pose counts in full. This reuses the same judgement the diversity buffer
/// makes for gallery expansion (AR-019) — which embeddings on a track are
/// mutually distinct — rather than inventing a second notion of novelty.
///
/// Owned by TrackRegistry rather than left to callers. A caller that forgot to
/// discount, or applied it twice, would silently produce confident wrong
/// answers, and the registry is the one place where all evidence converges.
///
/// **Similarity enters as a calibrated probability, never a raw cosine**
/// (AR-024): "is this the same view" is a decision, and a bare cosine threshold
/// means something different for every model and every face size.
#include "types.hpp"
#include <algorithm>
#include <cmath>
#include <functional>
#include <vector>
class EvidenceDiscounter {
public:
/// cosine similarity → P(same view). Supplied by the caller so the
/// calibration fitted for the active embedder is used (AR-023/AR-024).
using Calibrate = std::function<float(float)>;
struct Config {
int max_views{8}; ///< distinct views remembered per track
float admit_below{0.6f}; ///< P(same view) under this ⇒ a new view
float floor{0.0f}; ///< minimum weight for a redundant observation
};
// Two constructors rather than a defaulted argument: `Config{}` as a default
// argument would reference Config's own member initializers before the
// enclosing class is complete, which is ill-formed.
explicit EvidenceDiscounter(Calibrate cal)
: cal_(std::move(cal)), cfg_() {}
EvidenceDiscounter(Calibrate cal, Config cfg)
: cal_(std::move(cal)), cfg_(cfg) {}
/// Weight in [0,1] for one observation, updating `views` when the
/// observation is novel enough to count as a distinct look at the subject.
///
/// The first observation on a track always counts in full: there is nothing
/// for it to be redundant with.
float weight(std::vector<Embedding>& views, const Embedding& e) const {
if (views.empty()) {
views.push_back(e);
return 1.0f;
}
float p_same = 0.0f;
for (const auto& v : views)
p_same = std::max(p_same, cal_(cosine_similarity(v, e)));
// Weight is the probability this is *not* a repeat of something already
// counted. A near-duplicate contributes ~0; an unseen pose ~1.
const float w = std::max(cfg_.floor, 1.0f - p_same);
if (p_same < cfg_.admit_below &&
static_cast<int>(views.size()) < cfg_.max_views) {
views.push_back(e);
}
return w;
}
private:
Calibrate cal_;
Config cfg_;
};
src/face_utils.hpp
+1
View File
@@ -1,4 +1,5 @@
#pragma once
/// TRACES: AR-005 | SR-002
#include "types.hpp"
#include <opencv2/calib3d.hpp>
src/gallery/embedder_stamp.cpp
+326
View File
@@ -0,0 +1,326 @@
/// TRACES: GR-004 | SR-001
#include "embedder_stamp.hpp"
#include "types.hpp"
#include <algorithm>
#include <cstdlib>
#include <cstring>
#include <filesystem>
#include <fstream>
#include <iostream>
#include <map>
#include <mutex>
#include <sstream>
#include <stdexcept>
#include <vector>
namespace fs = std::filesystem;
// ── SHA-256 (FIPS 180-4) ──────────────────────────────────────────────────────
// Self-contained rather than pulled from OpenSSL: the gallery library already
// links OpenCV, HDF5, FFmpeg and a GPU backend, and the unit tests deliberately
// link none of those crypto stacks. ~80 lines of table-driven code is cheaper
// than another find_package that CI has to satisfy on an Intel N100.
namespace {
struct Sha256 {
uint32_t h[8] = {0x6a09e667u, 0xbb67ae85u, 0x3c6ef372u, 0xa54ff53au,
0x510e527fu, 0x9b05688cu, 0x1f83d9abu, 0x5be0cd19u};
uint64_t len = 0;
uint8_t buf[64]{};
size_t buf_n = 0;
static uint32_t ror(uint32_t x, int n) { return (x >> n) | (x << (32 - n)); }
void block(const uint8_t* p) {
static const uint32_t k[64] = {
0x428a2f98u,0x71374491u,0xb5c0fbcfu,0xe9b5dba5u,0x3956c25bu,0x59f111f1u,
0x923f82a4u,0xab1c5ed5u,0xd807aa98u,0x12835b01u,0x243185beu,0x550c7dc3u,
0x72be5d74u,0x80deb1feu,0x9bdc06a7u,0xc19bf174u,0xe49b69c1u,0xefbe4786u,
0x0fc19dc6u,0x240ca1ccu,0x2de92c6fu,0x4a7484aau,0x5cb0a9dcu,0x76f988dau,
0x983e5152u,0xa831c66du,0xb00327c8u,0xbf597fc7u,0xc6e00bf3u,0xd5a79147u,
0x06ca6351u,0x14292967u,0x27b70a85u,0x2e1b2138u,0x4d2c6dfcu,0x53380d13u,
0x650a7354u,0x766a0abbu,0x81c2c92eu,0x92722c85u,0xa2bfe8a1u,0xa81a664bu,
0xc24b8b70u,0xc76c51a3u,0xd192e819u,0xd6990624u,0xf40e3585u,0x106aa070u,
0x19a4c116u,0x1e376c08u,0x2748774cu,0x34b0bcb5u,0x391c0cb3u,0x4ed8aa4au,
0x5b9cca4fu,0x682e6ff3u,0x748f82eeu,0x78a5636fu,0x84c87814u,0x8cc70208u,
0x90befffau,0xa4506cebu,0xbef9a3f7u,0xc67178f2u};
uint32_t w[64];
for (int i = 0; i < 16; ++i)
w[i] = (uint32_t(p[i * 4]) << 24) | (uint32_t(p[i * 4 + 1]) << 16) |
(uint32_t(p[i * 4 + 2]) << 8) | uint32_t(p[i * 4 + 3]);
for (int i = 16; i < 64; ++i) {
uint32_t s0 = ror(w[i - 15], 7) ^ ror(w[i - 15], 18) ^ (w[i - 15] >> 3);
uint32_t s1 = ror(w[i - 2], 17) ^ ror(w[i - 2], 19) ^ (w[i - 2] >> 10);
w[i] = w[i - 16] + s0 + w[i - 7] + s1;
}
uint32_t a = h[0], b = h[1], c = h[2], d = h[3];
uint32_t e = h[4], f = h[5], g = h[6], hh = h[7];
for (int i = 0; i < 64; ++i) {
uint32_t S1 = ror(e, 6) ^ ror(e, 11) ^ ror(e, 25);
uint32_t ch = (e & f) ^ (~e & g);
uint32_t t1 = hh + S1 + ch + k[i] + w[i];
uint32_t S0 = ror(a, 2) ^ ror(a, 13) ^ ror(a, 22);
uint32_t mj = (a & b) ^ (a & c) ^ (b & c);
uint32_t t2 = S0 + mj;
hh = g; g = f; f = e; e = d + t1;
d = c; c = b; b = a; a = t1 + t2;
}
h[0] += a; h[1] += b; h[2] += c; h[3] += d;
h[4] += e; h[5] += f; h[6] += g; h[7] += hh;
}
void update(const uint8_t* p, size_t n) {
len += n;
while (n) {
size_t take = std::min(n, size_t(64) - buf_n);
std::memcpy(buf + buf_n, p, take);
buf_n += take; p += take; n -= take;
if (buf_n == 64) { block(buf); buf_n = 0; }
}
}
std::string hex() {
uint64_t bits = len * 8;
uint8_t pad = 0x80;
update(&pad, 1);
uint8_t zero = 0;
while (buf_n != 56) update(&zero, 1);
uint8_t tail[8];
for (int i = 0; i < 8; ++i) tail[i] = uint8_t(bits >> (56 - i * 8));
// update() would re-count these into len, but len is already frozen in bits.
std::memcpy(buf + buf_n, tail, 8);
block(buf);
buf_n = 0;
static const char* d = "0123456789abcdef";
std::string out;
out.reserve(64);
for (int i = 0; i < 8; ++i)
for (int s = 28; s >= 0; s -= 4)
out += d[(h[i] >> s) & 0xF];
return out;
}
};
// (path, mtime, size) → digest. Hashing a 250 MB ONNX is cheap but not free, and
// the optimizer constructs many networks in one process against the same model.
std::mutex g_hash_mu;
std::map<std::string, std::string> g_hash_cache;
std::string short_hash(const std::string& hex) {
return hex.size() > 12 ? hex.substr(0, 12) + "…" : hex;
}
} // namespace
std::string sha256_hex(const std::string& bytes) {
Sha256 s;
s.update(reinterpret_cast<const uint8_t*>(bytes.data()), bytes.size());
return s.hex();
}
std::string sha256_file_hex(const std::string& path) {
if (path.empty()) return "";
std::error_code ec;
auto size = fs::file_size(path, ec);
if (ec) return "";
auto mtime = fs::last_write_time(path, ec);
if (ec) return "";
std::ostringstream key;
key << path << '|' << size << '|'
<< mtime.time_since_epoch().count();
{
std::lock_guard<std::mutex> lk(g_hash_mu);
auto it = g_hash_cache.find(key.str());
if (it != g_hash_cache.end()) return it->second;
}
std::ifstream f(path, std::ios::binary);
if (!f) return "";
Sha256 s;
std::vector<char> chunk(1 << 20);
while (f) {
f.read(chunk.data(), static_cast<std::streamsize>(chunk.size()));
std::streamsize got = f.gcount();
if (got > 0) s.update(reinterpret_cast<const uint8_t*>(chunk.data()),
static_cast<size_t>(got));
}
std::string hex = s.hex();
std::lock_guard<std::mutex> lk(g_hash_mu);
g_hash_cache[key.str()] = hex;
return hex;
}
// ── EmbedderStamp ─────────────────────────────────────────────────────────────
std::string EmbedderStamp::describe() const {
std::string name = model_name.empty() ? "<unnamed model>" : model_name;
if (model_sha256.empty())
return name + " (sha256 unavailable)";
return name + " (sha256 " + short_hash(model_sha256) + ")";
}
EmbedderStamp make_embedder_stamp(const std::string& model_path) {
EmbedderStamp s;
if (model_path.empty()) return s;
s.model_name = fs::path(model_path).filename().string();
s.model_sha256 = sha256_file_hex(model_path);
if (s.model_sha256.empty())
std::cerr << "[gallery] cannot hash embedder model " << model_path
<< " — model binding falls back to filename only (GR-004)\n";
return s;
}
bool require_gallery_stamp_from_env() {
const char* v = std::getenv("SAE_REQUIRE_GALLERY_STAMP");
return v && *v && std::strcmp(v, "0") != 0;
}
// ── Comparison ────────────────────────────────────────────────────────────────
StampCheck compare_embedder_stamps(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc,
const std::string& embedder_desc) {
StampCheck out;
std::ostringstream m;
// The gallery predates GR-004 (or was written by a tool that does not stamp).
if (built_with.empty()) {
out.verdict = StampVerdict::unstamped;
m << "gallery '" << gallery_desc << "' carries no embedder stamp (GR-004).\n"
<< " gallery was built with : UNKNOWN — this file predates model binding\n"
<< " embedder now loaded : " << loading_with.describe()
<< " [" << embedder_desc << "]\n"
<< " If these are not the same model every similarity from this run is\n"
<< " meaningless but will look plausible. Rebuild or re-stamp the gallery\n"
<< " (scripts/stamp_gallery.py), or run with SAE_REQUIRE_GALLERY_STAMP=1 to\n"
<< " make this a hard error.";
out.message = m.str();
return out;
}
// Gallery is stamped but we cannot say what is about to embed.
if (loading_with.empty()) {
out.verdict = StampVerdict::unknown_embedder;
m << "cannot identify the embedder being used against gallery '"
<< gallery_desc << "' (GR-004).\n"
<< " gallery was built with : " << built_with.describe() << "\n"
<< " embedder now loaded : UNKNOWN [" << embedder_desc << "]\n"
<< " The binding cannot be checked, so it is not being checked.";
out.message = m.str();
return out;
}
const bool have_both_hashes =
!built_with.model_sha256.empty() && !loading_with.model_sha256.empty();
// Embedding width disagreeing is a mismatch on its own terms — different
// spaces entirely, and it will not even be caught by a cosine that "looks fine".
if (built_with.embed_dim != loading_with.embed_dim) {
out.verdict = StampVerdict::mismatch;
m << "gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
<< " gallery was built with : " << built_with.describe()
<< ", dim=" << built_with.embed_dim << " [" << gallery_desc << "]\n"
<< " embedder now loaded : " << loading_with.describe()
<< ", dim=" << loading_with.embed_dim << " [" << embedder_desc << "]\n"
<< " Embedding dimensions differ; these are not the same space.";
out.message = m.str();
return out;
}
if (have_both_hashes) {
if (built_with.model_sha256 == loading_with.model_sha256) {
out.verdict = StampVerdict::match;
m << "embedder binding verified: " << built_with.describe();
if (built_with.model_name != loading_with.model_name)
m << " (gallery recorded it as '" << built_with.model_name
<< "', loaded from '" << loading_with.model_name
<< "' — same bytes, renamed file)";
out.message = m.str();
return out;
}
out.verdict = StampVerdict::mismatch;
m << "gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
<< " gallery was built with : " << built_with.model_name
<< " sha256=" << built_with.model_sha256 << "\n"
<< " [" << gallery_desc << "]\n"
<< " embedder now loaded : " << loading_with.model_name
<< " sha256=" << loading_with.model_sha256 << "\n"
<< " [" << embedder_desc << "]\n"
<< " Cosine similarities between embeddings from different models are\n"
<< " meaningless but look plausible. Rebuild the gallery with the loaded\n"
<< " model, or point the embedder at the model the gallery was built with.";
out.message = m.str();
return out;
}
// One side has no hash (e.g. a TRT deployment with the .onnx absent). Names
// are all we have; agreeing on them is evidence, not proof.
if (!built_with.model_name.empty() &&
built_with.model_name == loading_with.model_name) {
out.verdict = StampVerdict::weak_match;
m << "embedder binding UNPROVEN for gallery '" << gallery_desc << "' (GR-004).\n"
<< " gallery was built with : " << built_with.describe() << "\n"
<< " embedder now loaded : " << loading_with.describe()
<< " [" << embedder_desc << "]\n"
<< " Filenames agree but at least one SHA-256 is unavailable, so an\n"
<< " in-place re-export under the same name would not be detected.";
out.message = m.str();
return out;
}
out.verdict = StampVerdict::mismatch;
m << "gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
<< " gallery was built with : " << built_with.describe()
<< " [" << gallery_desc << "]\n"
<< " embedder now loaded : " << loading_with.describe()
<< " [" << embedder_desc << "]\n"
<< " Cosine similarities between embeddings from different models are\n"
<< " meaningless but look plausible. Rebuild the gallery with the loaded\n"
<< " model, or point the embedder at the model the gallery was built with.";
out.message = m.str();
return out;
}
void enforce_embedder_stamp(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc,
const std::string& embedder_desc,
bool require_stamp) {
const bool strict = require_stamp || require_gallery_stamp_from_env();
StampCheck chk = compare_embedder_stamps(built_with, loading_with,
gallery_desc, embedder_desc);
if (chk.fatal(strict)) {
if (chk.verdict != StampVerdict::mismatch)
throw std::runtime_error(chk.message +
"\n (fatal because SAE_REQUIRE_GALLERY_STAMP / --require-gallery-stamp is set)");
throw std::runtime_error(chk.message);
}
if (chk.verdict == StampVerdict::match) {
std::cerr << "[gallery] " << chk.message << "\n";
} else {
std::cerr << "\n[gallery] ***** WARNING (GR-004) *****\n"
<< chk.message << "\n"
<< "[gallery] ****************************\n\n";
}
}
void verify_gallery_embedder(const ActorGallery& gallery,
const std::string& gallery_path,
const std::string& arcface_model_path,
bool require_stamp) {
enforce_embedder_stamp(gallery.embedder,
make_embedder_stamp(arcface_model_path),
gallery_path,
arcface_model_path.empty() ? "no --arcface given"
: arcface_model_path,
require_stamp);
}
src/gallery/embedder_stamp.hpp
+115
View File
@@ -0,0 +1,115 @@
#pragma once
/// TRACES: GR-004 | SR-001
//
// Gallery ↔ embedder binding.
//
// A gallery is only valid for the embedder that built it. Cosine similarities
// between embeddings from two different models are meaningless but *look*
// plausible — nothing crashes, nothing is obviously wrong, and every number
// measured downstream is quietly garbage. So the embedder's identity is stamped
// into the gallery at build time and checked by every consumer at load time.
//
// ── What identifies an embedder ───────────────────────────────────────────────
// Two fields, carried together:
//
// model_name basename of the model file, e.g. "LVFace-B_Glint360K.onnx"
// model_sha256 hex SHA-256 of that file's bytes
//
// The hash is what *decides*; the name is what a human *reads*. Neither alone is
// enough:
//
// • A name alone is a promise, not a fact. Models get re-exported, re-quantised
// and overwritten in place under an unchanged filename — which is precisely
// the case where the weights differ and nothing else does. A name-only stamp
// is blind to exactly the failure it exists to catch.
// • A hash alone is correct but unreadable: "expected 3f2a… got 9c1b…" tells an
// operator nothing about what to do next.
//
// SHA-256 over the file bytes is derived from the artefact rather than asserted
// about it, is stable across machines and filesystems, and needs no registry to
// be kept up to date. Cost is ~0.1 s for a 250 MB ONNX, paid once per process
// (results are memoised on path+mtime+size), which is noise next to model load.
//
// ── Degraded and legacy cases ─────────────────────────────────────────────────
// A TRT-backend deployment may run from a prebuilt .engine with the source .onnx
// absent, so the hash cannot be computed. Then the name is compared alone and the
// result is reported as a *weak* match — believed, not proven.
//
// Galleries built before GR-004 carry no stamp at all. They warn loudly rather
// than fail, because the state is unknown rather than known-bad, and because
// hard-failing every pre-existing gallery would make the check something people
// route around rather than trust. Set require_stamp (or SAE_REQUIRE_GALLERY_STAMP=1)
// to promote "unknown" to a hard error — that is the mode measurement work runs in.
//
// A *mismatch* is always fatal, in every mode, with no bypass.
#include <cstdint>
#include <string>
struct EmbedderStamp {
std::string model_name; // basename of the model file
std::string model_sha256; // lowercase hex SHA-256 of the file's bytes ("" = unavailable)
int32_t embed_dim{512};
bool empty() const { return model_name.empty() && model_sha256.empty(); }
// "LVFace-B_Glint360K.onnx (sha256 3f2a1c4d…)" — for error messages.
std::string describe() const;
};
// Identify the model at `model_path`. Missing/unreadable file → name filled from
// the path, hash left empty (the weak-match path). Empty path → empty stamp.
EmbedderStamp make_embedder_stamp(const std::string& model_path);
enum class StampVerdict {
match, // hashes agree — binding proven
weak_match, // names agree, no hash on one side — believed, unproven
unstamped, // gallery predates GR-004 / was written without a stamp
unknown_embedder, // gallery is stamped but the loaded embedder can't be identified
mismatch, // proven different models — always fatal
};
struct StampCheck {
StampVerdict verdict{StampVerdict::match};
std::string message; // human-readable, names BOTH sides
// A mismatch is fatal unconditionally. The three "cannot prove it" verdicts
// are fatal only in strict mode.
bool fatal(bool require_stamp) const {
return verdict == StampVerdict::mismatch ||
(require_stamp && verdict != StampVerdict::match);
}
};
// Pure comparison — no file I/O, no model loading. This is the unit under test.
// `gallery_desc`/`embedder_desc` are only used to make the message locatable
// (a gallery path, a dump path, "the embedder being loaded", …).
StampCheck compare_embedder_stamps(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc = "gallery",
const std::string& embedder_desc = "embedder");
// Apply the comparison: throw std::runtime_error on a fatal verdict, otherwise
// log to stderr. `require_stamp` is OR-ed with SAE_REQUIRE_GALLERY_STAMP.
void enforce_embedder_stamp(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc,
const std::string& embedder_desc,
bool require_stamp);
// Convenience for the common consumer shape: "I loaded this gallery and I am
// about to embed with this model file." Hashes the model, then enforces.
struct ActorGallery;
void verify_gallery_embedder(const ActorGallery& gallery,
const std::string& gallery_path,
const std::string& arcface_model_path,
bool require_stamp);
// SAE_REQUIRE_GALLERY_STAMP=1 → treat an unprovable binding as fatal.
bool require_gallery_stamp_from_env();
// Lowercase hex SHA-256. Exposed so a test can pin the digest against the
// published vectors, which is what guarantees the C++ and Python (hashlib)
// stamps of the same file agree.
std::string sha256_hex(const std::string& bytes);
std::string sha256_file_hex(const std::string& path); // "" if unreadable
src/gallery/gallery_builder.cpp
+7
View File
@@ -1,5 +1,6 @@
#include "gallery_builder.hpp"
#include "config.hpp"
#include "embedder_stamp.hpp"
#include "face_utils.hpp"
#include "inference/face_detector.hpp"
#include "inference/face_embedder.hpp"
@@ -41,6 +42,12 @@ ActorGallery build_gallery(const BuildConfig& cfg) {
ActorGallery gallery;
/// TRACES: GR-004 | SR-001
// Stamp before the first embedding exists, so there is no window in which a
// gallery holds vectors without recording what produced them.
gallery.embedder = make_embedder_stamp(cfg.arcface_model);
std::cerr << "[build_gallery] embedder: " << gallery.embedder.describe() << "\n";
for (const auto& actor_dir : fs::directory_iterator(cfg.gallery_root)) {
if (!actor_dir.is_directory()) continue;
src/gallery/gallery_calibration.hpp
+29
View File
@@ -1,4 +1,5 @@
#pragma once
/// TRACES: AR-023 | SR-002
#include "types.hpp"
#include <algorithm>
@@ -8,6 +9,7 @@
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <functional>
#include <iostream>
#include <stdexcept>
#include <string>
@@ -48,6 +50,33 @@ struct GalleryCalibration {
}
};
/// TRACES: AR-023, AR-024 | SR-002
///
/// cosine → P(same person). The one probability space the pipeline reasons in.
///
/// Handed to every stage that has to decide whether two embeddings are the same
/// person — track association (AR-007), evidence discounting (AR-025), identity
/// matching — so a threshold of 0.5 means the same thing in all of them. A stage
/// that thresholded a raw cosine instead would be using a number that means
/// something different for every model, gallery and face size (AR-024).
///
/// **No prior term.** `log_prior_odds` adjusts for the gallery's base rate, which
/// is a question about *which of N actors*; association asks whether two faces
/// are one person, where the balanced fit is the right answer. Passing the
/// matcher's prior here would silently bias tracking by the size of the cast.
inline std::function<float(float)> same_person_probability(const GalleryCalibration& cal) {
if (!cal.valid) {
// Loud, because the failure mode is invisible: an untuned sigmoid still
// returns plausible probabilities, and every threshold downstream of it
// is then a guess wearing a calibrated number's clothes.
std::cerr << "[calibration] WARNING: no fitted calibration — association and "
"evidence weighting fall back to the untuned default sigmoid "
"(a=" << cal.a << ", b=" << cal.b << "). Probabilities are "
"not meaningful for this embedder.\n";
}
return [cal](float similarity) { return cal.probability(similarity); };
}
// Fit a logistic sigmoid to gallery pair similarities.
// Positive pairs: same actor, different reference images.
// Negative pairs: different actors (all cross-actor embedding pairs).
src/gallery/gallery_store.cpp
+40
View File
@@ -79,6 +79,21 @@ static ActorGallery load_gallery_hdf5(const std::string& path) {
gallery.actors.push_back(std::move(actor));
}
/// TRACES: GR-004 | SR-001
// Absent /embedder group == a gallery written before model binding existed.
// It stays readable; verify_gallery_embedder() decides what that means.
if (file.nameExists("embedder")) {
H5::Group eg = file.openGroup("embedder");
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
if (eg.attrExists("model_name"))
eg.openAttribute("model_name").read(str, gallery.embedder.model_name);
if (eg.attrExists("model_sha256"))
eg.openAttribute("model_sha256").read(str, gallery.embedder.model_sha256);
if (eg.attrExists("embed_dim"))
eg.openAttribute("embed_dim").read(H5::PredType::NATIVE_INT32,
&gallery.embedder.embed_dim);
}
if (file.nameExists("calibration")) {
H5::Group cal = file.openGroup("calibration");
cal.openAttribute("a").read(H5::PredType::NATIVE_FLOAT, &gallery.calib_a);
@@ -149,6 +164,20 @@ static void save_gallery_hdf5(const std::string& path, const ActorGallery& galle
write_str_dataset(file, "name", name);
write_str_dataset(file, "source_images", src_images);
/// TRACES: GR-004 | SR-001
// Bind the file to the embedder that produced its vectors. Written only when
// known — an empty stamp must round-trip as "unstamped", not as a stamp
// claiming an unnamed model.
if (!gallery.embedder.empty()) {
H5::Group eg = file.createGroup("embedder");
H5::DataSpace scalar(H5S_SCALAR);
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
eg.createAttribute("model_name", str, scalar).write(str, gallery.embedder.model_name);
eg.createAttribute("model_sha256", str, scalar).write(str, gallery.embedder.model_sha256);
eg.createAttribute("embed_dim", H5::PredType::NATIVE_INT32, scalar)
.write(H5::PredType::NATIVE_INT32, &gallery.embedder.embed_dim);
}
if (gallery.calib_hash != 0) {
H5::Group cal = file.createGroup("calibration");
H5::DataSpace scalar(H5S_SCALAR);
@@ -186,6 +215,17 @@ ActorGallery load_gallery(const std::string& path) {
<< std::chrono::duration<double>(t1 - t0).count() << "s\n";
ActorGallery gallery;
/// TRACES: GR-004 | SR-001
// Optional top-level "embedder" object, matching the HDF5 /embedder group.
// Written by the JSON-era helper scripts; absent in anything older.
if (j.contains("embedder") && j.at("embedder").is_object()) {
const auto& je = j.at("embedder");
gallery.embedder.model_name = je.value("model_name", "");
gallery.embedder.model_sha256 = je.value("model_sha256", "");
gallery.embedder.embed_dim = je.value("embed_dim", 512);
}
for (const auto& ja : j.at("actors")) {
ActorGallery::Actor actor;
actor.imdb_id = ja.value("imdb_id", "");
src/gallery/gallery_store.hpp
+6
View File
@@ -12,6 +12,11 @@
// /count int32 [A] number of refs for actor a
// /imdb_id /tmdb_id /jellyfin_id /name : variable-length string [A]
// /source_images : variable-length string [N], parallel to /embeddings rows
// /embedder/model_name : scalar var-len string attr — embedder file basename
// /embedder/model_sha256 : scalar var-len string attr — SHA-256 of that file
// /embedder/embed_dim : scalar int32 attr
// The GR-004 model binding. Absent group == unstamped
// (pre-GR-004 file); see gallery/embedder_stamp.hpp.
// /calibration/a, /b : scalar float32 attrs — Platt-sigmoid P(match|sim) fit
// /calibration/valid : scalar int8 attr (0/1)
// /calibration/hash : scalar uint64 attr — hash of the embeddings the fit
@@ -19,6 +24,7 @@
//
// Legacy JSON format (read-only):
// {
// "embedder": {"model_name": "...", "model_sha256": "...", "embed_dim": 512},
// "actors": [
// {
// "imdb_id": "nm0000093", // optional, "" if unknown
src/kpn_bindings.cpp
+28 -2
View File
@@ -15,6 +15,7 @@
#include "types.hpp"
#include "config.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/face_tracker_node.hpp"
#include "nodes/identity_matcher_node.hpp"
@@ -163,6 +164,9 @@ static Config config_from_dict(nb::dict d) {
getd("anneal_sec", cfg.anneal_sec);
// gallery expansion (usually off for sweeps; expose so it can be toggled)
if (d.contains("expand_gallery")) cfg.expand_gallery = nb::cast<bool>(d["expand_gallery"]);
/// TRACES: GR-004 | SR-001
if (d.contains("require_gallery_stamp"))
cfg.require_gallery_stamp = nb::cast<bool>(d["require_gallery_stamp"]);
return cfg;
}
@@ -210,8 +214,17 @@ NB_MODULE(sae_kpn, m) {
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "config"_a, "capacity"_a = 16);
/// TRACES: GR-004 | SR-001
// embedder_model / embedder_sha256 identify whatever produced the embeddings
// that will be fed in. In a replay those come from the dump's own stamp (see
// scripts/optimizer/SCHEMA.md), because there is no live embedder in the
// network — the dump *is* the embedder as far as this gallery is concerned.
// Passing neither leaves the binding unverifiable, which warns loudly and is
// fatal under SAE_REQUIRE_GALLERY_STAMP.
m.def("add_identity_matcher", [](Net& net, std::string name, std::string gallery_path,
nb::dict cfg_dict, std::size_t cap) {
nb::dict cfg_dict, std::size_t cap,
std::string embedder_model,
std::string embedder_sha256) {
Config cfg = config_from_dict(cfg_dict);
cfg.gallery_path = gallery_path; // needed to persist refreshed calibration back
// Cache loaded galleries by path so a threshold sweep (many networks, same
@@ -222,11 +235,24 @@ NB_MODULE(sae_kpn, m) {
if (it == cache.end())
it = cache.emplace(gallery_path,
std::make_shared<ActorGallery>(load_gallery(gallery_path))).first;
// Checked on every construction, not only on the cache miss: the same
// process may replay several dumps against one cached gallery.
EmbedderStamp feeding;
feeding.model_name = std::move(embedder_model);
feeding.model_sha256 = std::move(embedder_sha256);
enforce_embedder_stamp(it->second->embedder, feeding, gallery_path,
feeding.model_name.empty()
? "embeddings fed into this network"
: feeding.model_name,
cfg.require_gallery_stamp);
auto node = std::make_shared<kpn::ObjectVariantNodeWrapper<
IdentityMatcherFunc, SaeVariant, kpn::in<"tracked">, kpn::out<"matched">>>(
cap, *it->second, cfg);
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "gallery"_a, "config"_a, "capacity"_a = 16);
}, "net"_a, "name"_a, "gallery"_a, "config"_a, "capacity"_a = 16,
"embedder_model"_a = "", "embedder_sha256"_a = "");
m.def("add_scene_tracker", [](Net& net, std::string name, nb::dict cfg_dict, std::size_t cap) {
Config cfg = config_from_dict(cfg_dict);
src/main.cpp
+57 -7
View File
@@ -1,5 +1,9 @@
// scene_analyze — identify actors in a movie using a KPN pipeline
//
// TRACES: DP-001, DP-002 | PR-004
// One analysis core; the CLI is a front-end over it and must not fork pipeline
// logic. Other deployment modes (DP-003, DP-004) wrap this same core.
//
// KPN topology (release build):
//
// [frame_source] ──Frame──► [face_detector] ──SceneFrame──► [face_aligner]
@@ -45,6 +49,7 @@
#include "config.hpp"
#include "types.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/frame_source_node.hpp"
#include "nodes/camera_position_change_detector_node.hpp"
@@ -114,6 +119,7 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--detector")) cfg.detector_model = next();
else if (arg("--detector-engine")) cfg.detector_engine = next();
else if (arg("--arcface")) cfg.arcface_model = next();
else if (arg("--require-gallery-stamp")) cfg.require_gallery_stamp = true;
else if (arg("--arcface-engine")) cfg.arcface_engine = next();
else if (arg("--conf")) cfg.detector_conf = std::stof(next());
else if (arg("--max-faces")) cfg.max_faces = std::stoi(next());
@@ -122,10 +128,8 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--ratio-ceil")) cfg.match_ratio_ceil = std::stof(next());
else if (arg("--track-alpha")) cfg.track_alpha = std::stof(next());
else if (arg("--track-min-iou")) cfg.track_min_iou = std::stof(next());
else if (arg("--track-max-embed")) cfg.track_max_embed_dist = std::stof(next());
else if (arg("--track-max-missing")) cfg.track_max_frames_missing = std::stoi(next());
else if (arg("--cut-revive-sim")) cfg.cut_revive_sim = std::stof(next());
else if (arg("--cut-inactive-max")) cfg.cut_inactive_max_frames = std::stoi(next());
else if (arg("--track-min-prob")) cfg.track_assoc_min_prob = std::stof(next());
else if (arg("--track-extinction")) cfg.track_extinction_sec = std::stod(next());
else if (arg("--anneal")) cfg.anneal_sec = std::stod(next());
else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
@@ -167,6 +171,11 @@ int main(int argc, char** argv) {
ActorGallery gallery;
try {
gallery = load_gallery(cfg.gallery_path);
/// TRACES: GR-004 | SR-001
// Hard startup error before a single frame is decoded: a gallery built
// with another embedder yields plausible-looking, meaningless matches.
verify_gallery_embedder(gallery, cfg.gallery_path, cfg.arcface_model,
cfg.require_gallery_stamp);
} catch (const std::exception& e) {
std::cerr << "Gallery error: " << e.what() << "\n";
return 1;
@@ -183,10 +192,36 @@ int main(int argc, char** argv) {
FaceDetectorFunc detector_fn{cfg};
FaceAlignerFunc aligner_fn;
EmbedderFunc embedder_fn{cfg};
FaceTrackerFunc ftracker_fn{cfg};
// Constructed before the tracker: it fits (or loads) the calibration, and
// the tracker must decide in that same probability space (AR-024).
IdentityMatcherFunc matcher_fn {gallery, cfg};
/// TRACES: AR-007, AR-008, AR-012, AR-024 | SR-002
// The registry is created here and shared, not owned by a node: track state
// is not a stage in the stream, it is state several stages read and write,
// and its final answer is only known when a track dies.
auto same_person = same_person_probability(matcher_fn.calibration());
TrackRegistry::Config reg_cfg;
reg_cfg.extinction_sec = cfg.track_extinction_sec;
auto registry = std::make_shared<TrackRegistry>(
reg_cfg, EvidenceDiscounter(same_person));
matcher_fn.set_registry(registry);
FaceTrackerFunc ftracker_fn{cfg, registry, same_person};
SceneTrackerFunc tracker_fn {cfg};
ResultSinkFunc sink_fn {cfg, done};
/// TRACES: AR-012, AR-016, IR-002, IR-003 | SR-002
// A reaped track goes straight to the aggregator, so the registry holds only
// live tracks and its size is bounded by concurrent on-screen faces rather
// than growing with the film.
registry->on_track_dead([&sink_fn](const DeadTrack& d) { sink_fn.add_claim(d); });
// AR-016: a film ends with faces on screen and those tracks have not timed
// out. Without this flush the closing scene's cast is silently never
// emitted — a loss that reads as a recognition miss, not a bookkeeping bug.
sink_fn.set_pre_write_hook([registry](double last_ts) { registry->flush(last_ts); });
#ifdef SAE_DEBUG
DebugRendererFunc debug_fn {cfg};
#endif
@@ -247,15 +282,30 @@ int main(int argc, char** argv) {
net.stop();
net.print_diagnostics();
/// TRACES: AR-004 | SR-002
// A dropped frame does not degrade a result, it silently changes one —
// the output is a claim about footage that was never analysed, and
// nothing in the file says so. Since AR-004 made data pushes block, a
// drop can no longer happen on the data path, so any drop here means
// either that fix regressed (it lives in the KPN submodule, one line,
// easy to lose in an update) or a channel was disabled mid-run.
//
// Reporting it in a footer and exiting 0 made both invisible: the run
// "succeeded" and the truth file looked complete. Fail instead.
bool dropped = false;
{
std::lock_guard<std::mutex> lk(event_mtx);
if (!overflow_counts.empty()) {
std::cerr << "[main] dropped frames (channel overflow):\n";
dropped = true;
std::cerr << "[main] ERROR: frames were dropped (channel overflow):\n";
for (const auto& [name, count] : overflow_counts)
std::cerr << " " << name << ": " << count << "\n";
std::cerr << "[main] The output would describe footage that was never "
"analysed. Refusing to report success.\n";
}
}
return node_crashed.load(std::memory_order_acquire) ? 1 : 0;
if (node_crashed.load(std::memory_order_acquire)) return 1;
return dropped ? 2 : 0;
};
// ── Build static network and run ──────────────────────────────────────────
src/nodes/embedding_dump_node.hpp
+14 -2
View File
@@ -1,6 +1,8 @@
#pragma once
/// TRACES: VR-001 | PR-002
#include "types.hpp"
#include "config.hpp"
#include "gallery/embedder_stamp.hpp"
#include <H5Cpp.h>
@@ -25,7 +27,13 @@ struct EmbeddingDumpFunc {
: path_(cfg.dump_embeddings_path), movie_(cfg.movie_path),
sample_fps_(cfg.sample_fps), done_(done)
{
std::cerr << "[embedding_dump] writing " << path_ << "\n";
/// TRACES: GR-004 | SR-001
// A dump is a bag of embeddings with no model attached, replayed against a
// gallery hours or weeks later — the same silent cross-model hazard as the
// gallery itself, so it carries the same stamp.
stamp_ = make_embedder_stamp(cfg.arcface_model);
std::cerr << "[embedding_dump] writing " << path_
<< " embedder: " << stamp_.describe() << "\n";
}
void operator()(EmbeddedSceneFrame ef) {
@@ -91,6 +99,9 @@ private:
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
auto mv = file.createAttribute("movie", str, scalar);
mv.write(str, movie_);
/// TRACES: GR-004 | SR-001
file.createAttribute("embedder_model", str, scalar).write(str, stamp_.model_name);
file.createAttribute("embedder_sha256", str, scalar).write(str, stamp_.model_sha256);
H5::Group frames = file.createGroup("frames");
write_vec(frames, "timestamp_sec", ts_, H5::PredType::NATIVE_DOUBLE);
@@ -110,7 +121,8 @@ private:
<< conf_.size() << " faces → " << path_ << "\n";
}
std::string path_, movie_;
std::string path_, movie_;
EmbedderStamp stamp_;
float sample_fps_;
std::atomic<bool>& done_;
std::atomic<bool> written_{false};
src/nodes/face_detector_node.hpp
+1
View File
@@ -1,4 +1,5 @@
#pragma once
/// TRACES: AR-001 | SR-002
#include "config.hpp"
#include "inference/face_detector.hpp"
src/nodes/face_tracker_node.hpp
+169 -162
View File
@@ -1,102 +1,124 @@
#pragma once
/// TRACES: AR-007, AR-008, AR-024 | SR-002
///
/// FaceTrackerFunc — KPN node that links face detections into tracks.
///
/// **The registry is the tracker's state.** The node owns no track map of its
/// own: it drives `TrackRegistry` through a `FrameScope` and reads the same
/// `Track` objects everything else reads. Two parallel copies could disagree,
/// and every divergence would surface as a wrong presence window rather than as
/// a crash — silently, and only in the output.
///
/// **One candidate pool** (AR-008). `last_seen` alone distinguishes a track that
/// is on screen from one that is dormant, and it only affects whether IoU means
/// anything. There is no parked pool and no revival branch: re-associating a
/// track whose face was lost — across a cut or not — is ordinary inter-frame
/// association, and it falls out of the embedding comparison already being done.
///
/// Assignment cost (track i, detection j):
///
/// p = P(same person | cosine(track mean, detection)) ← calibrated
/// alpha = base weight, or 0 when position carries no information
/// cost = alpha·(1 − IoU) + (1 − alpha)·(1 − p)
///
/// gated to INF unless the pair is admissible on position *or* on identity.
///
/// **alpha is frame- and track-dependent** (AR-007). It falls to 0 —
/// embedding only — when either:
/// - the frame is flagged `is_cut` / `is_scene_boundary`: the viewpoint
/// changed, so the same person is at a new position; or
/// - the track is dormant (`last_seen` set): time has passed since its box was
/// last observed, so that box is stale regardless of cuts.
/// Both are the same statement — spatial continuity is broken — arrived at from
/// two directions, which is why they collapse into one rule rather than two
/// branches.
///
/// **Everything is thresholded in probability space** (AR-024). The cosine goes
/// through the calibration before it is compared to anything; the raw-cosine
/// constants `track_max_embed_dist` and `cut_revive_sim` are retired.
#include "types.hpp"
#include "config.hpp"
#include "track_registry.hpp"
#include <algorithm>
#include <cmath>
#include <functional>
#include <iostream>
#include <limits>
#include <map>
#include <memory>
#include <stdexcept>
#include <string_view>
#include <vector>
// ── FaceTrackerFunc ───────────────────────────────────────────────────────────
// KPN node: links face detections across consecutive frames using the Hungarian
// algorithm on a combined spatial (IoU) + embedding (cosine distance) cost.
//
// Each track accumulates a running directional mean of its ArcFace embeddings
// (averaged then re-normalised to the unit sphere), used as the embedding side
// of the assignment cost below for more stable track continuity.
//
// Assignment cost (track i, detection j):
// cost = alpha * (1 - IoU) + (1-alpha) * min(cosine_dist/2, 1)
// Gated to INF when IoU < min_iou AND cosine_dist > max_embed_dist.
//
// Unmatched tracks have their frames_missing counter incremented; they are
// expired once frames_missing > max_frames_missing.
//
// Cross-cut re-association. A camera-angle change (Frame::is_cut, set by
// camera_position_change_detector) destroys spatial (IoU) continuity — the same
// person reappears at a new position — but not identity. On a cut the tracker
// does NOT discard its tracks; it parks them in an inactive pool keyed by their
// last-frame raw embedding. A post-cut detection whose raw cosine similarity to
// a parked track's last-frame embedding is ≥ cut_revive_sim revives that track:
// the original track_id, mean embedding and n_frames are restored (only the bbox
// jumps to the new detection), so identity continuity survives the cut. Parked
// tracks left unrevived for cut_inactive_max_frames are finally dropped.
struct FaceTrackerFunc {
static constexpr std::string_view label() { return "face_tracker"; }
struct TrackState {
cv::Rect2f bbox;
Embedding mean_emb{};
Embedding last_emb{}; // raw embedding of the most recent matched frame
int n_frames{0};
int frames_missing{0};
};
/// cosine similarity → P(same person). Supplied by the caller so the fit
/// belonging to the active embedder is used (AR-023/AR-024) — the same
/// pattern, and normally the same function object, as
/// `EvidenceDiscounter::Calibrate`.
using Calibrate = std::function<float(float)>;
explicit FaceTrackerFunc(const Config& cfg)
: alpha_(cfg.track_alpha)
/// The registry is a constructor argument, not an option: a tracker without
/// one would have to keep its own tracks, which is the defect this replaces.
FaceTrackerFunc(const Config& cfg,
std::shared_ptr<TrackRegistry> registry,
Calibrate calibrate)
: registry_(std::move(registry))
, calibrate_(std::move(calibrate))
, alpha_base_(cfg.track_alpha)
, min_iou_(cfg.track_min_iou)
, max_embed_dist_(cfg.track_max_embed_dist)
, max_missing_(cfg.track_max_frames_missing)
, revive_sim_(cfg.cut_revive_sim)
, inactive_max_(cfg.cut_inactive_max_frames)
, min_assoc_prob_(cfg.track_assoc_min_prob)
{
std::cerr << "[face_tracker] alpha=" << alpha_
if (!registry_)
throw std::invalid_argument("face_tracker: registry must not be null");
if (!calibrate_)
throw std::invalid_argument("face_tracker: a calibration is required — "
"association is decided in probability space");
std::cerr << "[face_tracker] alpha_base=" << alpha_base_
<< " min_iou=" << min_iou_
<< " max_embed_dist=" << max_embed_dist_
<< " max_missing=" << max_missing_
<< " cut_revive_sim=" << revive_sim_
<< " cut_inactive_max=" << inactive_max_ << "\n";
<< " min_assoc_prob=" << min_assoc_prob_ << "\n";
}
TrackedSceneFrame operator()(EmbeddedSceneFrame ef) {
if (ef.source.eof) {
tracks_.clear();
inactive_.clear();
// Deliberately does *not* flush the registry. The identity matcher
// runs downstream and its votes for the final frames are still in
// flight; reaping here would drop them (they would land on ids that
// no longer exist and show up as dropped_votes). AR-016's flush
// belongs at the pipeline's termination point, after the last vote.
boxes_.clear();
TrackedSceneFrame out;
out.source = std::move(ef.source);
return out;
}
const int n_det = static_cast<int>(ef.embeddings.size());
const double t = ef.source.timestamp_sec;
const int n_det = static_cast<int>(ef.embeddings.size());
// Camera-angle change: park active tracks instead of destroying them so
// they can be revived by identity (raw last-frame embedding cosine) once
// the same people reappear from the new angle.
if (ef.source.is_cut && !tracks_.empty()) {
std::cerr << "[face_tracker] cut — parking " << tracks_.size()
<< " track(s) into inactive pool\n";
for (auto& [tid, ts] : tracks_) {
ts.frames_missing = 0; // repurpose as time-since-parked counter
inactive_[tid] = std::move(ts);
}
tracks_.clear();
}
// Unconditional: the clock must advance on frames with no detections
// too, or a track only dies when some unrelated face happens to appear
// and a film that ends mid-track never closes it (AR-013).
auto scope = registry_->begin_frame(t);
// Age the inactive pool every frame and drop tracks parked too long.
for (auto it = inactive_.begin(); it != inactive_.end(); ) {
it->second.frames_missing++;
it = (it->second.frames_missing > inactive_max_)
? inactive_.erase(it) : std::next(it);
}
// One pool (AR-008) — on-screen and dormant tracks compete together.
std::vector<Track*> cands = scope.candidates();
const int n_trk = static_cast<int>(cands.size());
// Snapshot active track IDs so the map can be modified safely below
std::vector<int> tids;
tids.reserve(tracks_.size());
for (auto& [tid, _] : tracks_) tids.push_back(tid);
const int n_trk = static_cast<int>(tids.size());
prune_boxes(cands);
std::vector<Spatial*> sp(n_trk);
for (int ti = 0; ti < n_trk; ++ti)
sp[ti] = &boxes_.try_emplace(cands[ti]->id, Spatial{{}, t, false})
.first->second;
// AR-007 — the frame half of the frame-dependent weighting. Both flags
// say the same thing to the tracker: whatever was at that position is
// not there any more.
const bool viewpoint_change =
ef.source.is_cut || ef.source.is_scene_boundary;
// ── Cost matrix [n_trk × n_det] ──────────────────────────────────────
constexpr float INF_COST = 1e6f;
@@ -104,109 +126,108 @@ struct FaceTrackerFunc {
std::vector<float>(n_det, INF_COST));
for (int ti = 0; ti < n_trk; ++ti) {
const TrackState& ts = tracks_[tids[ti]];
// Spatial continuity holds only for a track that was on screen, whose
// box we have actually observed, on a frame that did not change the
// viewpoint. Otherwise the box is stale and IoU is noise.
const bool spatial_meaningful =
sp[ti]->observed && cands[ti]->on_screen() && !viewpoint_change;
const float alpha = spatial_meaningful ? alpha_base_ : 0.f;
for (int di = 0; di < n_det; ++di) {
float iou_v = iou(ts.bbox, ef.faces[di].bbox);
float emb_d = (ts.n_frames > 0)
? 1.f - cosine_similarity(ts.mean_emb, ef.embeddings[di])
: 1.f;
if (iou_v < min_iou_ && emb_d > max_embed_dist_) continue;
float s = 1.f - iou_v;
float e = std::min(emb_d * 0.5f, 1.f);
cost[ti][di] = alpha_ * s + (1.f - alpha_) * e;
// AR-024 — the cosine is converted before it is used for
// anything, including the gate below.
const float p = calibrate_(
cosine_similarity(cands[ti]->mean, ef.embeddings[di]));
const float iou_v = spatial_meaningful
? iou(sp[ti]->bbox, ef.faces[di].bbox) : 0.f;
// Either signal on its own can admit a link: a face that moved a
// little but whose embedding degraded (blur, profile turn) is
// still linkable on position, and a face that jumped across the
// frame is still linkable on identity. Neither ⇒ no link.
const bool spatial_ok = spatial_meaningful && iou_v >= min_iou_;
const bool identity_ok = p >= min_assoc_prob_;
if (!spatial_ok && !identity_ok) continue;
cost[ti][di] = alpha * (1.f - iou_v) + (1.f - alpha) * (1.f - p);
}
}
// ── Hungarian assignment ──────────────────────────────────────────────
// ── Hungarian assignment ─────────────────────────────────────────────
std::vector<int> assign(n_trk, -1);
if (n_trk > 0 && n_det > 0)
assign = hungarian(cost, n_trk, n_det);
// ── Build output frame ────────────────────────────────────────────────
// ── Build output frame ───────────────────────────────────────────────
TrackedSceneFrame out;
out.source = ef.source;
out.faces = ef.faces;
out.crops = ef.crops;
out.embeddings = ef.embeddings;
out.source = ef.source;
out.faces = ef.faces;
out.crops = ef.crops;
out.embeddings = ef.embeddings;
out.track_ids.assign(n_det, -1);
std::vector<bool> det_matched(n_det, false);
// Update matched tracks
for (int ti = 0; ti < n_trk; ++ti) {
int di = assign[ti];
bool valid = (di >= 0 && di < n_det && cost[ti][di] < INF_COST * 0.5f);
TrackState& ts = tracks_[tids[ti]];
const int di = assign[ti];
const int id = cands[ti]->id;
const bool valid =
(di >= 0 && di < n_det && cost[ti][di] < INF_COST * 0.5f);
if (!valid) {
ts.frames_missing++;
// Only a track that *was* on screen can become lost, and it
// becomes lost as of its last sighting, never as of now — the
// gap after the final sighting is never claimed (AR-013). A
// track already dormant is left alone so its extinction clock
// keeps running from the right instant.
if (cands[ti]->on_screen()) scope.mark_lost(id, sp[ti]->last_ts);
continue;
}
update_mean(ts.mean_emb, ts.n_frames, ef.embeddings[di]);
ts.last_emb = ef.embeddings[di];
ts.bbox = ef.faces[di].bbox;
ts.n_frames++;
ts.frames_missing = 0;
det_matched[di] = true;
out.track_ids[di] = tids[ti];
scope.mark_seen(id, t, ef.embeddings[di]);
sp[ti]->bbox = ef.faces[di].bbox;
sp[ti]->last_ts = t;
sp[ti]->observed = true;
det_matched[di] = true;
out.track_ids[di] = id;
}
// Handle unmatched detections: first try to revive a parked track by
// identity (raw last-frame embedding cosine), else start a fresh track.
for (int di = 0; di < n_det; ++di) {
if (det_matched[di]) continue;
int tid = revive_from_inactive(ef.embeddings[di]);
if (tid >= 0) {
// Restore the parked track: keep its identity statistics
// (mean_emb, n_frames), jump the bbox to the new detection.
TrackState ts = std::move(inactive_[tid]);
inactive_.erase(tid);
update_mean(ts.mean_emb, ts.n_frames, ef.embeddings[di]);
ts.last_emb = ef.embeddings[di];
ts.bbox = ef.faces[di].bbox;
ts.n_frames++;
ts.frames_missing = 0;
tracks_[tid] = std::move(ts);
out.track_ids[di] = tid;
std::cerr << "[face_tracker] revived track " << tid
<< " across cut\n";
continue;
}
tid = next_id_++;
TrackState ts;
ts.bbox = ef.faces[di].bbox;
ts.mean_emb = ef.embeddings[di];
ts.last_emb = ef.embeddings[di];
ts.n_frames = 1;
tracks_[tid] = ts;
out.track_ids[di] = tid;
}
// Expire stale tracks
for (auto it = tracks_.begin(); it != tracks_.end(); ) {
it = (it->second.frames_missing > max_missing_)
? tracks_.erase(it) : std::next(it);
const int id = scope.create(t, ef.embeddings[di]);
boxes_[id] = Spatial{ef.faces[di].bbox, t, true};
out.track_ids[di] = id;
}
// No reaping here: begin_frame's tick owns the extinction sweep, so
// there is exactly one place a track can die.
return out;
}
private:
// Pick the parked track whose last-frame embedding is most similar to emb,
// returning its id if that raw cosine similarity clears revive_sim_, else -1.
// The caller removes the returned track from the pool, so a later detection in
// the same frame cannot claim it again.
int revive_from_inactive(const Embedding& emb) const {
int best_tid = -1;
float best_sim = revive_sim_; // threshold is the bar to beat (inclusive)
for (const auto& [tid, ts] : inactive_) {
float sim = cosine_similarity(ts.last_emb, emb);
if (sim >= best_sim) { best_sim = sim; best_tid = tid; }
// subsequent ties keep the later id; harmless, all clear the threshold
// ── Spatial annotation ───────────────────────────────────────────────────
// The one piece of per-track state the registry does not hold, because it is
// not about presence: where the face was, and when it was last seen there.
// Keyed by registry track id and pruned against `candidates()` every frame,
// so it cannot outlive or contradict the registry — it annotates the pool
// rather than duplicating it.
struct Spatial {
cv::Rect2f bbox{};
double last_ts{0.0}; ///< timestamp of the last frame this track matched
bool observed{false}; ///< false until a detection has been assigned
};
// Drop boxes for ids the registry no longer has. `candidates()` is the
// authority on what exists; anything else is a leak (and, for a reused id,
// would be a stale box attached to a different person).
void prune_boxes(const std::vector<Track*>& cands) {
if (boxes_.size() == cands.size()) return; // common case: nothing died
std::map<int, Spatial> kept;
for (const Track* t : cands) {
auto it = boxes_.find(t->id);
if (it != boxes_.end()) kept.emplace(t->id, it->second);
}
return best_tid;
boxes_.swap(kept);
}
// IoU of two axis-aligned bounding boxes
@@ -220,17 +241,6 @@ private:
return inter / (a.width * a.height + b.width * b.height - inter);
}
// Online directional mean: average then re-normalise to unit sphere
static void update_mean(Embedding& mean, int n_prev, const Embedding& emb) {
float norm_sq = 0.f;
for (int k = 0; k < 512; ++k) {
mean[k] = (mean[k] * n_prev + emb[k]) / (n_prev + 1);
norm_sq += mean[k] * mean[k];
}
float inv = 1.f / std::sqrt(norm_sq);
for (int k = 0; k < 512; ++k) mean[k] *= inv;
}
// O(n³) potential-based Hungarian algorithm (Jonker-Volgenant / Kuhn-Munkres).
// Returns assign[row] = col (0-indexed), or -1 when row is matched to a
// padded virtual column (i.e., unmatched). Rectangular matrices are padded
@@ -292,13 +302,10 @@ private:
return ans;
}
std::map<int, TrackState> tracks_;
std::map<int, TrackState> inactive_; // parked across a cut, keyed by track id
int next_id_{0};
float alpha_;
std::shared_ptr<TrackRegistry> registry_;
Calibrate calibrate_;
std::map<int, Spatial> boxes_; ///< track id → where it was, when
float alpha_base_;
float min_iou_;
float max_embed_dist_;
int max_missing_;
float revive_sim_;
int inactive_max_;
float min_assoc_prob_;
};
src/nodes/identity_matcher_node.hpp
+29
View File
@@ -5,6 +5,7 @@
#include "gallery/gallery_store.hpp"
#include "gallery/gallery_calibration.hpp"
#include "gallery/track_gallery.hpp"
#include "track_registry.hpp"
#include <cstdint>
#include <cstring>
@@ -107,6 +108,20 @@ struct IdentityMatcherFunc {
sim_engine_ = make_similarity_engine(host_gallery.data(), n_gallery_, kMaxFaces);
}
/// TRACES: AR-023, AR-024 | SR-002
/// The fitted sigmoid. Exposed because the matcher is where it gets fitted
/// (and cached back to the gallery), but it is not the matcher's private
/// property: track association and evidence weighting must threshold in the
/// *same* probability space, or a "0.5" in one stage and a "0.5" in another
/// mean different things. See `same_person_probability`.
const GalleryCalibration& calibration() const { return cal_; }
/// TRACES: AR-012, AR-025 | SR-002
/// Where per-frame identity evidence reaches the registry. Optional: with no
/// registry attached the matcher behaves exactly as before, which keeps the
/// replay harness and the unit tests working unchanged.
void set_registry(std::shared_ptr<TrackRegistry> r) { registry_ = std::move(r); }
// Runtime setter — lets a persistent pipeline be reused across a threshold sweep
// without rebuilding the (expensive, gallery-resident) matcher. The gallery,
// calibration and GPU sim-engine stay put; only the accept threshold changes.
@@ -224,6 +239,19 @@ struct IdentityMatcherFunc {
// face (annex already folded in above); the track's diversity buffer
// keeps the gallery-far views and promotes them once the track is
// confirmed. No-op unless --expand-gallery is set.
// TRACES: AR-012, AR-025 | SR-002
// Every scored face is evidence, not only the accepted ones: a run of
// near-misses for one actor is itself informative, and discarding it
// would make ownership depend on a per-frame threshold the redesign
// exists to stop relying on. The registry discounts for correlation
// and decides ownership from the accumulated posterior (AR-025).
if (registry_ && best_actor >= 0 && tf.track_ids[fi] >= 0) {
const float p = cal_.valid
? cal_.probability(best_s, log_prior_odds_)
: std::max(0.f, best_s);
registry_->observe(tf.track_ids[fi], best_actor, p, tf.embeddings[fi]);
}
track_gallery_.observe(tf.track_ids[fi], tf.embeddings[fi],
best_actor, best_s, accept, tf.crops[fi]);
@@ -247,4 +275,5 @@ private:
std::unique_ptr<ISimilarityEngine> sim_engine_;
TrackGallery track_gallery_;
std::shared_ptr<TrackRegistry> registry_;
};
src/nodes/result_sink_node.hpp
+86 -34
View File
@@ -1,6 +1,8 @@
#pragma once
/// TRACES: IR-001 | SR-003
#include "types.hpp"
#include "config.hpp"
#include "track_registry.hpp"
#include <nlohmann/json.hpp>
#include <algorithm>
@@ -9,6 +11,8 @@
#include <fstream>
#include <iostream>
#include <map>
#include <mutex>
#include <functional>
#include <string>
#include <vector>
@@ -42,10 +46,26 @@ using json = nlohmann::json;
struct ResultSinkFunc {
static constexpr std::string_view label() { return "result_sink"; }
/// TRACES: AR-012, AR-017, IR-002 | SR-002, SR-003
/// A finished presence claim from the registry. Called from inside the
/// registry's reap while it holds its own lock, so this must stay a cheap
/// push and must never re-enter the registry.
void add_claim(const DeadTrack& d) {
if (d.actor_idx < 0) return; // never owned: nothing to claim
std::lock_guard<std::mutex> g(claims_mu_);
claims_.push_back(d);
}
ResultSinkFunc(const Config& cfg, std::atomic<bool>& done)
: cfg_(cfg), done_(done)
{}
/// TRACES: AR-016 | SR-002
/// Runs immediately before the output is written, with the last timestamp
/// seen. Used to flush tracks still live at EOF, which have not timed out
/// and would otherwise never be emitted.
void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }
void operator()(SceneAnnotation sa) {
if (sa.eof) {
flush();
@@ -58,12 +78,20 @@ struct ResultSinkFunc {
<< " unknowns=" << count_unknown(sa.visible_actors)
<< std::flush;
for (const auto& ia : sa.visible_actors) {
if (ia.actor_idx < 0) continue;
auto& m = actor_meta_[ia.actor_idx];
if (m.name.empty())
m = {ia.name, ia.imdb_id, ia.tmdb_id, ia.jellyfin_id};
}
last_ts_ = sa.timestamp_sec;
frames_.push_back(std::move(sa));
}
// Write accumulated results and signal done. Safe to call more than once.
void flush() {
if (written_.exchange(true)) return;
if (pre_write_) pre_write_(last_ts_);
write_output();
done_.store(true, std::memory_order_release);
}
@@ -71,7 +99,7 @@ struct ResultSinkFunc {
private:
// Bump when the minimal/standard output JSON structure changes in a way
// the Jellyfin plugin needs to detect.
static constexpr int kSchemaVersion = 1;
static constexpr int kSchemaVersion = 2; // SR-003 coordinated bump
static int count_known(const std::vector<IdentifiedActor>& v) {
int n = 0;
@@ -91,10 +119,19 @@ private:
if (cfg_.verbosity == Verbosity::xray) {
root = build_xray();
} else {
/// TRACES: IR-002 | SR-003
/// schema_version 2, per jRay/SPEC.md JR-002. anneal_sec is REMOVED
/// rather than zeroed: a field naming a mechanism the pipeline no
/// longer has is actively misleading, and would outlive everyone who
/// remembers why it reads 0. extinction_sec succeeds it as the
/// parameter that actually shapes window extent.
root["schema_version"] = kSchemaVersion;
root["movie"] = cfg_.movie_path;
root["sample_fps"] = cfg_.sample_fps;
root["anneal_sec"] = cfg_.anneal_sec;
root["movie"] = cfg_.movie_path;
root["extraction"] = {
{"sample_fps", cfg_.sample_fps},
{"extinction_sec", cfg_.track_extinction_sec},
{"gallery_scope", cfg_.gallery_scope},
};
root["actors"] = build_epochs();
if (cfg_.verbosity == Verbosity::standard)
root["frames"] = build_standard();
@@ -109,42 +146,45 @@ private:
std::cerr << "[result_sink] done.\n";
}
struct Window {
double start{0.0};
double end{0.0};
float belief{0.f}; ///< the posterior that justified the claim (AR-017)
};
struct ActorWindow {
std::string name, imdb_id, tmdb_id, jellyfin_id;
std::vector<std::pair<double, double>> scenes; // [start_sec, end_sec]
std::vector<Window> scenes;
};
struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };
// Core logic: merge per-frame detections into annealed [start, end] windows.
/// TRACES: AR-012, IR-002 | SR-002
/// A claim already IS a window — `[first_seen, last_seen]` of a track the
/// actor owned. There is no annealing pass: `anneal_sec` existed to bridge
/// gaps between isolated accepted frames, and a track that survives its own
/// gaps leaves it nothing to do (see the AR-012 withdrawal note).
std::vector<ActorWindow> build_actor_windows() {
struct Info { std::string name, imdb_id, tmdb_id, jellyfin_id; };
std::map<int, Info> actor_info;
std::map<int, std::vector<double>> timestamps;
std::lock_guard<std::mutex> g(claims_mu_);
for (const auto& frame : frames_) {
for (const auto& ia : frame.visible_actors) {
if (ia.actor_idx < 0) continue;
actor_info[ia.actor_idx] = {ia.name, ia.imdb_id, ia.tmdb_id, ia.jellyfin_id};
timestamps[ia.actor_idx].push_back(frame.timestamp_sec);
std::map<int, ActorWindow> by_actor;
for (const auto& c : claims_) {
auto& aw = by_actor[c.actor_idx];
if (aw.name.empty()) {
auto it = actor_meta_.find(c.actor_idx);
if (it != actor_meta_.end()) {
aw.name = it->second.name;
aw.imdb_id = it->second.imdb_id;
aw.tmdb_id = it->second.tmdb_id;
aw.jellyfin_id = it->second.jellyfin_id;
}
}
aw.scenes.push_back({c.first_seen, c.last_seen, c.belief});
}
std::vector<ActorWindow> result;
for (auto& [idx, ts_vec] : timestamps) {
ActorWindow aw;
aw.name = actor_info[idx].name;
aw.imdb_id = actor_info[idx].imdb_id;
aw.tmdb_id = actor_info[idx].tmdb_id;
aw.jellyfin_id = actor_info[idx].jellyfin_id;
double win_start = ts_vec[0], win_end = ts_vec[0];
for (size_t i = 1; i < ts_vec.size(); ++i) {
if (ts_vec[i] - win_end > cfg_.anneal_sec) {
aw.scenes.push_back({win_start, win_end});
win_start = ts_vec[i];
}
win_end = ts_vec[i];
}
aw.scenes.push_back({win_start, win_end});
for (auto& [idx, aw] : by_actor) {
std::sort(aw.scenes.begin(), aw.scenes.end(),
[](const Window& a, const Window& b) { return a.start < b.start; });
result.push_back(std::move(aw));
}
return result;
@@ -153,9 +193,16 @@ private:
json build_epochs() {
json actors = json::array();
for (const auto& aw : build_actor_windows()) {
// Objects, not float pairs: a window carries the belief that
// justified it and the route by which it was identified (AR-017),
// so a consumer can caveat or filter rather than treating every
// window as equally certain.
json windows = json::array();
for (const auto& [s, e] : aw.scenes)
windows.push_back({s, e});
for (const auto& w : aw.scenes)
windows.push_back({{"start", w.start},
{"end", w.end},
{"belief", w.belief},
{"route", "live"}});
json ja;
ja["name"] = aw.name;
ja["imdb_id"] = aw.imdb_id;
@@ -173,9 +220,9 @@ private:
json build_xray() {
std::map<int, std::vector<std::string>> xray;
for (const auto& aw : build_actor_windows()) {
for (const auto& [start, end] : aw.scenes) {
int t0 = static_cast<int>(std::floor(start));
int t1 = static_cast<int>(std::ceil(end));
for (const auto& w : aw.scenes) {
int t0 = static_cast<int>(std::floor(w.start));
int t1 = static_cast<int>(std::ceil(w.end));
for (int t = t0; t <= t1; ++t)
xray[t].push_back(aw.name);
}
@@ -226,4 +273,9 @@ private:
std::atomic<bool>& done_;
std::atomic<bool> written_{false};
std::vector<SceneAnnotation> frames_;
std::function<void(double)> pre_write_;
double last_ts_{0.0};
std::mutex claims_mu_;
std::vector<DeadTrack> claims_;
std::map<int, ActorMeta> actor_meta_; ///< actor_idx → identity keys
};
src/scene_preview.cpp
+8 -1
View File
@@ -21,6 +21,7 @@
#include "config.hpp"
#include "types.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/frame_source_node.hpp"
#include "nodes/camera_position_change_detector_node.hpp"
@@ -76,6 +77,7 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--detector-engine")) cfg.detector_engine = next();
else if (arg("--arcface")) cfg.arcface_model = next();
else if (arg("--arcface-engine")) cfg.arcface_engine = next();
else if (arg("--require-gallery-stamp")) cfg.require_gallery_stamp = true;
else if (arg("--conf")) cfg.detector_conf = std::stof(next());
else if (arg("--max-faces")) cfg.max_faces = std::stoi(next());
else if (arg("--min-face-px")) cfg.min_face_px = std::stof(next());
@@ -126,7 +128,12 @@ int main(int argc, char** argv) {
}
ActorGallery gallery;
try { gallery = load_gallery(cfg.gallery_path); }
try {
gallery = load_gallery(cfg.gallery_path);
/// TRACES: GR-004 | SR-001
verify_gallery_embedder(gallery, cfg.gallery_path, cfg.arcface_model,
cfg.require_gallery_stamp);
}
catch (const std::exception& e) {
std::cerr << "Gallery error: " << e.what() << "\n";
return 1;
src/track_registry.hpp
+335
View File
@@ -0,0 +1,335 @@
#pragma once
/// TRACES: AR-012, AR-013, AR-014, AR-015, AR-016, AR-017 | SR-002
///
/// TrackRegistry — the single owner of track state and of presence.
///
/// Presence follows **track extent**, not per-frame recognition (AR-012): a
/// window is `[first_seen, last_seen]` of a track an actor owns, so it starts
/// when the actor appeared rather than when the recogniser first succeeded.
///
/// `last_seen` carries the entire liveness state (AR-013):
///
/// unset → on screen now
/// set → went off screen at that timestamp, still revivable
/// reaped → emitted to the aggregator and erased
///
/// There is no missing-frame counter and no expired flag; the optional *is* the
/// state machine, and it subsumes what was previously a two-pool split in the
/// tracker (active vs. parked-across-a-cut).
///
/// **Interior gaps are claimed, the trailing cool-down is not.** A face lost at
/// t1 and re-associated at t2 within the timeout never closed its track, so the
/// actor is present across [t1, t2] — correct, since someone briefly occluded or
/// off-camera has not left the scene. But a track that dies ends its window at
/// `last_seen`, never at the moment of death. That asymmetry is what removes the
/// over-claim the retired `extinction_sec` keep-alive produced.
///
/// The registry is created in `main` and shared by `shared_ptr`; it is *not* a
/// KPN node. Ownership is not a stage in the stream — it is state several stages
/// read and write, whose final answer is only known when a track dies.
#include "types.hpp"
#include "evidence_discount.hpp"
#include <cmath>
#include <cstdint>
#include <functional>
#include <map>
#include <mutex>
#include <optional>
#include <string>
#include <vector>
// ── DeadTrack ────────────────────────────────────────────────────────────────
// A finished presence claim, emitted exactly once when a track is reaped or
// flushed. Immutable by construction: it carries everything needed to justify
// itself (AR-017), with no back-reference into registry state.
struct DeadTrack {
int track_id{-1};
double first_seen{0.0};
double last_seen{0.0}; ///< always the last sighting, never the death time
int actor_idx{-1}; ///< -1 when the track was never owned
float belief{0.0f}; ///< accumulated posterior for actor_idx
int observations{0}; ///< evidence updates that landed on this track
float effective_obs{0.f}; ///< sum of discounted weights — the evidence that counted
};
// ── Track ────────────────────────────────────────────────────────────────────
struct Track {
int id{-1};
double first_seen{0.0};
std::optional<double> last_seen; ///< unset ⇒ on screen
std::optional<int> actor; ///< set once a posterior crosses
std::map<int, float> belief; ///< actor_idx → accumulated log-odds
Embedding mean{}; ///< running directional mean
std::vector<Embedding> views; ///< distinct looks, for AR-025 discounting
float discounted_weight{0.f}; ///< sum of applied weights
int n_obs{0};
bool on_screen() const { return !last_seen.has_value(); }
};
// ── TrackRegistry ────────────────────────────────────────────────────────────
class TrackRegistry {
public:
using DeadTrackFn = std::function<void(const DeadTrack&)>;
struct Config {
double extinction_sec{5.0}; ///< how long a lost track stays revivable
float ownership_logodds{2.0f}; ///< belief needed to own a track (~0.88 posterior)
};
/// The discounter is a constructor argument rather than an option: there is
/// no correct way to accumulate per-frame evidence without it.
TrackRegistry(Config cfg, EvidenceDiscounter discounter)
: cfg_(cfg), discounter_(std::move(discounter)) {}
void on_track_dead(DeadTrackFn fn) { on_dead_ = std::move(fn); }
// ── Frame scope ──────────────────────────────────────────────────────────
// The tracker mutates registry state across a whole association pass, so
// that pass must be atomic as a unit — per-call locking would let another
// thread observe a half-updated frame. FrameScope holds the lock for its
// lifetime and exposes the mutating operations without re-locking.
class FrameScope {
public:
FrameScope(TrackRegistry& reg, double now)
: reg_(reg), lock_(reg.mu_) { reg_.tick_locked(now); }
/// All live tracks — **one pool**. `last_seen` tells the caller whether
/// IoU is meaningful; a dormant track is matched on embedding alone.
/// There is no separate revival path (AR-008).
std::vector<Track*> candidates() {
std::vector<Track*> out;
out.reserve(reg_.tracks_.size());
for (auto& [id, t] : reg_.tracks_) out.push_back(&t);
return out;
}
int create(double t, const Embedding& e) { return reg_.create_locked(t, e); }
void mark_seen(int id, double t, const Embedding& e){ reg_.mark_seen_locked(id, t, e); }
void mark_lost(int id, double last_on_screen) { reg_.mark_lost_locked(id, last_on_screen); }
private:
TrackRegistry& reg_;
std::unique_lock<std::mutex> lock_;
};
FrameScope begin_frame(double now) { return FrameScope(*this, now); }
/// Advance the clock and reap. Called every sampled frame **whether or not
/// it had detections** — without it a track only dies when some other face
/// happens to appear, and a film ending mid-track never closes.
void tick(double now) { std::lock_guard g(mu_); tick_locked(now); }
// ── Evidence ─────────────────────────────────────────────────────────────
/// Fold one observation into a track's belief (AR-025).
///
/// `posterior` is a **calibrated probability**, never a raw cosine
/// (AR-024) — the registry converts it to log-odds itself, so the
/// accumulation cannot be fed an uncalibrated number by a careless caller.
///
/// Correlation discounting is applied **here**, not by the caller.
/// Consecutive frames of one track are near-identical, and accumulating
/// them as independent evidence drives the posterior to certainty on what
/// is effectively a single measurement. Leaving that to callers would mean
/// a forgotten or doubly-applied discount produces confident wrong answers
/// silently; the registry is the one place all evidence converges, so it is
/// the one place the correction belongs.
///
/// A vote for a track that has already been reaped is dropped and counted:
/// a nonzero `dropped_votes()` means the timeout is shorter than the
/// matcher's lag, which is a real misconfiguration and must not be silent.
void observe(int track_id, int actor_idx, float posterior, const Embedding& e) {
std::lock_guard g(mu_);
auto it = tracks_.find(track_id);
if (it == tracks_.end()) { ++dropped_votes_; return; }
Track& t = it->second;
const float w = discounter_.weight(t.views, e);
t.belief[actor_idx] += w * logit(posterior);
t.discounted_weight += w;
++t.n_obs;
const int best = argmax_belief(t);
const float best_lo = t.belief[best];
if (best_lo < cfg_.ownership_logodds) return;
if (!t.actor.has_value()) {
claim_locked(t, best);
return;
}
if (*t.actor != best) {
// AR-014 — belief swapped A→B. Not a correction: a track_id almost
// certainly carried across a viewpoint change onto a different
// person. Two non-twins both clearing the threshold on one face is
// not realistic; a track spanning two people is. Continuing would
// emit one window blending both, so close here and start afresh.
split_locked(t, best);
}
}
/// Snapshot read: tally and verdict under one lock. Reading them separately
/// would let a track be both unowned and owned within a single promotion
/// decision, since the matcher may be voting concurrently.
std::optional<int> owner(int track_id) const {
std::lock_guard g(mu_);
auto it = tracks_.find(track_id);
return it == tracks_.end() ? std::nullopt : it->second.actor;
}
// ── Termination ──────────────────────────────────────────────────────────
/// Emit every still-live track and empty the registry (AR-016). A film ends
/// with faces on screen and those tracks have not timed out, so without this
/// the closing scene's cast is silently never emitted — a loss that presents
/// as a recognition miss rather than a bookkeeping bug.
///
/// Idempotent: calling it twice emits nothing the second time.
void flush(double final_ts) {
std::lock_guard g(mu_);
for (auto& [id, t] : tracks_) emit_locked(t, t.last_seen.value_or(final_ts));
tracks_.clear();
}
// ── Diagnostics ──────────────────────────────────────────────────────────
// These measure how often tracking is silently wrong, which nothing in the
// pipeline currently reveals.
int dropped_votes() const { std::lock_guard g(mu_); return dropped_votes_; }
int belief_swaps() const { std::lock_guard g(mu_); return belief_swaps_; }
int actor_conflicts() const { std::lock_guard g(mu_); return actor_conflicts_; }
std::size_t live() const { std::lock_guard g(mu_); return tracks_.size(); }
private:
// ── Locked internals ─────────────────────────────────────────────────────
void tick_locked(double now) {
for (auto it = tracks_.begin(); it != tracks_.end(); ) {
const auto& ls = it->second.last_seen;
if (ls && (now - *ls) > cfg_.extinction_sec) {
emit_locked(it->second, *ls);
it = tracks_.erase(it);
} else {
++it;
}
}
}
int create_locked(double t, const Embedding& e) {
const int id = next_id_++;
Track tr;
tr.id = id;
tr.first_seen = t;
tr.mean = e;
tr.n_obs = 0;
tracks_.emplace(id, std::move(tr));
return id;
}
void mark_seen_locked(int id, double t, const Embedding& e) {
auto it = tracks_.find(id);
if (it == tracks_.end()) return;
Track& tr = it->second;
tr.last_seen.reset(); // back on screen; the gap is absorbed
update_mean(tr, e);
(void)t;
}
void mark_lost_locked(int id, double last_on_screen) {
auto it = tracks_.find(id);
if (it == tracks_.end()) return;
it->second.last_seen = last_on_screen;
}
void claim_locked(Track& t, int actor) {
// AR-015 — if another live track already owns this actor, at least one
// is wrong: a person cannot be in two places at once. The cause is the
// same as a belief swap — a missed camera or scene change. Detected on
// the update that causes it via the reverse index, not by scanning.
auto seen = owner_index_.find(actor);
if (seen != owner_index_.end() && seen->second != t.id
&& tracks_.count(seen->second)) {
++actor_conflicts_;
}
t.actor = actor;
owner_index_[actor] = t.id;
}
void split_locked(Track& t, int new_actor) {
++belief_swaps_;
const double boundary = t.last_seen.value_or(t.first_seen);
emit_locked(t, boundary);
// The successor inherits the embedding and the belief that caused the
// swap, and starts at the swap frame — so the two windows abut without
// overlapping and neither blends the two people.
Track next;
next.id = next_id_++;
next.first_seen = boundary;
next.mean = t.mean;
next.belief[new_actor] = t.belief[new_actor];
next.n_obs = 1;
const int old_id = t.id;
Track stash = std::move(next);
tracks_.erase(old_id);
const int nid = stash.id;
tracks_.emplace(nid, std::move(stash));
claim_locked(tracks_.at(nid), new_actor);
}
void emit_locked(Track& t, double end_ts) {
if (!on_dead_) return;
DeadTrack d;
d.track_id = t.id;
d.first_seen = t.first_seen;
d.last_seen = end_ts;
d.observations = t.n_obs;
d.effective_obs = t.discounted_weight;
if (t.actor) {
d.actor_idx = *t.actor;
d.belief = logistic(t.belief[*t.actor]);
auto oi = owner_index_.find(*t.actor);
if (oi != owner_index_.end() && oi->second == t.id) owner_index_.erase(oi);
}
on_dead_(d);
}
static int argmax_belief(const Track& t) {
int best = -1;
float hi = -1e30f;
for (const auto& [a, lo] : t.belief) if (lo > hi) { hi = lo; best = a; }
return best;
}
static void update_mean(Track& t, const Embedding& e) {
// Directional mean: accumulate then re-normalise to the unit sphere, so
// cosine against it stays a plain dot product.
double norm = 0.0;
for (int i = 0; i < 512; ++i) {
t.mean[i] = t.mean[i] * static_cast<float>(t.n_obs ? t.n_obs : 1) + e[i];
norm += static_cast<double>(t.mean[i]) * t.mean[i];
}
norm = norm > 0 ? std::sqrt(norm) : 1.0;
for (int i = 0; i < 512; ++i) t.mean[i] = static_cast<float>(t.mean[i] / norm);
}
static float logistic(float z) {
return z >= 0 ? 1.f / (1.f + std::exp(-z))
: std::exp(z) / (1.f + std::exp(z));
}
static float logit(float p) {
const float eps = 1e-6f;
p = std::min(1.f - eps, std::max(eps, p));
return std::log(p / (1.f - p));
}
Config cfg_;
EvidenceDiscounter discounter_;
mutable std::mutex mu_;
std::map<int, Track> tracks_;
std::map<int, int> owner_index_; ///< actor_idx → live track_id (AR-015)
DeadTrackFn on_dead_;
int next_id_{0};
int dropped_votes_{0};
int belief_swaps_{0};
int actor_conflicts_{0};
};
src/types.hpp
+7
View File
@@ -6,6 +6,8 @@
#include <opencv2/core.hpp>
#include "gallery/embedder_stamp.hpp"
// ── Embedding ─────────────────────────────────────────────────────────────────
// 512-dim L2-normalised ArcFace embedding
using Embedding = std::array<float, 512>;
@@ -136,6 +138,11 @@ struct ActorGallery {
};
std::vector<Actor> actors;
/// TRACES: GR-004 | SR-001
// Which embedder produced every embedding above. Empty == the file predates
// model binding; see gallery/embedder_stamp.hpp for what is checked and why.
EmbedderStamp embedder;
// Cached Platt-sigmoid calibration (see gallery/gallery_calibration.hpp),
// stored alongside the gallery in HDF5 so it never needs recomputing
// unless the reference embeddings actually change. calib_valid=false and
tests/CMakeLists.txt
+12 -1
View File
@@ -21,21 +21,32 @@ add_executable(sae_tests
test_face_utils.cpp
test_track_gallery.cpp
test_face_tracker.cpp
test_track_registry.cpp
test_replay_fixtures.cpp
test_audio_signature.cpp
${CMAKE_SOURCE_DIR}/src/backends/gemm_backend.cpp
${CMAKE_SOURCE_DIR}/src/gallery/gallery_store.cpp
${CMAKE_SOURCE_DIR}/src/audio_signature.cpp
${CMAKE_SOURCE_DIR}/src/gallery/embedder_stamp.cpp
)
target_include_directories(sae_tests PRIVATE ${CMAKE_SOURCE_DIR}/src)
# SAE_GEMM_CPU: build the CPU reference GEMM regardless of the main backend.
# SAE_MODELS_DIR: config.hpp (pulled in by track_gallery.hpp) bakes model paths.
# SAE_TEST_FIXTURES_DIR: the audio golden vector is read from the source tree,
# not copied, so the file the plugin repo shares is the file under test.
target_compile_definitions(sae_tests PRIVATE
SAE_GEMM_CPU
SAE_MODELS_DIR="${SAE_MODELS_DIR}")
SAE_MODELS_DIR="${SAE_MODELS_DIR}"
SAE_TEST_FIXTURES_DIR="${CMAKE_CURRENT_SOURCE_DIR}/fixtures")
# gallery_store.cpp + gallery_calibration.hpp use nlohmann/json and HDF5
# (galleries are HDF5-native, see src/gallery/gallery_store.cpp); face_utils.hpp
# and the calibration GEMM pull in OpenCV (calib3d/imgproc/core) via types.hpp.
# ffmpeg_libs: audio_signature.cpp decodes the golden fixture (avformat/avcodec/
# avutil/swresample). Still GPU-free — the audio path is pure CPU.
target_link_libraries(sae_tests PRIVATE
Catch2::Catch2WithMain
nlohmann_json::nlohmann_json
ffmpeg_libs
${OpenCV_LIBS}
${HDF5_CXX_LIBRARIES})
target_include_directories(sae_tests PRIVATE ${HDF5_INCLUDE_DIRS})
tests/fixtures/audio/jray_audio_v1_golden.json
Vendored
+194
View File
@@ -0,0 +1,194 @@
{
"_": "Golden vector for the JRay v1 audio signature (JRay-public-server SPEC.md \u00a73). Shared verbatim between scene-actor-extraction (C++) and the jRay Jellyfin plugin (C#) so the two implementations can be proven bit-identical. IR-004, IR-005, IR-007, IR-008.",
"version": "v1",
"signature": "v1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAeHx8eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh8fHzk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5V1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dycnJycnJycnJycnJycnJycnJycnJycnJycnJycnMPDgwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKytFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRWNjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2Njfn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5/GxoZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGTc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3UlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU1JsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbAoLCwoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCwsLJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSVDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ15eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eX19eeXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXkXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFzExMTExMTExMTExMTExMTExMTExMTExMTExMTExT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09qampqampqampqampqampqampqampqampqampqamsHBwUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyM+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4/PlhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYd3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3ExMRERERERERERERERERERERERERERERERERERERES8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpLS0plZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZQMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0eHh44ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4OFdXV1ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWV1dXcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXEPDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDyoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKysqRERERERERERERERERERERERERERERERERERERERjY2NjY2NjY2NjYw==",
"frame_count": 1288,
"media": {
"file": "jray_audio_v1_tone.flac",
"generator": "make_fixture.py",
"container": "FLAC (lossless \u2014 decodes to exactly the PCM make_fixture.py emits)",
"duration_sec": 120.0,
"sample_rate": 11025,
"channels": 1,
"sample_format": "s16",
"sha256": "912ecd426cd426dccb37753e0249694227619c701cb9f533502b37da0fbe8096",
"bytes": 585142
},
"decoded_window": {
"_": "Checksums of the 120 s centre window after downmix to mono and resample to 11025 Hz, i.e. exactly the stream `ffmpeg -ss <mid-60> -t 120 -i <file> -vn -ac 1 -ar 11025 -f f32le -` produces. Check these first: a mismatch here is a decode problem, not a DSP one.",
"samples": 1323000,
"f32le_fnv1a64": "0x1ef7899cd4d12662",
"s16le_fnv1a64": "0xf824fa56f125c0dc"
},
"params": {
"window_sec": 120.0,
"window_centre": "runtime/2, i.e. samples from runtime/2 - 60 s; truncated to exactly 1323000 samples",
"min_duration_sec": 120.0,
"min_duration_rule": "IR-007 \u2014 below this emit NO signature and apply no sync offset",
"sample_rate": 11025,
"channels": 1,
"arithmetic": "IEEE-754 double throughout; float32 is not sufficient",
"sample_scale": "s16 * (1/32768), FFmpeg's native s16->flt",
"frame_size": 4096,
"hop_size": 1024,
"frame_count_rule": "1 + (n_samples - 4096) / 1024, integer division; whole frames only",
"window_fn": "Hann, PERIODIC: w[n] = 0.5 * (1 - cos(2*pi*n/4096))",
"transform": "radix-2 DIT complex FFT over the 4096 real samples (imag=0), no normalisation",
"magnitude": "sqrt(re^2 + im^2), linear",
"band_lo_hz": 300.0,
"band_hi_hz": 3000.0,
"num_bands": 32,
"band_edges": "edge[b] = 300 * (3000/300)^(b/32), b = 0..32",
"band_bins": "band b owns FFT bins [k_lo[b], k_lo[b+1]) with k_lo[b] = ceil(edge[b] * 4096 / 11025); see band_fft_bins",
"band_value": "MEAN of the linear magnitudes in the band (not sum, not max)",
"peak_bin": "argmax over the 32 band values; ties resolve to the LOWEST index",
"energy_metric": "E = mean magnitude over all FFT bins 112..1114, i.e. the whole 300-3000 Hz band",
"energy_reference": "upper median of E over all frames: sorted[n/2], no averaging of the two middle values",
"energy_ratio": "r = log10((E + 1e-12) / (E_ref + 1e-12))",
"energy_class_edges": [
-0.6,
-0.2,
0.2
],
"energy_class": "0 if r < -0.6, 1 if r < -0.2, 2 if r < 0.2, else 3",
"byte_layout": "bit7 = 0 (reserved), bits6..2 = 5-bit band index, bits1..0 = 2-bit energy class; byte = (band << 2) | class",
"base64": "standard alphabet A-Za-z0-9+/ with '=' padding",
"prefix": "v1:"
},
"band_fft_bins": [
[
112,
120
],
[
120,
129
],
[
129,
139
],
[
139,
149
],
[
149,
160
],
[
160,
172
],
[
172,
185
],
[
185,
199
],
[
199,
213
],
[
213,
229
],
[
229,
246
],
[
246,
265
],
[
265,
285
],
[
285,
306
],
[
306,
328
],
[
328,
353
],
[
353,
379
],
[
379,
408
],
[
408,
438
],
[
438,
471
],
[
471,
506
],
[
506,
543
],
[
543,
584
],
[
584,
627
],
[
627,
674
],
[
674,
724
],
[
724,
778
],
[
778,
836
],
[
836,
899
],
[
899,
966
],
[
966,
1038
],
[
1038,
1115
]
],
"notes": [
"The server spec fixes the window, rate, STFT geometry, band and the 5+2 bit packing. Everything under params beyond that (Hann periodicity, band aggregation, the energy-class definition, tie-breaking, base64 alphabet) is pinned HERE for v1 \u2014 the spec does not constrain it, and two implementations that guess differently produce non-matching signatures.",
"Decision margins on this fixture: the two strongest bands are within 1.3% on the closest frame, and the closest frame to an energy-class edge is 3.6e-3 away in log10. Both are many orders of magnitude above double-precision FFT differences, so any two correct double- precision implementations agree; a float32 implementation is not guaranteed to.",
"Coverage: all 32 bands and all 4 energy classes appear in the golden signature.",
"Robustness observed on this fixture: identical peak-bin sequence after a stereo/44100 Hz round trip and after AAC 128 kbit/s re-encoding."
]
}
tests/fixtures/audio/jray_audio_v1_tone.flac
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/audio/make_fixture.py
Vendored
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""Regenerate the JRay audio-signature golden fixture.
python3 make_fixture.py # writes jray_audio_v1_tone.flac here
This is the *source of truth* for the fixture media: `jray_audio_v1_tone.flac`
is a lossless FLAC encoding of exactly the PCM this script emits, so any repo
that wants to check its own audio-signature implementation against the golden
vector in `jray_audio_v1_golden.json` can regenerate the input from scratch and
confirm it is byte-identical (the golden file records `pcm_fnv1a64`, a hash of
the decoded 16-bit samples).
Deliberately dependency-free (no numpy) and written in plain arithmetic so it
ports to any language in ~20 lines.
Signal — 120.000 s, mono, 11025 Hz, 16-bit signed PCM:
* split into segments of 32768 samples (~2.97 s), 40.4 segments in total;
* segment `s` carries one sine at the geometric centre of log-band
`(s * 7) mod 32` of the 300-3000 Hz band, so all 32 bands are exercised;
* its amplitude walks a golden-ratio low-discrepancy sequence over
[10^-1.55, 10^-0.02] so frame energies spread continuously across ~1.5
decades and all four energy classes are exercised, without a dense cluster
of frames sitting on a class boundary;
* phase is carried across segment boundaries (no clicks);
* a constant, far quieter 777 Hz tone sits underneath so no frame is
degenerate;
* samples are quantised with floor(x * 32767 + 0.5).
Why FLAC and not WAV: 120 s of 11025 Hz 16-bit PCM is 2.6 MB and does not
compress in git. FLAC is lossless — FFmpeg decodes it to exactly the PCM
written here — and is ~3.5x smaller. `--wav` writes the uncompressed original
if you want to diff it.
"""
import math
import struct
import subprocess
import sys
import os
SAMPLE_RATE = 11025
DURATION_SEC = 120.0
SEGMENT = 32768 # samples per tone segment
BAND_STRIDE = 7 # coprime with 32 -> visits every band
BAND_LO_HZ = 300.0
BAND_HI_HZ = 3000.0
NUM_BANDS = 32
AMP_LOG_MIN = -1.55 # 10^-1.55 ~= 0.028
AMP_LOG_SPAN = 1.53 # up to 10^-0.02 ~= 0.955
PHI_FRAC = 0.6180339887498949
BG_HZ = 777.0
BG_AMP = 0.004
OUT_FLAC = "jray_audio_v1_tone.flac"
OUT_WAV = "jray_audio_v1_tone.wav"
def generate():
"""Return the 120 s signal as a list of int16 sample values."""
n = int(round(SAMPLE_RATE * DURATION_SEC))
out = [0] * n
phase = 0.0
two_pi = 2.0 * math.pi
for start in range(0, n, SEGMENT):
s = start // SEGMENT
end = min(n, start + SEGMENT)
band = (s * BAND_STRIDE) % NUM_BANDS
# geometric centre of log-band `band`
freq = BAND_LO_HZ * (BAND_HI_HZ / BAND_LO_HZ) ** ((band + 0.5) / NUM_BANDS)
amp = 10.0 ** (AMP_LOG_MIN + AMP_LOG_SPAN * ((s * PHI_FRAC) % 1.0))
step = two_pi * freq / SAMPLE_RATE
for k in range(end - start):
i = start + k
x = amp * math.sin(phase + step * k)
x += BG_AMP * math.sin(two_pi * BG_HZ * i / SAMPLE_RATE)
if x > 1.0:
x = 1.0
elif x < -1.0:
x = -1.0
out[i] = int(math.floor(x * 32767.0 + 0.5))
phase = (phase + step * (end - start)) % two_pi
return out
def write_wav(path, samples):
data = struct.pack("<%dh" % len(samples), *samples)
hdr = b"RIFF" + struct.pack("<I", 36 + len(data)) + b"WAVE"
hdr += b"fmt " + struct.pack("<IHHIIHH", 16, 1, 1, SAMPLE_RATE,
SAMPLE_RATE * 2, 2, 16)
hdr += b"data" + struct.pack("<I", len(data))
with open(path, "wb") as fh:
fh.write(hdr + data)
def main():
here = os.path.dirname(os.path.abspath(__file__))
samples = generate()
wav = os.path.join(here, OUT_WAV)
write_wav(wav, samples)
if "--wav" in sys.argv:
print("wrote", wav)
return
flac = os.path.join(here, OUT_FLAC)
# -compression_level 12 is deterministic for a given libFLAC/ffmpeg build;
# only the container bytes vary, never the decoded PCM.
subprocess.run(["ffmpeg", "-nostdin", "-v", "error", "-y", "-i", wav,
"-c:a", "flac", "-compression_level", "12", flac],
check=True)
os.remove(wav)
print("wrote", flac)
if __name__ == "__main__":
main()
tests/fixtures/dumps/bali_13.h5
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/dumps/bali_27.h5
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/dumps/bali_28.h5
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/dumps/bali_31.h5
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/dumps/bali_46.h5
Vendored
BIN
View File
Binary file not shown.
tests/test_audio_signature.cpp
+359
View File
@@ -0,0 +1,359 @@
// Unit tests for the JRay v1 audio signature (src/audio_signature.*).
//
/// TRACES: UT-101, UT-102, UT-103, UT-104 | IR-004, IR-005, IR-007, IR-008
//
// The headline test is the golden vector: a deterministic tone fixture checked
// into tests/fixtures/audio/ together with the signature it must produce. That
// fixture is the artefact shared with the jRay plugin repo, and it is what
// makes "both producers agree bit-for-bit" a checked claim rather than an
// assertion (IR-005).
//
// GPU-free, model-free, no network. Pure CPU DSP plus an FFmpeg decode of a
// 585 KB file — which is precisely why this is the right cross-repo check: it
// runs anywhere, including the N100 CI host.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "audio_signature.hpp"
#include <nlohmann/json.hpp>
#include <cmath>
#include <cstdint>
#include <filesystem>
#include <fstream>
#include <stdexcept>
#include <string>
#include <vector>
using namespace sae::audio;
namespace fs = std::filesystem;
namespace {
const std::string kFixtureDir = SAE_TEST_FIXTURES_DIR "/audio";
const std::string kGoldenPath = kFixtureDir + "/jray_audio_v1_golden.json";
const std::string kMediaPath = kFixtureDir + "/jray_audio_v1_tone.flac";
const nlohmann::json& golden() {
static const nlohmann::json j = [] {
std::ifstream in(kGoldenPath);
if (!in.good())
throw std::runtime_error("golden fixture not found: " + kGoldenPath);
nlohmann::json parsed;
in >> parsed;
return parsed;
}();
return j;
}
std::uint64_t hex64(const std::string& s) {
return std::stoull(s, nullptr, 16);
}
// The fixture is 120 s of audio: decoding and signing it is the expensive part
// of this file, so both results are computed once and shared. Every test below
// still asserts against the on-disk golden values, not against each other.
const std::optional<std::vector<float>>& fixture_window() {
static const std::optional<std::vector<float>> w = decode_centre_window(kMediaPath);
return w;
}
const std::optional<std::string>& fixture_signature() {
static const std::optional<std::string> s = compute_signature(kMediaPath);
return s;
}
// ── Minimal WAV writer, so the short-media and resample cases need no fixture ─
// 16-bit PCM, interleaved.
struct TempWav {
fs::path path;
explicit TempWav(const std::string& name)
: path(fs::temp_directory_path() / ("sae_audio_test_" + name + ".wav")) {}
~TempWav() { std::error_code ec; fs::remove(path, ec); }
void write(const std::vector<std::int16_t>& samples, int rate, int channels) const {
const std::uint32_t bytes = static_cast<std::uint32_t>(samples.size() * 2);
const std::uint32_t byte_rate = static_cast<std::uint32_t>(rate * channels * 2);
std::ofstream out(path, std::ios::binary);
auto u32 = [&](std::uint32_t v) { out.write(reinterpret_cast<const char*>(&v), 4); };
auto u16 = [&](std::uint16_t v) { out.write(reinterpret_cast<const char*>(&v), 2); };
out.write("RIFF", 4); u32(36 + bytes); out.write("WAVE", 4);
out.write("fmt ", 4); u32(16); u16(1); u16(static_cast<std::uint16_t>(channels));
u32(static_cast<std::uint32_t>(rate)); u32(byte_rate);
u16(static_cast<std::uint16_t>(channels * 2)); u16(16);
out.write("data", 4); u32(bytes);
out.write(reinterpret_cast<const char*>(samples.data()), bytes);
}
};
// A plain 1 kHz tone, mono, at the signature's own rate.
std::vector<std::int16_t> tone(double seconds, int rate = kSampleRate) {
const std::size_t n = static_cast<std::size_t>(std::llround(seconds * rate));
std::vector<std::int16_t> s(n);
for (std::size_t i = 0; i < n; ++i)
s[i] = static_cast<std::int16_t>(std::llround(
20000.0 * std::sin(2.0 * 3.14159265358979323846 * 1000.0 * double(i) / rate)));
return s;
}
std::vector<std::uint8_t> base64_decode(const std::string& in) {
auto val = [](char c) -> int {
if (c >= 'A' && c <= 'Z') return c - 'A';
if (c >= 'a' && c <= 'z') return c - 'a' + 26;
if (c >= '0' && c <= '9') return c - '0' + 52;
if (c == '+') return 62;
if (c == '/') return 63;
return -1;
};
std::vector<std::uint8_t> out;
std::uint32_t acc = 0;
int bits = 0;
for (char c : in) {
const int v = val(c);
if (v < 0) continue; // '=' padding
acc = (acc << 6) | static_cast<std::uint32_t>(v);
bits += 6;
if (bits >= 8) {
bits -= 8;
out.push_back(static_cast<std::uint8_t>((acc >> bits) & 0xFF));
}
}
return out;
}
} // namespace
// ── UT-101 — the golden vector ──────────────────────────────────────────────
/// TRACES: UT-101 | IR-004, IR-005, IR-008
TEST_CASE("signature of the golden fixture matches the recorded value exactly",
"[audio_signature][golden]") {
REQUIRE(fs::exists(kMediaPath));
const std::optional<std::string>& sig = fixture_signature();
REQUIRE(sig.has_value());
CHECK(*sig == golden()["signature"].get<std::string>());
}
/// TRACES: UT-101 | IR-005
TEST_CASE("decoded centre window matches the recorded PCM checksum",
"[audio_signature][golden]") {
// Checked separately from the signature so a codec-level difference is
// distinguishable from a DSP-level one: if this passes and the signature
// test fails, the DSP diverged; if this fails, the decode did.
const std::optional<std::vector<float>>& mono = fixture_window();
REQUIRE(mono.has_value());
CHECK(mono->size() == golden()["decoded_window"]["samples"].get<std::size_t>());
CHECK(fnv1a64(mono->data(), mono->size() * sizeof(float)) ==
hex64(golden()["decoded_window"]["f32le_fnv1a64"].get<std::string>()));
}
/// TRACES: UT-101 | IR-004
TEST_CASE("log-spaced band table matches the recorded one", "[audio_signature][golden]") {
// The band->FFT-bin table is the part of the construction most likely to
// drift between two implementations, so it is pinned independently of the
// signature it produces.
const auto& tbl = band_fft_bins();
const auto& want = golden()["band_fft_bins"];
REQUIRE(want.size() == tbl.size());
for (std::size_t b = 0; b < tbl.size(); ++b) {
CHECK(tbl[b].first == want[b][0].get<int>());
CHECK(tbl[b].second == want[b][1].get<int>());
CHECK(tbl[b].second > tbl[b].first); // no empty band
if (b) CHECK(tbl[b].first == tbl[b - 1].second); // contiguous, no overlap
}
}
/// TRACES: UT-101 | IR-004, IR-008
TEST_CASE("signature is well-formed: v1 prefix, 1288 frames, structural bytes",
"[audio_signature][golden]") {
const std::optional<std::string>& sig = fixture_signature();
REQUIRE(sig.has_value());
// IR-008 — the signature carries its own version, separate from
// schema_version, so a future DSP change is detectable rather than silently
// producing non-matching signatures.
REQUIRE(sig->rfind(kVersionPrefix, 0) == 0);
const std::vector<std::uint8_t> bytes = base64_decode(sig->substr(3));
CHECK(bytes.size() == kExpectedFrames);
CHECK(bytes.size() == golden()["frame_count"].get<std::size_t>());
// The server validates this structure on upload (server SPEC §3): each byte
// is a 5-bit band index plus a 2-bit energy class, so bit 7 is always clear
// and arbitrary bytes are invalid. That is what keeps the field from being
// a payload channel.
bool bands_seen[kNumBands] = {};
bool classes_seen[4] = {};
for (std::uint8_t b : bytes) {
REQUIRE((b & 0x80) == 0);
bands_seen[(b >> 2) & 0x1F] = true;
classes_seen[b & 0x03] = true;
}
// The fixture is built to exercise the whole output alphabet — if it ever
// stops doing so, the golden vector has become a weaker check than it looks.
for (bool seen : bands_seen) CHECK(seen);
for (bool seen : classes_seen) CHECK(seen);
}
// ── UT-102 — IR-007, media shorter than the window ──────────────────────────
/// TRACES: UT-102 | IR-007
TEST_CASE("media shorter than 120 s emits no signature", "[audio_signature][short]") {
// The window runtime/2 ± 60 s underflows, so there is no signature and no
// sync offset downstream. Both producers must apply the identical rule or
// they diverge on exactly the short items most likely to be misidentified.
SECTION("30 s") {
TempWav w("short30");
w.write(tone(30.0), kSampleRate, 1);
CHECK_FALSE(compute_signature(w.path.string()).has_value());
CHECK_FALSE(decode_centre_window(w.path.string()).has_value());
}
SECTION("just under the boundary") {
TempWav w("short11999");
w.write(tone(119.99), kSampleRate, 1);
CHECK_FALSE(compute_signature(w.path.string()).has_value());
}
}
/// TRACES: UT-102 | IR-007
TEST_CASE("media of exactly 120 s emits a full-length signature",
"[audio_signature][short]") {
TempWav w("exact120");
w.write(tone(120.0), kSampleRate, 1);
const std::optional<std::string> sig = compute_signature(w.path.string());
REQUIRE(sig.has_value());
CHECK(base64_decode(sig->substr(3)).size() == kExpectedFrames);
}
/// TRACES: UT-102 | IR-007
TEST_CASE("unreadable media degrades to no signature rather than failing",
"[audio_signature][short]") {
// UR-9 is an enhancement and must never be able to break a fetch.
CHECK_FALSE(compute_signature("/nonexistent/definitely-not-here.mkv").has_value());
}
/// TRACES: UT-102 | IR-004
TEST_CASE("the window is taken from the centre, not the head",
"[audio_signature][centre]") {
// Sampling from the centre is the whole reason the construction avoids the
// head and tail (logos, cold opens, credits), so it needs its own check:
// wrap the fixture's own 120 s in 90 s of silence either side and the
// signature of the 300 s file must be the golden value, byte for byte.
// Nothing else pins the seek offset — a head-anchored window would pass
// every other test in this file.
const std::optional<std::vector<float>>& mono = fixture_window();
REQUIRE(mono.has_value());
const std::size_t pad = 90 * kSampleRate;
std::vector<std::int16_t> padded(pad * 2 + mono->size(), 0);
for (std::size_t i = 0; i < mono->size(); ++i)
padded[pad + i] = static_cast<std::int16_t>(std::llround(double((*mono)[i]) * 32768.0));
TempWav w("centred300");
w.write(padded, kSampleRate, 1);
const std::optional<std::string> sig = compute_signature(w.path.string());
REQUIRE(sig.has_value());
CHECK(*sig == golden()["signature"].get<std::string>());
}
// ── UT-103 — downmix and resample ───────────────────────────────────────────
/// TRACES: UT-103 | IR-004
TEST_CASE("stereo, non-native sample rate yields the same peak-bin sequence",
"[audio_signature][resample]") {
// The golden fixture is already mono at 11025 Hz so the golden vector does
// not depend on the resampler's version. This case exercises the path that
// real media takes — downmix plus resample — by rebuilding the fixture's own
// audio as 22050 Hz stereo and checking the peak bins survive it.
const std::optional<std::vector<float>>& mono = fixture_window();
REQUIRE(mono.has_value());
std::vector<std::int16_t> stereo;
stereo.reserve(mono->size() * 4);
for (float f : *mono) {
const auto s = static_cast<std::int16_t>(std::llround(double(f) * 32768.0));
stereo.push_back(s); stereo.push_back(s); // sample 1, L/R
stereo.push_back(s); stereo.push_back(s); // sample 2 (zero-order hold)
}
TempWav w("stereo22050");
w.write(stereo, 2 * kSampleRate, 2);
const std::optional<std::string> sig = compute_signature(w.path.string());
REQUIRE(sig.has_value());
const std::vector<std::uint8_t> got = base64_decode(sig->substr(3));
const std::vector<std::uint8_t> want =
base64_decode(golden()["signature"].get<std::string>().substr(3));
REQUIRE(got.size() == want.size());
std::size_t agree = 0;
for (std::size_t i = 0; i < got.size(); ++i)
agree += ((got[i] >> 2) == (want[i] >> 2)) ? 1 : 0;
// The server treats ≥ 0.85 as the `audio` match tier; this path scores 1.0
// in practice, and the margin is left for libswresample version drift.
CHECK(double(agree) / double(got.size()) >= 0.85);
}
// ── UT-104 — the pure DSP surface ───────────────────────────────────────────
/// TRACES: UT-104 | IR-004
TEST_CASE("pack_frames uses whole frames only", "[audio_signature][dsp]") {
CHECK(pack_frames(std::vector<float>(kFrameSize - 1, 0.f)).empty());
CHECK(pack_frames(std::vector<float>(kFrameSize, 0.f)).size() == 1);
CHECK(pack_frames(std::vector<float>(kFrameSize + kHopSize - 1, 0.f)).size() == 1);
CHECK(pack_frames(std::vector<float>(kFrameSize + kHopSize, 0.f)).size() == 2);
// The full 120 s window is 1288 frames — asserted as a constant rather than
// by running the DSP over 1.3 M zeros, which is the same claim for free.
CHECK(kWindowSamples == 1323000u);
CHECK(kExpectedFrames == 1288u);
CHECK_FALSE(signature_from_mono(std::vector<float>(kFrameSize - 1, 0.f)).has_value());
}
/// TRACES: UT-104 | IR-004
TEST_CASE("a pure tone lands in the band that contains it", "[audio_signature][dsp]") {
// 1000 Hz sits in log-band floor(32 * log10(1000/300)) = 16.
const int expect = static_cast<int>(std::floor(
kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));
std::vector<float> mono(kWindowSamples / 100);
for (std::size_t i = 0; i < mono.size(); ++i)
mono[i] = static_cast<float>(0.5 * std::sin(
2.0 * 3.14159265358979323846 * 1000.0 * double(i) / kSampleRate));
const std::vector<std::uint8_t> packed = pack_frames(mono);
REQUIRE_FALSE(packed.empty());
for (std::uint8_t b : packed) CHECK(((b >> 2) & 0x1F) == expect);
}
/// TRACES: UT-104 | IR-004
TEST_CASE("signature is invariant to overall gain", "[audio_signature][dsp]") {
// Loudness normalisation between two releases of the same cut must not
// change the signature — that is why the energy class is relative.
std::vector<float> a(kWindowSamples / 50);
for (std::size_t i = 0; i < a.size(); ++i) {
const double t = double(i) / kSampleRate;
a[i] = static_cast<float>(0.4 * std::sin(2.0 * 3.14159265358979323846 * 640.0 * t) +
0.2 * std::sin(2.0 * 3.14159265358979323846 * 1900.0 * t) *
std::sin(2.0 * 3.14159265358979323846 * 0.7 * t));
}
std::vector<float> b(a.size());
for (std::size_t i = 0; i < a.size(); ++i) b[i] = a[i] * 0.25f;
CHECK(pack_frames(a) == pack_frames(b));
}
/// TRACES: UT-104 | IR-004
TEST_CASE("base64 encoder matches the standard alphabet and padding",
"[audio_signature][dsp]") {
auto enc = [](const std::string& s) {
return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());
};
CHECK(enc("") == "");
CHECK(enc("f") == "Zg==");
CHECK(enc("fo") == "Zm8=");
CHECK(enc("foo") == "Zm9v");
CHECK(enc("foob") == "Zm9vYg==");
CHECK(enc("fooba") == "Zm9vYmE=");
CHECK(enc("foobar") == "Zm9vYmFy");
const std::uint8_t all[] = {0xFB, 0xFF, 0xBF}; // exercises '+' and '/'
CHECK(base64_encode(all, 3) == "+/+/");
}
tests/test_face_tracker.cpp
+117 -53
View File
@@ -13,8 +13,12 @@
#include "config.hpp"
#include "nodes/face_tracker_node.hpp"
#include "types.hpp"
#include "track_registry.hpp"
#include "evidence_discount.hpp"
#include <algorithm>
#include <cmath>
#include <memory>
namespace {
@@ -55,85 +59,145 @@ EmbeddedSceneFrame frame(double t, float x, float y, const Embedding& emb,
return ef;
}
Config tracker_cfg() {
Config cfg;
cfg.cut_revive_sim = 0.50f;
cfg.cut_inactive_max_frames = 5;
return cfg;
}
// Build a tracker over a fresh registry. The registry IS the tracker's state
// now (AR-008), so a test constructs both together and can inspect either.
struct Rig {
std::shared_ptr<TrackRegistry> reg;
FaceTrackerFunc ft;
explicit Rig(double extinction = 30.0, float assoc_min_prob = 0.5f)
: reg(std::make_shared<TrackRegistry>(
[extinction] {
TrackRegistry::Config c;
c.extinction_sec = extinction;
return c;
}(),
EvidenceDiscounter([](float cos) { return std::max(0.f, cos); })))
, ft([&] {
Config c;
c.track_assoc_min_prob = assoc_min_prob;
return c;
}(),
reg,
// Trivial calibration: cosine passed through as P(same). Real runs use
// the fit belonging to the active embedder (AR-023/AR-024).
[](float cos) { return std::max(0.f, cos); })
{}
int track_of(EmbeddedSceneFrame f) { return ft(std::move(f)).track_ids[0]; }
};
} // namespace
TEST_CASE("track id is stable across ordinary frames", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
// ── AR-008 — one pool, ordinary association ──────────────────────────────────
TEST_CASE("track id is stable across ordinary frames", "[face_tracker][AR-008]") {
Rig r;
Embedding e = axis(0);
int id0 = ft(frame(0.0, 10, 10, e)).track_ids[0];
int id1 = ft(frame(1.0, 11, 10, e)).track_ids[0]; // overlaps → same track
int id0 = r.track_of(frame(0.0, 10, 10, e));
int id1 = r.track_of(frame(1.0, 11, 10, e)); // overlaps → same track
CHECK(id0 >= 0);
CHECK(id1 == id0);
}
TEST_CASE("cut revives the same track id for a matching identity", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
TEST_CASE("a face lost across a cut and re-associated is the SAME track",
"[face_tracker][AR-008]") {
// Previously this was a distinct "revival" path guarded by a raw-cosine
// constant. There is no such path now: a dormant track is an ordinary
// association candidate, and continuity falls out of the embedding match.
Rig r;
// Pre-cut: establish a track for a person whose embedding is near-identical
// across the cut (sim well above cut_revive_sim), but whose box jumps so IoU
// is 0 — the ordinary spatial path cannot re-link it.
Embedding pre = at_sim(0, 1, 0.99f);
int id_pre = ft(frame(0.0, 10, 10, pre)).track_ids[0];
int id_pre = r.track_of(frame(0.0, 10, 10, pre));
REQUIRE(id_pre >= 0);
Embedding post = at_sim(0, 1, 0.98f); // cos(diff) ≈ 0.9997 > 0.50
auto out = ft(frame(1.0, 300, 300, post, /*is_cut=*/true));
CHECK(out.track_ids[0] == id_pre); // revived, not a fresh id
// Box jumps so IoU is zero — only the embedding can link it.
Embedding post = at_sim(0, 1, 0.98f);
CHECK(r.track_of(frame(1.0, 300, 300, post, /*is_cut=*/true)) == id_pre);
}
TEST_CASE("cut starts a fresh track when identity does not match", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
int id_pre = ft(frame(0.0, 10, 10, axis(0))).track_ids[0];
TEST_CASE("a cut starts a fresh track when identity does not match",
"[face_tracker][AR-008]") {
Rig r;
int id_pre = r.track_of(frame(0.0, 10, 10, axis(0)));
REQUIRE(id_pre >= 0);
// Post-cut face is orthogonal (sim 0 < cut_revive_sim) and spatially disjoint
// → no revival, brand-new id.
auto out = ft(frame(1.0, 300, 300, axis(5), /*is_cut=*/true));
CHECK(out.track_ids[0] != id_pre);
CHECK(out.track_ids[0] >= 0);
// Orthogonal embedding and disjoint box: nothing links them.
int id_post = r.track_of(frame(1.0, 300, 300, axis(5), /*is_cut=*/true));
CHECK(id_post != id_pre);
CHECK(id_post >= 0);
}
TEST_CASE("parked track expires after cut_inactive_max_frames", "[face_tracker]") {
Config cfg = tracker_cfg();
cfg.cut_inactive_max_frames = 2;
FaceTrackerFunc ft(cfg);
// ── AR-007 — a cut makes association ignore position ─────────────────────────
TEST_CASE("on a cut, identity follows the embedding rather than the box",
"[face_tracker][AR-007]") {
// Two people swap screen positions across a cut while keeping their faces.
// If IoU still carried weight the ids would follow the boxes and swap; with
// alpha driven to embedding-only on a cut, they must follow the faces.
Rig r;
Embedding a = at_sim(0, 1, 0.99f);
Embedding b = at_sim(2, 3, 0.99f);
EmbeddedSceneFrame f0;
f0.source.timestamp_sec = 0.0;
f0.faces = {face_at(10, 10), face_at(300, 300)};
f0.crops = {cv::Mat(), cv::Mat()};
f0.embeddings = {a, b};
auto out0 = r.ft(std::move(f0));
const int id_a = out0.track_ids[0];
const int id_b = out0.track_ids[1];
REQUIRE(id_a >= 0);
REQUIRE(id_b >= 0);
REQUIRE(id_a != id_b);
// Same two people, positions exchanged, on a cut frame.
EmbeddedSceneFrame f1;
f1.source.timestamp_sec = 1.0;
f1.source.is_cut = true;
f1.faces = {face_at(300, 300), face_at(10, 10)};
f1.crops = {cv::Mat(), cv::Mat()};
f1.embeddings = {a, b};
auto out1 = r.ft(std::move(f1));
CHECK(out1.track_ids[0] == id_a); // A kept its id despite moving to B's box
CHECK(out1.track_ids[1] == id_b);
}
// ── AR-013 — extinction replaces the parked-pool frame counter ───────────────
TEST_CASE("a track past the extinction window is gone, not revived",
"[face_tracker][AR-013]") {
// The old design aged a parked pool in frames, which silently changed
// meaning with sample_fps. Extinction is in seconds and lives in the
// registry, so the tracker no longer counts anything.
Rig r(/*extinction=*/2.0);
Embedding person = at_sim(0, 1, 0.99f);
int id_pre = ft(frame(0.0, 10, 10, person)).track_ids[0];
int id_pre = r.track_of(frame(0.0, 10, 10, person));
REQUIRE(id_pre >= 0);
// Cut with an unrelated face parks id_pre; then let the pool age past its
// limit with more unrelated, spatially-disjoint faces (each ages the pool by
// one). By the time the person returns, id_pre must be gone.
ft(frame(1.0, 300, 300, axis(7), /*is_cut=*/true)); // park (age 1)
ft(frame(2.0, 300, 300, axis(7))); // age 2
ft(frame(3.0, 300, 300, axis(7))); // age 3 → id_pre dropped
// Unrelated faces elsewhere while the clock runs well past extinction.
r.track_of(frame(1.0, 300, 300, axis(7), /*is_cut=*/true));
r.track_of(frame(10.0, 300, 300, axis(7)));
auto out = ft(frame(4.0, 10, 10, person)); // same identity returns
CHECK(out.track_ids[0] != id_pre); // too late — fresh id
CHECK(r.track_of(frame(11.0, 10, 10, person)) != id_pre);
}
TEST_CASE("eof clears active and parked tracks", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
Embedding person = at_sim(0, 1, 0.99f);
int id_pre = ft(frame(0.0, 10, 10, person)).track_ids[0];
ft(frame(1.0, 300, 300, axis(7), /*is_cut=*/true)); // park id_pre
TEST_CASE("a track within the extinction window is still a candidate",
"[face_tracker][AR-013]") {
Rig r(/*extinction=*/30.0);
Embedding person = at_sim(0, 1, 0.99f);
int id_pre = r.track_of(frame(0.0, 10, 10, person));
r.track_of(frame(1.0, 300, 300, axis(7), /*is_cut=*/true));
// Back inside the window: the same person continues the same track, so the
// gap is absorbed into one window rather than splitting it.
CHECK(r.track_of(frame(3.0, 10, 10, person)) == id_pre);
}
TEST_CASE("eof is forwarded", "[face_tracker]") {
Rig r;
EmbeddedSceneFrame eof;
eof.source.eof = true;
auto out = ft(std::move(eof));
CHECK(out.source.eof);
// After eof the pools are empty: the returning identity must get a fresh id,
// not the parked one.
auto out2 = ft(frame(2.0, 10, 10, person));
CHECK(out2.track_ids[0] != id_pre);
CHECK(r.ft(std::move(eof)).source.eof);
}
tests/test_gallery_store.cpp
+251 -1
View File
@@ -1,13 +1,17 @@
// Unit tests for gallery (de)serialisation: HDF5 round-trip fidelity (the only
// format save_gallery writes), legacy JSON read back-compat (optional field
// defaults, the legacy "jellyfin_person_id" fallback). GPU-free, model-free.
// defaults, the legacy "jellyfin_person_id" fallback), and the GR-004 embedder
// stamp. GPU-free, model-free — the stamp tests exercise the comparison logic
// with synthetic stamps and never load an ONNX, so they run on CI's Intel N100.
#include <catch2/catch_test_macros.hpp>
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "types.hpp"
#include <cstdio>
#include <fstream>
#include <stdexcept>
#include <string>
#include <nlohmann/json.hpp>
@@ -27,6 +31,22 @@ Embedding make_embedding(float base) {
return e;
}
// A stamp built by hand — no ONNX is read, so these tests never need a model.
EmbedderStamp stamp(const std::string& name, const std::string& sha, int32_t dim = 512) {
EmbedderStamp s;
s.model_name = name;
s.model_sha256 = sha;
s.embed_dim = dim;
return s;
}
const std::string kShaA(64, 'a');
const std::string kShaB(64, 'b');
bool mentions(const std::string& haystack, const std::string& needle) {
return haystack.find(needle) != std::string::npos;
}
} // namespace
TEST_CASE("gallery save/load round-trips actors and embeddings", "[gallery]") {
@@ -153,3 +173,233 @@ TEST_CASE("load_gallery reads the legacy jellyfin_person_id key", "[gallery]") {
TEST_CASE("load_gallery throws on a missing file", "[gallery]") {
CHECK_THROWS(load_gallery("/nonexistent/path/gallery.json"));
}
// ── GR-004: gallery ↔ embedder binding ───────────────────────────────────────
// Verification plan row GR-004/T1: "Mismatched embedder → hard startup error;
// error names both sides." The comparison is a pure function over two stamps, so
// none of this needs a GPU, an ONNX, or even a file.
/// TRACES: GR-004 | SR-001
TEST_CASE("gallery save/load round-trips the embedder stamp", "[gallery][GR-004]") {
ActorGallery g;
ActorGallery::Actor a;
a.name = "Stamped Actor";
a.embeddings = {make_embedding(0.3f)};
g.actors.push_back(a);
g.embedder = stamp("LVFace-B_Glint360K.onnx", kShaA);
TempFile tf("gallery_stamped.h5");
save_gallery(tf.path, g);
ActorGallery loaded = load_gallery(tf.path);
CHECK(loaded.embedder.model_name == "LVFace-B_Glint360K.onnx");
CHECK(loaded.embedder.model_sha256 == kShaA);
CHECK(loaded.embedder.embed_dim == 512);
CHECK_FALSE(loaded.embedder.empty());
}
/// TRACES: GR-004 | SR-001
TEST_CASE("a gallery written without a stamp loads as unstamped", "[gallery][GR-004]") {
// The back-compat case: pre-GR-004 files have no /embedder group at all. The
// absence must survive the round trip as an absence — a stamp naming no model
// would read as "checked and fine" to every consumer.
ActorGallery g;
ActorGallery::Actor a;
a.name = "Legacy Actor";
a.embeddings = {make_embedding(0.f)};
g.actors.push_back(a);
TempFile tf("gallery_unstamped.h5");
save_gallery(tf.path, g);
ActorGallery loaded = load_gallery(tf.path);
CHECK(loaded.embedder.empty());
}
/// TRACES: GR-004 | SR-001
TEST_CASE("legacy JSON galleries carry an optional embedder stamp", "[gallery][GR-004]") {
nlohmann::json j;
j["embedder"] = {{"model_name", "arcface_w600k_r50.onnx"},
{"model_sha256", kShaB},
{"embed_dim", 512}};
j["actors"] = nlohmann::json::array();
nlohmann::json ja;
ja["name"] = "JSON Actor";
ja["embeddings"] = nlohmann::json::array();
ja["embeddings"].push_back(std::vector<float>(512, 0.1f));
j["actors"].push_back(ja);
TempFile tf("gallery_json_stamp.json");
{ std::ofstream out(tf.path); out << j.dump(); }
ActorGallery g = load_gallery(tf.path);
CHECK(g.embedder.model_name == "arcface_w600k_r50.onnx");
CHECK(g.embedder.model_sha256 == kShaB);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("matching embedder stamps pass", "[gallery][GR-004]") {
auto chk = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaA));
CHECK(chk.verdict == StampVerdict::match);
CHECK_FALSE(chk.fatal(false));
CHECK_FALSE(chk.fatal(true)); // a proven match is never fatal, even in strict mode
CHECK_NOTHROW(enforce_embedder_stamp(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaA),
"g.h5", "model.onnx", true));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("the hash decides, not the filename", "[gallery][GR-004]") {
// Same bytes under a different filename is the SAME model — a renamed or
// relocated file must not be treated as a different one.
auto same = compare_embedder_stamps(stamp("lvface.onnx", kShaA),
stamp("LVFace-B_Glint360K.onnx", kShaA));
CHECK(same.verdict == StampVerdict::match);
// Different bytes under the SAME filename is a DIFFERENT model — this is the
// in-place re-export a name-only stamp would miss entirely, and the reason the
// stamp carries a hash at all.
auto differ = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaB));
CHECK(differ.verdict == StampVerdict::mismatch);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("mismatched embedder is fatal and names both sides", "[gallery][GR-004]") {
const auto built = stamp("LVFace-B_Glint360K.onnx", kShaA);
const auto loaded = stamp("arcface_w600k_r50.onnx", kShaB);
auto chk = compare_embedder_stamps(built, loaded, "cast.h5", "models/r50.onnx");
REQUIRE(chk.verdict == StampVerdict::mismatch);
CHECK(chk.fatal(false)); // no bypass: a mismatch is fatal in every mode
CHECK(chk.fatal(true));
// Both sides must be identifiable from the message alone.
CHECK(mentions(chk.message, "LVFace-B_Glint360K.onnx"));
CHECK(mentions(chk.message, "arcface_w600k_r50.onnx"));
CHECK(mentions(chk.message, kShaA));
CHECK(mentions(chk.message, kShaB));
CHECK(mentions(chk.message, "cast.h5"));
CHECK(mentions(chk.message, "models/r50.onnx"));
// ...and it must reach the caller as an error, not a log line.
CHECK_THROWS_AS(enforce_embedder_stamp(built, loaded, "cast.h5",
"models/r50.onnx", false),
std::runtime_error);
try {
enforce_embedder_stamp(built, loaded, "cast.h5", "models/r50.onnx", false);
FAIL("mismatch must throw");
} catch (const std::runtime_error& e) {
const std::string what = e.what();
CHECK(mentions(what, "LVFace-B_Glint360K.onnx"));
CHECK(mentions(what, "arcface_w600k_r50.onnx"));
}
}
/// TRACES: GR-004 | SR-001
TEST_CASE("differing embedding width is a mismatch", "[gallery][GR-004]") {
auto chk = compare_embedder_stamps(stamp("a.onnx", kShaA, 512),
stamp("a.onnx", kShaA, 256));
CHECK(chk.verdict == StampVerdict::mismatch);
CHECK(mentions(chk.message, "512"));
CHECK(mentions(chk.message, "256"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("an unstamped gallery warns by default and fails under strict",
"[gallery][GR-004]") {
// Decision recorded in src/gallery/embedder_stamp.hpp: unstamped is UNKNOWN,
// not known-bad, and every pre-GR-004 gallery is unstamped. Hard-failing them
// all would make the check something people disable rather than trust; so it
// warns loudly, names the risk, and is promotable to fatal for measurement runs.
EmbedderStamp none;
auto chk = compare_embedder_stamps(none, stamp("model.onnx", kShaA), "old.h5");
REQUIRE(chk.verdict == StampVerdict::unstamped);
CHECK_FALSE(chk.fatal(false));
CHECK(chk.fatal(true));
CHECK(mentions(chk.message, "old.h5"));
CHECK(mentions(chk.message, "model.onnx")); // the loaded side is still named
CHECK(mentions(chk.message, "UNKNOWN")); // ...and the gallery side is honest
CHECK_NOTHROW(enforce_embedder_stamp(none, stamp("model.onnx", kShaA),
"old.h5", "model.onnx", false));
CHECK_THROWS_AS(enforce_embedder_stamp(none, stamp("model.onnx", kShaA),
"old.h5", "model.onnx", true),
std::runtime_error);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("an unidentifiable embedder against a stamped gallery is not silent",
"[gallery][GR-004]") {
// e.g. a replay whose dump predates GR-004: we know what built the gallery but
// not what produced the vectors being fed in. Unverifiable, so it must not
// report success.
auto chk = compare_embedder_stamps(stamp("model.onnx", kShaA), EmbedderStamp{},
"g.h5", "old dump.h5");
CHECK(chk.verdict == StampVerdict::unknown_embedder);
CHECK_FALSE(chk.fatal(false));
CHECK(chk.fatal(true));
CHECK(mentions(chk.message, "model.onnx"));
CHECK(mentions(chk.message, "old dump.h5"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("name-only agreement is a weak match, not a clean pass", "[gallery][GR-004]") {
// A TRT deployment can run from a prebuilt .engine with the .onnx absent, so
// no hash is computable. Names agreeing is evidence, not proof.
auto weak = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", ""));
CHECK(weak.verdict == StampVerdict::weak_match);
CHECK_FALSE(weak.fatal(false));
CHECK(weak.fatal(true));
// Names disagreeing with no hash available is still a mismatch — the weaker
// evidence is enough to convict, just not to acquit.
auto bad = compare_embedder_stamps(stamp("lvface.onnx", ""),
stamp("arcface.onnx", ""));
CHECK(bad.verdict == StampVerdict::mismatch);
CHECK(mentions(bad.message, "lvface.onnx"));
CHECK(mentions(bad.message, "arcface.onnx"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("sha256 matches the published vectors", "[gallery][GR-004]") {
// Pins the in-tree FIPS 180-4 implementation against the standard vectors.
// This is what guarantees the C++ stamp and the Python (hashlib) stamp in
// scripts/sae_gallery.py agree on the same model file — without it the two
// halves of GR-004 could silently diverge and every check would be a mismatch.
CHECK(sha256_hex("") ==
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855");
CHECK(sha256_hex("abc") ==
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad");
CHECK(sha256_hex("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq") ==
"248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1");
// Multi-block input, exercising the length-padding path past 64 bytes.
CHECK(sha256_hex(std::string(1000, 'a')) ==
"41edece42d63e8d9bf515a9ba6932e1c20cbc9f5a5d134645adb5db1b9737ea3");
}
/// TRACES: GR-004 | SR-001
TEST_CASE("make_embedder_stamp hashes a real file and degrades gracefully",
"[gallery][GR-004]") {
// Stands in for an ONNX: the stamp does not care what the bytes mean.
TempFile tf("fake_model.onnx");
{ std::ofstream out(tf.path, std::ios::binary); out << "abc"; }
EmbedderStamp s = make_embedder_stamp(tf.path);
CHECK(s.model_sha256 ==
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad");
CHECK_FALSE(s.model_name.empty());
CHECK(s.model_name.find('/') == std::string::npos); // basename, not full path
// A model path that does not exist still yields a comparable name-only stamp
// rather than an empty one, which is what keeps engine-only deployments usable.
EmbedderStamp missing = make_embedder_stamp("/nonexistent/models/foo.onnx");
CHECK(missing.model_name == "foo.onnx");
CHECK(missing.model_sha256.empty());
CHECK_FALSE(missing.empty());
CHECK(make_embedder_stamp("").empty());
}
tests/test_replay_fixtures.cpp
+258
View File
@@ -0,0 +1,258 @@
// Replay tests — the real tracker and registry driven from committed fixtures.
//
// TRACES: AR-012, AR-013, AR-004, VR-001, VR-002 | IT-001
//
// Tier T2: composition, not units. The registry tests construct awkward states
// directly; these check that the pieces behave when wired together and fed real
// footage — 480x360 public-domain clips at 5 fps, with the cuts, gaps and
// crowded frames that actual film produces and synthetic input does not.
//
// No GPU and no model: the fixtures are HDF5 dumps taken after embedding, so
// everything here is CPU maths. That is what lets this run on the CI host at
// all (see docs/requirements.md, "CI never calls a model").
//
// Driving the node functors directly rather than through a KPN network is
// deliberate: functors are plain objects, so there are no threads, no channels
// and no scheduling — the same input gives the same output every time, which is
// exactly what a fixture-based test needs.
#include <catch2/catch_test_macros.hpp>
#include "config.hpp"
#include "evidence_discount.hpp"
#include "nodes/face_tracker_node.hpp"
#include "track_registry.hpp"
#include "types.hpp"
#include <H5Cpp.h>
#include <algorithm>
#include <cmath>
#include <memory>
#include <string>
#include <vector>
namespace {
// ── Fixture reader ───────────────────────────────────────────────────────────
// The flat/ragged layout of scripts/optimizer/SCHEMA.md: per-face arrays
// concatenated, with a per-frame index table pointing into them.
struct Dump {
std::vector<double> ts;
std::vector<uint8_t> is_cut;
std::vector<int64_t> face_offset;
std::vector<int32_t> face_count;
std::vector<Embedding> emb;
std::vector<float> bbox; // 4 per face
std::string embedder;
std::size_t frames() const { return ts.size(); }
std::size_t faces() const { return emb.size(); }
};
template <typename T>
std::vector<T> read1d(H5::Group& g, const char* name, const H5::DataType& dt) {
H5::DataSet ds = g.openDataSet(name);
hsize_t n = 0;
ds.getSpace().getSimpleExtentDims(&n, nullptr);
std::vector<T> out(n);
if (n) ds.read(out.data(), dt);
return out;
}
Dump load(const std::string& path) {
H5::H5File f(path, H5F_ACC_RDONLY);
H5::Group frames = f.openGroup("frames");
H5::Group faces = f.openGroup("faces");
Dump d;
d.ts = read1d<double>(frames, "timestamp_sec", H5::PredType::NATIVE_DOUBLE);
d.is_cut = read1d<uint8_t>(frames, "is_cut", H5::PredType::NATIVE_UINT8);
d.face_offset = read1d<int64_t>(frames, "face_offset", H5::PredType::NATIVE_INT64);
d.face_count = read1d<int32_t>(frames, "face_count", H5::PredType::NATIVE_INT32);
H5::DataSet e = faces.openDataSet("embedding");
hsize_t dims[2]{0, 0};
e.getSpace().getSimpleExtentDims(dims, nullptr);
std::vector<float> flat(dims[0] * dims[1]);
if (!flat.empty()) e.read(flat.data(), H5::PredType::NATIVE_FLOAT);
d.emb.resize(dims[0]);
for (hsize_t i = 0; i < dims[0]; ++i)
std::copy_n(flat.begin() + i * dims[1], 512, d.emb[i].begin());
// bbox is 2-D [N,4]; reading it with the 1-D helper would size the buffer
// from the first extent only and then read four times that many floats.
{
H5::DataSet bs = faces.openDataSet("bbox");
hsize_t bd[2]{0, 0};
bs.getSpace().getSimpleExtentDims(bd, nullptr);
d.bbox.resize(bd[0] * bd[1]);
if (!d.bbox.empty()) bs.read(d.bbox.data(), H5::PredType::NATIVE_FLOAT);
}
// GR-004: the dump records which embedder produced it, so a replay cannot
// be silently scored against a gallery from a different model.
if (f.attrExists("embedder_model")) {
// Written as a variable-length string (embedding_dump_node.hpp:99), so
// the read must name the same type explicitly.
H5::StrType vlen(H5::PredType::C_S1, H5T_VARIABLE);
f.openAttribute("embedder_model").read(vlen, d.embedder);
}
return d;
}
std::string fixture(const char* name) {
return std::string(SAE_TEST_FIXTURES_DIR) + "/dumps/" + name;
}
// ── Harness ──────────────────────────────────────────────────────────────────
struct Replay {
std::vector<DeadTrack> claims;
std::vector<int> track_ids; // per face, in fixture order
std::size_t faces_seen{0};
};
Replay run(const Dump& d, double extinction = 10.0) {
Replay r;
TrackRegistry::Config rc;
rc.extinction_sec = extinction;
auto cal = [](float cos) { return std::max(0.f, cos); };
auto reg = std::make_shared<TrackRegistry>(rc, EvidenceDiscounter(cal));
reg->on_track_dead([&r](const DeadTrack& t) { r.claims.push_back(t); });
Config cfg;
cfg.track_assoc_min_prob = 0.5f;
FaceTrackerFunc ft(cfg, reg, cal);
for (std::size_t i = 0; i < d.frames(); ++i) {
EmbeddedSceneFrame ef;
ef.source.timestamp_sec = d.ts[i];
ef.source.is_cut = d.is_cut[i] != 0;
const int64_t off = d.face_offset[i];
const int32_t n = d.face_count[i];
for (int32_t k = 0; k < n; ++k) {
DetectedFace face;
const float* b = &d.bbox[(off + k) * 4];
face.bbox = cv::Rect2f(b[0], b[1], b[2], b[3]);
face.confidence = 1.0f;
ef.faces.push_back(face);
ef.crops.push_back(cv::Mat());
ef.embeddings.push_back(d.emb[off + k]);
}
r.faces_seen += static_cast<std::size_t>(n);
auto out = ft(std::move(ef));
for (int id : out.track_ids) r.track_ids.push_back(id);
}
reg->flush(d.ts.empty() ? 0.0 : d.ts.back());
return r;
}
} // namespace
// ── AR-004 / VR-001 — the fixtures are intact and self-describing ────────────
TEST_CASE("fixtures are complete and carry their embedder identity",
"[replay][AR-004][VR-001]") {
// Frame counts are exact rather than approximate. Before node outputs
// blocked on a full channel, generation lost most of a clip and what it
// lost depended on timing — these numbers could not have been asserted.
struct Expect { const char* file; std::size_t frames, faces; };
const Expect all[] = {
{"bali_13.h5", 385, 693},
{"bali_27.h5", 335, 335},
{"bali_28.h5", 345, 368},
{"bali_31.h5", 145, 203},
{"bali_46.h5", 385, 140},
};
for (const auto& x : all) {
INFO(x.file);
Dump d = load(fixture(x.file));
CHECK(d.frames() == x.frames);
CHECK(d.faces() == x.faces);
CHECK(d.embedder == "LVFace-B_Glint360K.onnx");
// face_offset must be contiguous: a gap means faces went missing
// between frames, which no consumer could detect.
int64_t running = 0;
for (std::size_t i = 0; i < d.frames(); ++i) {
REQUIRE(d.face_offset[i] == running);
running += d.face_count[i];
}
CHECK(static_cast<std::size_t>(running) == d.faces());
}
}
// ── VR-002 — replay is deterministic ─────────────────────────────────────────
TEST_CASE("replaying a fixture twice gives identical tracks", "[replay][VR-002]") {
// The property the whole fixture strategy rests on. If this fails, every
// golden output derived from a fixture is unreliable and the CI replay
// tier is worthless.
Dump d = load(fixture("bali_28.h5"));
Replay a = run(d);
Replay b = run(d);
REQUIRE(a.track_ids.size() == b.track_ids.size());
CHECK(a.track_ids == b.track_ids);
REQUIRE(a.claims.size() == b.claims.size());
for (std::size_t i = 0; i < a.claims.size(); ++i) {
CHECK(a.claims[i].first_seen == b.claims[i].first_seen);
CHECK(a.claims[i].last_seen == b.claims[i].last_seen);
}
}
// ── AR-012 / AR-013 — window invariants on real footage ──────────────────────
TEST_CASE("every face is assigned a track and every track closes",
"[replay][AR-012]") {
Dump d = load(fixture("bali_13.h5"));
Replay r = run(d);
CHECK(r.track_ids.size() == r.faces_seen);
for (int id : r.track_ids) CHECK(id >= 0); // nothing silently unassigned
// flush() must leave nothing behind: a track still open at EOF would be a
// window that never reaches the output.
CHECK(r.claims.size() > 0);
}
TEST_CASE("windows are well-formed and inside the clip", "[replay][AR-013]") {
for (const char* f : {"bali_13.h5", "bali_27.h5", "bali_28.h5",
"bali_31.h5", "bali_46.h5"}) {
INFO(f);
Dump d = load(fixture(f));
Replay r = run(d);
const double t0 = d.ts.front(), t1 = d.ts.back();
for (const auto& c : r.claims) {
// A window ends at the last sighting, never after it — so it can
// never extend past the footage that produced it.
CHECK(c.first_seen <= c.last_seen);
CHECK(c.first_seen >= t0);
CHECK(c.last_seen <= t1);
}
}
}
TEST_CASE("a longer extinction window yields fewer, longer tracks",
"[replay][AR-013]") {
// The timeout decides whether a gap is absorbed into one window or splits
// it in two, so lengthening it must merge tracks rather than multiply them.
// On sparse footage this is the difference the constant actually makes.
Dump d = load(fixture("bali_46.h5")); // 140 faces over 385 frames
Replay tight = run(d, /*extinction=*/1.0);
Replay loose = run(d, /*extinction=*/30.0);
CHECK(loose.claims.size() <= tight.claims.size());
}
// ── AR-007 — cuts are exercised by the corpus, not just by construction ──────
TEST_CASE("the cut-heavy fixture actually contains cuts", "[replay][AR-007]") {
// Guards the corpus rather than the code: if a regeneration produced a
// fixture with no cuts, the association tests above would still pass while
// silently testing nothing about viewpoint changes.
Dump d = load(fixture("bali_28.h5"));
const int cuts = std::count(d.is_cut.begin(), d.is_cut.end(), uint8_t{1});
CHECK(cuts >= 5);
}
tests/test_track_registry.cpp
+332
View File
@@ -0,0 +1,332 @@
// Unit tests for TrackRegistry (track_registry.hpp): presence as track extent.
//
// TRACES: AR-012, AR-013, AR-014, AR-015, AR-016, AR-017 | UT-001
//
// Pure, GPU-free, model-free — drives the registry directly with synthetic
// timestamps and evidence. Node functors and this registry are plain objects
// constructed outside the KPN network, so the awkward cases can be built
// exactly rather than hunted for in a clip: a gap one frame under the timeout,
// a belief swap, two live tracks converging on one actor, a film ending
// mid-track.
#include <catch2/catch_test_macros.hpp>
#include "track_registry.hpp"
#include "evidence_discount.hpp"
#include <algorithm>
#include <vector>
namespace {
Embedding axis(int slot) {
Embedding e{};
e[slot] = 1.0f;
return e;
}
// Collects the claims a registry emits, which is the whole observable output.
struct Sink {
std::vector<DeadTrack> claims;
void attach(TrackRegistry& r) {
r.on_track_dead([this](const DeadTrack& d) { claims.push_back(d); });
}
const DeadTrack* forActor(int a) const {
for (const auto& c : claims) if (c.actor_idx == a) return &c;
return nullptr;
}
};
// A discounter whose calibration is deliberately trivial, so the tests exercise
// registry behaviour rather than a fitted sigmoid.
EvidenceDiscounter disc() {
return EvidenceDiscounter([](float cos) { return std::max(0.f, cos); });
}
TrackRegistry::Config cfg(double extinction = 5.0, float own = 2.0f) {
TrackRegistry::Config c;
c.extinction_sec = extinction;
c.ownership_logodds = own;
return c;
}
} // namespace
// ── AR-012 — the change this whole redesign exists for ───────────────────────
TEST_CASE("window starts at first sighting, not at first recognition", "[registry][AR-012]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(10.0); id = f.create(10.0, axis(0)); }
// Seen for 20s but only recognised at the very end — the pose was wrong
// until then. This is the case the old per-frame design got wrong: it would
// have reported presence starting at 30, not 10.
for (double t = 11.0; t <= 30.0; t += 1.0) {
auto f = reg.begin_frame(t);
f.mark_seen(id, t, axis(0));
}
reg.observe(id, 7, 0.99f, axis(7));
{ auto f = reg.begin_frame(31.0); f.mark_lost(id, 30.0); }
reg.tick(40.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == 7);
CHECK(sink.claims[0].first_seen == 10.0); // ← not 30.0
CHECK(sink.claims[0].last_seen == 30.0);
}
// ── AR-013 — the asymmetry that removes the old over-claim ───────────────────
TEST_CASE("interior gaps are absorbed; the trailing cool-down is not",
"[registry][AR-013]") {
TrackRegistry reg(cfg(/*extinction=*/5.0), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 3, 0.99f, axis(3));
// Off screen at 10, back at 13 — inside the timeout, so the same track
// continues and the actor is claimed present *through* the gap.
{ auto f = reg.begin_frame(10.0); f.mark_lost(id, 10.0); }
{ auto f = reg.begin_frame(13.0); f.mark_seen(id, 13.0, axis(0)); }
CHECK(sink.claims.empty()); // nothing closed
CHECK(reg.live() == 1);
// Lost for good at 20. The window must end there, not at the death time.
{ auto f = reg.begin_frame(20.0); f.mark_lost(id, 20.0); }
reg.tick(20.0 + 5.0 + 0.001);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].first_seen == 0.0);
CHECK(sink.claims[0].last_seen == 20.0); // ← not 25.001
}
TEST_CASE("a gap past the timeout yields two tracks, not one", "[registry][AR-013]") {
TrackRegistry reg(cfg(/*extinction=*/5.0), disc());
Sink sink; sink.attach(reg);
int a;
{ auto f = reg.begin_frame(0.0); a = f.create(0.0, axis(0)); }
reg.observe(a, 1, 0.99f, axis(1));
{ auto f = reg.begin_frame(10.0); f.mark_lost(a, 10.0); }
reg.tick(30.0); // well past extinction
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].last_seen == 10.0);
// A face reappearing after the timeout is genuinely a new track: past the
// re-acquisition window there are no grounds to assert continuity.
int b;
{ auto f = reg.begin_frame(31.0); b = f.create(31.0, axis(0)); }
CHECK(b != a);
}
// ── AR-016 — the silent-loss guard ───────────────────────────────────────────
TEST_CASE("EOF flush closes tracks still on screen", "[registry][AR-016]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(100.0); id = f.create(100.0, axis(0)); }
reg.observe(id, 5, 0.99f, axis(5));
// A film almost always ends with faces on screen; these have not timed out.
reg.flush(/*final_ts=*/120.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == 5);
CHECK(sink.claims[0].last_seen == 120.0);
sink.claims.clear();
reg.flush(130.0);
CHECK(sink.claims.empty()); // idempotent
CHECK(reg.live() == 0);
}
TEST_CASE("flush closes a lost-but-unreaped track at its last sighting",
"[registry][AR-016]") {
TrackRegistry reg(cfg(/*extinction=*/60.0), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 2, 0.99f, axis(2));
{ auto f = reg.begin_frame(10.0); f.mark_lost(id, 10.0); }
reg.flush(/*final_ts=*/50.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].last_seen == 10.0); // last sighting, not EOF
}
// ── AR-014 — belief swap is a track boundary, not a correction ───────────────
TEST_CASE("belief swap closes one window and opens another", "[registry][AR-014]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 1, 0.99f, axis(1)); // owned by actor 1
{ auto f = reg.begin_frame(5.0); f.mark_lost(id, 5.0); }
// The swap must out-accumulate the incumbent, not merely tie it: one
// contrary observation is noise, and a tie leaves ownership where it is.
reg.observe(id, 2, 0.99f, axis(2));
reg.observe(id, 2, 0.99f, axis(3));
CHECK(reg.belief_swaps() == 1);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == 1);
CHECK(sink.claims[0].last_seen == 5.0); // closed at its last sighting
// The successor is a distinct track, so nothing blends the two people.
reg.flush(9.0);
const DeadTrack* second = sink.forActor(2);
REQUIRE(second != nullptr);
CHECK(second->track_id != id);
CHECK(second->first_seen == 5.0); // abuts, does not overlap
}
// ── AR-015 — identity contradiction as a cut detector ────────────────────────
TEST_CASE("two live tracks owned by one actor is counted", "[registry][AR-015]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int a, b;
{ auto f = reg.begin_frame(0.0); a = f.create(0.0, axis(0)); b = f.create(0.0, axis(1)); }
reg.observe(a, 9, 0.99f, axis(9));
CHECK(reg.actor_conflicts() == 0);
// One person cannot be in two places at once, so this is a missed camera or
// scene change that split them — detected on the update that causes it.
reg.observe(b, 9, 0.99f, axis(9));
CHECK(reg.actor_conflicts() == 1);
}
// ── AR-017 / diagnostics ─────────────────────────────────────────────────────
TEST_CASE("an unowned track emits no claim", "[registry][AR-012]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 4, 0.62f, axis(4)); // never clears the ownership threshold
{ auto f = reg.begin_frame(1.0); f.mark_lost(id, 1.0); }
reg.tick(100.0);
// Someone was there, but nothing can be claimed about who.
CHECK(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == -1);
}
TEST_CASE("claims carry the belief that justified them", "[registry][AR-017]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 6, 0.99f, axis(6));
reg.flush(1.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].belief > 0.9f); // logistic(4.0) ≈ 0.982
CHECK(sink.claims[0].observations == 1);
}
TEST_CASE("a vote for a reaped track is dropped and counted", "[registry][AR-013]") {
TrackRegistry reg(cfg(/*extinction=*/1.0), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
{ auto f = reg.begin_frame(1.0); f.mark_lost(id, 1.0); }
reg.tick(10.0); // reaped
// The matcher runs downstream of the tracker, so a late vote is expected.
// Silently ignoring it would hide a timeout shorter than the matcher's lag.
reg.observe(id, 3, 0.99f, axis(3));
CHECK(reg.dropped_votes() == 1);
}
TEST_CASE("a single-frame track yields a zero-length window", "[registry][AR-012]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(42.0); id = f.create(42.0, axis(0)); }
reg.observe(id, 8, 0.99f, axis(8));
{ auto f = reg.begin_frame(43.0); f.mark_lost(id, 42.0); }
reg.tick(100.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].first_seen == 42.0);
CHECK(sink.claims[0].last_seen == 42.0);
}
// ── AR-025 — correlated observations must not accumulate as independent ──────
TEST_CASE("repeated identical views do not reach the certainty of distinct ones",
"[registry][AR-025]") {
// Thirty frames of the same face at the same angle is not thirty pieces of
// evidence. Without discounting, log-odds accumulate linearly and the
// posterior saturates on what is effectively a single measurement.
TrackRegistry same(cfg(), disc());
TrackRegistry varied(cfg(), disc());
Sink s_same, s_varied;
s_same.attach(same);
s_varied.attach(varied);
int a, b;
{ auto f = same.begin_frame(0.0); a = f.create(0.0, axis(0)); }
{ auto f = varied.begin_frame(0.0); b = f.create(0.0, axis(0)); }
for (int i = 0; i < 8; ++i) {
same.observe(a, 1, 0.9f, axis(0)); // the identical view, every time
varied.observe(b, 1, 0.9f, axis(i + 1)); // a genuinely new look each time
}
same.flush(1.0);
varied.flush(1.0);
REQUIRE(s_same.claims.size() == 1);
REQUIRE(s_varied.claims.size() == 1);
// Same raw observation count, but only the varied track earned the evidence.
CHECK(s_same.claims[0].observations == s_varied.claims[0].observations);
CHECK(s_same.claims[0].effective_obs < s_varied.claims[0].effective_obs);
CHECK(s_same.claims[0].effective_obs < 2.0f); // ~one view's worth
}
TEST_CASE("the first observation on a track always counts in full",
"[registry][AR-025]") {
// There is nothing for it to be redundant with.
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 1, 0.9f, axis(0));
reg.flush(1.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].effective_obs == 1.0f);
}
TEST_CASE("the registry takes a probability, not a cosine", "[registry][AR-024]") {
// A posterior at the decision boundary must not move belief at all: 0.5
// carries no information either way, and its log-odds are zero. Feeding a
// raw cosine here would be silently wrong rather than obviously so, which
// is why the conversion lives inside the registry.
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 1, 0.5f, axis(0));
reg.flush(1.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == -1); // never owned
}
traceability.toml
+44
View File
@@ -0,0 +1,44 @@
# Traceability configuration for scene-actor-extraction.
#
# Read by the shared extractor (scripts/traceability/extract_traces.py), which
# is the same implementation every JRay component uses. Everything repo-specific
# lives here rather than in the tool; run `extract_traces.py
# --print-example-config` for the annotated schema.
#
# This file's directory is taken as the repo root, so the gate works from any
# subdirectory.
# The prefixes this repo's register defines. Nothing else enters the fraction:
# UT/IT are evidence for requirements, PR/SR belong to the system spec.
requirement_types = ["AR", "DP", "IR", "GR", "VR"]
# C++ pipeline plus the Python tooling, optimizer and validation scripts.
languages = ["cpp", "python"]
source_roots = ["src", "tests", "scripts", "experiments", "eval"]
# exclude_dirs is deliberately NOT set. The tool's defaults already exclude
# `vendor` (among __pycache__, external, build, node_modules and friends), which
# covers the submodule at scripts/vendor/jray-project — that is not this repo's
# code, and its parser tests carry literal TRACES: strings that would otherwise
# be credited here as coverage.
#
# Note the key REPLACES the defaults rather than adding to them, and matching is
# on path components, not prefixes: setting it to ["scripts/vendor"] both fails
# to match anything and silently drops every default exclusion.
# CI is an Intel N100 with no discrete GPU. T4 is deliberately absent: a
# requirement verifiable only on GPU hardware is reported as tagged but
# unexecuted and never counted as covered, because counting a test that cannot
# run is the same failure mode as JellyTau's 158% coverage bug.
ci_executable_tiers = ["T1", "T2", "T3", "static"]
# Threshold policy. 0 today because almost nothing is tagged yet - tags land as
# the pipeline is built. This is not a gate that cannot fail: orphan tags, a
# >100% ratio, a register that parses to nothing and an empty source scan are
# all hard failures already. Ratchet this up as tags land; never reset it down.
min_coverage = 0.0
# The system spec owning PR/SR is vendored per-component as a submodule. Point
# at it once that lands to turn on PR/SR orphan checking:
system_spec = "scripts/vendor/jray-project/SPEC.md"