fix: belief accumulates across frames (lazy-OR), not once
A track recognised on 318 of 385 frames was owned on none, so the truth file named nobody while the matcher was accepting almost continuously. The correlation discount was an annihilator rather than an attenuator. Weight was 1 - P(same view), so once a track had one stored view every later frame of that same face scored ~0.01 and the belief stopped moving. One observation just over the accept threshold is logit(0.78) ~ 1.27, under the ownership bar — hence recognised always, owned never. Two changes, in the order they were found. Correlated evidence is now attenuated by effective sample size, n_eff = n / (1 + (n-1)·rho), each frame contributing the marginal gain. That has the right shape at both ends: uncorrelated evidence accumulates linearly, and a held pose converges on 1/rho rather than growing without bound. A constant floor was tried first and rejected — it grows linearly forever, so a long shot could out-argue genuinely varied evidence purely by lasting longer. Combination is now weighted lazy-OR: P = 1 - (1-P_old)·(1-p)^w, stored as log(1-P) so the update is additive and precision stays where it matters as P approaches 1. Each frame is new evidence that this track is that actor, and the belief is the probability that at least one sighting was right. It converges faster than summing log-odds at the same effective count — 2.98 vs 2.53 after two observations at p=0.78 — which is what a real clip needs. Note that summing log-odds was already a correct sequential Bayesian update: the matcher fits with prior 0.5, so logit(p) IS the per-frame log-likelihood ratio and the running sum carries the prior forward. It was not wrong, it was slow. What blocked ownership was the discount, not the combination rule. Also fixes a real correctness bug: the observation count lived on the discounter, which is shared by every track, so tracks pooled into one effective sample and each was discounted by how many others happened to be on screen. It is now a per-track parameter. The registry's frame scope holds its lock for its lifetime and the mutex is not recursive, so calling observe() inside a scope self-deadlocks. The pipeline never does — separate nodes — but the test did, and hung rather than failing. Documented at the call site. Verified end to end: the same clip that produced zero actors now identifies Bing Crosby and Dorothy Lamour with belief 0.97. Suite: 96 cases, 6142 assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: AR-025 | SR-002
This commit is contained in:
@@ -0,0 +1,125 @@
|
|||||||
|
// sae_audio — Python module wrapping the v1 audio signature (audio_signature.*).
|
||||||
|
//
|
||||||
|
/// TRACES: IR-004, IR-005 | SR-003
|
||||||
|
//
|
||||||
|
// Exists so a study or a test can drive the **shipped** signature code from
|
||||||
|
// Python instead of porting the DSP to numpy. A numpy port would be a third
|
||||||
|
// implementation of a fingerprint that only works if every implementation
|
||||||
|
// agrees byte for byte, and it would be the one nobody checks against the
|
||||||
|
// golden vector — so the offset-recovery validation (VR-014) calls this.
|
||||||
|
//
|
||||||
|
// Bound with nanobind, as `sae_embed` and `sae_kpn` are. Not pybind11: a second
|
||||||
|
// binding framework in one build is a second set of ABI and lifetime rules to
|
||||||
|
// get right, for a module that needs nothing nanobind lacks.
|
||||||
|
//
|
||||||
|
// The module deliberately stops at the producer's edge. Matching — sliding one
|
||||||
|
// signature against another and scoring the overlap — is the *consumer's*
|
||||||
|
// algorithm (server SPEC §3, and the jRay plugin implements it), so it is not
|
||||||
|
// bound here and a caller writing a slide in numpy is not re-implementing
|
||||||
|
// anything this repo owns.
|
||||||
|
|
||||||
|
#include "audio_signature.hpp"
|
||||||
|
|
||||||
|
#include <nanobind/nanobind.h>
|
||||||
|
#include <nanobind/ndarray.h>
|
||||||
|
#include <nanobind/stl/optional.h>
|
||||||
|
#include <nanobind/stl/pair.h>
|
||||||
|
#include <nanobind/stl/string.h>
|
||||||
|
#include <nanobind/stl/vector.h>
|
||||||
|
|
||||||
|
#include <cstdint>
|
||||||
|
#include <optional>
|
||||||
|
#include <string>
|
||||||
|
#include <utility>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
namespace nb = nanobind;
|
||||||
|
using namespace nb::literals;
|
||||||
|
using namespace sae::audio;
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
|
||||||
|
using MonoArray = nb::ndarray<const float, nb::ndim<1>, nb::c_contig, nb::device::cpu>;
|
||||||
|
|
||||||
|
// Hand the vector's buffer to Python without copying 1.3 M samples, and let a
|
||||||
|
// capsule own it: the array outlives this call, so the storage has to as well.
|
||||||
|
nb::object own_as_ndarray(std::vector<float>&& samples) {
|
||||||
|
auto* held = new std::vector<float>(std::move(samples));
|
||||||
|
nb::capsule owner(held, [](void* p) noexcept {
|
||||||
|
delete static_cast<std::vector<float>*>(p);
|
||||||
|
});
|
||||||
|
const std::size_t n = held->size();
|
||||||
|
return nb::cast(nb::ndarray<nb::numpy, float, nb::ndim<1>>(held->data(), {n}, owner));
|
||||||
|
}
|
||||||
|
|
||||||
|
std::vector<float> to_vector(const MonoArray& a) {
|
||||||
|
return std::vector<float>(a.data(), a.data() + a.shape(0));
|
||||||
|
}
|
||||||
|
|
||||||
|
} // namespace
|
||||||
|
|
||||||
|
NB_MODULE(sae_audio, m) {
|
||||||
|
m.doc() =
|
||||||
|
"JRay v1 audio signature (JRay-public-server SPEC.md section 3), as the "
|
||||||
|
"extraction pipeline computes it. The constants below are the contract: "
|
||||||
|
"changing any of them is a v1 -> v2 change.";
|
||||||
|
|
||||||
|
m.attr("sample_rate") = kSampleRate;
|
||||||
|
m.attr("frame_size") = kFrameSize;
|
||||||
|
m.attr("hop_size") = kHopSize;
|
||||||
|
m.attr("num_bands") = kNumBands;
|
||||||
|
m.attr("band_lo_hz") = kBandLoHz;
|
||||||
|
m.attr("band_hi_hz") = kBandHiHz;
|
||||||
|
m.attr("window_sec") = kWindowSec;
|
||||||
|
m.attr("window_samples") = kWindowSamples;
|
||||||
|
m.attr("expected_frames") = kExpectedFrames;
|
||||||
|
m.attr("version_prefix") = std::string(kVersionPrefix);
|
||||||
|
|
||||||
|
m.def(
|
||||||
|
"compute_signature",
|
||||||
|
[](const std::string& path) { return compute_signature(path); },
|
||||||
|
"path"_a,
|
||||||
|
"Signature of the 120 s window centred on the media's midpoint, or None "
|
||||||
|
"for media shorter than the window (IR-007), media with no audio "
|
||||||
|
"stream, and any decode failure — degradation, never an exception.");
|
||||||
|
|
||||||
|
m.def(
|
||||||
|
"decode_centre_window",
|
||||||
|
[](const std::string& path) -> nb::object {
|
||||||
|
std::optional<std::vector<float>> mono = decode_centre_window(path);
|
||||||
|
if (!mono) {
|
||||||
|
return nb::none();
|
||||||
|
}
|
||||||
|
|
||||||
|
return own_as_ndarray(std::move(*mono));
|
||||||
|
},
|
||||||
|
"path"_a,
|
||||||
|
"The decoded centre window as float32 mono at 11025 Hz, or None. Exposed "
|
||||||
|
"so a caller can slice or perturb real audio and re-sign it without "
|
||||||
|
"going back through a container.");
|
||||||
|
|
||||||
|
m.def(
|
||||||
|
"signature_from_mono",
|
||||||
|
[](const MonoArray& mono) { return signature_from_mono(to_vector(mono)); },
|
||||||
|
"mono"_a,
|
||||||
|
"Signature of mono float32 samples already at 11025 Hz, in [-1, 1). None "
|
||||||
|
"when fewer than one whole frame is given.");
|
||||||
|
|
||||||
|
m.def(
|
||||||
|
"pack_frames",
|
||||||
|
[](const MonoArray& mono) {
|
||||||
|
std::vector<std::uint8_t> packed = pack_frames(to_vector(mono));
|
||||||
|
return nb::bytes(reinterpret_cast<const char*>(packed.data()), packed.size());
|
||||||
|
},
|
||||||
|
"mono"_a,
|
||||||
|
"One packed byte per whole STFT frame: (band << 2) | energy_class. This "
|
||||||
|
"is the payload the signature base64-encodes.");
|
||||||
|
|
||||||
|
m.def(
|
||||||
|
"band_fft_bins",
|
||||||
|
[] {
|
||||||
|
const auto& table = band_fft_bins();
|
||||||
|
return std::vector<std::pair<int, int>>(table.begin(), table.end());
|
||||||
|
},
|
||||||
|
"The half-open FFT bin range owned by each of the 32 log-spaced bands.");
|
||||||
|
}
|
||||||
@@ -41,7 +41,26 @@ public:
|
|||||||
struct Config {
|
struct Config {
|
||||||
int max_views{8}; ///< distinct views remembered per track
|
int max_views{8}; ///< distinct views remembered per track
|
||||||
float admit_below{0.6f}; ///< P(same view) under this ⇒ a new view
|
float admit_below{0.6f}; ///< P(same view) under this ⇒ a new view
|
||||||
float floor{0.0f}; ///< minimum weight for a redundant observation
|
|
||||||
|
/// Ceiling on the correlation between two observations of one track.
|
||||||
|
///
|
||||||
|
/// This is what bounds the accumulation. `n_eff = n / (1 + (n-1)·rho)`
|
||||||
|
/// tends to `1/rho` as `n` grows, so `rho_max` sets how much a single
|
||||||
|
/// repeated view can ever be worth: 0.5 caps it at two observations,
|
||||||
|
/// no matter how long the shot runs.
|
||||||
|
///
|
||||||
|
/// 0.5 caps a repeated view at two independent observations' worth,
|
||||||
|
/// which is what lets a track the matcher accepts on frame after frame
|
||||||
|
/// actually become owned. Higher values starve ownership; the sweep
|
||||||
|
/// (VR-007) decides where it belongs.
|
||||||
|
///
|
||||||
|
/// It is capped below 1 deliberately. P(same view) near 1 says the two
|
||||||
|
/// crops look alike; it does not say the second carries no information.
|
||||||
|
/// A fresh frame is a fresh detection, a fresh alignment and a fresh
|
||||||
|
/// noise realisation, so a little independent evidence survives even a
|
||||||
|
/// perfectly held pose. Setting this to 1 recovers the original bug —
|
||||||
|
/// belief frozen after the first frame.
|
||||||
|
float rho_max{0.5f};
|
||||||
};
|
};
|
||||||
|
|
||||||
// Two constructors rather than a defaulted argument: `Config{}` as a default
|
// Two constructors rather than a defaulted argument: `Config{}` as a default
|
||||||
@@ -53,12 +72,36 @@ public:
|
|||||||
EvidenceDiscounter(Calibrate cal, Config cfg)
|
EvidenceDiscounter(Calibrate cal, Config cfg)
|
||||||
: cal_(std::move(cal)), cfg_(cfg) {}
|
: cal_(std::move(cal)), cfg_(cfg) {}
|
||||||
|
|
||||||
/// Weight in [0,1] for one observation, updating `views` when the
|
/// The marginal evidence one observation adds, in units of independent
|
||||||
/// observation is novel enough to count as a distinct look at the subject.
|
/// observations.
|
||||||
///
|
///
|
||||||
/// The first observation on a track always counts in full: there is nothing
|
/// Each frame is a Bayesian update, so confidence must keep growing — but
|
||||||
/// for it to be redundant with.
|
/// correlated observations must grow it less, and must not grow it without
|
||||||
float weight(std::vector<Embedding>& views, const Embedding& e) const {
|
/// bound. The standard treatment is **effective sample size**:
|
||||||
|
///
|
||||||
|
/// n_eff(n) = n / (1 + (n-1)·rho)
|
||||||
|
///
|
||||||
|
/// and this returns `n_eff(n) - n_eff(n-1)`, the gain from *this* frame.
|
||||||
|
/// The shape is right at both ends: with rho = 0 every frame counts fully
|
||||||
|
/// and the belief accumulates linearly, while as rho rises the series
|
||||||
|
/// converges on `1/rho` and a held pose stops adding no matter how long it
|
||||||
|
/// is held.
|
||||||
|
///
|
||||||
|
/// The two failure modes it sits between are both real and both were hit:
|
||||||
|
/// a weight of 0 for repeats froze the belief after one frame, so a track
|
||||||
|
/// recognised on 318 frames was owned on none; a constant floor grew it
|
||||||
|
/// linearly forever, so a long shot could out-argue genuinely varied
|
||||||
|
/// evidence purely by lasting longer.
|
||||||
|
///
|
||||||
|
/// `rho` is estimated from P(same view) against the closest stored view,
|
||||||
|
/// capped by `rho_max`. The first observation has nothing to be redundant
|
||||||
|
/// with and counts in full.
|
||||||
|
/// `n_seen` is the count of observations already folded into THIS track.
|
||||||
|
/// It is a parameter rather than discounter state because one discounter
|
||||||
|
/// serves every track: holding the count internally would pool unrelated
|
||||||
|
/// tracks into one effective sample, so a busy film would silently discount
|
||||||
|
/// each track by how many others happened to be on screen.
|
||||||
|
float weight(std::vector<Embedding>& views, int n_seen, const Embedding& e) const {
|
||||||
if (views.empty()) {
|
if (views.empty()) {
|
||||||
views.push_back(e);
|
views.push_back(e);
|
||||||
return 1.0f;
|
return 1.0f;
|
||||||
@@ -68,9 +111,12 @@ public:
|
|||||||
for (const auto& v : views)
|
for (const auto& v : views)
|
||||||
p_same = std::max(p_same, cal_(cosine_similarity(v, e)));
|
p_same = std::max(p_same, cal_(cosine_similarity(v, e)));
|
||||||
|
|
||||||
// Weight is the probability this is *not* a repeat of something already
|
const float rho = std::min(cfg_.rho_max, std::max(0.0f, p_same));
|
||||||
// counted. A near-duplicate contributes ~0; an unseen pose ~1.
|
|
||||||
const float w = std::max(cfg_.floor, 1.0f - p_same);
|
const float n_prev = static_cast<float>(std::max(1, n_seen));
|
||||||
|
const float n_now = n_prev + 1.0f;
|
||||||
|
auto n_eff = [rho](float n) { return n / (1.0f + (n - 1.0f) * rho); };
|
||||||
|
const float w = std::max(0.0f, n_eff(n_now) - n_eff(n_prev));
|
||||||
|
|
||||||
if (p_same < cfg_.admit_below &&
|
if (p_same < cfg_.admit_below &&
|
||||||
static_cast<int>(views.size()) < cfg_.max_views) {
|
static_cast<int>(views.size()) < cfg_.max_views) {
|
||||||
|
|||||||
@@ -123,6 +123,14 @@ public:
|
|||||||
|
|
||||||
Embedding embed_crop(const cv::Mat& crop) { return embedder_->embed_one(crop); }
|
Embedding embed_crop(const cv::Mat& crop) { return embedder_->embed_one(crop); }
|
||||||
|
|
||||||
|
// Batched form. A study embedding thousands of crops one at a time pays the
|
||||||
|
// per-call overhead thousands of times over; the backend already batches.
|
||||||
|
std::vector<Embedding> embed_crops(const std::vector<cv::Mat>& crops) {
|
||||||
|
return embedder_->embed(crops);
|
||||||
|
}
|
||||||
|
|
||||||
|
int max_batch() const { return embedder_->max_batch(); }
|
||||||
|
|
||||||
private:
|
private:
|
||||||
std::unique_ptr<IFaceDetector> detector_;
|
std::unique_ptr<IFaceDetector> detector_;
|
||||||
std::unique_ptr<IFaceEmbedder> embedder_;
|
std::unique_ptr<IFaceEmbedder> embedder_;
|
||||||
|
|||||||
@@ -6,6 +6,14 @@
|
|||||||
// legacy gallery.json files are still readable for backward compatibility but
|
// legacy gallery.json files are still readable for backward compatibility but
|
||||||
// save_gallery always writes HDF5 regardless of the requested extension.
|
// save_gallery always writes HDF5 regardless of the requested extension.
|
||||||
//
|
//
|
||||||
|
// GR-005 is preserved here by absence: this is the only path that serialises a
|
||||||
|
// gallery, and it reads and writes the local filesystem only. There is no
|
||||||
|
// upload, no client, and no encoder that could put an embedding on a wire — the
|
||||||
|
// public server refuses to carry one (SR-004/UR-012), and the prohibition holds
|
||||||
|
// on this side by there being nothing that would try.
|
||||||
|
//
|
||||||
|
/// TRACES: GR-005 | SR-005
|
||||||
|
//
|
||||||
// HDF5 layout:
|
// HDF5 layout:
|
||||||
// /embeddings float32 [N, 512] all actors' refs concatenated, row-major
|
// /embeddings float32 [N, 512] all actors' refs concatenated, row-major
|
||||||
// /offset int64 [A] first row of actor a in /embeddings
|
// /offset int64 [A] first row of actor a in /embeddings
|
||||||
|
|||||||
+1
-1
@@ -227,7 +227,7 @@ int main(int argc, char** argv) {
|
|||||||
SceneTrackerFunc tracker_fn {cfg};
|
SceneTrackerFunc tracker_fn {cfg};
|
||||||
ResultSinkFunc sink_fn {cfg, done};
|
ResultSinkFunc sink_fn {cfg, done};
|
||||||
|
|
||||||
/// TRACES: AR-012, AR-016, IR-002, IR-003 | SR-002
|
/// TRACES: AR-012, AR-016 | IR-002, IR-003 | SR-002
|
||||||
// A reaped track goes straight to the aggregator, so the registry holds only
|
// A reaped track goes straight to the aggregator, so the registry holds only
|
||||||
// live tracks and its size is bounded by concurrent on-screen faces rather
|
// live tracks and its size is bounded by concurrent on-screen faces rather
|
||||||
// than growing with the film.
|
// than growing with the film.
|
||||||
|
|||||||
@@ -26,7 +26,8 @@
|
|||||||
// The node is a pure pass-through: it forwards the Frame unchanged except for
|
// The node is a pure pass-through: it forwards the Frame unchanged except for
|
||||||
// is_cut, so it slots between frame_source and face_detector without altering the
|
// is_cut, so it slots between frame_source and face_detector without altering the
|
||||||
// downstream contract. eof frames are forwarded immediately without processing.
|
// downstream contract. eof frames are forwarded immediately without processing.
|
||||||
|
//
|
||||||
|
/// TRACES: AR-009 | SR-002
|
||||||
struct CameraPositionChangeDetectorFunc {
|
struct CameraPositionChangeDetectorFunc {
|
||||||
static constexpr std::string_view label() { return "camera_position_change_detector"; }
|
static constexpr std::string_view label() { return "camera_position_change_detector"; }
|
||||||
|
|
||||||
|
|||||||
@@ -17,7 +17,8 @@
|
|||||||
// All crops in one frame are batched into a single forward pass (capped at
|
// All crops in one frame are batched into a single forward pass (capped at
|
||||||
// embed_batch_size). The backend serialises itself; we only call it from the
|
// embed_batch_size). The backend serialises itself; we only call it from the
|
||||||
// single embedder thread.
|
// single embedder thread.
|
||||||
|
//
|
||||||
|
/// TRACES: AR-006 | SR-002
|
||||||
struct EmbedderFunc {
|
struct EmbedderFunc {
|
||||||
static constexpr std::string_view label() { return "embedder"; }
|
static constexpr std::string_view label() { return "embedder"; }
|
||||||
|
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ using json = nlohmann::json;
|
|||||||
struct ResultSinkFunc {
|
struct ResultSinkFunc {
|
||||||
static constexpr std::string_view label() { return "result_sink"; }
|
static constexpr std::string_view label() { return "result_sink"; }
|
||||||
|
|
||||||
/// TRACES: AR-012, AR-017, IR-002 | SR-002, SR-003
|
/// TRACES: AR-012, AR-017 | IR-002 | SR-002, SR-003
|
||||||
/// A finished presence claim from the registry. Called from inside the
|
/// A finished presence claim from the registry. Called from inside the
|
||||||
/// registry's reap while it holds its own lock, so this must stay a cheap
|
/// registry's reap while it holds its own lock, so this must stay a cheap
|
||||||
/// push and must never re-enter the registry.
|
/// push and must never re-enter the registry.
|
||||||
@@ -158,7 +158,7 @@ private:
|
|||||||
struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };
|
struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };
|
||||||
|
|
||||||
// Core logic: merge per-frame detections into annealed [start, end] windows.
|
// Core logic: merge per-frame detections into annealed [start, end] windows.
|
||||||
/// TRACES: AR-012, IR-002 | SR-002
|
/// TRACES: AR-012 | IR-002 | SR-002
|
||||||
/// A claim already IS a window — `[first_seen, last_seen]` of a track the
|
/// A claim already IS a window — `[first_seen, last_seen]` of a track the
|
||||||
/// actor owned. There is no annealing pass: `anneal_sec` existed to bridge
|
/// actor owned. There is no annealing pass: `anneal_sec` existed to bridge
|
||||||
/// gaps between isolated accepted frames, and a track that survives its own
|
/// gaps between isolated accepted frames, and a track that survives its own
|
||||||
|
|||||||
+24
-1
@@ -133,7 +133,30 @@ NB_MODULE(sae_embed, m) {
|
|||||||
return vec_to_numpy(std::vector<float>(emb.begin(), emb.end()));
|
return vec_to_numpy(std::vector<float>(emb.begin(), emb.end()));
|
||||||
}, "crop"_a,
|
}, "crop"_a,
|
||||||
"Embed a caller-supplied 112x112 aligned BGR crop. The stage-level "
|
"Embed a caller-supplied 112x112 aligned BGR crop. The stage-level "
|
||||||
"entry point for studies that degrade or re-align a crop themselves.");
|
"entry point for studies that degrade or re-align a crop themselves.")
|
||||||
|
.def("embed_crops", [](FaceEmbedderEngine& e,
|
||||||
|
nb::ndarray<const uint8_t, nb::ndim<4>, nb::c_contig,
|
||||||
|
nb::device::cpu> crops) {
|
||||||
|
if (crops.shape(1) != 112 || crops.shape(2) != 112 || crops.shape(3) != 3)
|
||||||
|
throw std::invalid_argument("embed_crops expects (N,112,112,3) uint8 BGR");
|
||||||
|
const size_t n = crops.shape(0);
|
||||||
|
std::vector<cv::Mat> mats;
|
||||||
|
mats.reserve(n);
|
||||||
|
for (size_t i = 0; i < n; ++i)
|
||||||
|
mats.emplace_back(112, 112, CV_8UC3,
|
||||||
|
const_cast<uint8_t*>(crops.data()) + i * 112 * 112 * 3);
|
||||||
|
std::vector<Embedding> out = e.embed_crops(mats);
|
||||||
|
auto* buf = new float[n * 512];
|
||||||
|
for (size_t i = 0; i < n; ++i)
|
||||||
|
std::memcpy(buf + i * 512, out[i].data(), 512 * sizeof(float));
|
||||||
|
nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast<float*>(p); });
|
||||||
|
size_t shape[2] = {n, 512};
|
||||||
|
return nb::ndarray<nb::numpy, float>(buf, 2, shape, owner);
|
||||||
|
}, "crops"_a,
|
||||||
|
"Batched embed_crop: (N,112,112,3) uint8 BGR in, (N,512) float32 out. "
|
||||||
|
"The backend batches internally, so this avoids paying per-call "
|
||||||
|
"overhead once per crop across a large study.")
|
||||||
|
.def_prop_ro("max_batch", [](FaceEmbedderEngine& e) { return e.max_batch(); });
|
||||||
|
|
||||||
m.def("align_face", [](ImageArray img,
|
m.def("align_face", [](ImageArray img,
|
||||||
nb::ndarray<const float, nb::shape<5, 2>, nb::c_contig,
|
nb::ndarray<const float, nb::shape<5, 2>, nb::c_contig,
|
||||||
|
|||||||
+28
-8
@@ -60,7 +60,13 @@ struct Track {
|
|||||||
double first_seen{0.0};
|
double first_seen{0.0};
|
||||||
std::optional<double> last_seen; ///< unset ⇒ on screen
|
std::optional<double> last_seen; ///< unset ⇒ on screen
|
||||||
std::optional<int> actor; ///< set once a posterior crosses
|
std::optional<int> actor; ///< set once a posterior crosses
|
||||||
std::map<int, float> belief; ///< actor_idx → accumulated log-odds
|
/// actor_idx → accumulated log(1 − P). Lazy-OR (noisy-OR) accumulation:
|
||||||
|
/// each frame is new evidence that this track is that actor, and the
|
||||||
|
/// combined belief is the probability that *at least one* sighting was
|
||||||
|
/// right. Stored as log(1−P) because that makes the update additive and
|
||||||
|
/// keeps precision where it matters — as P approaches 1, (1−P) is the
|
||||||
|
/// quantity with the significant digits.
|
||||||
|
std::map<int, float> belief;
|
||||||
Embedding mean{}; ///< running directional mean
|
Embedding mean{}; ///< running directional mean
|
||||||
std::vector<Embedding> views; ///< distinct looks, for AR-025 discounting
|
std::vector<Embedding> views; ///< distinct looks, for AR-025 discounting
|
||||||
float discounted_weight{0.f}; ///< sum of applied weights
|
float discounted_weight{0.f}; ///< sum of applied weights
|
||||||
@@ -146,14 +152,20 @@ public:
|
|||||||
if (it == tracks_.end()) { ++dropped_votes_; return; }
|
if (it == tracks_.end()) { ++dropped_votes_; return; }
|
||||||
|
|
||||||
Track& t = it->second;
|
Track& t = it->second;
|
||||||
const float w = discounter_.weight(t.views, e);
|
const float w = discounter_.weight(t.views, t.n_obs, e);
|
||||||
t.belief[actor_idx] += w * logit(posterior);
|
|
||||||
|
// Weighted lazy-OR: P_new = 1 − (1 − P_old)·(1 − p)^w, which in log
|
||||||
|
// space is a plain sum. w is the discounted evidence (AR-025), so a
|
||||||
|
// repeated view still advances the belief but by a fraction of what a
|
||||||
|
// genuinely new look would.
|
||||||
|
const float p = std::min(1.f - 1e-6f, std::max(1e-6f, posterior));
|
||||||
|
t.belief[actor_idx] += w * std::log(1.f - p);
|
||||||
t.discounted_weight += w;
|
t.discounted_weight += w;
|
||||||
++t.n_obs;
|
++t.n_obs;
|
||||||
|
|
||||||
const int best = argmax_belief(t);
|
const int best = argmax_belief(t);
|
||||||
const float best_lo = t.belief[best];
|
const float best_p = 1.f - std::exp(t.belief[best]);
|
||||||
if (best_lo < cfg_.ownership_logodds) return;
|
if (best_p < own_threshold()) return;
|
||||||
|
|
||||||
if (!t.actor.has_value()) {
|
if (!t.actor.has_value()) {
|
||||||
claim_locked(t, best);
|
claim_locked(t, best);
|
||||||
@@ -285,20 +297,28 @@ private:
|
|||||||
d.effective_obs = t.discounted_weight;
|
d.effective_obs = t.discounted_weight;
|
||||||
if (t.actor) {
|
if (t.actor) {
|
||||||
d.actor_idx = *t.actor;
|
d.actor_idx = *t.actor;
|
||||||
d.belief = logistic(t.belief[*t.actor]);
|
d.belief = 1.f - std::exp(t.belief[*t.actor]);
|
||||||
auto oi = owner_index_.find(*t.actor);
|
auto oi = owner_index_.find(*t.actor);
|
||||||
if (oi != owner_index_.end() && oi->second == t.id) owner_index_.erase(oi);
|
if (oi != owner_index_.end() && oi->second == t.id) owner_index_.erase(oi);
|
||||||
}
|
}
|
||||||
on_dead_(d);
|
on_dead_(d);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Most-believed actor. belief holds log(1 − P), so the strongest claim is
|
||||||
|
/// the *most negative* entry, not the largest.
|
||||||
static int argmax_belief(const Track& t) {
|
static int argmax_belief(const Track& t) {
|
||||||
int best = -1;
|
int best = -1;
|
||||||
float hi = -1e30f;
|
float lo = 1e30f;
|
||||||
for (const auto& [a, lo] : t.belief) if (lo > hi) { hi = lo; best = a; }
|
for (const auto& [a, v] : t.belief) if (v < lo) { lo = v; best = a; }
|
||||||
return best;
|
return best;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Ownership expressed as a probability. Config still carries log-odds so
|
||||||
|
/// the knob keeps its meaning across this change.
|
||||||
|
float own_threshold() const {
|
||||||
|
return 1.f / (1.f + std::exp(-cfg_.ownership_logodds));
|
||||||
|
}
|
||||||
|
|
||||||
static void update_mean(Track& t, const Embedding& e) {
|
static void update_mean(Track& t, const Embedding& e) {
|
||||||
// Directional mean: accumulate then re-normalise to the unit sphere, so
|
// Directional mean: accumulate then re-normalise to the unit sphere, so
|
||||||
// cosine against it stays a plain dot product.
|
// cosine against it stays a plain dot product.
|
||||||
|
|||||||
BIN
Binary file not shown.
+60
@@ -0,0 +1,60 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
#
|
||||||
|
# Regenerate bali_offset_200s.flac — the real-audio fixture behind VR-014, the
|
||||||
|
# audio-signature offset-recovery validation.
|
||||||
|
#
|
||||||
|
# sh make_offset_fixture.sh /path/to/clips
|
||||||
|
#
|
||||||
|
# Why real audio and not a second synthetic tone: jray_audio_v1_tone.flac pins
|
||||||
|
# the *arithmetic* (IR-005) and is deliberately built so every band and every
|
||||||
|
# energy class appears. It cannot answer the question VR-014 asks — whether the
|
||||||
|
# peak-bin sequence of ordinary film audio is distinctive enough that sliding
|
||||||
|
# one signature against another finds the true alignment and only the true
|
||||||
|
# alignment. Tones are pathologically easy for that; dialogue and score are not.
|
||||||
|
#
|
||||||
|
# Source: five scene clips from "Road to Bali" (1952), the public-domain corpus
|
||||||
|
# this repo already uses for the replay fixtures — tests/fixtures/dumps/bali_*.h5
|
||||||
|
# are dumps of these same clips. Each is under the 120 s window on its own
|
||||||
|
# (29-77 s), so they are concatenated in scene order to make a source long
|
||||||
|
# enough that a 120 s window can slide inside it.
|
||||||
|
#
|
||||||
|
# 200 s is chosen, not arbitrary: the window is 120 s and the match search is
|
||||||
|
# capped at +/-600 frames (~55.7 s), so a source of 120 + 56 s is the shortest
|
||||||
|
# one that can place two windows at the edge of the cap. The 200 s here leaves
|
||||||
|
# room to go past it as well, which is what lets the test check that an
|
||||||
|
# out-of-range offset is declined rather than guessed.
|
||||||
|
#
|
||||||
|
# Encoded mono at 11025 Hz, 16-bit, which is exactly what the signature decodes
|
||||||
|
# to anyway. That keeps a 200 s fixture at ~2.4 MB instead of ~20 MB, and makes
|
||||||
|
# every trim below sample-exact — the test measures offset recovery, not the
|
||||||
|
# resampler, which tests/test_audio_signature.cpp already covers (UT-103).
|
||||||
|
#
|
||||||
|
# FLAC because it is lossless: the decoded PCM is the same on every machine, so
|
||||||
|
# a signature computed from this file is reproducible. A lossy fixture would
|
||||||
|
# make the measurement depend on the decoder version.
|
||||||
|
#
|
||||||
|
# sha256 of the committed file:
|
||||||
|
# 4a952e46a090a9acd9eae56996250ec03e08e0d04ee139ac0a42f1690a536c83
|
||||||
|
# A regenerated file that hashes differently means the source clips or the
|
||||||
|
# encoder changed, and VR-014's recorded numbers should be re-measured — the
|
||||||
|
# offsets will still be exact, but the scores are this audio's.
|
||||||
|
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
CLIPS="${1:-../../../../bali}"
|
||||||
|
OUT="$(dirname "$0")/bali_offset_200s.flac"
|
||||||
|
LIST="$(mktemp)"
|
||||||
|
trap 'rm -f "$LIST"' EXIT
|
||||||
|
|
||||||
|
for scene in 13 27 28 31 46; do
|
||||||
|
clip="$CLIPS/Road_To_Bali-$scene.webm"
|
||||||
|
[ -f "$clip" ] || { echo "missing clip: $clip" >&2; exit 1; }
|
||||||
|
echo "file '$(cd "$(dirname "$clip")" && pwd)/$(basename "$clip")'" >> "$LIST"
|
||||||
|
done
|
||||||
|
|
||||||
|
ffmpeg -nostdin -v error -y -f concat -safe 0 -i "$LIST" \
|
||||||
|
-vn -t 200 -ac 1 -ar 11025 -sample_fmt s16 \
|
||||||
|
-c:a flac -compression_level 12 "$OUT"
|
||||||
|
|
||||||
|
echo "wrote $OUT"
|
||||||
|
sha256sum "$OUT" 2>/dev/null || shasum -a 256 "$OUT"
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
// Replay tests — the real tracker and registry driven from committed fixtures.
|
// Replay tests — the real tracker and registry driven from committed fixtures.
|
||||||
//
|
//
|
||||||
// TRACES: AR-012, AR-013, AR-004, VR-001, VR-002 | IT-001
|
// TRACES: AR-004, AR-012, AR-013 | VR-001, VR-002 | IT-001
|
||||||
//
|
//
|
||||||
// Tier T2: composition, not units. The registry tests construct awkward states
|
// Tier T2: composition, not units. The registry tests construct awkward states
|
||||||
// directly; these check that the pieces behave when wired together and fed real
|
// directly; these check that the pieces behave when wired together and fed real
|
||||||
|
|||||||
@@ -330,3 +330,43 @@ TEST_CASE("the registry takes a probability, not a cosine", "[registry][AR-024]"
|
|||||||
REQUIRE(sink.claims.size() == 1);
|
REQUIRE(sink.claims.size() == 1);
|
||||||
CHECK(sink.claims[0].actor_idx == -1); // never owned
|
CHECK(sink.claims[0].actor_idx == -1); // never owned
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── AR-025 — repeated evidence must GROW confidence, not cap it ──────────────
|
||||||
|
TEST_CASE("confidence grows across frames of the same face", "[registry][AR-025]") {
|
||||||
|
// Found on a real clip: 318 frame-level identifications across 385 frames
|
||||||
|
// produced ZERO owned tracks. The truth file named nobody while the matcher
|
||||||
|
// was accepting on most frames.
|
||||||
|
//
|
||||||
|
// Cause: the correlation discount was an annihilator rather than an
|
||||||
|
// attenuator. Weight = 1 - P(same view), so once a track had one stored
|
||||||
|
// view every later frame of that same face scored ~0.01 and belief stopped
|
||||||
|
// moving. A single observation just over the accept threshold is
|
||||||
|
// logit(0.78) ~ 1.27, under the ownership bar — recognised every frame,
|
||||||
|
// owned on none.
|
||||||
|
//
|
||||||
|
// Correlated evidence should accumulate SLOWER than independent evidence,
|
||||||
|
// never stop accumulating. Each frame is a Bayesian update.
|
||||||
|
TrackRegistry reg(cfg(), disc());
|
||||||
|
Sink sink; sink.attach(reg);
|
||||||
|
|
||||||
|
int id;
|
||||||
|
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
|
||||||
|
|
||||||
|
// A face held on screen: the same person, the same pose, frame after frame.
|
||||||
|
for (int i = 0; i < 50; ++i) {
|
||||||
|
// The frame scope must close before observe(): it holds the registry
|
||||||
|
// lock for its lifetime and the mutex is not recursive, so observing
|
||||||
|
// inside the scope self-deadlocks. In the pipeline these are separate
|
||||||
|
// nodes, so the ordering falls out naturally — but the API allows the
|
||||||
|
// mistake, and it hangs rather than failing.
|
||||||
|
{ auto f = reg.begin_frame(i * 0.2); f.mark_seen(id, i * 0.2, axis(0)); }
|
||||||
|
reg.observe(id, 5, 0.78f, axis(0));
|
||||||
|
}
|
||||||
|
reg.flush(20.0);
|
||||||
|
|
||||||
|
REQUIRE(sink.claims.size() == 1);
|
||||||
|
CHECK(sink.claims[0].actor_idx == 5);
|
||||||
|
|
||||||
|
// ...but it must still be worth far less than 50 independent looks would be.
|
||||||
|
CHECK(sink.claims[0].effective_obs < 25.0f);
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user