feat: tracker on the registry — one pool, calibrated, frame-dependent

Merges feature/tracker-registry. See e9aea3f for the detail; in summary the
tracker no longer owns track state, association weights embedding over position
whenever position is uninformative, and every similarity comparison is a
calibrated probability.

Four constants retired: track_max_embed_dist, cut_revive_sim,
cut_inactive_max_frames, track_max_frames_missing.

Suite: 80 cases, 3250 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-007, AR-008, AR-024 | SR-002
This commit is contained in:
2026-07-31 10:01:31 +02:00
co-authored by Claude Opus 5
7 changed files with 359 additions and 237 deletions
+3 -3
View File
@@ -34,8 +34,8 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | Planned | | AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | Planned |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done | | AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done |
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done | | AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | In Progress | | AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done**`track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | Planned | | AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted |
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done | | AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Not started**`is_scene_boundary` has no producer; `SceneDetectorFunc` is a terminal sink and never annotates the frame | | AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Not started**`is_scene_boundary` has no producer; `SceneDetectorFunc` is a terminal sink and never annotates the frame |
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned | | AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
@@ -51,7 +51,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned | | AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned | | AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done | | AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** (registry boundary) — `observe()` takes a calibrated probability and converts to log-odds itself; retiring the raw-cosine constants is still open | | AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association and accumulation both in probability space; `track_max_embed_dist`, `cut_revive_sim` retired |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` | | AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress | | AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned | | AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
+18 -14
View File
@@ -94,21 +94,25 @@ struct Config {
float dense_scale{1.0f}; // dense-mode frame downscale (1 = off) float dense_scale{1.0f}; // dense-mode frame downscale (1 = off)
// ── Face tracking (frame-to-frame) ─────────────────────────────────────── // ── Face tracking (frame-to-frame) ───────────────────────────────────────
float track_alpha{0.4f}; // cost weight: 0=embedding only, 1=spatial only /// TRACES: AR-007, AR-008, AR-024 | SR-002
// track_alpha is the *base* weight, used on ordinary frames. It is
// frame-dependent (AR-007): on is_cut / is_scene_boundary, and for any track
// that is no longer on screen, it drops to 0 (embedding only), because
// position carries no information across a viewpoint change or a gap.
float track_alpha{0.4f}; // base cost weight: 0=embedding only, 1=spatial only
float track_min_iou{0.1f}; // IoU below which spatial link alone is rejected float track_min_iou{0.1f}; // IoU below which spatial link alone is rejected
float track_max_embed_dist{0.7f}; // cosine dist above which embedding link alone is rejected // Minimum P(same person) for an association to be admissible on appearance
int track_max_frames_missing{5}; // expire track after N consecutive missed frames // alone. This replaces track_max_embed_dist (a raw cosine distance, AR-024).
// 0.5 is not a tuned constant: it is the decision boundary. Below it the pair
// ── Cross-cut track re-association ──────────────────────────────────────── // is more likely two people than one, and no amount of IoU makes that a link
// A camera-angle change (Frame::is_cut) breaks spatial (IoU) continuity but // worth asserting on identity grounds.
// not identity: the same people are usually still on screen from a new angle. float track_assoc_min_prob{0.5f};
// Instead of destroying tracks on a cut, the tracker parks them in an // How long a track that has gone off screen stays available for association
// inactive pool. A post-cut detection whose raw cosine similarity to a parked // before the registry reaps it and emits its presence claim (AR-013).
// track's last-frame embedding is ≥ cut_revive_sim revives that track_id // Replaces track_max_frames_missing: a frame count silently changed meaning
// (identity continuity survives the cut); otherwise it starts a fresh track. // with sample_fps, and the same number had to be guessed twice (once for an
// Parked tracks that go unrevived for cut_inactive_max_frames are dropped. // ordinary miss, once for a cut). Seconds mean one thing at any sample rate.
float cut_revive_sim{0.50f}; // min raw cosine sim (last-frame emb) to revive across a cut double track_extinction_sec{5.0};
int cut_inactive_max_frames{5}; // drop a parked track after N frames without revival
// ── Scene tracking ──────────────────────────────────────────────────────── // ── Scene tracking ────────────────────────────────────────────────────────
// extinction_sec re-tuned by DE against X-Ray per-second presence, 4-film rep4 // extinction_sec re-tuned by DE against X-Ray per-second presence, 4-film rep4
+28
View File
@@ -9,6 +9,7 @@
#include <cstdlib> #include <cstdlib>
#include <cstring> #include <cstring>
#include <fstream> #include <fstream>
#include <functional>
#include <iostream> #include <iostream>
#include <stdexcept> #include <stdexcept>
#include <string> #include <string>
@@ -49,6 +50,33 @@ struct GalleryCalibration {
} }
}; };
/// TRACES: AR-023, AR-024 | SR-002
///
/// cosine → P(same person). The one probability space the pipeline reasons in.
///
/// Handed to every stage that has to decide whether two embeddings are the same
/// person — track association (AR-007), evidence discounting (AR-025), identity
/// matching — so a threshold of 0.5 means the same thing in all of them. A stage
/// that thresholded a raw cosine instead would be using a number that means
/// something different for every model, gallery and face size (AR-024).
///
/// **No prior term.** `log_prior_odds` adjusts for the gallery's base rate, which
/// is a question about *which of N actors*; association asks whether two faces
/// are one person, where the balanced fit is the right answer. Passing the
/// matcher's prior here would silently bias tracking by the size of the cast.
inline std::function<float(float)> same_person_probability(const GalleryCalibration& cal) {
if (!cal.valid) {
// Loud, because the failure mode is invisible: an untuned sigmoid still
// returns plausible probabilities, and every threshold downstream of it
// is then a guess wearing a calibrated number's clothes.
std::cerr << "[calibration] WARNING: no fitted calibration — association and "
"evidence weighting fall back to the untuned default sigmoid "
"(a=" << cal.a << ", b=" << cal.b << "). Probabilities are "
"not meaningful for this embedder.\n";
}
return [cal](float similarity) { return cal.probability(similarity); };
}
// Fit a logistic sigmoid to gallery pair similarities. // Fit a logistic sigmoid to gallery pair similarities.
// Positive pairs: same actor, different reference images. // Positive pairs: same actor, different reference images.
// Negative pairs: different actors (all cross-actor embedding pairs). // Negative pairs: different actors (all cross-actor embedding pairs).
+16 -5
View File
@@ -128,10 +128,8 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--ratio-ceil")) cfg.match_ratio_ceil = std::stof(next()); else if (arg("--ratio-ceil")) cfg.match_ratio_ceil = std::stof(next());
else if (arg("--track-alpha")) cfg.track_alpha = std::stof(next()); else if (arg("--track-alpha")) cfg.track_alpha = std::stof(next());
else if (arg("--track-min-iou")) cfg.track_min_iou = std::stof(next()); else if (arg("--track-min-iou")) cfg.track_min_iou = std::stof(next());
else if (arg("--track-max-embed")) cfg.track_max_embed_dist = std::stof(next()); else if (arg("--track-min-prob")) cfg.track_assoc_min_prob = std::stof(next());
else if (arg("--track-max-missing")) cfg.track_max_frames_missing = std::stoi(next()); else if (arg("--track-extinction")) cfg.track_extinction_sec = std::stod(next());
else if (arg("--cut-revive-sim")) cfg.cut_revive_sim = std::stof(next());
else if (arg("--cut-inactive-max")) cfg.cut_inactive_max_frames = std::stoi(next());
else if (arg("--anneal")) cfg.anneal_sec = std::stod(next()); else if (arg("--anneal")) cfg.anneal_sec = std::stod(next());
else if (arg("--expand-gallery")) cfg.expand_gallery = true; else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next()); else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
@@ -194,8 +192,21 @@ int main(int argc, char** argv) {
FaceDetectorFunc detector_fn{cfg}; FaceDetectorFunc detector_fn{cfg};
FaceAlignerFunc aligner_fn; FaceAlignerFunc aligner_fn;
EmbedderFunc embedder_fn{cfg}; EmbedderFunc embedder_fn{cfg};
FaceTrackerFunc ftracker_fn{cfg}; // Constructed before the tracker: it fits (or loads) the calibration, and
// the tracker must decide in that same probability space (AR-024).
IdentityMatcherFunc matcher_fn {gallery, cfg}; IdentityMatcherFunc matcher_fn {gallery, cfg};
/// TRACES: AR-007, AR-008, AR-012, AR-024 | SR-002
// The registry is created here and shared, not owned by a node: track state
// is not a stage in the stream, it is state several stages read and write,
// and its final answer is only known when a track dies.
auto same_person = same_person_probability(matcher_fn.calibration());
TrackRegistry::Config reg_cfg;
reg_cfg.extinction_sec = cfg.track_extinction_sec;
auto registry = std::make_shared<TrackRegistry>(
reg_cfg, EvidenceDiscounter(same_person));
FaceTrackerFunc ftracker_fn{cfg, registry, same_person};
SceneTrackerFunc tracker_fn {cfg}; SceneTrackerFunc tracker_fn {cfg};
ResultSinkFunc sink_fn {cfg, done}; ResultSinkFunc sink_fn {cfg, done};
#ifdef SAE_DEBUG #ifdef SAE_DEBUG
+164 -157
View File
@@ -1,102 +1,124 @@
#pragma once #pragma once
/// TRACES: AR-007, AR-008, AR-024 | SR-002
///
/// FaceTrackerFunc — KPN node that links face detections into tracks.
///
/// **The registry is the tracker's state.** The node owns no track map of its
/// own: it drives `TrackRegistry` through a `FrameScope` and reads the same
/// `Track` objects everything else reads. Two parallel copies could disagree,
/// and every divergence would surface as a wrong presence window rather than as
/// a crash — silently, and only in the output.
///
/// **One candidate pool** (AR-008). `last_seen` alone distinguishes a track that
/// is on screen from one that is dormant, and it only affects whether IoU means
/// anything. There is no parked pool and no revival branch: re-associating a
/// track whose face was lost — across a cut or not — is ordinary inter-frame
/// association, and it falls out of the embedding comparison already being done.
///
/// Assignment cost (track i, detection j):
///
/// p = P(same person | cosine(track mean, detection)) ← calibrated
/// alpha = base weight, or 0 when position carries no information
/// cost = alpha·(1 IoU) + (1 alpha)·(1 p)
///
/// gated to INF unless the pair is admissible on position *or* on identity.
///
/// **alpha is frame- and track-dependent** (AR-007). It falls to 0 —
/// embedding only — when either:
/// - the frame is flagged `is_cut` / `is_scene_boundary`: the viewpoint
/// changed, so the same person is at a new position; or
/// - the track is dormant (`last_seen` set): time has passed since its box was
/// last observed, so that box is stale regardless of cuts.
/// Both are the same statement — spatial continuity is broken — arrived at from
/// two directions, which is why they collapse into one rule rather than two
/// branches.
///
/// **Everything is thresholded in probability space** (AR-024). The cosine goes
/// through the calibration before it is compared to anything; the raw-cosine
/// constants `track_max_embed_dist` and `cut_revive_sim` are retired.
#include "types.hpp" #include "types.hpp"
#include "config.hpp" #include "config.hpp"
#include "track_registry.hpp"
#include <algorithm> #include <algorithm>
#include <cmath> #include <cmath>
#include <functional>
#include <iostream> #include <iostream>
#include <limits> #include <limits>
#include <map> #include <map>
#include <memory>
#include <stdexcept>
#include <string_view>
#include <vector> #include <vector>
// ── FaceTrackerFunc ───────────────────────────────────────────────────────────
// KPN node: links face detections across consecutive frames using the Hungarian
// algorithm on a combined spatial (IoU) + embedding (cosine distance) cost.
//
// Each track accumulates a running directional mean of its ArcFace embeddings
// (averaged then re-normalised to the unit sphere), used as the embedding side
// of the assignment cost below for more stable track continuity.
//
// Assignment cost (track i, detection j):
// cost = alpha * (1 - IoU) + (1-alpha) * min(cosine_dist/2, 1)
// Gated to INF when IoU < min_iou AND cosine_dist > max_embed_dist.
//
// Unmatched tracks have their frames_missing counter incremented; they are
// expired once frames_missing > max_frames_missing.
//
// Cross-cut re-association. A camera-angle change (Frame::is_cut, set by
// camera_position_change_detector) destroys spatial (IoU) continuity — the same
// person reappears at a new position — but not identity. On a cut the tracker
// does NOT discard its tracks; it parks them in an inactive pool keyed by their
// last-frame raw embedding. A post-cut detection whose raw cosine similarity to
// a parked track's last-frame embedding is ≥ cut_revive_sim revives that track:
// the original track_id, mean embedding and n_frames are restored (only the bbox
// jumps to the new detection), so identity continuity survives the cut. Parked
// tracks left unrevived for cut_inactive_max_frames are finally dropped.
struct FaceTrackerFunc { struct FaceTrackerFunc {
static constexpr std::string_view label() { return "face_tracker"; } static constexpr std::string_view label() { return "face_tracker"; }
struct TrackState { /// cosine similarity → P(same person). Supplied by the caller so the fit
cv::Rect2f bbox; /// belonging to the active embedder is used (AR-023/AR-024) — the same
Embedding mean_emb{}; /// pattern, and normally the same function object, as
Embedding last_emb{}; // raw embedding of the most recent matched frame /// `EvidenceDiscounter::Calibrate`.
int n_frames{0}; using Calibrate = std::function<float(float)>;
int frames_missing{0};
};
explicit FaceTrackerFunc(const Config& cfg) /// The registry is a constructor argument, not an option: a tracker without
: alpha_(cfg.track_alpha) /// one would have to keep its own tracks, which is the defect this replaces.
FaceTrackerFunc(const Config& cfg,
std::shared_ptr<TrackRegistry> registry,
Calibrate calibrate)
: registry_(std::move(registry))
, calibrate_(std::move(calibrate))
, alpha_base_(cfg.track_alpha)
, min_iou_(cfg.track_min_iou) , min_iou_(cfg.track_min_iou)
, max_embed_dist_(cfg.track_max_embed_dist) , min_assoc_prob_(cfg.track_assoc_min_prob)
, max_missing_(cfg.track_max_frames_missing)
, revive_sim_(cfg.cut_revive_sim)
, inactive_max_(cfg.cut_inactive_max_frames)
{ {
std::cerr << "[face_tracker] alpha=" << alpha_ if (!registry_)
throw std::invalid_argument("face_tracker: registry must not be null");
if (!calibrate_)
throw std::invalid_argument("face_tracker: a calibration is required — "
"association is decided in probability space");
std::cerr << "[face_tracker] alpha_base=" << alpha_base_
<< " min_iou=" << min_iou_ << " min_iou=" << min_iou_
<< " max_embed_dist=" << max_embed_dist_ << " min_assoc_prob=" << min_assoc_prob_ << "\n";
<< " max_missing=" << max_missing_
<< " cut_revive_sim=" << revive_sim_
<< " cut_inactive_max=" << inactive_max_ << "\n";
} }
TrackedSceneFrame operator()(EmbeddedSceneFrame ef) { TrackedSceneFrame operator()(EmbeddedSceneFrame ef) {
if (ef.source.eof) { if (ef.source.eof) {
tracks_.clear(); // Deliberately does *not* flush the registry. The identity matcher
inactive_.clear(); // runs downstream and its votes for the final frames are still in
// flight; reaping here would drop them (they would land on ids that
// no longer exist and show up as dropped_votes). AR-016's flush
// belongs at the pipeline's termination point, after the last vote.
boxes_.clear();
TrackedSceneFrame out; TrackedSceneFrame out;
out.source = std::move(ef.source); out.source = std::move(ef.source);
return out; return out;
} }
const double t = ef.source.timestamp_sec;
const int n_det = static_cast<int>(ef.embeddings.size()); const int n_det = static_cast<int>(ef.embeddings.size());
// Camera-angle change: park active tracks instead of destroying them so // Unconditional: the clock must advance on frames with no detections
// they can be revived by identity (raw last-frame embedding cosine) once // too, or a track only dies when some unrelated face happens to appear
// the same people reappear from the new angle. // and a film that ends mid-track never closes it (AR-013).
if (ef.source.is_cut && !tracks_.empty()) { auto scope = registry_->begin_frame(t);
std::cerr << "[face_tracker] cut — parking " << tracks_.size()
<< " track(s) into inactive pool\n";
for (auto& [tid, ts] : tracks_) {
ts.frames_missing = 0; // repurpose as time-since-parked counter
inactive_[tid] = std::move(ts);
}
tracks_.clear();
}
// Age the inactive pool every frame and drop tracks parked too long. // One pool (AR-008) — on-screen and dormant tracks compete together.
for (auto it = inactive_.begin(); it != inactive_.end(); ) { std::vector<Track*> cands = scope.candidates();
it->second.frames_missing++; const int n_trk = static_cast<int>(cands.size());
it = (it->second.frames_missing > inactive_max_)
? inactive_.erase(it) : std::next(it);
}
// Snapshot active track IDs so the map can be modified safely below prune_boxes(cands);
std::vector<int> tids; std::vector<Spatial*> sp(n_trk);
tids.reserve(tracks_.size()); for (int ti = 0; ti < n_trk; ++ti)
for (auto& [tid, _] : tracks_) tids.push_back(tid); sp[ti] = &boxes_.try_emplace(cands[ti]->id, Spatial{{}, t, false})
const int n_trk = static_cast<int>(tids.size()); .first->second;
// AR-007 — the frame half of the frame-dependent weighting. Both flags
// say the same thing to the tracker: whatever was at that position is
// not there any more.
const bool viewpoint_change =
ef.source.is_cut || ef.source.is_scene_boundary;
// ── Cost matrix [n_trk × n_det] ────────────────────────────────────── // ── Cost matrix [n_trk × n_det] ──────────────────────────────────────
constexpr float INF_COST = 1e6f; constexpr float INF_COST = 1e6f;
@@ -104,25 +126,39 @@ struct FaceTrackerFunc {
std::vector<float>(n_det, INF_COST)); std::vector<float>(n_det, INF_COST));
for (int ti = 0; ti < n_trk; ++ti) { for (int ti = 0; ti < n_trk; ++ti) {
const TrackState& ts = tracks_[tids[ti]]; // Spatial continuity holds only for a track that was on screen, whose
// box we have actually observed, on a frame that did not change the
// viewpoint. Otherwise the box is stale and IoU is noise.
const bool spatial_meaningful =
sp[ti]->observed && cands[ti]->on_screen() && !viewpoint_change;
const float alpha = spatial_meaningful ? alpha_base_ : 0.f;
for (int di = 0; di < n_det; ++di) { for (int di = 0; di < n_det; ++di) {
float iou_v = iou(ts.bbox, ef.faces[di].bbox); // AR-024 — the cosine is converted before it is used for
float emb_d = (ts.n_frames > 0) // anything, including the gate below.
? 1.f - cosine_similarity(ts.mean_emb, ef.embeddings[di]) const float p = calibrate_(
: 1.f; cosine_similarity(cands[ti]->mean, ef.embeddings[di]));
if (iou_v < min_iou_ && emb_d > max_embed_dist_) continue; const float iou_v = spatial_meaningful
float s = 1.f - iou_v; ? iou(sp[ti]->bbox, ef.faces[di].bbox) : 0.f;
float e = std::min(emb_d * 0.5f, 1.f);
cost[ti][di] = alpha_ * s + (1.f - alpha_) * e; // Either signal on its own can admit a link: a face that moved a
// little but whose embedding degraded (blur, profile turn) is
// still linkable on position, and a face that jumped across the
// frame is still linkable on identity. Neither ⇒ no link.
const bool spatial_ok = spatial_meaningful && iou_v >= min_iou_;
const bool identity_ok = p >= min_assoc_prob_;
if (!spatial_ok && !identity_ok) continue;
cost[ti][di] = alpha * (1.f - iou_v) + (1.f - alpha) * (1.f - p);
} }
} }
// ── Hungarian assignment ───────────────────────────────────────────── // ── Hungarian assignment ─────────────────────────────────────────────
std::vector<int> assign(n_trk, -1); std::vector<int> assign(n_trk, -1);
if (n_trk > 0 && n_det > 0) if (n_trk > 0 && n_det > 0)
assign = hungarian(cost, n_trk, n_det); assign = hungarian(cost, n_trk, n_det);
// ── Build output frame ─────────────────────────────────────────────── // ── Build output frame ───────────────────────────────────────────────
TrackedSceneFrame out; TrackedSceneFrame out;
out.source = ef.source; out.source = ef.source;
out.faces = ef.faces; out.faces = ef.faces;
@@ -132,81 +168,66 @@ struct FaceTrackerFunc {
std::vector<bool> det_matched(n_det, false); std::vector<bool> det_matched(n_det, false);
// Update matched tracks
for (int ti = 0; ti < n_trk; ++ti) { for (int ti = 0; ti < n_trk; ++ti) {
int di = assign[ti]; const int di = assign[ti];
bool valid = (di >= 0 && di < n_det && cost[ti][di] < INF_COST * 0.5f); const int id = cands[ti]->id;
TrackState& ts = tracks_[tids[ti]]; const bool valid =
(di >= 0 && di < n_det && cost[ti][di] < INF_COST * 0.5f);
if (!valid) { if (!valid) {
ts.frames_missing++; // Only a track that *was* on screen can become lost, and it
// becomes lost as of its last sighting, never as of now — the
// gap after the final sighting is never claimed (AR-013). A
// track already dormant is left alone so its extinction clock
// keeps running from the right instant.
if (cands[ti]->on_screen()) scope.mark_lost(id, sp[ti]->last_ts);
continue; continue;
} }
update_mean(ts.mean_emb, ts.n_frames, ef.embeddings[di]);
ts.last_emb = ef.embeddings[di];
ts.bbox = ef.faces[di].bbox;
ts.n_frames++;
ts.frames_missing = 0;
det_matched[di] = true;
out.track_ids[di] = tids[ti]; scope.mark_seen(id, t, ef.embeddings[di]);
sp[ti]->bbox = ef.faces[di].bbox;
sp[ti]->last_ts = t;
sp[ti]->observed = true;
det_matched[di] = true;
out.track_ids[di] = id;
} }
// Handle unmatched detections: first try to revive a parked track by
// identity (raw last-frame embedding cosine), else start a fresh track.
for (int di = 0; di < n_det; ++di) { for (int di = 0; di < n_det; ++di) {
if (det_matched[di]) continue; if (det_matched[di]) continue;
const int id = scope.create(t, ef.embeddings[di]);
int tid = revive_from_inactive(ef.embeddings[di]); boxes_[id] = Spatial{ef.faces[di].bbox, t, true};
if (tid >= 0) { out.track_ids[di] = id;
// Restore the parked track: keep its identity statistics
// (mean_emb, n_frames), jump the bbox to the new detection.
TrackState ts = std::move(inactive_[tid]);
inactive_.erase(tid);
update_mean(ts.mean_emb, ts.n_frames, ef.embeddings[di]);
ts.last_emb = ef.embeddings[di];
ts.bbox = ef.faces[di].bbox;
ts.n_frames++;
ts.frames_missing = 0;
tracks_[tid] = std::move(ts);
out.track_ids[di] = tid;
std::cerr << "[face_tracker] revived track " << tid
<< " across cut\n";
continue;
}
tid = next_id_++;
TrackState ts;
ts.bbox = ef.faces[di].bbox;
ts.mean_emb = ef.embeddings[di];
ts.last_emb = ef.embeddings[di];
ts.n_frames = 1;
tracks_[tid] = ts;
out.track_ids[di] = tid;
}
// Expire stale tracks
for (auto it = tracks_.begin(); it != tracks_.end(); ) {
it = (it->second.frames_missing > max_missing_)
? tracks_.erase(it) : std::next(it);
} }
// No reaping here: begin_frame's tick owns the extinction sweep, so
// there is exactly one place a track can die.
return out; return out;
} }
private: private:
// Pick the parked track whose last-frame embedding is most similar to emb, // ── Spatial annotation ───────────────────────────────────────────────────
// returning its id if that raw cosine similarity clears revive_sim_, else -1. // The one piece of per-track state the registry does not hold, because it is
// The caller removes the returned track from the pool, so a later detection in // not about presence: where the face was, and when it was last seen there.
// the same frame cannot claim it again. // Keyed by registry track id and pruned against `candidates()` every frame,
int revive_from_inactive(const Embedding& emb) const { // so it cannot outlive or contradict the registry — it annotates the pool
int best_tid = -1; // rather than duplicating it.
float best_sim = revive_sim_; // threshold is the bar to beat (inclusive) struct Spatial {
for (const auto& [tid, ts] : inactive_) { cv::Rect2f bbox{};
float sim = cosine_similarity(ts.last_emb, emb); double last_ts{0.0}; ///< timestamp of the last frame this track matched
if (sim >= best_sim) { best_sim = sim; best_tid = tid; } bool observed{false}; ///< false until a detection has been assigned
// subsequent ties keep the later id; harmless, all clear the threshold };
// Drop boxes for ids the registry no longer has. `candidates()` is the
// authority on what exists; anything else is a leak (and, for a reused id,
// would be a stale box attached to a different person).
void prune_boxes(const std::vector<Track*>& cands) {
if (boxes_.size() == cands.size()) return; // common case: nothing died
std::map<int, Spatial> kept;
for (const Track* t : cands) {
auto it = boxes_.find(t->id);
if (it != boxes_.end()) kept.emplace(t->id, it->second);
} }
return best_tid; boxes_.swap(kept);
} }
// IoU of two axis-aligned bounding boxes // IoU of two axis-aligned bounding boxes
@@ -220,17 +241,6 @@ private:
return inter / (a.width * a.height + b.width * b.height - inter); return inter / (a.width * a.height + b.width * b.height - inter);
} }
// Online directional mean: average then re-normalise to unit sphere
static void update_mean(Embedding& mean, int n_prev, const Embedding& emb) {
float norm_sq = 0.f;
for (int k = 0; k < 512; ++k) {
mean[k] = (mean[k] * n_prev + emb[k]) / (n_prev + 1);
norm_sq += mean[k] * mean[k];
}
float inv = 1.f / std::sqrt(norm_sq);
for (int k = 0; k < 512; ++k) mean[k] *= inv;
}
// O(n³) potential-based Hungarian algorithm (Jonker-Volgenant / Kuhn-Munkres). // O(n³) potential-based Hungarian algorithm (Jonker-Volgenant / Kuhn-Munkres).
// Returns assign[row] = col (0-indexed), or -1 when row is matched to a // Returns assign[row] = col (0-indexed), or -1 when row is matched to a
// padded virtual column (i.e., unmatched). Rectangular matrices are padded // padded virtual column (i.e., unmatched). Rectangular matrices are padded
@@ -292,13 +302,10 @@ private:
return ans; return ans;
} }
std::map<int, TrackState> tracks_; std::shared_ptr<TrackRegistry> registry_;
std::map<int, TrackState> inactive_; // parked across a cut, keyed by track id Calibrate calibrate_;
int next_id_{0}; std::map<int, Spatial> boxes_; ///< track id → where it was, when
float alpha_; float alpha_base_;
float min_iou_; float min_iou_;
float max_embed_dist_; float min_assoc_prob_;
int max_missing_;
float revive_sim_;
int inactive_max_;
}; };
+8
View File
@@ -107,6 +107,14 @@ struct IdentityMatcherFunc {
sim_engine_ = make_similarity_engine(host_gallery.data(), n_gallery_, kMaxFaces); sim_engine_ = make_similarity_engine(host_gallery.data(), n_gallery_, kMaxFaces);
} }
/// TRACES: AR-023, AR-024 | SR-002
/// The fitted sigmoid. Exposed because the matcher is where it gets fitted
/// (and cached back to the gallery), but it is not the matcher's private
/// property: track association and evidence weighting must threshold in the
/// *same* probability space, or a "0.5" in one stage and a "0.5" in another
/// mean different things. See `same_person_probability`.
const GalleryCalibration& calibration() const { return cal_; }
// Runtime setter — lets a persistent pipeline be reused across a threshold sweep // Runtime setter — lets a persistent pipeline be reused across a threshold sweep
// without rebuilding the (expensive, gallery-resident) matcher. The gallery, // without rebuilding the (expensive, gallery-resident) matcher. The gallery,
// calibration and GPU sim-engine stay put; only the accept threshold changes. // calibration and GPU sim-engine stay put; only the accept threshold changes.
+117 -53
View File
@@ -13,8 +13,12 @@
#include "config.hpp" #include "config.hpp"
#include "nodes/face_tracker_node.hpp" #include "nodes/face_tracker_node.hpp"
#include "types.hpp" #include "types.hpp"
#include "track_registry.hpp"
#include "evidence_discount.hpp"
#include <algorithm>
#include <cmath> #include <cmath>
#include <memory>
namespace { namespace {
@@ -55,85 +59,145 @@ EmbeddedSceneFrame frame(double t, float x, float y, const Embedding& emb,
return ef; return ef;
} }
Config tracker_cfg() { // Build a tracker over a fresh registry. The registry IS the tracker's state
Config cfg; // now (AR-008), so a test constructs both together and can inspect either.
cfg.cut_revive_sim = 0.50f; struct Rig {
cfg.cut_inactive_max_frames = 5; std::shared_ptr<TrackRegistry> reg;
return cfg; FaceTrackerFunc ft;
}
explicit Rig(double extinction = 30.0, float assoc_min_prob = 0.5f)
: reg(std::make_shared<TrackRegistry>(
[extinction] {
TrackRegistry::Config c;
c.extinction_sec = extinction;
return c;
}(),
EvidenceDiscounter([](float cos) { return std::max(0.f, cos); })))
, ft([&] {
Config c;
c.track_assoc_min_prob = assoc_min_prob;
return c;
}(),
reg,
// Trivial calibration: cosine passed through as P(same). Real runs use
// the fit belonging to the active embedder (AR-023/AR-024).
[](float cos) { return std::max(0.f, cos); })
{}
int track_of(EmbeddedSceneFrame f) { return ft(std::move(f)).track_ids[0]; }
};
} // namespace } // namespace
TEST_CASE("track id is stable across ordinary frames", "[face_tracker]") { // ── AR-008 — one pool, ordinary association ──────────────────────────────────
FaceTrackerFunc ft(tracker_cfg()); TEST_CASE("track id is stable across ordinary frames", "[face_tracker][AR-008]") {
Rig r;
Embedding e = axis(0); Embedding e = axis(0);
int id0 = ft(frame(0.0, 10, 10, e)).track_ids[0]; int id0 = r.track_of(frame(0.0, 10, 10, e));
int id1 = ft(frame(1.0, 11, 10, e)).track_ids[0]; // overlaps → same track int id1 = r.track_of(frame(1.0, 11, 10, e)); // overlaps → same track
CHECK(id0 >= 0); CHECK(id0 >= 0);
CHECK(id1 == id0); CHECK(id1 == id0);
} }
TEST_CASE("cut revives the same track id for a matching identity", "[face_tracker]") { TEST_CASE("a face lost across a cut and re-associated is the SAME track",
FaceTrackerFunc ft(tracker_cfg()); "[face_tracker][AR-008]") {
// Previously this was a distinct "revival" path guarded by a raw-cosine
// constant. There is no such path now: a dormant track is an ordinary
// association candidate, and continuity falls out of the embedding match.
Rig r;
// Pre-cut: establish a track for a person whose embedding is near-identical
// across the cut (sim well above cut_revive_sim), but whose box jumps so IoU
// is 0 — the ordinary spatial path cannot re-link it.
Embedding pre = at_sim(0, 1, 0.99f); Embedding pre = at_sim(0, 1, 0.99f);
int id_pre = ft(frame(0.0, 10, 10, pre)).track_ids[0]; int id_pre = r.track_of(frame(0.0, 10, 10, pre));
REQUIRE(id_pre >= 0); REQUIRE(id_pre >= 0);
Embedding post = at_sim(0, 1, 0.98f); // cos(diff) ≈ 0.9997 > 0.50 // Box jumps so IoU is zero — only the embedding can link it.
auto out = ft(frame(1.0, 300, 300, post, /*is_cut=*/true)); Embedding post = at_sim(0, 1, 0.98f);
CHECK(out.track_ids[0] == id_pre); // revived, not a fresh id CHECK(r.track_of(frame(1.0, 300, 300, post, /*is_cut=*/true)) == id_pre);
} }
TEST_CASE("cut starts a fresh track when identity does not match", "[face_tracker]") { TEST_CASE("a cut starts a fresh track when identity does not match",
FaceTrackerFunc ft(tracker_cfg()); "[face_tracker][AR-008]") {
Rig r;
int id_pre = ft(frame(0.0, 10, 10, axis(0))).track_ids[0]; int id_pre = r.track_of(frame(0.0, 10, 10, axis(0)));
REQUIRE(id_pre >= 0); REQUIRE(id_pre >= 0);
// Post-cut face is orthogonal (sim 0 < cut_revive_sim) and spatially disjoint // Orthogonal embedding and disjoint box: nothing links them.
// → no revival, brand-new id. int id_post = r.track_of(frame(1.0, 300, 300, axis(5), /*is_cut=*/true));
auto out = ft(frame(1.0, 300, 300, axis(5), /*is_cut=*/true)); CHECK(id_post != id_pre);
CHECK(out.track_ids[0] != id_pre); CHECK(id_post >= 0);
CHECK(out.track_ids[0] >= 0);
} }
TEST_CASE("parked track expires after cut_inactive_max_frames", "[face_tracker]") { // ── AR-007 — a cut makes association ignore position ─────────────────────────
Config cfg = tracker_cfg(); TEST_CASE("on a cut, identity follows the embedding rather than the box",
cfg.cut_inactive_max_frames = 2; "[face_tracker][AR-007]") {
FaceTrackerFunc ft(cfg); // Two people swap screen positions across a cut while keeping their faces.
// If IoU still carried weight the ids would follow the boxes and swap; with
// alpha driven to embedding-only on a cut, they must follow the faces.
Rig r;
Embedding a = at_sim(0, 1, 0.99f);
Embedding b = at_sim(2, 3, 0.99f);
EmbeddedSceneFrame f0;
f0.source.timestamp_sec = 0.0;
f0.faces = {face_at(10, 10), face_at(300, 300)};
f0.crops = {cv::Mat(), cv::Mat()};
f0.embeddings = {a, b};
auto out0 = r.ft(std::move(f0));
const int id_a = out0.track_ids[0];
const int id_b = out0.track_ids[1];
REQUIRE(id_a >= 0);
REQUIRE(id_b >= 0);
REQUIRE(id_a != id_b);
// Same two people, positions exchanged, on a cut frame.
EmbeddedSceneFrame f1;
f1.source.timestamp_sec = 1.0;
f1.source.is_cut = true;
f1.faces = {face_at(300, 300), face_at(10, 10)};
f1.crops = {cv::Mat(), cv::Mat()};
f1.embeddings = {a, b};
auto out1 = r.ft(std::move(f1));
CHECK(out1.track_ids[0] == id_a); // A kept its id despite moving to B's box
CHECK(out1.track_ids[1] == id_b);
}
// ── AR-013 — extinction replaces the parked-pool frame counter ───────────────
TEST_CASE("a track past the extinction window is gone, not revived",
"[face_tracker][AR-013]") {
// The old design aged a parked pool in frames, which silently changed
// meaning with sample_fps. Extinction is in seconds and lives in the
// registry, so the tracker no longer counts anything.
Rig r(/*extinction=*/2.0);
Embedding person = at_sim(0, 1, 0.99f); Embedding person = at_sim(0, 1, 0.99f);
int id_pre = ft(frame(0.0, 10, 10, person)).track_ids[0]; int id_pre = r.track_of(frame(0.0, 10, 10, person));
REQUIRE(id_pre >= 0); REQUIRE(id_pre >= 0);
// Cut with an unrelated face parks id_pre; then let the pool age past its // Unrelated faces elsewhere while the clock runs well past extinction.
// limit with more unrelated, spatially-disjoint faces (each ages the pool by r.track_of(frame(1.0, 300, 300, axis(7), /*is_cut=*/true));
// one). By the time the person returns, id_pre must be gone. r.track_of(frame(10.0, 300, 300, axis(7)));
ft(frame(1.0, 300, 300, axis(7), /*is_cut=*/true)); // park (age 1)
ft(frame(2.0, 300, 300, axis(7))); // age 2
ft(frame(3.0, 300, 300, axis(7))); // age 3 → id_pre dropped
auto out = ft(frame(4.0, 10, 10, person)); // same identity returns CHECK(r.track_of(frame(11.0, 10, 10, person)) != id_pre);
CHECK(out.track_ids[0] != id_pre); // too late — fresh id
} }
TEST_CASE("eof clears active and parked tracks", "[face_tracker]") { TEST_CASE("a track within the extinction window is still a candidate",
FaceTrackerFunc ft(tracker_cfg()); "[face_tracker][AR-013]") {
Embedding person = at_sim(0, 1, 0.99f); Rig r(/*extinction=*/30.0);
int id_pre = ft(frame(0.0, 10, 10, person)).track_ids[0];
ft(frame(1.0, 300, 300, axis(7), /*is_cut=*/true)); // park id_pre
Embedding person = at_sim(0, 1, 0.99f);
int id_pre = r.track_of(frame(0.0, 10, 10, person));
r.track_of(frame(1.0, 300, 300, axis(7), /*is_cut=*/true));
// Back inside the window: the same person continues the same track, so the
// gap is absorbed into one window rather than splitting it.
CHECK(r.track_of(frame(3.0, 10, 10, person)) == id_pre);
}
TEST_CASE("eof is forwarded", "[face_tracker]") {
Rig r;
EmbeddedSceneFrame eof; EmbeddedSceneFrame eof;
eof.source.eof = true; eof.source.eof = true;
auto out = ft(std::move(eof)); CHECK(r.ft(std::move(eof)).source.eof);
CHECK(out.source.eof);
// After eof the pools are empty: the returning identity must get a fresh id,
// not the parked one.
auto out2 = ft(frame(2.0, 10, 10, person));
CHECK(out2.track_ids[0] != id_pre);
} }