dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(replay): the whole replay chain is C++, including the sink

Browse Source 
The sae_kpn module has not compiled since the AR-007/AR-008 tracker redesign,
and was switched off at the build rather than patched because the fix is a
restructuring. Two failures, one cause.

It did not compile: `add_face_tracker` built FaceTrackerFunc from a Config
alone, and the tracker has required a TrackRegistry and a calibration since
association moved into probability space.

And presence was rebuilt in Python. `replay.py::build_minimal` merged
per-frame detections into windows by annealing gaps, which is what the
pipeline did before AR-012. The sink builds a window from a TrackRegistry
claim instead — the extent of a track an actor owned, starting when they
appeared rather than when recognition first succeeded. Those answer different
questions, so every sweep was tuning against a contract the shipped code had
stopped honouring.

Both follow from the seam being a factory per node. The chain has a
construction order — the matcher fits the calibration, the registry needs a
discounter built from it, the tracker needs both, and the sink needs the
registry's claims — and independent factories cannot express it, so the
tracker kept being built against a signature that no longer existed. One
`add_pipeline` mirrors main.cpp exactly and is now the only way to build the
chain, so the ordering cannot be got wrong again from Python. DP-001 is the
requirement behind it: a replay harness is a front-end, and its job is to
supply frames and read the result, not to re-derive presence.

Lifetimes needed a home. ResultSinkFunc holds `const Config&` and
`std::atomic<bool>&`, which under main() are locals in a frame outliving the
pipeline; there is no such frame when the network is built and torn down from
Python. ReplaySession owns both for the network's lifetime, keyed by network
and released explicitly — a sweep builds one network per replay and the sink
retains every annotation, so holding them forever would grow with films x
configs. Getting this wrong presented as an empty output_path: the sink
announced `[result_sink] writing ` and wrote nothing.

test_sae_kpn.py is ported rather than left behind. It called all three removed
factories and asserted on SceneAnnotations read back per frame; neither half
survives, so it now waits on pipeline_done and asserts on the file the sink
writes. Verified against gallery_lvface.h5: three frames through the real
chain, timestamps 0/1/2, truth file written. EOF is a control token the sink
flushes on and does not record, so three inputs give three frames, never four.

SAE_BUILD_KPN_BINDINGS goes back to ON.

TRACES: VR-011, VR-002 | DP-001 | PR-002
This commit is contained in:
dtourolle
2026-08-05 19:40:25 +02:00
parent 1141172b04
commit 48332d2041
6 changed files with 414 additions and 229 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/kpn_bindings.cpp
+199 -47
View File
@@ -1,11 +1,38 @@
// sae_kpn — run the real downstream pipeline nodes (face_tracker, identity_matcher,
// frame_annotation) inside a Python-assembled KPN network, fed by a Python HDF5 replay
// source. Lets a parameter sweep re-run the exact C++ matching/tracking logic over
// dumped embeddings — no video decode, no GPU — with different Config knobs each run.
// sae_kpn — run the real downstream pipeline inside a Python-assembled KPN
// network, fed by a Python HDF5 replay source. Lets a parameter sweep re-run the
// exact C++ tracking/matching/presence logic over dumped embeddings — no video
// decode, no GPU — with different Config knobs each run.
//
/// TRACES: VR-011, VR-002 | PR-002
//
// **The whole chain is C++, including the sink.** That is the VR-011 change and
// it is the point of the requirement: replay must drive the real nodes, not a
// reimplementation. Two things were wrong before.
//
// 1. It did not compile. `add_face_tracker` built `FaceTrackerFunc` from a
// Config alone, and the tracker has required a TrackRegistry and a
// calibration since AR-007/AR-008 moved association into probability
// space. Any .so in a stale build/ predates that.
//
// 2. Presence was rebuilt in Python. `replay.py::build_minimal` merged
// per-frame detections into windows by annealing gaps — which is what the
// pipeline did before AR-012. The sink now builds a window from a
// TrackRegistry claim: the extent of a track an actor owned, starting when
// they appeared rather than when recognition first succeeded. Those answer
// different questions, so every sweep was tuning against a contract the
// shipped code had stopped honouring.
//
// Both had the same root cause, which is why this is one binding and not three.
// The chain has a construction ORDER — the matcher fits the calibration, the
// registry needs a discounter built from it, the tracker needs both, and the
// sink needs the registry's claims — and a factory-per-node API cannot express
// it. `add_pipeline` mirrors main.cpp exactly and is the only way to build the
// chain, so the ordering cannot be got wrong again from Python.
//
// Boundary types (cross the Python seam):
// EmbeddedSceneFrame IN (built by the Python replay source from HDF5 arrays)
// SceneAnnotation OUT (read by the Python sink → presence JSON)
// SceneAnnotation OUT (optional tee for per-frame debug rendering only —
// the presence output is written by the C++ sink)
// Intermediate types (TrackedSceneFrame, MatchedSceneFrame) flow C++→C++ only, but
// still need channel factories + converters registered so PyNetwork can wire them.
@@ -20,6 +47,9 @@
#include "nodes/face_tracker_node.hpp"
#include "nodes/identity_matcher_node.hpp"
#include "nodes/frame_annotation_node.hpp"
#include "nodes/result_sink_node.hpp"
#include "track_registry.hpp"
#include "evidence_discount.hpp"
#include <nanobind/nanobind.h>
#include <nanobind/ndarray.h>
@@ -27,6 +57,8 @@
#include <nanobind/stl/vector.h>
#include <nanobind/stl/map.h>
#include <atomic>
#include <map>
#include <memory>
#include <optional>
#include <variant>
@@ -34,10 +66,52 @@
namespace nb = nanobind;
using namespace nb::literals;
// ── ReplaySession ─────────────────────────────────────────────────────────────
/// TRACES: VR-011 | PR-002
/// State the network's nodes reference but do not own.
///
/// ResultSinkFunc holds `std::atomic<bool>&`, exactly as it does under main(),
/// where it is a stack local in a function that outlives the pipeline. There is
/// no such frame here -- the network is built and torn down from Python -- so
/// the flag lives in a session held for the network's lifetime and released
/// explicitly. The registry is here for the same reason: the sink's claim
/// callback captures it.
struct ReplaySession {
/// Owns the Config, and must. ResultSinkFunc holds `const Config&` -- under
/// main() that is a stack local in a frame which outlives the pipeline, so
/// the reference is fine there. There is no such frame here: the network is
/// built inside a binding call and torn down from Python, so a Config local
/// to add_pipeline dies the moment it returns and the sink is left reading
/// freed memory. It presented as an empty output_path -- the sink announced
/// `[result_sink] writing ` and wrote nothing.
Config cfg;
std::atomic<bool> done{false};
std::shared_ptr<TrackRegistry> registry;
};
// Function-local static so ordering against other translation units cannot bite.
inline std::map<void*, std::shared_ptr<ReplaySession>>& sessions() {
static std::map<void*, std::shared_ptr<ReplaySession>> s;
return s;
}
// The variant spanning every type that flows on a channel in the replay chain.
using SaeVariant = std::variant<EmbeddedSceneFrame, TrackedSceneFrame,
MatchedSceneFrame, SceneAnnotation>;
// ── Node wrapper aliases ──────────────────────────────────────────────────────
// Named once so add_pipeline and the runtime setters cannot disagree about a
// node's port names: a mismatch there is a dynamic_cast that returns null, i.e.
// a runtime setter that silently does nothing.
using MatcherWrap = kpn::ObjectVariantNodeWrapper<
IdentityMatcherFunc, SaeVariant, kpn::in<"tracked">, kpn::out<"matched">>;
using TrackerWrap = kpn::ObjectVariantNodeWrapper<
FaceTrackerFunc, SaeVariant, kpn::in<"embedded">, kpn::out<"tracked">>;
using AnnotWrap = kpn::ObjectVariantNodeWrapper<
FrameAnnotationFunc, SaeVariant, kpn::in<"matched">, kpn::out<"annotation">>;
using SinkWrap = kpn::ObjectVariantNodeWrapper<
ResultSinkFunc, SaeVariant, kpn::in<"annotation">, kpn::out<>>;
// ── Converters ─────────────────────────────────────────────────────────────────
// Only EmbeddedSceneFrame (in) and SceneAnnotation (out) actually cross the seam;
// the two intermediates get identity-ish stubs (never converted in practice) so the
@@ -183,6 +257,28 @@ static Config config_from_dict(nb::dict d) {
/// TRACES: GR-004 | SR-001
if (d.contains("require_gallery_stamp"))
cfg.require_gallery_stamp = nb::cast<bool>(d["require_gallery_stamp"]);
/// TRACES: VR-011, IR-001 | PR-002, SR-003
// The sink is a real node in this network now, so it needs the two things
// that decide what it writes and where. Both used to be irrelevant here
// because the replay never had a sink -- Python rebuilt presence instead,
// which is the reimplementation VR-002 forbids and VR-011 removes.
if (d.contains("output_path"))
cfg.output_path = nb::cast<std::string>(d["output_path"]);
if (d.contains("verbosity")) {
const int v = nb::cast<int>(d["verbosity"]);
cfg.verbosity = v == 2 ? Verbosity::xray
: v == 1 ? Verbosity::standard
: Verbosity::minimal;
}
// Reported verbatim in the truth file's extraction block, so a replayed
// manifest says which gallery scope produced it (IR-002).
if (d.contains("gallery_scope"))
cfg.gallery_scope = nb::cast<std::string>(d["gallery_scope"]);
if (d.contains("sample_fps"))
cfg.sample_fps = nb::cast<float>(d["sample_fps"]);
if (d.contains("movie_path"))
cfg.movie_path = nb::cast<std::string>(d["movie_path"]);
return cfg;
}
@@ -222,38 +318,51 @@ NB_MODULE(sae_kpn, m) {
std::move(outs), cap);
}, "net"_a, "name"_a, "callable"_a, "inputs"_a, "outputs"_a, "capacity"_a = 5);
// ── Real node factories ─────────────────────────────────────────────────────
m.def("add_face_tracker", [](Net& net, std::string name, nb::dict cfg_dict, std::size_t cap) {
// ── The pipeline ────────────────────────────────────────────────────────────
/// TRACES: VR-011, VR-002 | DP-001 | PR-002, PR-004
///
/// One call builds the whole downstream chain, in the one order that works:
///
/// matcher (fits the calibration)
/// -> registry (needs a discounter built from it)
/// -> tracker (needs both)
/// -> frame_annotation
/// -> result_sink (needs the registry's claims)
///
/// This replaces add_face_tracker / add_identity_matcher / add_frame_annotation.
/// They were separate because the network is assembled node by node from
/// Python -- and that is exactly how the seam broke: the tracker's dependency
/// on a calibration that only exists once the matcher is built cannot be
/// expressed as three independent factories, so the tracker factory kept
/// constructing FaceTrackerFunc{cfg} against a signature that no longer
/// existed. A binding that cannot represent the order will eventually be
/// called in the wrong one.
///
/// DP-001 -- "modes are front-ends and must not fork pipeline logic" -- is
/// the requirement this serves. The replay harness is a front-end. Its job is
/// to supply frames and read the result, not to re-derive presence.
m.def("add_pipeline", [](Net& net, std::string gallery_path, nb::dict cfg_dict,
std::size_t cap, std::string embedder_model,
std::string embedder_sha256) {
Config cfg = config_from_dict(cfg_dict);
auto node = std::make_shared<kpn::ObjectVariantNodeWrapper<
FaceTrackerFunc, SaeVariant, kpn::in<"embedded">, kpn::out<"tracked">>>(cap, cfg);
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "config"_a, "capacity"_a = 16);
cfg.gallery_path = gallery_path; // so a refreshed calibration persists back
/// TRACES: GR-004 | SR-001
// embedder_model / embedder_sha256 identify whatever produced the embeddings
// that will be fed in. In a replay those come from the dump's own stamp (see
// scripts/optimizer/SCHEMA.md), because there is no live embedder in the
// network — the dump *is* the embedder as far as this gallery is concerned.
// Passing neither leaves the binding unverifiable, which warns loudly and is
// fatal under SAE_REQUIRE_GALLERY_STAMP.
m.def("add_identity_matcher", [](Net& net, std::string name, std::string gallery_path,
nb::dict cfg_dict, std::size_t cap,
std::string embedder_model,
std::string embedder_sha256) {
Config cfg = config_from_dict(cfg_dict);
cfg.gallery_path = gallery_path; // needed to persist refreshed calibration back
// Cache loaded galleries by path so a threshold sweep (many networks, same
// gallery) pays the ~24s JSON parse only once. The matcher holds a const
// ref; the cache keeps the gallery alive for the process lifetime.
// gallery) pays the parse once. The matcher holds a const ref; the cache
// keeps the gallery alive for the process lifetime.
static std::map<std::string, std::shared_ptr<ActorGallery>> cache;
auto it = cache.find(gallery_path);
if (it == cache.end())
it = cache.emplace(gallery_path,
std::make_shared<ActorGallery>(load_gallery(gallery_path))).first;
// Checked on every construction, not only on the cache miss: the same
// process may replay several dumps against one cached gallery.
/// TRACES: GR-004 | SR-001
// embedder_model / embedder_sha256 identify whatever produced the
// embeddings that will be fed in. In a replay those come from the dump's
// own stamp: there is no live embedder here, so the dump *is* the
// embedder as far as this gallery is concerned. Checked on every
// construction, not only on a cache miss -- one process may replay
// several dumps against one cached gallery.
EmbedderStamp feeding;
feeding.model_name = std::move(embedder_model);
feeding.model_sha256 = std::move(embedder_sha256);
@@ -263,31 +372,74 @@ NB_MODULE(sae_kpn, m) {
: feeding.model_name,
cfg.require_gallery_stamp);
auto node = std::make_shared<kpn::ObjectVariantNodeWrapper<
IdentityMatcherFunc, SaeVariant, kpn::in<"tracked">, kpn::out<"matched">>>(
cap, *it->second, cfg);
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "gallery"_a, "config"_a, "capacity"_a = 16,
// 1. Matcher first: its constructor fits (or loads) the calibration.
auto matcher = std::make_shared<MatcherWrap>(cap, *it->second, cfg);
// 2. The calibration every other stage must decide in (AR-024).
auto same_person = same_person_probability(matcher->functor().calibration());
// 3. Registry + discounter, from Config (AR-025).
TrackRegistry::Config reg_cfg;
reg_cfg.track_extinction_sec = cfg.track_extinction_sec;
reg_cfg.ownership_logodds = cfg.ownership_logodds;
EvidenceDiscounter::Config disc_cfg;
disc_cfg.max_views = cfg.evidence_max_views;
disc_cfg.admit_below = cfg.evidence_admit_below;
disc_cfg.rho_max = cfg.evidence_rho_max;
auto registry = std::make_shared<TrackRegistry>(
reg_cfg, EvidenceDiscounter(same_person, disc_cfg));
matcher->functor().set_registry(registry);
// 4. Tracker, which needs both.
auto tracker = std::make_shared<TrackerWrap>(cap, cfg, registry, same_person);
// 5. Projection, stateless.
auto annot = std::make_shared<AnnotWrap>(cap);
// 6. The real sink. `done` outlives the network via the session below;
// ResultSinkFunc holds it by reference, as it does in main.cpp.
auto session = std::make_shared<ReplaySession>();
session->cfg = cfg; // the sink holds this by reference
session->registry = registry;
auto sink = std::make_shared<SinkWrap>(cap, session->cfg, session->done);
/// TRACES: AR-012, AR-016 | IR-003 | SR-002
// The claim path, identical to main.cpp's. Without the flush hook every
// track still live at EOF is silently dropped -- which in a replay is
// most of the closing scene, and reads as a recognition miss rather than
// as a missing wire.
ResultSinkFunc& sink_fn = sink->functor();
registry->on_track_dead([&sink_fn](const DeadTrack& d) { sink_fn.add_claim(d); });
sink_fn.set_pre_write_hook([registry](double last_ts) { registry->flush(last_ts); });
net.add("tracker", tracker);
net.add("matcher", matcher);
net.add("annotation", annot);
net.add("sink", sink);
// Keyed by network so release_pipeline can free it. Not a leak-by-design:
// a sweep builds one network per replay, and the sink accumulates every
// annotation, so holding these forever would grow with films x configs.
sessions()[&net] = session;
}, "net"_a, "gallery"_a, "config"_a, "capacity"_a = 16,
"embedder_model"_a = "", "embedder_sha256"_a = "");
/// TRACES: AR-012, AR-013 | SR-002
// Was add_scene_tracker, backed by the extinction-timer state machine. The
// node is gone (see frame_annotation_node.hpp) and so is the timer; this
// projects a matched frame into the same SceneAnnotation the Python sink
// already reads, so the seam's output type is unchanged. It takes no config
// because it has no state to configure -- which is the point.
m.def("add_frame_annotation", [](Net& net, std::string name, std::size_t cap) {
auto node = std::make_shared<kpn::ObjectVariantNodeWrapper<
FrameAnnotationFunc, SaeVariant, kpn::in<"matched">, kpn::out<"annotation">>>(cap);
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "capacity"_a = 16);
/// Drop the session for a network. Idempotent. Call after net.stop(); not
/// calling it holds one registry and one sink's accumulated frames per
/// replay, which a long sweep will notice.
m.def("release_pipeline", [](Net& net) { sessions().erase(&net); }, "net"_a);
/// True once the sink has written its output. The sink flushes on the EOF
/// annotation, so a caller that reads the file before this is racing it.
m.def("pipeline_done", [](Net& net) {
auto it = sessions().find(&net);
return it != sessions().end()
&& it->second->done.load(std::memory_order_acquire);
}, "net"_a);
// ── Runtime setters (persistent-pipeline reuse across a threshold sweep) ─────
// Build the network once, then change thresholds between replays — no rebuild,
// no teardown (which is where the ROCm deadlock lives), no gallery reload.
using MatcherWrap = kpn::ObjectVariantNodeWrapper<
IdentityMatcherFunc, SaeVariant, kpn::in<"tracked">, kpn::out<"matched">>;
m.def("set_prob_threshold", [](Net& net, std::string name, float t) {
auto* w = dynamic_cast<MatcherWrap*>(net.node_ptr(name));
if (!w) throw std::runtime_error("set_prob_threshold: '" + name + "' is not an identity_matcher");