Files
scene-actor-extraction/src/main.cpp
T
dtourolleandClaude Opus 5 e9aea3fc41 feat: tracker owns no state; association is frame-dependent and calibrated
Three requirements land together because they cannot be separated. The
cross-cut revival branch was the only user of cut_revive_sim, so retiring that
raw cosine forces the pool collapse, and collapsing the pool removes the only
caller of the constant. Splitting them would have produced an intermediate
commit whose only purpose was to be split.

AR-008 — FaceTrackerFunc no longer keeps its own tracks_/inactive_ maps; it
holds a shared_ptr<TrackRegistry> and operates on it directly. Two parallel
copies of track state could disagree, and every divergence would surface as a
wrong presence window with nothing to indicate it. There is now ONE candidate
pool: last_seen alone says whether IoU is meaningful. The park/revive path is
deleted outright — matching a dormant track is ordinary inter-frame
association, and continuity falls out of the embedding comparison the tracker
already did rather than being a mechanism of its own.

AR-007 — track_alpha becomes the base weight for ordinary frames only.
Association drops to embedding-only when position carries no information:
on is_cut or is_scene_boundary, because the viewpoint changed, and for a
dormant track, because time has passed since its box was last valid. The second
case matters as much as the first and had no equivalent before.

AR-024 — association cost is a calibrated probability, never a raw cosine. The
tracker takes the calibration belonging to the active embedder, the same
function object EvidenceDiscounter uses. track_max_embed_dist becomes
track_assoc_min_prob, which means the same thing for every model, gallery and
face size, where a bare cosine threshold did not.

Retired: track_max_embed_dist, cut_revive_sim, cut_inactive_max_frames, and
track_max_frames_missing — the last superseded by the registry's extinction
window. That one is worth naming: a frame count silently changed meaning with
sample_fps, so the same configuration behaved differently at 1 fps and 5 fps.
Extinction is in seconds and lives in one place.

Tests rewritten rather than deleted. The old cases asserted revival by raw
cosine; the same behaviours are now asserted through the registry — a face lost
across a cut and re-associated is the SAME track, one unbroken window, and a
face returning past the extinction window is not. Added the case AR-007 exists
for: two people swap screen positions across a cut while keeping their faces,
and identity must follow the embedding rather than the box.

Suite: 80 cases, 3250 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-007, AR-008, AR-024 | SR-002
2026-07-31 09:58:27 +02:00

371 lines
21 KiB
C++

// scene_analyze — identify actors in a movie using a KPN pipeline
//
// TRACES: DP-001, DP-002 | PR-004
// One analysis core; the CLI is a front-end over it and must not fork pipeline
// logic. Other deployment modes (DP-003, DP-004) wrap this same core.
//
// KPN topology (release build):
//
// [frame_source] ──Frame──► [face_detector] ──SceneFrame──► [face_aligner]
// ──AlignedSceneFrame──► [embedder] ──EmbeddedSceneFrame──►
// [identity_matcher] ──MatchedSceneFrame──► [scene_tracker]
// ──SceneAnnotation──► [result_sink]
//
// Debug build (SAE_DEBUG=1):
// [identity_matcher] output fans out to both [scene_tracker] AND [debug_renderer].
// FanoutNode<MatchedSceneFrame, 2> is auto-inserted by make_network().
//
// Usage:
// scene_analyze --movie <path> --gallery <gallery.json> [options]
//
// Options:
// --output <path> output JSON (default: annotations.json)
// --fps <N> sample rate in frames/sec (default: 1.0)
// --verbosity <0|1|2> 0=minimal, 1=standard, 2=jellyfin-xray (default: 0)
// --match-threshold <f> cosine dist threshold (default: 0.45)
// --extinction <f> actor extinction window in seconds (default: 5.0)
// --detector <path> override SCRFD detector model path
// --arcface <path> override ArcFace model path
// --scene-detect enable TransNetV2 shot-boundary detection (dense decode;
// writes <output>.scenes.json). Off by default.
// --scene-detector <path> override TransNetV2 .onnx model path
// --scene-detector-engine <path> pre-built TransNetV2 TRT engine (TRT backend)
// --scene-threshold <f> boundary sigmoid prob above this → cut (default: 0.60)
// --scene-stride <N> frames between TransNetV2 windows (default: 50, ≤100)
// --scene-decode-fps <f> dense decode rate in scene-detect mode (default: 12;
// 0 = native fps). Lower = faster, coarser boundaries.
// --dense-scale <f> downscale decoded frames in scene-detect mode (0<f≤1,
// default 1=off). Speeds decode; keep ≥0.5 on 1080p.
// --max-faces <N> max faces kept per frame (default: 10)
// --expand-gallery enable per-film gallery expansion from track continuity
// --expand-buffer <N> per-track diversity buffer size (default: 20)
// --expand-novelty-sim <f> promote only views with best sim < f (default: 0.55)
// --expand-spread-max <f> reject track if buffer spread > f (default: 0.60)
// --expand-min-anchor <N> accepted frames before a track confirms (default: 3)
// --expand-debug-dir <p> dump promoted mugshots + embeddings here (SAE_DEBUG)
// (SAE_DEBUG only)
// --debug-dir <path> debug frames output dir (default: debug_frames)
// --crop-context <f> bbox expansion factor for context crops (default: 1.5)
#include "config.hpp"
#include "types.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/frame_source_node.hpp"
#include "nodes/camera_position_change_detector_node.hpp"
#include "nodes/face_detector_node.hpp"
#include "nodes/face_aligner_node.hpp"
#include "nodes/embedder_node.hpp"
#include "nodes/face_tracker_node.hpp"
#include "nodes/identity_matcher_node.hpp"
#include "nodes/scene_tracker_node.hpp"
#include "nodes/scene_detector_node.hpp"
#include "nodes/result_sink_node.hpp"
#include "nodes/embedding_dump_node.hpp"
#ifdef SAE_DEBUG
#include "nodes/debug_renderer_node.hpp"
#endif
#include <kpn/kpn.hpp>
#include <atomic>
#include <chrono>
#include <cstring>
#include <iostream>
#include <map>
#include <mutex>
#include <stdexcept>
#include <string>
#include <string_view>
#include <thread>
// ── CLI parsing ───────────────────────────────────────────────────────────────
static Config parse_args(int argc, char** argv) {
Config cfg;
cfg.detector_model = kDefaultDetectorModel;
cfg.arcface_model = kDefaultArcfaceModel;
cfg.scene_model = kDefaultSceneModel;
cfg.output_path = "annotations.json";
for (int i = 1; i < argc; ++i) {
auto arg = [&](const char* flag) { return std::strcmp(argv[i], flag) == 0; };
auto next = [&]() -> std::string {
if (++i >= argc) throw std::runtime_error(std::string("missing arg after ") + argv[i-1]);
return argv[i];
};
if (arg("--movie")) cfg.movie_path = next();
else if (arg("--gallery")) cfg.gallery_path = next();
else if (arg("--output")) cfg.output_path = next();
else if (arg("--dump-embeddings")) cfg.dump_embeddings_path = next();
else if (arg("--fps")) cfg.sample_fps = std::stof(next());
else if (arg("--max-decode-fps")) cfg.max_decode_fps = std::stof(next());
else if (arg("--start")) cfg.start_sec = std::stod(next());
else if (arg("--end")) cfg.end_sec = std::stod(next());
else if (arg("--cut-threshold")) cfg.cut_threshold = std::stof(next());
else if (arg("--scene-detect")) cfg.scene_detect = true;
else if (arg("--scene-detector")) cfg.scene_model = next();
else if (arg("--scene-detector-engine")) cfg.scene_engine = next();
else if (arg("--scene-threshold")) cfg.scene_threshold = std::stof(next());
else if (arg("--scene-stride")) cfg.scene_stride = std::stoi(next());
else if (arg("--scene-decode-fps")) cfg.scene_decode_fps = std::stof(next());
else if (arg("--dense-scale")) cfg.dense_scale = std::stof(next());
else if (arg("--verbosity")) { int v = std::stoi(next()); cfg.verbosity = v == 2 ? Verbosity::xray : v == 1 ? Verbosity::standard : Verbosity::minimal; }
else if (arg("--prior")) cfg.match_prior = std::stof(next());
else if (arg("--prob-threshold")) cfg.prob_threshold = std::stof(next());
else if (arg("--match-threshold")) cfg.match_threshold = std::stof(next());
else if (arg("--extinction")) cfg.extinction_sec = std::stod(next());
else if (arg("--detector")) cfg.detector_model = next();
else if (arg("--detector-engine")) cfg.detector_engine = next();
else if (arg("--arcface")) cfg.arcface_model = next();
else if (arg("--require-gallery-stamp")) cfg.require_gallery_stamp = true;
else if (arg("--arcface-engine")) cfg.arcface_engine = next();
else if (arg("--conf")) cfg.detector_conf = std::stof(next());
else if (arg("--max-faces")) cfg.max_faces = std::stoi(next());
else if (arg("--min-face-px")) cfg.min_face_px = std::stof(next());
else if (arg("--ratio")) cfg.match_ratio = std::stof(next());
else if (arg("--ratio-ceil")) cfg.match_ratio_ceil = std::stof(next());
else if (arg("--track-alpha")) cfg.track_alpha = std::stof(next());
else if (arg("--track-min-iou")) cfg.track_min_iou = std::stof(next());
else if (arg("--track-min-prob")) cfg.track_assoc_min_prob = std::stof(next());
else if (arg("--track-extinction")) cfg.track_extinction_sec = std::stod(next());
else if (arg("--anneal")) cfg.anneal_sec = std::stod(next());
else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
else if (arg("--expand-novelty-sim")) cfg.expand_novelty_sim = std::stof(next());
else if (arg("--expand-spread-max")) cfg.expand_track_spread_max = std::stof(next());
else if (arg("--expand-min-anchor")) cfg.expand_min_anchor_frames = std::stoi(next());
else if (arg("--expand-debug-dir")) cfg.expand_debug_dir = next();
else if (arg("--trt-cache")) cfg.trt.cache_dir = next();
else if (arg("--trt-fp16")) cfg.trt.fp16 = true;
else if (arg("--no-trt-fp16")) cfg.trt.fp16 = false;
else if (arg("--trt-int8")) cfg.trt.int8 = true;
else if (arg("--embed-batch")) cfg.embed_batch_size = std::stoi(next());
#ifdef SAE_DEBUG
else if (arg("--debug-dir")) cfg.debug_dir = next();
else if (arg("--crop-context")) cfg.crop_context = std::stof(next());
#endif
else {
std::cerr << "[warn] unknown flag: " << argv[i] << "\n";
}
}
if (cfg.movie_path.empty()) throw std::runtime_error("--movie is required");
if (cfg.gallery_path.empty()) throw std::runtime_error("--gallery is required");
return cfg;
}
// ── Main ──────────────────────────────────────────────────────────────────────
int main(int argc, char** argv) {
Config cfg;
try {
cfg = parse_args(argc, argv);
} catch (const std::exception& e) {
std::cerr << "Usage error: " << e.what() << "\n";
return 1;
}
// Load actor gallery
ActorGallery gallery;
try {
gallery = load_gallery(cfg.gallery_path);
/// TRACES: GR-004 | SR-001
// Hard startup error before a single frame is decoded: a gallery built
// with another embedder yields plausible-looking, meaningless matches.
verify_gallery_embedder(gallery, cfg.gallery_path, cfg.arcface_model,
cfg.require_gallery_stamp);
} catch (const std::exception& e) {
std::cerr << "Gallery error: " << e.what() << "\n";
return 1;
}
std::cerr << "[main] gallery loaded: " << gallery.actors.size() << " actors\n";
// ── Construct node functors ───────────────────────────────────────────────
std::atomic<bool> done{false}; // set by result_sink (face branch)
std::atomic<bool> scene_done{true}; // set by scene_detector; true when disabled
FrameSourceFunc source_fn {cfg};
CameraPositionChangeDetectorFunc campos_fn {cfg};
FaceDetectorFunc detector_fn{cfg};
FaceAlignerFunc aligner_fn;
EmbedderFunc embedder_fn{cfg};
// Constructed before the tracker: it fits (or loads) the calibration, and
// the tracker must decide in that same probability space (AR-024).
IdentityMatcherFunc matcher_fn {gallery, cfg};
/// TRACES: AR-007, AR-008, AR-012, AR-024 | SR-002
// The registry is created here and shared, not owned by a node: track state
// is not a stage in the stream, it is state several stages read and write,
// and its final answer is only known when a track dies.
auto same_person = same_person_probability(matcher_fn.calibration());
TrackRegistry::Config reg_cfg;
reg_cfg.extinction_sec = cfg.track_extinction_sec;
auto registry = std::make_shared<TrackRegistry>(
reg_cfg, EvidenceDiscounter(same_person));
FaceTrackerFunc ftracker_fn{cfg, registry, same_person};
SceneTrackerFunc tracker_fn {cfg};
ResultSinkFunc sink_fn {cfg, done};
#ifdef SAE_DEBUG
DebugRendererFunc debug_fn {cfg};
#endif
// ── Wrap in KPN ObjectNodes ───────────────────────────────────────────────
// Queue sizes tuned to the pipeline's speed profile:
// embedder (16ms) is the slowest GPU node — buffer before it must be largest
// to prevent face_aligner pool overflows and frame drops.
kpn::ObjectNode<FrameSourceFunc, kpn::in<>, kpn::out<"raw">, "frame_source", 0> source (source_fn, 32);
kpn::ObjectNode<CameraPositionChangeDetectorFunc, kpn::in<"raw">, kpn::out<"frame">, "camera_pos", 0> campos (campos_fn, 32);
kpn::ObjectNode<FaceDetectorFunc, kpn::in<"frame">, kpn::out<"scene">, "face_detector", 0> detector (detector_fn, 64);
kpn::ObjectNode<FaceAlignerFunc, kpn::in<"scene">, kpn::out<"aligned">, "face_aligner", 0> aligner (aligner_fn, 64);
kpn::ObjectNode<EmbedderFunc, kpn::in<"aligned">, kpn::out<"embedded">, "embedder", 0> embedder (embedder_fn, 32);
kpn::ObjectNode<FaceTrackerFunc, kpn::in<"embedded">, kpn::out<"tracked">, "face_tracker", 0> ftracker (ftracker_fn, 16);
kpn::ObjectNode<IdentityMatcherFunc, kpn::in<"tracked">, kpn::out<"matched">, "identity_matcher", 0> matcher (matcher_fn, 16);
kpn::ObjectNode<SceneTrackerFunc, kpn::in<"matched">, kpn::out<"annotation">, "scene_tracker", 0> tracker (tracker_fn, 16);
kpn::ObjectNode<ResultSinkFunc, kpn::in<"annotation">,kpn::out<>, "result_sink", 0> sink (sink_fn, 16);
// ── Pipeline observability + run loop (topology-agnostic) ──────────────────
// Factored into a lambda so the two topologies (with/without the scene-detect
// branch) share identical event handling, wait loop, and teardown. Any
// make_network result type binds to `Net&&`.
std::mutex event_mtx;
std::map<std::string, long> overflow_counts;
std::atomic<bool> node_crashed{false};
auto run_net = [&](auto&& net) -> int {
// Tally per-node channel overflow, and treat any non-result_sink Closed
// event as a crash so the wait loop below can't hang on `done` forever.
net.set_event_handler(
[&](std::string_view node_name, kpn::NodeEvent ev,
std::chrono::steady_clock::time_point) {
if (ev == kpn::NodeEvent::Overflow) {
std::lock_guard<std::mutex> lk(event_mtx);
++overflow_counts[std::string(node_name)];
} else { // NodeEvent::Closed
if (node_name == "result_sink" && done.load(std::memory_order_acquire))
return;
std::cerr << "[main] node '" << node_name
<< "' stopped unexpectedly — aborting pipeline\n";
node_crashed.store(true, std::memory_order_release);
}
});
std::cerr << "[main] starting pipeline…\n";
net.start();
// Wait until BOTH terminal branches finish: result_sink (face pipeline)
// and, when enabled, scene_detector (the dense TransNetV2 branch, which
// runs much slower and must not be torn down mid-stream). scene_done is
// pre-set true when scene detection is disabled.
while ((!done.load(std::memory_order_acquire) ||
!scene_done.load(std::memory_order_acquire)) &&
!node_crashed.load(std::memory_order_acquire))
std::this_thread::sleep_for(std::chrono::milliseconds(100));
net.stop();
net.print_diagnostics();
{
std::lock_guard<std::mutex> lk(event_mtx);
if (!overflow_counts.empty()) {
std::cerr << "[main] dropped frames (channel overflow):\n";
for (const auto& [name, count] : overflow_counts)
std::cerr << " " << name << ": " << count << "\n";
}
}
return node_crashed.load(std::memory_order_acquire) ? 1 : 0;
};
// ── Build static network and run ──────────────────────────────────────────
// Common face-analysis chain (campos → … → sink) is identical in all cases;
// the scene-detect branch and the debug fanout are spliced on conditionally.
// Topology:
// plain: source → campos → detector → … → sink
// scene-detect: source ─┬→ campos → decimate(filter) → detector → … → sink
// └→ scene_detector (TransNetV2 sink → scenes.json)
// The fanout after `source` is auto-inserted by make_network when its output
// feeds two edges. In dense mode campos still sees native-rate frames (so it
// detects angle changes correctly); a FilterNode then thins to sample_fps
// before face detection.
#ifdef SAE_DEBUG
kpn::ObjectNode<DebugRendererFunc, kpn::in<"matched">, kpn::out<>, "debug_renderer", 1> debug_node(debug_fn, 16);
#define SAE_DEBUG_EDGE , kpn::edge(matcher.output<"matched">(), debug_node.input<"matched">())
#else
#define SAE_DEBUG_EDGE
#endif
int rc = 0;
if (!cfg.dump_embeddings_path.empty()) {
// Dump-only topology: run the expensive front half and tee the embedder
// output to an HDF5 dump for offline sweep replay (sae_kpn). Downstream
// matching is skipped — the sweep re-runs it from the dump.
EmbeddingDumpFunc dump_fn{cfg, done};
kpn::ObjectNode<EmbeddingDumpFunc, kpn::in<"embedded">, kpn::out<>, "embedding_dump", 0>
dump_node(dump_fn, 32);
auto net = kpn::make_network(
kpn::edge(source.output<"raw">(), campos.input<"raw">()),
kpn::edge(campos.output<"frame">(), detector.input<"frame">()),
kpn::edge(detector.output<"scene">(), aligner.input<"scene">()),
kpn::edge(aligner.output<"aligned">(), embedder.input<"aligned">()),
kpn::edge(embedder.output<"embedded">(), dump_node.input<"embedded">())
);
return run_net(std::move(net));
}
if (cfg.scene_detect) {
scene_done.store(false, std::memory_order_release); // now a real terminal branch
SceneDetectorFunc scene_fn{cfg, scene_done};
kpn::ObjectNode<SceneDetectorFunc, kpn::in<"dense">, kpn::out<>, "scene_detector", 0>
scene_node(scene_fn, 128);
// Decimator: keep frames on the sample_fps cadence, drop the rest.
// eof always passes so downstream shuts down cleanly. Stateful — one
// instance, mutable via shared_ptr so the std::function stays copyable.
auto decim_state = std::make_shared<double>(-1e18);
const double interval = 1.0 / cfg.sample_fps;
auto decimate = kpn::make_filter<Frame>(
[decim_state, interval](const Frame& f) {
if (f.eof) return true;
if (f.timestamp_sec - *decim_state >= interval - 1e-6) {
*decim_state = f.timestamp_sec;
return true;
}
return false;
}, 32);
auto net = kpn::make_network(
kpn::edge(source.output<"raw">(), campos.input<"raw">()),
kpn::edge(source.output<"raw">(), scene_node.input<"dense">()),
kpn::edge(campos.output<"frame">(), decimate.input<0>()),
kpn::edge(decimate.output<0>(), detector.input<"frame">()),
kpn::edge(detector.output<"scene">(), aligner.input<"scene">()),
kpn::edge(aligner.output<"aligned">(), embedder.input<"aligned">()),
kpn::edge(embedder.output<"embedded">(), ftracker.input<"embedded">()),
kpn::edge(ftracker.output<"tracked">(), matcher.input<"tracked">()),
kpn::edge(matcher.output<"matched">(), tracker.input<"matched">()),
kpn::edge(tracker.output<"annotation">(), sink.input<"annotation">())
SAE_DEBUG_EDGE
);
rc = run_net(std::move(net));
} else {
auto net = kpn::make_network(
kpn::edge(source.output<"raw">(), campos.input<"raw">()),
kpn::edge(campos.output<"frame">(), detector.input<"frame">()),
kpn::edge(detector.output<"scene">(), aligner.input<"scene">()),
kpn::edge(aligner.output<"aligned">(), embedder.input<"aligned">()),
kpn::edge(embedder.output<"embedded">(), ftracker.input<"embedded">()),
kpn::edge(ftracker.output<"tracked">(), matcher.input<"tracked">()),
kpn::edge(matcher.output<"matched">(), tracker.input<"matched">()),
kpn::edge(tracker.output<"annotation">(), sink.input<"annotation">())
SAE_DEBUG_EDGE
);
rc = run_net(std::move(net));
}
#undef SAE_DEBUG_EDGE
return rc;
}