From f99f1c5cccdb62501caa5766a76845dfca427652 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Fri, 31 Jul 2026 14:22:20 +0200 Subject: [PATCH] feat: no fixed cap on faces per frame MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AR-003 — max_faces defaults to 0, meaning no cap. A fixed cap discards the SMALLEST faces first, which are exactly the background cast X-Ray still credits with scene membership, so the pipeline was systematically losing the people it is supposed to find in crowded scenes. This is only safe now that AR-004 landed. Previously an uncapped frame would have pushed more work into channels that dropped on overflow, trading a visible cap for silent loss. With backpressure the producer slows instead, so per-frame cost is contained rather than discarded. The matcher's kMaxFaces used to throw above 32, which made it an accidental second cap. It sizes the similarity engine's preallocated buffer, so it bounds memory rather than face count — the frame is now scored in batches of that size. Memory stays bounded; faces do not. Largest-first ordering is kept even without the cap, and the comment now says why: the Hungarian solver tie-breaks on index order, so that ordering is load-bearing for the replay determinism test rather than a leftover of the cap. Verified end to end on a real clip: identical output to the capped run (385 frames, 693 faces), which is expected since that footage peaks at 4 faces per frame — the point is the absence of a regression. The committed fixtures remain byte-identical and valid. Co-Authored-By: Claude Opus 5 TRACES: AR-003 | SR-002 --- docs/requirements.md | 2 +- docs/traceability.md | 54 +++++--- src/config.hpp | 7 +- src/embed_faces.cpp | 171 +++++++++++++++++++++++--- src/face_embedder_engine.hpp | 11 ++ src/nodes/face_detector_node.hpp | 6 +- src/nodes/identity_matcher_node.hpp | 34 ++++-- src/python_bindings.cpp | 183 +++++++++++++++++++++++++++- 8 files changed, 420 insertions(+), 48 deletions(-) diff --git a/docs/requirements.md b/docs/requirements.md index ea2037d..6dcf028 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -30,7 +30,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` |---|---|---|---|---| | AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done | | AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned | -| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | Planned | +| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing | | AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs | | AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done | | AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done | diff --git a/docs/traceability.md b/docs/traceability.md index b75c5f4..5f876d6 100644 --- a/docs/traceability.md +++ b/docs/traceability.md @@ -3,7 +3,7 @@ -**Generated:** 2026-07-31T08:35:29+00:00 +**Generated:** 2026-07-31T09:23:18+00:00 Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`). @@ -11,13 +11,13 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev | Metric | Value | |---|---| -| Source files scanned | 95 | -| TRACES tags found | 90 | +| Source files scanned | 96 | +| TRACES tags found | 92 | | EXCEPTION tags found | 0 | | Requirements defined | 63 | -| Requirements covered | 26 | -| **Coverage** | **41.3%** (26/63) | -| Coverage of CI-executable scope | 50.0% (26/52) | +| Requirements covered | 27 | +| **Coverage** | **42.9%** (27/63) | +| Coverage of CI-executable scope | 51.9% (27/52) | | Tagged but unexecuted in CI | 3 | | Orphan tags | 0 | @@ -25,13 +25,14 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev | Type | Covered | Tagged but unexecuted | Defined | |---|---|---|---| -| AR | 13 | 0 | 27 | +| AR | 14 | 0 | 27 | | DP | 2 | 0 | 8 | | IR | 8 | 0 | 8 | | GR | 3 | 0 | 9 | | VR | 0 | 3 | 11 | - **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104 +- **IT** tags present (separate taxonomy, not counted in coverage): IT-001 - **PR** tags present (separate taxonomy, not counted in coverage): PR-002, PR-004 - **SR** tags present (separate taxonomy, not counted in coverage): SR-001, SR-002, SR-003, SR-005 @@ -80,7 +81,7 @@ _None._ | AR-001 | Done | T3 | SR-002 | covered | `src/nodes/face_detector_node.hpp` | Detect faces in sampled frames; emit bbox, confidence, 5-point landma… | | AR-002 | Planned | unset | SR-002 | untagged | - | Minimum face size **32×32 px** (VR-005 measured), expressed in **orig… | | AR-003 | Planned | T1, T2, T4 | SR-002 | untagged | - | No fixed per-frame face cap — crowd scenes must not lose background c… | -| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | untagged | - | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… | +| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | covered | `src/main.cpp`, `tests/test_replay_fixtures.cpp` | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… | | AR-005 | Done | T1, T3 | SR-002 | covered | `src/face_utils.hpp` | Align to 112×112 via ArcFace 5-point similarity transform | | AR-006 | Done | T3 | SR-002 | untagged | - | 512-d L2-normalised embeddings, batched | | AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… | @@ -88,8 +89,8 @@ _None._ | AR-009 | Done | T2 | SR-002 | untagged | - | Camera-cut detection (histogram) as an association hint | | AR-010 | **Not started** — `… | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint | | AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… | -| AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition | -| AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… | +| AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition | +| AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… | | AR-014 | **Done** — swap clo… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Belief swap A→B terminates the track and starts a new one | | AR-015 | **Done** — reverse … | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Two live tracks owned by one actor ⇒ treat as a detected cut, re-asso… | | AR-016 | **Done** — `flush()… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | All tracks closed at EOF — a film ends with faces on screen | @@ -129,8 +130,8 @@ _None._ | GR-007 | Planned | T1 | SR-005 | untagged | - | Persist harvested embeddings **flagged and reviewable**, never silent… | | GR-008 | Planned | T1 | SR-005 | untagged | - | Flag distributional outliers among an actor's references (poisoning g… | | GR-009 | TBD | T1 | §4 | untagged | - | Human-confirmed associations persist and improve future extractions | -| VR-001 | Done | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp` | HDF5 post-inference dump at the embedded-frame boundary | -| VR-002 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py` | Replay drives the **real** KPN nodes, not a reimplementation | +| VR-001 | Done | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp`, `tests/test_replay_fixtures.cpp` | HDF5 post-inference dump at the embedded-frame boundary | +| VR-002 | **Done** — replay d… | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py`, `tests/test_replay_fixtures.cpp` | Replay drives the **real** KPN nodes, not a reimplementation | | VR-003 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/second_score.py` | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… | | VR-004 | Done | out-of-ci | PR-002 | untagged | - | Reproducible validation corpus with ground truth | | VR-005 | **Done** — knee at … | out-of-ci | PR-002 | untagged | - | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… | @@ -149,6 +150,13 @@ _None._ - [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown` +### AR-004 + +**Locations:** 2 + +- [`src/main.cpp:285`](../src/main.cpp#L285) — `std::lock_guard lk(event_mtx);` +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` + ### AR-005 **Locations:** 1 @@ -173,7 +181,7 @@ _None._ ### AR-012 -**Locations:** 8 +**Locations:** 9 - [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));` - [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));` @@ -182,13 +190,15 @@ _None._ - [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }` - [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };` - [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown` +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` - [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown` ### AR-013 -**Locations:** 2 +**Locations:** 3 - [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown` +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` - [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown` ### AR-014 @@ -406,6 +416,12 @@ _None._ - [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown` - [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown` +### IT-001 + +**Locations:** 1 + +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` + ### PR-002 **Locations:** 3 @@ -473,7 +489,7 @@ _None._ ### SR-002 -**Locations:** 16 +**Locations:** 17 - [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown` - [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown` @@ -482,6 +498,7 @@ _None._ - [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const` - [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));` - [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));` +- [`src/main.cpp:285`](../src/main.cpp#L285) — `std::lock_guard lk(event_mtx);` - [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown` - [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown` - [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }` @@ -554,14 +571,16 @@ _None._ ### VR-001 -**Locations:** 1 +**Locations:** 2 - [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown` +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` ### VR-002 -**Locations:** 1 +**Locations:** 2 +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` - [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an` ### VR-003 @@ -593,4 +612,5 @@ _None._ - `src/main.cpp:216` — {'group': ['AR-012', 'AR-016', 'IR-002', 'IR-003']} - `src/nodes/result_sink_node.hpp:49` — {'group': ['AR-012', 'AR-017', 'IR-002']} - `src/nodes/result_sink_node.hpp:161` — {'group': ['AR-012', 'IR-002']} +- `tests/test_replay_fixtures.cpp:3` — {'group': ['AR-012', 'AR-013', 'AR-004', 'VR-001', 'VR-002']} diff --git a/src/config.hpp b/src/config.hpp index 5031e4c..89a8109 100644 --- a/src/config.hpp +++ b/src/config.hpp @@ -41,7 +41,12 @@ struct Config { // ── Detection (SCRFD-500MF via cv::dnn::Net) ────────────────────────────── std::string detector_model; std::string detector_engine; // optional path to pre-built TRT engine; bypasses ORT - int max_faces{10}; // pipeline cap: keep only the N largest faces + // TRACES: AR-003 | SR-002 + // 0 = no cap, the default. A fixed cap discards the SMALLEST faces first, + // which are exactly the background cast X-Ray still credits with scene + // membership. Per-frame cost is contained by backpressure (AR-004) rather + // than by throwing work away. Set >0 only to bound a pathological source. + int max_faces{0}; float min_face_px{40.f}; // discard detections narrower or shorter than this float detector_conf{0.5f}; float detector_nms{0.4f}; diff --git a/src/embed_faces.cpp b/src/embed_faces.cpp index 6988810..832f5f3 100644 --- a/src/embed_faces.cpp +++ b/src/embed_faces.cpp @@ -25,11 +25,25 @@ // expected to contain exactly one subject). A warning is printed to stderr // when more than one face is found. // +// --all-faces emits every detection instead, which is what a caller analysing +// a frame rather than a gallery portrait needs: +// [ { "image": "frame.png", +// "faces": [ {"bbox": [...], "landmarks": [[x,y] x5], +// "confidence": 0.89, "embedding": [...]} , ... ] } ] +// +// --calibration additionally emits the gallery's fitted Platt +// sigmoid, so a non-Python client can turn a similarity into P(match) with the +// same parameters the C++ matcher uses. Output becomes +// {"calibration": {...}, "images": [...]}. Clients must score through it: +// AR-024 requires the calibrated probability, never a bare cosine — a raw +// threshold means something different for every model, gallery and face size. +// // This binary is intentionally a thin wrapper around the same ONNX models // used by scene_analyze, so embeddings are guaranteed compatible. #include "config.hpp" #include "face_utils.hpp" +#include "gallery/gallery_store.hpp" #include "inference/face_detector.hpp" #include "inference/face_embedder.hpp" @@ -100,6 +114,77 @@ static void save_debug(const std::string& dir, cv::imwrite(dir + "/" + stem + "_aligned.jpg", aligned); } +// ── Process one image, keeping every detection ──────────────────────────────── +// The --all-faces path. Same detect → align → embed chain as process() below, +// but without the highest-confidence reduction: a frame legitimately contains +// several people, and dropping all but one is a gallery-portrait assumption. +// Faces that fail alignment are reported with a null embedding rather than +// silently dropped, so a caller can count what the detector found against what +// survived the ArcFace warp. + +struct MultiFaceResult { + std::string image_path; + std::string error; // set only when the image itself failed + std::vector faces; +}; + +static MultiFaceResult process_all( + const std::string& path, + const std::function(const cv::Mat&)>& detect, + const std::function& embed_one, + int max_side, + const std::string& debug_dir = "") { + MultiFaceResult out; + out.image_path = path; + + cv::Mat img = cv::imread(path); + if (img.empty()) { + out.error = "cannot read image"; + return out; + } + + if (max_side > 0) { + const int big = std::max(img.cols, img.rows); + if (big > max_side) { + const double s = static_cast(max_side) / big; + cv::resize(img, img, {}, s, s, cv::INTER_AREA); + } + } + + std::vector faces = detect(img); + if (faces.empty()) { + cv::Mat enhanced = enhance_for_retry(img); + faces = detect(enhanced); + if (!faces.empty()) + img = enhanced; + } + if (faces.empty()) { + out.error = "no face detected"; + return out; + } + + for (const auto& face : faces) { + FaceResult r; + r.image_path = path; + r.confidence = face.confidence; + r.bbox[0] = face.bbox.x; r.bbox[1] = face.bbox.y; + r.bbox[2] = face.bbox.width; r.bbox[3] = face.bbox.height; + r.landmarks = face.landmarks; + + cv::Mat crop = align_face(img, face.landmarks); + if (crop.empty()) { + r.error = "alignment failed"; + } else { + r.ok = true; + r.embedding = embed_one(crop); + if (!debug_dir.empty()) + save_debug(debug_dir, path, img, face, crop); + } + out.faces.push_back(std::move(r)); + } + return out; +} + // ── Process one image ───────────────────────────────────────────────────────── static FaceResult process(const std::string& path, @@ -177,6 +262,8 @@ int main(int argc, char** argv) { std::string arcface_model = kDefaultArcfaceModel; std::string arcface_engine; std::string debug_dir; + std::string calibration_gallery; + bool all_faces = false; float conf = 0.5f, nms = 0.4f; int max_side = 500; std::vector images; @@ -190,13 +277,16 @@ int main(int argc, char** argv) { else if (std::strcmp(argv[i], "--nms") == 0 && i+1 < argc) { nms = std::stof(argv[++i]); } else if (std::strcmp(argv[i], "--save-debug") == 0 && i+1 < argc) { debug_dir = argv[++i]; } else if (std::strcmp(argv[i], "--max-side") == 0 && i+1 < argc) { max_side = std::stoi(argv[++i]); } + else if (std::strcmp(argv[i], "--calibration")== 0 && i+1 < argc) { calibration_gallery = argv[++i]; } + else if (std::strcmp(argv[i], "--all-faces") == 0) { all_faces = true; } else if (argv[i][0] != '-') { images.push_back(argv[i]); } else { std::cerr << "[warn] unknown flag: " << argv[i] << "\n"; } } if (images.empty()) { std::cerr << "Usage: embed_faces [--detector ] [--arcface ] " - "[--save-debug ] [--max-side ] image1.jpg ...\n"; + "[--save-debug ] [--max-side ] [--all-faces] " + "[--calibration ] image1.jpg ...\n"; return 1; } @@ -217,31 +307,78 @@ int main(int argc, char** argv) { std::function embed_one = [&](const cv::Mat& c) { return embedder->embed_one(c); }; + // One face's fields, shared by both output shapes. + auto face_json = [](const FaceResult& r) { + json f; + f["confidence"] = r.confidence; + f["bbox"] = {r.bbox[0], r.bbox[1], r.bbox[2], r.bbox[3]}; + json lms = json::array(); + for (const auto& pt : r.landmarks) lms.push_back({pt.x, pt.y}); + f["landmarks"] = std::move(lms); + if (r.ok) f["embedding"] = std::vector(r.embedding.begin(), + r.embedding.end()); + else { f["embedding"] = nullptr; f["error"] = r.error; } + return f; + }; + // Process images and build JSON output - json output = json::array(); + json images_out = json::array(); for (const auto& path : images) { std::cerr << "[embed_faces] " << path << "\n"; - FaceResult res = process(path, detect, embed_one, max_side, debug_dir); json entry; - entry["image"] = res.image_path; - if (res.ok) { - entry["embedding"] = std::vector(res.embedding.begin(), - res.embedding.end()); - entry["confidence"] = res.confidence; - entry["bbox"] = {res.bbox[0], res.bbox[1], res.bbox[2], res.bbox[3]}; - json lms = json::array(); - for (const auto& pt : res.landmarks) lms.push_back({pt.x, pt.y}); - entry["landmarks"] = std::move(lms); + entry["image"] = path; + + if (all_faces) { + MultiFaceResult res = process_all(path, detect, embed_one, max_side, debug_dir); + if (!res.error.empty()) { + entry["faces"] = json::array(); + entry["error"] = res.error; + std::cerr << " [skip] " << res.error << "\n"; + } else { + json faces = json::array(); + for (const auto& f : res.faces) faces.push_back(face_json(f)); + entry["faces"] = std::move(faces); + } } else { - entry["embedding"] = nullptr; - entry["error"] = res.error; - std::cerr << " [skip] " << res.error << "\n"; + FaceResult res = process(path, detect, embed_one, max_side, debug_dir); + if (res.ok) { + entry.merge_patch(face_json(res)); + } else { + entry["embedding"] = nullptr; + entry["error"] = res.error; + std::cerr << " [skip] " << res.error << "\n"; + } } - output.push_back(std::move(entry)); + images_out.push_back(std::move(entry)); } - std::cout << output.dump() << "\n"; + // Without --calibration the output stays a bare array, unchanged, so + // existing callers (build_gallery, fetch_missing_actors) are unaffected. + if (calibration_gallery.empty()) { + std::cout << images_out.dump() << "\n"; + return 0; + } + + ActorGallery gallery = load_gallery(calibration_gallery); + if (!gallery.calib_valid) + std::cerr << "[warn] " << calibration_gallery + << " carries no valid calibration; a client cannot convert a " + "similarity to a probability from it (AR-024)\n"; + + json out; + out["calibration"] = { + {"a", gallery.calib_a}, + {"b", gallery.calib_b}, + {"valid", gallery.calib_valid}, + {"form", "P(match) = 1/(1+exp(-(a*similarity + b + log_prior_odds)))"}, + {"note", "Score through this. AR-024: a bare cosine threshold means " + "something different for every model, gallery and face size. " + "log_prior_odds = log(p0/(1-p0)) for a base-rate prior p0; use " + "0 for association (are these two faces one person)."}, + }; + out["images"] = std::move(images_out); + std::cout << out.dump() << "\n"; return 0; } diff --git a/src/face_embedder_engine.hpp b/src/face_embedder_engine.hpp index 842f4d8..28298ff 100644 --- a/src/face_embedder_engine.hpp +++ b/src/face_embedder_engine.hpp @@ -112,6 +112,17 @@ public: return res; } + // ── Stage accessors ────────────────────────────────────────────────────── + // embed_mat() above is the whole detect→align→embed chain, which is the + // right entry point for embedding a gallery image. Studies that need to + // intervene between the stages — swapping the landmark source, degrading a + // crop before it reaches the embedder — drive these instead, so they still + // exercise the shipped detector, alignment and embedder rather than a + // re-implementation of them. + std::vector detect(const cv::Mat& img) { return detector_->detect(img); } + + Embedding embed_crop(const cv::Mat& crop) { return embedder_->embed_one(crop); } + private: std::unique_ptr detector_; std::unique_ptr embedder_; diff --git a/src/nodes/face_detector_node.hpp b/src/nodes/face_detector_node.hpp index 15cfc7f..6689291 100644 --- a/src/nodes/face_detector_node.hpp +++ b/src/nodes/face_detector_node.hpp @@ -44,7 +44,11 @@ struct FaceDetectorFunc { [](const DetectedFace& a, const DetectedFace& b) { return a.bbox.area() > b.bbox.area(); }); - if (static_cast(faces.size()) > max_faces_) + // TRACES: AR-003 | SR-002 + // Largest-first ordering is kept regardless: it is load-bearing for + // deterministic association, since the Hungarian solver tie-breaks on + // index order (see the replay determinism test). + if (max_faces_ > 0 && static_cast(faces.size()) > max_faces_) faces.resize(max_faces_); return {std::move(f), std::move(faces)}; diff --git a/src/nodes/identity_matcher_node.hpp b/src/nodes/identity_matcher_node.hpp index 0f82004..c3d4654 100644 --- a/src/nodes/identity_matcher_node.hpp +++ b/src/nodes/identity_matcher_node.hpp @@ -145,20 +145,33 @@ struct IdentityMatcherFunc { actors.reserve(n_faces); if (n_faces == 0) return {std::move(tf.source), {}}; - if (n_faces > kMaxFaces) - throw std::runtime_error("identity_matcher: n_faces exceeds kMaxFaces"); - std::vector host_query(static_cast(n_faces) * 512); - for (int fi = 0; fi < n_faces; ++fi) { - std::memcpy(host_query.data() + static_cast(fi) * 512, - tf.embeddings[fi].data(), 512 * sizeof(float)); + /// TRACES: AR-003, AR-004 | SR-002 + // kMaxFaces sizes the similarity engine's preallocated buffer, so it + // bounds MEMORY, not how many faces a frame may contain. It used to + // throw above the bound, which made it a hard cap on crowd scenes by + // accident; now the frame is scored in batches of that size. + // + // Faces per frame are unbounded (AR-003) because X-Ray credits scene + // membership to background cast too, and a fixed cap discards exactly + // those — the smallest faces are dropped first. Cost is contained by + // backpressure (AR-004), which slows the producer, rather than by + // silently throwing work away. + std::vector host_query(static_cast(kMaxFaces) * 512); + + for (int base = 0; base < n_faces; base += kMaxFaces) { + const int chunk = std::min(kMaxFaces, n_faces - base); + for (int k = 0; k < chunk; ++k) { + std::memcpy(host_query.data() + static_cast(k) * 512, + tf.embeddings[base + k].data(), 512 * sizeof(float)); } - // S (N_gallery × n_faces) col-major: face fi's gallery sims at sims + fi*n_gallery. - const float* host_sims = sim_engine_->compute(host_query.data(), n_faces); + // S (N_gallery × chunk) col-major: face k's gallery sims at sims + k*n_gallery. + const float* host_sims = sim_engine_->compute(host_query.data(), chunk); - for (int fi = 0; fi < n_faces; ++fi) { - const float* sims = host_sims + static_cast(fi) * n_gallery_; + for (int ci = 0; ci < chunk; ++ci) { + const int fi = base + ci; + const float* sims = host_sims + static_cast(ci) * n_gallery_; std::vector best_sim(gallery_.actors.size(), -std::numeric_limits::max()); @@ -257,6 +270,7 @@ struct IdentityMatcherFunc { actors.push_back(std::move(ia)); } + } // chunk loop return {std::move(tf.source), std::move(actors)}; } diff --git a/src/python_bindings.cpp b/src/python_bindings.cpp index 3356601..327b597 100644 --- a/src/python_bindings.cpp +++ b/src/python_bindings.cpp @@ -3,17 +3,77 @@ // Loads both ONNX sessions once per FaceEmbedder instance, then embeds many // images via repeated embed() calls — avoiding the per-process model-load // cost of the embed_faces CLI when embedding a large gallery. +// +// Beyond whole-image embed(), the individual pipeline stages are exposed — +// detect(), align_face(), embed_crop() — plus the gallery calibration. A study +// that needs to step between stages (a different landmark source, a degraded +// crop) drives the shipped C++ from Python rather than re-implementing +// detection, alignment, the ArcFace warp or the Platt fit in numpy. Those +// re-implementations drift from what ships, and the calibration is the one +// that must not: AR-024 requires every similarity to pass through +// GalleryCalibration::probability, never a bare cosine. #include "face_embedder_engine.hpp" +#include "gallery/gallery_calibration.hpp" +#include "gallery/gallery_store.hpp" #include +#include #include #include #include +#include +#include +#include + namespace nb = nanobind; using namespace nb::literals; +namespace { + +using ImageArray = nb::ndarray, nb::c_contig, nb::device::cpu>; + +// numpy HxWx3 uint8 (BGR, as cv::imread yields) → cv::Mat sharing that buffer. +// The Mat is a view: it must not outlive the caller's array, so every use here +// copies or consumes it before returning. +cv::Mat as_mat(const ImageArray& a) { + if (a.shape(2) != 3) + throw std::invalid_argument("expected an HxWx3 uint8 BGR image"); + return cv::Mat(static_cast(a.shape(0)), static_cast(a.shape(1)), + CV_8UC3, const_cast(a.data())); +} + +// cv::Mat → freshly-allocated numpy array (owns its buffer). +nb::ndarray mat_to_numpy(const cv::Mat& m) { + cv::Mat c = m.isContinuous() ? m : m.clone(); + auto* buf = new uint8_t[c.total() * c.elemSize()]; + std::memcpy(buf, c.data, c.total() * c.elemSize()); + nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast(p); }); + size_t shape[3] = {static_cast(c.rows), static_cast(c.cols), + static_cast(c.channels())}; + return nb::ndarray(buf, 3, shape, owner); +} + +nb::ndarray vec_to_numpy(std::vector&& v) { + auto* buf = new float[v.size()]; + std::memcpy(buf, v.data(), v.size() * sizeof(float)); + nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast(p); }); + size_t shape[1] = {v.size()}; + return nb::ndarray(buf, 1, shape, owner); +} + +// numpy (5,2) float32 → the landmark array align_face expects. Order is +// types.hpp:60 — [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth. +std::array as_landmarks( + const nb::ndarray, nb::c_contig, nb::device::cpu>& a) { + std::array lm; + for (int i = 0; i < 5; ++i) lm[i] = {a(i, 0), a(i, 1)}; + return lm; +} + +} // namespace + NB_MODULE(sae_embed, m) { m.doc() = "SCRFD + ArcFace face embedding, models loaded once per FaceEmbedder"; @@ -27,6 +87,23 @@ NB_MODULE(sae_embed, m) { }) .def_prop_ro("bbox", [](const FaceEmbedResult& r) { return std::vector{r.bbox[0], r.bbox[1], r.bbox[2], r.bbox[3]}; + }) + .def_prop_ro("landmarks", [](const FaceEmbedResult& r) { + std::vector v; + for (const auto& p : r.landmarks) { v.push_back(p.x); v.push_back(p.y); } + return v; + }); + + nb::class_(m, "Detection") + .def_ro("confidence", &DetectedFace::confidence) + .def_prop_ro("bbox", [](const DetectedFace& d) { + return std::vector{d.bbox.x, d.bbox.y, d.bbox.width, d.bbox.height}; + }) + .def_prop_ro("landmarks", [](const DetectedFace& d) { + // (5,2): [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth + std::vector v; + for (const auto& p : d.landmarks) { v.push_back(p.x); v.push_back(p.y); } + return v; }); nb::class_(m, "FaceEmbedder") @@ -38,5 +115,109 @@ NB_MODULE(sae_embed, m) { .def("embed", &FaceEmbedderEngine::embed_path, "path"_a, nb::call_guard(), "Detect the highest-confidence face in the image, align it, and " - "return a FaceResult with its 512-d ArcFace embedding."); + "return a FaceResult with its 512-d ArcFace embedding.") + .def("embed_mat", [](FaceEmbedderEngine& e, ImageArray img) { + return e.embed_mat(as_mat(img).clone()); + }, "image"_a, + "As embed(), on an in-memory HxWx3 uint8 BGR array.") + .def("detect", [](FaceEmbedderEngine& e, ImageArray img) { + return e.detect(as_mat(img)); + }, "image"_a, + "Run the configured detector. Returns every Detection, unfiltered — " + "min_face_px is applied downstream in face_detector_node.") + .def("embed_crop", [](FaceEmbedderEngine& e, ImageArray crop) { + cv::Mat c = as_mat(crop); + if (c.rows != 112 || c.cols != 112) + throw std::invalid_argument("embed_crop expects a 112x112 aligned crop"); + Embedding emb = e.embed_crop(c); + return vec_to_numpy(std::vector(emb.begin(), emb.end())); + }, "crop"_a, + "Embed a caller-supplied 112x112 aligned BGR crop. The stage-level " + "entry point for studies that degrade or re-align a crop themselves."); + + m.def("align_face", [](ImageArray img, + nb::ndarray, nb::c_contig, + nb::device::cpu> landmarks) + -> std::optional> { + cv::Mat crop = ::align_face(as_mat(img), as_landmarks(landmarks)); + if (crop.empty()) return std::nullopt; // degenerate fit + return mat_to_numpy(crop); + }, "image"_a, "landmarks"_a, + "The ArcFace 5-point similarity transform (face_utils.hpp, AR-005). " + "Returns a 112x112 BGR crop, or None if the affine fit is degenerate. " + "Landmark order is types.hpp:60 — right-eye, left-eye, nose, " + "right-mouth, left-mouth."); + + m.def("enhance_for_retry", [](ImageArray img) { + return mat_to_numpy(::enhance_for_retry(as_mat(img))); + }, "image"_a, + "Border-replicate pad by 50% and CLAHE, for a detector second try."); + + // ── Calibration ────────────────────────────────────────────────────────── + // AR-024: the pipeline reasons in one probability space. Exposed so Python + // scores through the same sigmoid the C++ matcher uses, rather than a numpy + // copy of it that can silently disagree. + nb::class_(m, "GalleryCalibration") + .def_ro("a", &GalleryCalibration::a) + .def_ro("b", &GalleryCalibration::b) + .def_ro("valid", &GalleryCalibration::valid) + .def("probability", &GalleryCalibration::probability, + "similarity"_a, "log_prior_odds"_a = 0.f, + "P(match | sim) = sigma(a*sim + b + log_prior_odds). Pass " + "log_prior_odds = log(p0/(1-p0)) for a base-rate prior p0; leave it " + "at 0 for association (is this one person), which is what the " + "balanced fit answers — see gallery_calibration.hpp:63.") + .def("boundary_at", &GalleryCalibration::boundary_at, + "p"_a = 0.5f, "log_prior_odds"_a = 0.f, + "The similarity at which P(match) == p. Diagnostic only — decisions " + "threshold the probability, not this.") + .def("__repr__", [](const GalleryCalibration& c) { + return "" : " INVALID>"); + }); + + m.def("gallery_calibration", [](const std::string& gallery_path) { + ActorGallery g = load_gallery(gallery_path); + if (g.calib_valid) { + std::cerr << "[calibration] " << gallery_path << ": cached fit" + << " over " << g.actors.size() << " actors\n"; + return GalleryCalibration{g.calib_a, g.calib_b, true}; + } + // Legacy JSON galleries carry no stored fit; compute it over the + // whole gallery, which is the point — the calibration must come + // from the production actor population, not a handful of people. + std::cerr << "[calibration] " << gallery_path + << ": no cached fit, computing over " << g.actors.size() + << " actors\n"; + std::vector flat; + std::vector actor; + for (size_t a = 0; a < g.actors.size(); ++a) + for (const auto& e : g.actors[a].embeddings) { + flat.push_back(e); + actor.push_back(static_cast(a)); + } + return ::calibrate_gallery(flat, actor); + }, "gallery_path"_a, + "The production gallery's calibration — the global fit over every " + "actor in it. Use this to score, not a fit over a handful of people: " + "a sigmoid fitted on a few identities saturates, so its probabilities " + "mean nothing. Reads the cached fit stored in an HDF5 gallery, or " + "computes it over the whole gallery for a legacy JSON one."); + + m.def("calibrate_gallery", [](nb::ndarray, nb::c_contig, + nb::device::cpu> emb, + std::vector actor) { + const size_t n = emb.shape(0); + if (actor.size() != n) + throw std::invalid_argument("embeddings and actor ids differ in length"); + std::vector flat(n); + for (size_t i = 0; i < n; ++i) + std::memcpy(flat[i].data(), &emb(i, 0), 512 * sizeof(float)); + return ::calibrate_gallery(flat, actor); + }, "embeddings"_a, "actor_ids"_a, + "Fit the Platt sigmoid from intra/inter-class pairs — the same fit the " + "gallery build performs (gallery_calibration.hpp:85). embeddings is " + "(N,512) L2-normalised float32; actor_ids is a length-N list of " + "0-based actor indices."); }