diff --git a/docs/requirements.md b/docs/requirements.md index ea2037d..6dcf028 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -30,7 +30,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` |---|---|---|---|---| | AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done | | AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned | -| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | Planned | +| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing | | AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs | | AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done | | AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done | diff --git a/docs/traceability.md b/docs/traceability.md index b75c5f4..5f876d6 100644 --- a/docs/traceability.md +++ b/docs/traceability.md @@ -3,7 +3,7 @@ -**Generated:** 2026-07-31T08:35:29+00:00 +**Generated:** 2026-07-31T09:23:18+00:00 Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`). @@ -11,13 +11,13 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev | Metric | Value | |---|---| -| Source files scanned | 95 | -| TRACES tags found | 90 | +| Source files scanned | 96 | +| TRACES tags found | 92 | | EXCEPTION tags found | 0 | | Requirements defined | 63 | -| Requirements covered | 26 | -| **Coverage** | **41.3%** (26/63) | -| Coverage of CI-executable scope | 50.0% (26/52) | +| Requirements covered | 27 | +| **Coverage** | **42.9%** (27/63) | +| Coverage of CI-executable scope | 51.9% (27/52) | | Tagged but unexecuted in CI | 3 | | Orphan tags | 0 | @@ -25,13 +25,14 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev | Type | Covered | Tagged but unexecuted | Defined | |---|---|---|---| -| AR | 13 | 0 | 27 | +| AR | 14 | 0 | 27 | | DP | 2 | 0 | 8 | | IR | 8 | 0 | 8 | | GR | 3 | 0 | 9 | | VR | 0 | 3 | 11 | - **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104 +- **IT** tags present (separate taxonomy, not counted in coverage): IT-001 - **PR** tags present (separate taxonomy, not counted in coverage): PR-002, PR-004 - **SR** tags present (separate taxonomy, not counted in coverage): SR-001, SR-002, SR-003, SR-005 @@ -80,7 +81,7 @@ _None._ | AR-001 | Done | T3 | SR-002 | covered | `src/nodes/face_detector_node.hpp` | Detect faces in sampled frames; emit bbox, confidence, 5-point landma… | | AR-002 | Planned | unset | SR-002 | untagged | - | Minimum face size **32×32 px** (VR-005 measured), expressed in **orig… | | AR-003 | Planned | T1, T2, T4 | SR-002 | untagged | - | No fixed per-frame face cap — crowd scenes must not lose background c… | -| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | untagged | - | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… | +| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | covered | `src/main.cpp`, `tests/test_replay_fixtures.cpp` | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… | | AR-005 | Done | T1, T3 | SR-002 | covered | `src/face_utils.hpp` | Align to 112×112 via ArcFace 5-point similarity transform | | AR-006 | Done | T3 | SR-002 | untagged | - | 512-d L2-normalised embeddings, batched | | AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… | @@ -88,8 +89,8 @@ _None._ | AR-009 | Done | T2 | SR-002 | untagged | - | Camera-cut detection (histogram) as an association hint | | AR-010 | **Not started** — `… | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint | | AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… | -| AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition | -| AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… | +| AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition | +| AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… | | AR-014 | **Done** — swap clo… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Belief swap A→B terminates the track and starts a new one | | AR-015 | **Done** — reverse … | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Two live tracks owned by one actor ⇒ treat as a detected cut, re-asso… | | AR-016 | **Done** — `flush()… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | All tracks closed at EOF — a film ends with faces on screen | @@ -129,8 +130,8 @@ _None._ | GR-007 | Planned | T1 | SR-005 | untagged | - | Persist harvested embeddings **flagged and reviewable**, never silent… | | GR-008 | Planned | T1 | SR-005 | untagged | - | Flag distributional outliers among an actor's references (poisoning g… | | GR-009 | TBD | T1 | §4 | untagged | - | Human-confirmed associations persist and improve future extractions | -| VR-001 | Done | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp` | HDF5 post-inference dump at the embedded-frame boundary | -| VR-002 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py` | Replay drives the **real** KPN nodes, not a reimplementation | +| VR-001 | Done | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp`, `tests/test_replay_fixtures.cpp` | HDF5 post-inference dump at the embedded-frame boundary | +| VR-002 | **Done** — replay d… | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py`, `tests/test_replay_fixtures.cpp` | Replay drives the **real** KPN nodes, not a reimplementation | | VR-003 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/second_score.py` | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… | | VR-004 | Done | out-of-ci | PR-002 | untagged | - | Reproducible validation corpus with ground truth | | VR-005 | **Done** — knee at … | out-of-ci | PR-002 | untagged | - | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… | @@ -149,6 +150,13 @@ _None._ - [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown` +### AR-004 + +**Locations:** 2 + +- [`src/main.cpp:285`](../src/main.cpp#L285) — `std::lock_guard lk(event_mtx);` +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` + ### AR-005 **Locations:** 1 @@ -173,7 +181,7 @@ _None._ ### AR-012 -**Locations:** 8 +**Locations:** 9 - [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));` - [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));` @@ -182,13 +190,15 @@ _None._ - [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }` - [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };` - [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown` +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` - [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown` ### AR-013 -**Locations:** 2 +**Locations:** 3 - [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown` +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` - [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown` ### AR-014 @@ -406,6 +416,12 @@ _None._ - [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown` - [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown` +### IT-001 + +**Locations:** 1 + +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` + ### PR-002 **Locations:** 3 @@ -473,7 +489,7 @@ _None._ ### SR-002 -**Locations:** 16 +**Locations:** 17 - [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown` - [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown` @@ -482,6 +498,7 @@ _None._ - [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const` - [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));` - [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));` +- [`src/main.cpp:285`](../src/main.cpp#L285) — `std::lock_guard lk(event_mtx);` - [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown` - [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown` - [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }` @@ -554,14 +571,16 @@ _None._ ### VR-001 -**Locations:** 1 +**Locations:** 2 - [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown` +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` ### VR-002 -**Locations:** 1 +**Locations:** 2 +- [`tests/test_replay_fixtures.cpp:3`](../tests/test_replay_fixtures.cpp#L3) — `Unknown` - [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an` ### VR-003 @@ -593,4 +612,5 @@ _None._ - `src/main.cpp:216` — {'group': ['AR-012', 'AR-016', 'IR-002', 'IR-003']} - `src/nodes/result_sink_node.hpp:49` — {'group': ['AR-012', 'AR-017', 'IR-002']} - `src/nodes/result_sink_node.hpp:161` — {'group': ['AR-012', 'IR-002']} +- `tests/test_replay_fixtures.cpp:3` — {'group': ['AR-012', 'AR-013', 'AR-004', 'VR-001', 'VR-002']} diff --git a/src/config.hpp b/src/config.hpp index 5031e4c..89a8109 100644 --- a/src/config.hpp +++ b/src/config.hpp @@ -41,7 +41,12 @@ struct Config { // ── Detection (SCRFD-500MF via cv::dnn::Net) ────────────────────────────── std::string detector_model; std::string detector_engine; // optional path to pre-built TRT engine; bypasses ORT - int max_faces{10}; // pipeline cap: keep only the N largest faces + // TRACES: AR-003 | SR-002 + // 0 = no cap, the default. A fixed cap discards the SMALLEST faces first, + // which are exactly the background cast X-Ray still credits with scene + // membership. Per-frame cost is contained by backpressure (AR-004) rather + // than by throwing work away. Set >0 only to bound a pathological source. + int max_faces{0}; float min_face_px{40.f}; // discard detections narrower or shorter than this float detector_conf{0.5f}; float detector_nms{0.4f}; diff --git a/src/embed_faces.cpp b/src/embed_faces.cpp index 6988810..832f5f3 100644 --- a/src/embed_faces.cpp +++ b/src/embed_faces.cpp @@ -25,11 +25,25 @@ // expected to contain exactly one subject). A warning is printed to stderr // when more than one face is found. // +// --all-faces emits every detection instead, which is what a caller analysing +// a frame rather than a gallery portrait needs: +// [ { "image": "frame.png", +// "faces": [ {"bbox": [...], "landmarks": [[x,y] x5], +// "confidence": 0.89, "embedding": [...]} , ... ] } ] +// +// --calibration additionally emits the gallery's fitted Platt +// sigmoid, so a non-Python client can turn a similarity into P(match) with the +// same parameters the C++ matcher uses. Output becomes +// {"calibration": {...}, "images": [...]}. Clients must score through it: +// AR-024 requires the calibrated probability, never a bare cosine — a raw +// threshold means something different for every model, gallery and face size. +// // This binary is intentionally a thin wrapper around the same ONNX models // used by scene_analyze, so embeddings are guaranteed compatible. #include "config.hpp" #include "face_utils.hpp" +#include "gallery/gallery_store.hpp" #include "inference/face_detector.hpp" #include "inference/face_embedder.hpp" @@ -100,6 +114,77 @@ static void save_debug(const std::string& dir, cv::imwrite(dir + "/" + stem + "_aligned.jpg", aligned); } +// ── Process one image, keeping every detection ──────────────────────────────── +// The --all-faces path. Same detect → align → embed chain as process() below, +// but without the highest-confidence reduction: a frame legitimately contains +// several people, and dropping all but one is a gallery-portrait assumption. +// Faces that fail alignment are reported with a null embedding rather than +// silently dropped, so a caller can count what the detector found against what +// survived the ArcFace warp. + +struct MultiFaceResult { + std::string image_path; + std::string error; // set only when the image itself failed + std::vector faces; +}; + +static MultiFaceResult process_all( + const std::string& path, + const std::function(const cv::Mat&)>& detect, + const std::function& embed_one, + int max_side, + const std::string& debug_dir = "") { + MultiFaceResult out; + out.image_path = path; + + cv::Mat img = cv::imread(path); + if (img.empty()) { + out.error = "cannot read image"; + return out; + } + + if (max_side > 0) { + const int big = std::max(img.cols, img.rows); + if (big > max_side) { + const double s = static_cast(max_side) / big; + cv::resize(img, img, {}, s, s, cv::INTER_AREA); + } + } + + std::vector faces = detect(img); + if (faces.empty()) { + cv::Mat enhanced = enhance_for_retry(img); + faces = detect(enhanced); + if (!faces.empty()) + img = enhanced; + } + if (faces.empty()) { + out.error = "no face detected"; + return out; + } + + for (const auto& face : faces) { + FaceResult r; + r.image_path = path; + r.confidence = face.confidence; + r.bbox[0] = face.bbox.x; r.bbox[1] = face.bbox.y; + r.bbox[2] = face.bbox.width; r.bbox[3] = face.bbox.height; + r.landmarks = face.landmarks; + + cv::Mat crop = align_face(img, face.landmarks); + if (crop.empty()) { + r.error = "alignment failed"; + } else { + r.ok = true; + r.embedding = embed_one(crop); + if (!debug_dir.empty()) + save_debug(debug_dir, path, img, face, crop); + } + out.faces.push_back(std::move(r)); + } + return out; +} + // ── Process one image ───────────────────────────────────────────────────────── static FaceResult process(const std::string& path, @@ -177,6 +262,8 @@ int main(int argc, char** argv) { std::string arcface_model = kDefaultArcfaceModel; std::string arcface_engine; std::string debug_dir; + std::string calibration_gallery; + bool all_faces = false; float conf = 0.5f, nms = 0.4f; int max_side = 500; std::vector images; @@ -190,13 +277,16 @@ int main(int argc, char** argv) { else if (std::strcmp(argv[i], "--nms") == 0 && i+1 < argc) { nms = std::stof(argv[++i]); } else if (std::strcmp(argv[i], "--save-debug") == 0 && i+1 < argc) { debug_dir = argv[++i]; } else if (std::strcmp(argv[i], "--max-side") == 0 && i+1 < argc) { max_side = std::stoi(argv[++i]); } + else if (std::strcmp(argv[i], "--calibration")== 0 && i+1 < argc) { calibration_gallery = argv[++i]; } + else if (std::strcmp(argv[i], "--all-faces") == 0) { all_faces = true; } else if (argv[i][0] != '-') { images.push_back(argv[i]); } else { std::cerr << "[warn] unknown flag: " << argv[i] << "\n"; } } if (images.empty()) { std::cerr << "Usage: embed_faces [--detector ] [--arcface ] " - "[--save-debug ] [--max-side ] image1.jpg ...\n"; + "[--save-debug ] [--max-side ] [--all-faces] " + "[--calibration ] image1.jpg ...\n"; return 1; } @@ -217,31 +307,78 @@ int main(int argc, char** argv) { std::function embed_one = [&](const cv::Mat& c) { return embedder->embed_one(c); }; + // One face's fields, shared by both output shapes. + auto face_json = [](const FaceResult& r) { + json f; + f["confidence"] = r.confidence; + f["bbox"] = {r.bbox[0], r.bbox[1], r.bbox[2], r.bbox[3]}; + json lms = json::array(); + for (const auto& pt : r.landmarks) lms.push_back({pt.x, pt.y}); + f["landmarks"] = std::move(lms); + if (r.ok) f["embedding"] = std::vector(r.embedding.begin(), + r.embedding.end()); + else { f["embedding"] = nullptr; f["error"] = r.error; } + return f; + }; + // Process images and build JSON output - json output = json::array(); + json images_out = json::array(); for (const auto& path : images) { std::cerr << "[embed_faces] " << path << "\n"; - FaceResult res = process(path, detect, embed_one, max_side, debug_dir); json entry; - entry["image"] = res.image_path; - if (res.ok) { - entry["embedding"] = std::vector(res.embedding.begin(), - res.embedding.end()); - entry["confidence"] = res.confidence; - entry["bbox"] = {res.bbox[0], res.bbox[1], res.bbox[2], res.bbox[3]}; - json lms = json::array(); - for (const auto& pt : res.landmarks) lms.push_back({pt.x, pt.y}); - entry["landmarks"] = std::move(lms); + entry["image"] = path; + + if (all_faces) { + MultiFaceResult res = process_all(path, detect, embed_one, max_side, debug_dir); + if (!res.error.empty()) { + entry["faces"] = json::array(); + entry["error"] = res.error; + std::cerr << " [skip] " << res.error << "\n"; + } else { + json faces = json::array(); + for (const auto& f : res.faces) faces.push_back(face_json(f)); + entry["faces"] = std::move(faces); + } } else { - entry["embedding"] = nullptr; - entry["error"] = res.error; - std::cerr << " [skip] " << res.error << "\n"; + FaceResult res = process(path, detect, embed_one, max_side, debug_dir); + if (res.ok) { + entry.merge_patch(face_json(res)); + } else { + entry["embedding"] = nullptr; + entry["error"] = res.error; + std::cerr << " [skip] " << res.error << "\n"; + } } - output.push_back(std::move(entry)); + images_out.push_back(std::move(entry)); } - std::cout << output.dump() << "\n"; + // Without --calibration the output stays a bare array, unchanged, so + // existing callers (build_gallery, fetch_missing_actors) are unaffected. + if (calibration_gallery.empty()) { + std::cout << images_out.dump() << "\n"; + return 0; + } + + ActorGallery gallery = load_gallery(calibration_gallery); + if (!gallery.calib_valid) + std::cerr << "[warn] " << calibration_gallery + << " carries no valid calibration; a client cannot convert a " + "similarity to a probability from it (AR-024)\n"; + + json out; + out["calibration"] = { + {"a", gallery.calib_a}, + {"b", gallery.calib_b}, + {"valid", gallery.calib_valid}, + {"form", "P(match) = 1/(1+exp(-(a*similarity + b + log_prior_odds)))"}, + {"note", "Score through this. AR-024: a bare cosine threshold means " + "something different for every model, gallery and face size. " + "log_prior_odds = log(p0/(1-p0)) for a base-rate prior p0; use " + "0 for association (are these two faces one person)."}, + }; + out["images"] = std::move(images_out); + std::cout << out.dump() << "\n"; return 0; } diff --git a/src/face_embedder_engine.hpp b/src/face_embedder_engine.hpp index 842f4d8..28298ff 100644 --- a/src/face_embedder_engine.hpp +++ b/src/face_embedder_engine.hpp @@ -112,6 +112,17 @@ public: return res; } + // ── Stage accessors ────────────────────────────────────────────────────── + // embed_mat() above is the whole detect→align→embed chain, which is the + // right entry point for embedding a gallery image. Studies that need to + // intervene between the stages — swapping the landmark source, degrading a + // crop before it reaches the embedder — drive these instead, so they still + // exercise the shipped detector, alignment and embedder rather than a + // re-implementation of them. + std::vector detect(const cv::Mat& img) { return detector_->detect(img); } + + Embedding embed_crop(const cv::Mat& crop) { return embedder_->embed_one(crop); } + private: std::unique_ptr detector_; std::unique_ptr embedder_; diff --git a/src/nodes/face_detector_node.hpp b/src/nodes/face_detector_node.hpp index 15cfc7f..6689291 100644 --- a/src/nodes/face_detector_node.hpp +++ b/src/nodes/face_detector_node.hpp @@ -44,7 +44,11 @@ struct FaceDetectorFunc { [](const DetectedFace& a, const DetectedFace& b) { return a.bbox.area() > b.bbox.area(); }); - if (static_cast(faces.size()) > max_faces_) + // TRACES: AR-003 | SR-002 + // Largest-first ordering is kept regardless: it is load-bearing for + // deterministic association, since the Hungarian solver tie-breaks on + // index order (see the replay determinism test). + if (max_faces_ > 0 && static_cast(faces.size()) > max_faces_) faces.resize(max_faces_); return {std::move(f), std::move(faces)}; diff --git a/src/nodes/identity_matcher_node.hpp b/src/nodes/identity_matcher_node.hpp index 0f82004..c3d4654 100644 --- a/src/nodes/identity_matcher_node.hpp +++ b/src/nodes/identity_matcher_node.hpp @@ -145,20 +145,33 @@ struct IdentityMatcherFunc { actors.reserve(n_faces); if (n_faces == 0) return {std::move(tf.source), {}}; - if (n_faces > kMaxFaces) - throw std::runtime_error("identity_matcher: n_faces exceeds kMaxFaces"); - std::vector host_query(static_cast(n_faces) * 512); - for (int fi = 0; fi < n_faces; ++fi) { - std::memcpy(host_query.data() + static_cast(fi) * 512, - tf.embeddings[fi].data(), 512 * sizeof(float)); + /// TRACES: AR-003, AR-004 | SR-002 + // kMaxFaces sizes the similarity engine's preallocated buffer, so it + // bounds MEMORY, not how many faces a frame may contain. It used to + // throw above the bound, which made it a hard cap on crowd scenes by + // accident; now the frame is scored in batches of that size. + // + // Faces per frame are unbounded (AR-003) because X-Ray credits scene + // membership to background cast too, and a fixed cap discards exactly + // those — the smallest faces are dropped first. Cost is contained by + // backpressure (AR-004), which slows the producer, rather than by + // silently throwing work away. + std::vector host_query(static_cast(kMaxFaces) * 512); + + for (int base = 0; base < n_faces; base += kMaxFaces) { + const int chunk = std::min(kMaxFaces, n_faces - base); + for (int k = 0; k < chunk; ++k) { + std::memcpy(host_query.data() + static_cast(k) * 512, + tf.embeddings[base + k].data(), 512 * sizeof(float)); } - // S (N_gallery × n_faces) col-major: face fi's gallery sims at sims + fi*n_gallery. - const float* host_sims = sim_engine_->compute(host_query.data(), n_faces); + // S (N_gallery × chunk) col-major: face k's gallery sims at sims + k*n_gallery. + const float* host_sims = sim_engine_->compute(host_query.data(), chunk); - for (int fi = 0; fi < n_faces; ++fi) { - const float* sims = host_sims + static_cast(fi) * n_gallery_; + for (int ci = 0; ci < chunk; ++ci) { + const int fi = base + ci; + const float* sims = host_sims + static_cast(ci) * n_gallery_; std::vector best_sim(gallery_.actors.size(), -std::numeric_limits::max()); @@ -257,6 +270,7 @@ struct IdentityMatcherFunc { actors.push_back(std::move(ia)); } + } // chunk loop return {std::move(tf.source), std::move(actors)}; } diff --git a/src/python_bindings.cpp b/src/python_bindings.cpp index 3356601..327b597 100644 --- a/src/python_bindings.cpp +++ b/src/python_bindings.cpp @@ -3,17 +3,77 @@ // Loads both ONNX sessions once per FaceEmbedder instance, then embeds many // images via repeated embed() calls — avoiding the per-process model-load // cost of the embed_faces CLI when embedding a large gallery. +// +// Beyond whole-image embed(), the individual pipeline stages are exposed — +// detect(), align_face(), embed_crop() — plus the gallery calibration. A study +// that needs to step between stages (a different landmark source, a degraded +// crop) drives the shipped C++ from Python rather than re-implementing +// detection, alignment, the ArcFace warp or the Platt fit in numpy. Those +// re-implementations drift from what ships, and the calibration is the one +// that must not: AR-024 requires every similarity to pass through +// GalleryCalibration::probability, never a bare cosine. #include "face_embedder_engine.hpp" +#include "gallery/gallery_calibration.hpp" +#include "gallery/gallery_store.hpp" #include +#include #include #include #include +#include +#include +#include + namespace nb = nanobind; using namespace nb::literals; +namespace { + +using ImageArray = nb::ndarray, nb::c_contig, nb::device::cpu>; + +// numpy HxWx3 uint8 (BGR, as cv::imread yields) → cv::Mat sharing that buffer. +// The Mat is a view: it must not outlive the caller's array, so every use here +// copies or consumes it before returning. +cv::Mat as_mat(const ImageArray& a) { + if (a.shape(2) != 3) + throw std::invalid_argument("expected an HxWx3 uint8 BGR image"); + return cv::Mat(static_cast(a.shape(0)), static_cast(a.shape(1)), + CV_8UC3, const_cast(a.data())); +} + +// cv::Mat → freshly-allocated numpy array (owns its buffer). +nb::ndarray mat_to_numpy(const cv::Mat& m) { + cv::Mat c = m.isContinuous() ? m : m.clone(); + auto* buf = new uint8_t[c.total() * c.elemSize()]; + std::memcpy(buf, c.data, c.total() * c.elemSize()); + nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast(p); }); + size_t shape[3] = {static_cast(c.rows), static_cast(c.cols), + static_cast(c.channels())}; + return nb::ndarray(buf, 3, shape, owner); +} + +nb::ndarray vec_to_numpy(std::vector&& v) { + auto* buf = new float[v.size()]; + std::memcpy(buf, v.data(), v.size() * sizeof(float)); + nb::capsule owner(buf, [](void* p) noexcept { delete[] static_cast(p); }); + size_t shape[1] = {v.size()}; + return nb::ndarray(buf, 1, shape, owner); +} + +// numpy (5,2) float32 → the landmark array align_face expects. Order is +// types.hpp:60 — [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth. +std::array as_landmarks( + const nb::ndarray, nb::c_contig, nb::device::cpu>& a) { + std::array lm; + for (int i = 0; i < 5; ++i) lm[i] = {a(i, 0), a(i, 1)}; + return lm; +} + +} // namespace + NB_MODULE(sae_embed, m) { m.doc() = "SCRFD + ArcFace face embedding, models loaded once per FaceEmbedder"; @@ -27,6 +87,23 @@ NB_MODULE(sae_embed, m) { }) .def_prop_ro("bbox", [](const FaceEmbedResult& r) { return std::vector{r.bbox[0], r.bbox[1], r.bbox[2], r.bbox[3]}; + }) + .def_prop_ro("landmarks", [](const FaceEmbedResult& r) { + std::vector v; + for (const auto& p : r.landmarks) { v.push_back(p.x); v.push_back(p.y); } + return v; + }); + + nb::class_(m, "Detection") + .def_ro("confidence", &DetectedFace::confidence) + .def_prop_ro("bbox", [](const DetectedFace& d) { + return std::vector{d.bbox.x, d.bbox.y, d.bbox.width, d.bbox.height}; + }) + .def_prop_ro("landmarks", [](const DetectedFace& d) { + // (5,2): [0] right-eye [1] left-eye [2] nose [3] right-mouth [4] left-mouth + std::vector v; + for (const auto& p : d.landmarks) { v.push_back(p.x); v.push_back(p.y); } + return v; }); nb::class_(m, "FaceEmbedder") @@ -38,5 +115,109 @@ NB_MODULE(sae_embed, m) { .def("embed", &FaceEmbedderEngine::embed_path, "path"_a, nb::call_guard(), "Detect the highest-confidence face in the image, align it, and " - "return a FaceResult with its 512-d ArcFace embedding."); + "return a FaceResult with its 512-d ArcFace embedding.") + .def("embed_mat", [](FaceEmbedderEngine& e, ImageArray img) { + return e.embed_mat(as_mat(img).clone()); + }, "image"_a, + "As embed(), on an in-memory HxWx3 uint8 BGR array.") + .def("detect", [](FaceEmbedderEngine& e, ImageArray img) { + return e.detect(as_mat(img)); + }, "image"_a, + "Run the configured detector. Returns every Detection, unfiltered — " + "min_face_px is applied downstream in face_detector_node.") + .def("embed_crop", [](FaceEmbedderEngine& e, ImageArray crop) { + cv::Mat c = as_mat(crop); + if (c.rows != 112 || c.cols != 112) + throw std::invalid_argument("embed_crop expects a 112x112 aligned crop"); + Embedding emb = e.embed_crop(c); + return vec_to_numpy(std::vector(emb.begin(), emb.end())); + }, "crop"_a, + "Embed a caller-supplied 112x112 aligned BGR crop. The stage-level " + "entry point for studies that degrade or re-align a crop themselves."); + + m.def("align_face", [](ImageArray img, + nb::ndarray, nb::c_contig, + nb::device::cpu> landmarks) + -> std::optional> { + cv::Mat crop = ::align_face(as_mat(img), as_landmarks(landmarks)); + if (crop.empty()) return std::nullopt; // degenerate fit + return mat_to_numpy(crop); + }, "image"_a, "landmarks"_a, + "The ArcFace 5-point similarity transform (face_utils.hpp, AR-005). " + "Returns a 112x112 BGR crop, or None if the affine fit is degenerate. " + "Landmark order is types.hpp:60 — right-eye, left-eye, nose, " + "right-mouth, left-mouth."); + + m.def("enhance_for_retry", [](ImageArray img) { + return mat_to_numpy(::enhance_for_retry(as_mat(img))); + }, "image"_a, + "Border-replicate pad by 50% and CLAHE, for a detector second try."); + + // ── Calibration ────────────────────────────────────────────────────────── + // AR-024: the pipeline reasons in one probability space. Exposed so Python + // scores through the same sigmoid the C++ matcher uses, rather than a numpy + // copy of it that can silently disagree. + nb::class_(m, "GalleryCalibration") + .def_ro("a", &GalleryCalibration::a) + .def_ro("b", &GalleryCalibration::b) + .def_ro("valid", &GalleryCalibration::valid) + .def("probability", &GalleryCalibration::probability, + "similarity"_a, "log_prior_odds"_a = 0.f, + "P(match | sim) = sigma(a*sim + b + log_prior_odds). Pass " + "log_prior_odds = log(p0/(1-p0)) for a base-rate prior p0; leave it " + "at 0 for association (is this one person), which is what the " + "balanced fit answers — see gallery_calibration.hpp:63.") + .def("boundary_at", &GalleryCalibration::boundary_at, + "p"_a = 0.5f, "log_prior_odds"_a = 0.f, + "The similarity at which P(match) == p. Diagnostic only — decisions " + "threshold the probability, not this.") + .def("__repr__", [](const GalleryCalibration& c) { + return "" : " INVALID>"); + }); + + m.def("gallery_calibration", [](const std::string& gallery_path) { + ActorGallery g = load_gallery(gallery_path); + if (g.calib_valid) { + std::cerr << "[calibration] " << gallery_path << ": cached fit" + << " over " << g.actors.size() << " actors\n"; + return GalleryCalibration{g.calib_a, g.calib_b, true}; + } + // Legacy JSON galleries carry no stored fit; compute it over the + // whole gallery, which is the point — the calibration must come + // from the production actor population, not a handful of people. + std::cerr << "[calibration] " << gallery_path + << ": no cached fit, computing over " << g.actors.size() + << " actors\n"; + std::vector flat; + std::vector actor; + for (size_t a = 0; a < g.actors.size(); ++a) + for (const auto& e : g.actors[a].embeddings) { + flat.push_back(e); + actor.push_back(static_cast(a)); + } + return ::calibrate_gallery(flat, actor); + }, "gallery_path"_a, + "The production gallery's calibration — the global fit over every " + "actor in it. Use this to score, not a fit over a handful of people: " + "a sigmoid fitted on a few identities saturates, so its probabilities " + "mean nothing. Reads the cached fit stored in an HDF5 gallery, or " + "computes it over the whole gallery for a legacy JSON one."); + + m.def("calibrate_gallery", [](nb::ndarray, nb::c_contig, + nb::device::cpu> emb, + std::vector actor) { + const size_t n = emb.shape(0); + if (actor.size() != n) + throw std::invalid_argument("embeddings and actor ids differ in length"); + std::vector flat(n); + for (size_t i = 0; i < n; ++i) + std::memcpy(flat[i].data(), &emb(i, 0), 512 * sizeof(float)); + return ::calibrate_gallery(flat, actor); + }, "embeddings"_a, "actor_ids"_a, + "Fit the Platt sigmoid from intra/inter-class pairs — the same fit the " + "gallery build performs (gallery_calibration.hpp:85). embeddings is " + "(N,512) L2-normalised float32; actor_ids is a length-N list of " + "0-based actor indices."); }