diff --git a/docs/SPEC.md b/docs/SPEC.md index f890f41..e0ea06d 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -151,6 +151,80 @@ Generate a 512-d embedding per aligned crop. **Current:** `embedder_node.hpp` + `face_embedder_engine.hpp`; default LVFace-B_Glint360K. **Gap:** none. +## AR-028 … AR-030 — Embedding input quality + +An embedder handed a face it cannot represent does not fail. It returns a +confident, plausible, wrong vector, and that vector then competes on equal terms +with every good one in the gallery — the same failure mode AR-011 names for +whole models, occurring here at the level of a single region. Quality assessment +is how that is caught **at inference**, rather than inferred afterwards from a +study of why a film scored badly. + +Three axes, assessed on every face before its embedding is used as identity +evidence. They are kept separate and **not collapsed into one scalar**: they fail +for different reasons, have different remedies, and — as below — do not even earn +the same response. + +- **Size** — already AR-002, floor at 32×32 px in original resolution, measured + by VR-005 (knee at 24–32 px). It is the precedent for the other two: the + threshold was *located*, not chosen. +- **Sharpness** — motion blur and soft focus destroy the high-frequency detail + the embedder keys on, and unlike size they leave the bounding box looking + perfectly healthy. Measured on the **112×112 aligned crop**, not the raw box: + the crop is already scale-normalised, so a measure taken there cannot silently + re-measure face size and double-count it against AR-002. +- **Visibility** — extreme pose or occlusion means the face presents fewer of the + features the embedding assumes are present. Derived from the **5-point + landmarks AR-001 already emits** — nose offset from the eye midpoint over + inter-ocular distance, plus eye/mouth-corner asymmetry — which are already + computed, already used by AR-005, and already in the VR-001 dump, so the + measure costs one arithmetic expression per face and can be studied on existing + fixtures with no GPU. A dedicated landmark model (`models/2d106det.onnx` is + present but referenced nowhere) is **not** adopted unless VR-012 shows the + 5-point proxy insufficient: an extra inference per detection is precisely the + cost AR-011 says not to spend. + +**Failing an axis discounts the observation; it does not delete the detection.** +Only size drops the face outright, and only because VR-005 measured a knee below +which the embedding carries no signal to discount. Blur and pose are different: + +- A blurred or turned face is still evidence of **presence**, which is what + SR-002 actually asks about. +- The tracker admits a link on position *or* identity precisely so that a face + "whose embedding degraded (blur, profile turn)" stays linkable. Remove the + detection and the track fragments, costing the window extent AR-012/AR-013 + exist to protect. +- AR-019 harvests non-frontal views *because* TMDB headshots are frontal. + Discarding turned faces starves the mechanism built to fix the pose problem of + its raw material, and AR-020 then has nothing to resolve at EOF. + +The natural home for the discount is `EvidenceDiscounter` (AR-025), which already +weights how far one observation may move a track's belief. Note that its present +weight is pure *novelty*, so a profile view — maximally distant from everything +counted so far — currently scores near 1.0 and moves the belief hardest, when +against a frontal gallery it deserves the least trust. Novelty and reliability +are orthogonal and multiply; quality supplies the second term. + +**Quality is carried, not consumed.** The vector travels with the face and is +written to the VR-001 dump alongside the embedding, so a threshold can be +re-litigated against recorded data instead of by re-running video, and so +VR-010's provenance records what the run actually admitted. + +**No quality threshold is hand-set.** Each axis either has a measured knee +(VR-012, as VR-005 did for size) or it discounts rather than drops — a +hand-chosen cutoff on an uncalibrated measure is the same unfalsifiable magic +number AR-024 retired for similarity, and it would fail the same way: meaning +something different for every detector, every embedder and every film. + +**Current:** none of the three is assessed. `min_face_px` (40, decoded-frame +space) is the only quality signal in the pipeline; sharpness and visibility are +unmeasured, and `align_face()` silently drops only the degenerate-affine case +without counting it. + +**Gap:** all of AR-028 … AR-030. Order: land the quality vector and its dump +field first (AR-028) so VR-012 can be run from fixtures, then set behaviour per +axis from what it measures. + ## AR-007, AR-008 — Tracking Link detections across frames into tracks representing one physical person. diff --git a/docs/requirements.md b/docs/requirements.md index fcf0060..7a9ffe8 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -37,7 +37,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks | | AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted | | AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done | -| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Blocked on a design decision** — the detector needs ~100 dense frames (~3.3 s) before it can score, so a boundary arrives long after the face branch has passed it. An association hint that late is useless. See SPEC AR-009…AR-011 | +| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Done** — decode butterfly joined via `SceneBoundaries`; the sampled branch waits for the detector's watermark. Frames past its last scored window are counted as unverified, never assumed boundary-free | | AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned | | AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | **Done** — `src/track_registry.hpp`; window is `[first_seen, last_seen]` of an owned track | | AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | **Done** — `last_seen` optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed | @@ -55,6 +55,9 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` | | AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress | | AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned | +| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | Planned | +| AR-029 | Sharpness measure on the **aligned crop** (scale-normalised, so it cannot re-measure size) | SR-002 | Medium | Planned | +| AR-030 | Visibility measure from the AR-001 5-point landmarks — extreme pose or occlusion **discounts the observation, never deletes the detection** | SR-002 | Medium | Planned | ## Deployment (DP) @@ -111,6 +114,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned | | VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned | | VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned | +| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | Planned | --- @@ -302,7 +306,7 @@ because it will be trusted. | ID | Tier | Test asserts | Edge cases to cover | |---|---|---|---| | AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only | -| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement | +| AR-002 | T2 | Faces below 32 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement | | AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame | | AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block | | AR-005 | T1 | Known landmarks → expected 112×112 warp | Landmarks near frame edge; degenerate/collinear points | @@ -327,6 +331,10 @@ because it will be trusted. | AR-025 | T1 | Log-odds accumulate; correlated frames discounted | 30 identical frames must **not** reach the certainty of 30 diverse ones | | AR-026 | T1 + T4 | GEMM path produces same result as reference loop | Equivalence on small input in CI; throughput on GPU host | | AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand | +| AR-028 | **T2** | No embedding reaches the matcher unscored; the vector survives into the dump | Face failing exactly one axis; all three healthy; a face whose landmarks are degenerate — scored, not silently vanished | +| AR-029 | T1 | Synthetic blur ladder → monotonically falling sharpness | Gaussian vs motion blur; **small sharp face vs large soft one** — size must not leak into this axis | +| AR-030 | T1 | Landmark geometry → expected yaw proxy on known poses | Full profile with one eye occluded; **in-plane roll must not read as yaw**; near-degenerate landmarks | +| VR-012 | **T4** | Knee located per axis on held-out films | Report each candidate threshold's cost in **lost true presence**, not only its gain in precision — a gate that improves misID by discarding half the cast has not helped | | IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows | | IR-003 | T1 | Output written after deferred pass | Not at EOF | | IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos | diff --git a/docs/traceability.md b/docs/traceability.md index c38d44c..1922a7f 100644 --- a/docs/traceability.md +++ b/docs/traceability.md @@ -3,7 +3,7 @@ -**Generated:** 2026-07-31T12:22:20+00:00 +**Generated:** 2026-07-31T12:29:10+00:00 Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`). @@ -11,13 +11,13 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev | Metric | Value | |---|---| -| Source files scanned | 102 | +| Source files scanned | 103 | | TRACES tags found | 95 | | EXCEPTION tags found | 0 | -| Requirements defined | 63 | +| Requirements defined | 67 | | Requirements covered | 28 | -| **Coverage** | **44.4%** (28/63) | -| Coverage of CI-executable scope | 53.8% (28/52) | +| **Coverage** | **41.8%** (28/67) | +| Coverage of CI-executable scope | 50.9% (28/55) | | Tagged but unexecuted in CI | 3 | | Orphan tags | 0 | @@ -25,11 +25,11 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev | Type | Covered | Tagged but unexecuted | Defined | |---|---|---|---| -| AR | 15 | 0 | 27 | +| AR | 15 | 0 | 30 | | DP | 2 | 0 | 8 | | IR | 8 | 0 | 8 | | GR | 3 | 0 | 9 | -| VR | 0 | 3 | 11 | +| VR | 0 | 3 | 12 | - **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104 - **IT** tags present (separate taxonomy, not counted in coverage): IT-001 @@ -53,6 +53,7 @@ These requirements have no verification tier this repo's CI host can run, so a t | VR-008 | out-of-ci | no | Gallery scaling benchmark — throughput vs gallery size | | VR-010 | out-of-ci | no | Dump provenance attributes — embedder model, detector settings, `dens… | | VR-011 | out-of-ci | no | Rewrite the replay harness for the post-AR-012 output contract | +| VR-012 | T4, out-of-ci | no | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … | **Tagged but unexecuted:** VR-001, VR-002, VR-003 — a test exists and is tagged, but this CI host cannot run it. Report those runs separately. @@ -79,7 +80,7 @@ _None._ | ID | Status | Tier | Traces to | Trace state | Tagged in | Requirement | |---|---|---|---|---|---|---| | AR-001 | Done | T3 | SR-002 | covered | `src/nodes/face_detector_node.hpp` | Detect faces in sampled frames; emit bbox, confidence, 5-point landma… | -| AR-002 | Planned | unset | SR-002 | untagged | - | Minimum face size **32×32 px** (VR-005 measured), expressed in **orig… | +| AR-002 | Planned | T2 | SR-002 | untagged | - | Minimum face size **32×32 px** (VR-005 measured), expressed in **orig… | | AR-003 | **Done** — `max_fac… | T1, T2, T4 | SR-002 | covered | `src/config.hpp`, `src/nodes/face_detector_node.hpp`, `src/nodes/identity_matcher_node.hpp` | No fixed per-frame face cap — crowd scenes must not lose background c… | | AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_replay_fixtures.cpp` | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… | | AR-005 | Done | T1, T3 | SR-002 | covered | `src/face_utils.hpp` | Align to 112×112 via ArcFace 5-point similarity transform | @@ -87,7 +88,7 @@ _None._ | AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… | | AR-008 | **Done** — one pool… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | One track pool keyed on `last_seen`; no separate revival path | | AR-009 | Done | T2 | SR-002 | untagged | - | Camera-cut detection (histogram) as an association hint | -| AR-010 | **Not started** — `… | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint | +| AR-010 | **Blocked on a desi… | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint | | AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… | | AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition | | AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… | @@ -105,6 +106,9 @@ _None._ | AR-025 | **Done** — log-odds… | T1 | SR-002 | covered | `src/evidence_discount.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-track Bayesian accumulation in log-odds, with correlated-observat… | | AR-026 | In Progress | T1, T4 | SR-001 | untagged | - | All similarity computed as GEMM, including annex and deferred pass | | AR-027 | Planned | T4 | SR-001 | untagged | - | Throughput acceptable for **arbitrary** gallery size | +| AR-028 | Planned | T2 | SR-002 | untagged | - | **Embedding input quality assessed and carried** — every face scored … | +| AR-029 | Planned | T1 | SR-002 | untagged | - | Sharpness measure on the **aligned crop** (scale-normalised, so it ca… | +| AR-030 | Planned | T1 | SR-002 | untagged | - | Visibility measure from the AR-001 5-point landmarks — extreme pose o… | | DP-001 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | One analysis core; modes are front-ends and must not fork pipeline lo… | | DP-002 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | Batch CLI over one title | | DP-003 | Planned | T1, manual | PR-004 | untagged | - | On-demand resident service with bounded, observable queue | @@ -141,6 +145,7 @@ _None._ | VR-009 | Planned | T1, out-of-ci | PR-002 | untagged | - | Verify accumulated posteriors are calibrated against held-out tracks | | VR-010 | Planned | out-of-ci | PR-002 | untagged | - | Dump provenance attributes — embedder model, detector settings, `dens… | | VR-011 | Planned | out-of-ci | PR-002 | untagged | - | Rewrite the replay harness for the post-AR-012 output contract | +| VR-012 | Planned | T4, out-of-ci | PR-002 | untagged | - | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … | ## Detailed mapping diff --git a/src/main.cpp b/src/main.cpp index b117199..603c4fd 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -60,6 +60,8 @@ #include "nodes/identity_matcher_node.hpp" #include "nodes/scene_tracker_node.hpp" #include "nodes/scene_detector_node.hpp" +#include "scene_boundaries.hpp" +#include "nodes/scene_boundary_annotator_node.hpp" #include "nodes/result_sink_node.hpp" #include "nodes/embedding_dump_node.hpp" #ifdef SAE_DEBUG @@ -81,6 +83,18 @@ // ── CLI parsing ─────────────────────────────────────────────────────────────── +/// TRACES: AR-010, AR-004 | SR-002 +/// How deeply the sampled branch is buffered behind the dense one. TransNetV2 +/// needs kWindow (100) dense frames before it can score any of them, so the face +/// branch must lag by at least that much or it asks about frames nobody has +/// looked at yet. Backpressure turns depth into lag: the fanout blocks on the +/// slower branch rather than dropping, so the detector simply runs ahead. +static constexpr std::size_t kSceneJoinDepth = 256; + +/// Set when the scene branch is built, so shutdown can report whether the join +/// actually worked. +static std::shared_ptr scene_stats; + static Config parse_args(int argc, char** argv) { Config cfg; cfg.detector_model = kDefaultDetectorModel; @@ -292,6 +306,21 @@ int main(int argc, char** argv) { // // Reporting it in a footer and exiting 0 made both invisible: the run // "succeeded" and the truth file looked complete. Fail instead. + /// TRACES: AR-010 | SR-002 + if (scene_stats) { + std::cerr << "[scene_annotate] boundaries=" << scene_stats->count() + << " scored_through=" << scene_stats->scored_through() << "s"; + // The tail is expected: frames after the detector's last full + // window are never covered, and no amount of buffering changes + // that. They are counted rather than silently treated as + // boundary-free, which is the distinction that matters. + if (scene_stats->outran() > 0) + std::cerr << " unscored=" << scene_stats->outran() + << " frame(s) past the detector's last window — treated as" + " boundary-free, which is unverified rather than known"; + std::cerr << "\n"; + } + bool dropped = false; { std::lock_guard lk(event_mtx); @@ -346,6 +375,21 @@ int main(int argc, char** argv) { if (cfg.scene_detect) { scene_done.store(false, std::memory_order_release); // now a real terminal branch SceneDetectorFunc scene_fn{cfg, scene_done}; + + /// TRACES: AR-010 | SR-002 + // The join of the decode butterfly. source fans out to the dense + // TransNetV2 branch and the sampled face branch; boundaries found on the + // first have to reach the second, and cannot ride the frames because the + // branches run in parallel. + // + // TransNetV2 buffers kWindow frames before it can score any of them, so + // the face branch must lag by at least that much or it will ask about + // frames nobody has looked at yet. Channel depth is what creates the lag: + // with backpressure (AR-004) the fanout blocks on the slower branch, so + // a deep face-branch channel lets the detector run ahead by its window + // rather than dropping anything. + auto boundaries = std::make_shared(); + scene_fn.set_boundaries(boundaries); kpn::ObjectNode, kpn::out<>, "scene_detector", 0> scene_node(scene_fn, 128); @@ -362,13 +406,34 @@ int main(int argc, char** argv) { return true; } return false; - }, 32); + }, kSceneJoinDepth); + + /// TRACES: AR-010 | SR-002 + // Stamp is_scene_boundary from the detector's published verdict. tol is + // half a sample interval: the two branches sample at different rates, so + // a boundary found on a dense frame rarely lands exactly on a sampled + // one, and half an interval attributes it to the nearest sampled frame + // and no further. + // + // outran() counts frames that arrived before the detector had scored + // them. Nonzero means the join depth is too shallow for the window, and + // those frames were annotated from an incomplete verdict — which would + // otherwise look exactly like "no boundary here". + SceneBoundaryAnnotatorFunc annotate_fn{boundaries, 0.5 / cfg.sample_fps}; + kpn::ObjectNode, kpn::out<"frame">, + "scene_annotate", 0> annotate(annotate_fn, kSceneJoinDepth); + + // Reported at shutdown: without this the join is unverifiable, and an + // annotator that never fired looks identical to footage with no + // boundaries. + scene_stats = boundaries; auto net = kpn::make_network( kpn::edge(source.output<"raw">(), campos.input<"raw">()), kpn::edge(source.output<"raw">(), scene_node.input<"dense">()), kpn::edge(campos.output<"frame">(), decimate.input<0>()), - kpn::edge(decimate.output<0>(), detector.input<"frame">()), + kpn::edge(decimate.output<0>(), annotate.input<"frame">()), + kpn::edge(annotate.output<"frame">(), detector.input<"frame">()), kpn::edge(detector.output<"scene">(), aligner.input<"scene">()), kpn::edge(aligner.output<"aligned">(), embedder.input<"aligned">()), kpn::edge(embedder.output<"embedded">(), ftracker.input<"embedded">()), diff --git a/src/nodes/scene_boundary_annotator_node.hpp b/src/nodes/scene_boundary_annotator_node.hpp new file mode 100644 index 0000000..664aa05 --- /dev/null +++ b/src/nodes/scene_boundary_annotator_node.hpp @@ -0,0 +1,67 @@ +#pragma once +/// TRACES: AR-010 | SR-002 +/// +/// SceneBoundaryAnnotatorFunc — the join of the decode butterfly. +/// +/// `source` fans out to two branches: dense frames to TransNetV2, sampled frames +/// to face detection. A boundary found on the first has to reach the second, and +/// cannot ride along in the frame because the branches run in parallel. +/// +/// This node sits on the sampled branch and stamps `Frame::is_scene_boundary` +/// from the detector's published verdict. +/// +/// **It only works because the sampled branch lags.** TransNetV2 buffers +/// `kWindow` frames before it can score any of them, so this node must not reach +/// a frame before the detector has an opinion about it. Channel depth creates +/// that lag: with backpressure (AR-004) the fanout blocks on the slower branch, +/// so a deep channel here lets the detector run ahead by its window instead of +/// anything being dropped. +/// +/// When the lag is insufficient the node **counts it** rather than guessing. +/// Annotating an unscored frame as boundary-free is indistinguishable from a +/// genuine "no boundary here", and that is the failure that makes a downstream +/// test pass while verifying nothing. + +#include "scene_boundaries.hpp" +#include "types.hpp" + +#include +#include +#include + +struct SceneBoundaryAnnotatorFunc { + static constexpr std::string_view label() { return "scene_annotate"; } + + /// `tol` is half a sample interval. The branches sample at different rates, + /// so a boundary found on a dense frame rarely lands exactly on a sampled + /// one; half an interval attributes it to the nearest sampled frame and no + /// further. + SceneBoundaryAnnotatorFunc(std::shared_ptr b, double tol) + : bounds_(std::move(b)), tol_(tol) {} + + Frame operator()(Frame f) { + if (f.eof || !bounds_) return f; + + // Wait for the detector's verdict to cover this frame. Channel depth + // alone cannot provide the lag: it holds frames back only when the + // consumer is slower, and this branch is orders of magnitude faster per + // frame than TransNetV2. Blocking here is what makes the join real. + // + // Safe under backpressure because the branches are independent: this + // node stalling does not stop the detector consuming dense frames, and + // the fanout keeps feeding it. + if (!bounds_->wait_until_scored(f.timestamp_sec)) { + // The detector finished without covering this frame — the tail after + // its last full window. Unknown, not negative; counted so it cannot + // pass for "no boundary here". + bounds_->note_outran(); + return f; + } + f.is_scene_boundary = bounds_->is_boundary(f.timestamp_sec, tol_); + return f; + } + +private: + std::shared_ptr bounds_; + double tol_{0.0}; +}; diff --git a/src/nodes/scene_detector_node.hpp b/src/nodes/scene_detector_node.hpp index 8ec954a..d694e67 100644 --- a/src/nodes/scene_detector_node.hpp +++ b/src/nodes/scene_detector_node.hpp @@ -1,6 +1,9 @@ #pragma once #include "types.hpp" #include "config.hpp" +#include "scene_boundaries.hpp" + +#include #include "inference/scene_detector.hpp" #include @@ -30,6 +33,11 @@ struct SceneDetectorFunc { static constexpr std::string_view label() { return "scene_detector"; } + /// TRACES: AR-010 | SR-002 + /// Publish each window's verdict as it is scored, so the face branch — held + /// back by channel depth — can consult it for frames it has not reached yet. + void set_boundaries(std::shared_ptr b) { shared_ = std::move(b); } + SceneDetectorFunc(const Config& cfg, std::atomic& done) : detector_(make_scene_detector(cfg)) , threshold_(cfg.scene_threshold) @@ -51,6 +59,9 @@ struct SceneDetectorFunc { void operator()(Frame f) { if (f.eof) { flush_remaining(); + // Release anyone waiting on the join: the tail frames after the last + // full window will never be covered, so waiting for them would hang. + if (shared_) shared_->finish(); write_output(); done_.store(true, std::memory_order_release); return; @@ -82,6 +93,7 @@ private: // otherwise skip the leading guard already covered by the previous window. const int lo = (window_base_ == 0) ? 0 : guard_; const int hi = ISceneDetector::kWindow - guard_; + std::vector fresh; for (int i = lo; i < hi; ++i) { if (probs[i] <= threshold_) continue; // Local maximum → the boundary frame (avoid a run of high scores @@ -89,9 +101,19 @@ private: const bool peak = (i == 0 || probs[i] >= probs[i-1]) && (i == kLast_() || probs[i] >= probs[i+1]); - if (peak) + if (peak) { boundaries_.push_back({times_[i], probs[i]}); + fresh.push_back(times_[i]); + } } + + /// TRACES: AR-010 | SR-002 + // Publish with a watermark: everything up to times_[hi-1] now has a + // final verdict. The face branch consults this for frames it has not + // reached yet, and the watermark is what lets it tell "no boundary + // here" from "not scored yet". + if (shared_ && hi > lo) + shared_->publish(fresh, times_[hi - 1]); } // At EOF the tail (< kWindow frames) never formed a full window. Pad it out @@ -107,14 +129,24 @@ private: std::vector probs = detector_->detect_window(win); const int lo = (window_base_ == 0) ? 0 : guard_; + std::vector fresh; for (int i = lo; i < n; ++i) { // only real (non-padded) frames if (probs[i] <= threshold_) continue; const bool peak = (i == 0 || probs[i] >= probs[i-1]) && (i == n - 1 || probs[i] >= probs[i+1]); - if (peak) + if (peak) { boundaries_.push_back({times_[i], probs[i]}); + fresh.push_back(times_[i]); + } } + + /// TRACES: AR-010 | SR-002 + // Publish the tail too. Without this the final frames — everything after + // the last full window — reach the join with no verdict and are treated + // as boundary-free without evidence, which is precisely the ambiguity + // the watermark exists to prevent. + if (shared_ && n > 0) shared_->publish(fresh, times_[n - 1]); } void write_output() { @@ -176,4 +208,5 @@ private: int64_t window_base_{0}; // frame index of images_.front() std::vector boundaries_; bool written_{false}; + std::shared_ptr shared_; ///< AR-010 join point }; diff --git a/src/scene_boundaries.hpp b/src/scene_boundaries.hpp new file mode 100644 index 0000000..c2f8002 --- /dev/null +++ b/src/scene_boundaries.hpp @@ -0,0 +1,126 @@ +#pragma once +/// TRACES: AR-010 | SR-002 +/// +/// SceneBoundaries — the join point of the decode butterfly. +/// +/// The topology forks after decode: one branch runs TransNetV2 over dense +/// frames, the other runs face detection over the sampled cadence. Boundaries +/// found on the first branch have to reach the second, and they cannot be +/// carried in the frames themselves because the branches are parallel. +/// +/// **Why this needs a watermark.** TransNetV2 buffers `kWindow` frames before it +/// can score any of them, so at any instant the detector has an opinion about +/// everything up to some time T and nothing after it. Without recording T, a +/// consumer asking "is there a boundary at t?" cannot distinguish *no* from +/// *not yet* — and those demand opposite behaviour. Silently treating unscored +/// frames as boundary-free is exactly the class of failure that makes a +/// verification pass vacuously. +/// +/// The consumer is held back by channel depth (see main.cpp) so that by the time +/// it pulls a frame, the detector has already scored past it. `scored_through()` +/// is what lets that assumption be *checked* rather than assumed. + +#include +#include +#include +#include + +class SceneBoundaries { +public: + /// Peaks closer than this are one boundary. Matches the dedup scenes.json + /// applies, so the two views agree. + static constexpr double kMergeSec = 0.04; + + /// Called by the scene detector as each window is scored. `through` is the + /// timestamp up to which its verdict is now final. + void publish(const std::vector& ts, double through) { + { + std::lock_guard g(mu_); + // Dedup on insert, matching what scenes.json does at write time. A run + // of adjacent high-scoring frames is one boundary, not several, and + // leaving them raw made this view report 357 where the file said 13 — + // the same event counted many times. Harmless for is_boundary(), which + // absorbs them in its tolerance, but a count nobody can reconcile with + // the output file is a bad diagnostic. + bounds_.insert(bounds_.end(), ts.begin(), ts.end()); + std::sort(bounds_.begin(), bounds_.end()); + bounds_.erase(std::unique(bounds_.begin(), bounds_.end(), + [](double a, double b) { return b - a < kMergeSec; }), + bounds_.end()); + scored_through_ = std::max(scored_through_, through); + } + cv_.notify_all(); + } + + /// True if a boundary falls within `tol` of `t`. + /// + /// `tol` exists because the two branches sample at different rates: a + /// boundary found on a dense frame rarely lands exactly on a sampled one. + /// Half a sample interval is the natural width — it attributes the boundary + /// to the nearest sampled frame and no further. + bool is_boundary(double t, double tol) const { + std::lock_guard g(mu_); + auto it = std::lower_bound(bounds_.begin(), bounds_.end(), t - tol); + return it != bounds_.end() && *it <= t + tol; + } + + /// The timestamp through which the detector's verdict is final. A consumer + /// past this point is asking about frames nobody has looked at yet. + double scored_through() const { + std::lock_guard g(mu_); + return scored_through_; + } + + /// Block until the detector's verdict covers `t`, or it finishes. + /// + /// Channel depth alone does NOT create the required lag: it only holds + /// frames back when the consumer is slower, and the face branch is roughly + /// four orders of magnitude faster per frame than TransNetV2. So the join + /// has to wait explicitly. + /// + /// Returns false if the detector finished without ever covering `t`, which + /// happens for the tail frames after its last full window. The caller must + /// distinguish that from a genuine "no boundary" rather than assuming. + bool wait_until_scored(double t) const { + std::unique_lock lk(mu_); + cv_.wait(lk, [&] { return finished_ || scored_through_ >= t; }); + return scored_through_ >= t; + } + + /// Called when the detector will publish nothing further. Without this the + /// join would deadlock on the tail: those frames are never covered by a full + /// window, so waiting for them would wait forever. + void finish() { + { + std::lock_guard g(mu_); + finished_ = true; + } + cv_.notify_all(); + } + + std::size_t count() const { + std::lock_guard g(mu_); + return bounds_.size(); + } + + /// Consumers that outran the detector. Nonzero means the face branch is not + /// buffered deeply enough for the detector's window, so some frames were + /// annotated from an incomplete verdict — a real misconfiguration, and one + /// that would otherwise be invisible. + void note_outran() const { + std::lock_guard g(mu_); + ++outran_; + } + std::size_t outran() const { + std::lock_guard g(mu_); + return outran_; + } + +private: + mutable std::mutex mu_; + mutable std::condition_variable cv_; + bool finished_{false}; + std::vector bounds_; + double scored_through_{-1.0}; + mutable std::size_t outran_{0}; +};