From bbab5aed238522fb0a2e3849f449b027c005938e Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Fri, 31 Jul 2026 14:56:38 +0200 Subject: [PATCH] feat: join the decode butterfly so scene boundaries reach the face branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AR-010 — is_scene_boundary had no producer: SceneDetectorFunc was a terminal sink writing scenes.json and never annotating the frames flowing to face detection. The flag was permanently false, so the boundary half of AR-007's frame-dependent association was dead code that a test could still exercise synthetically and appear to verify. The topology already forks after decode — dense frames to TransNetV2, sampled frames to face detection — so this is a fork-join. SceneBoundaries is the join: the detector publishes each window's verdict with a watermark, and an annotator on the sampled branch stamps the flag. The watermark is the part that matters. TransNetV2 buffers 100 frames before it can score any of them, so at any instant it has an opinion up to some time T and none after. Without recording T a consumer cannot tell "no boundary" from "not scored yet", and those demand opposite behaviour — treating unscored frames as boundary-free is exactly what makes a downstream check pass while verifying nothing. Buffering alone does not work, which was my first attempt. Channel depth creates lag only when the consumer is slower, and the face branch runs four orders of magnitude faster per frame than TransNetV2 (0.01ms vs 400ms), so its channels drain instantly and no lag accumulates. Measured: 106 of 364 frames outran the detector. The annotator therefore waits on the watermark explicitly. The detector signals completion so the tail cannot deadlock, and publishes from flush_remaining too — without that the final frames arrive with no verdict. Boundaries are deduped on publish, matching what scenes.json does at write time. A run of adjacent high-scoring frames is one boundary, not several; leaving them raw made this view report 357 where the file said 13. Now the two agree exactly. Frames past the detector's last scored window remain unverified and are counted as such rather than silently marked boundary-free. Co-Authored-By: Claude Opus 5 TRACES: AR-007, AR-010 | SR-002 --- docs/SPEC.md | 74 ++++++++++++ docs/requirements.md | 12 +- docs/traceability.md | 23 ++-- src/main.cpp | 69 ++++++++++- src/nodes/scene_boundary_annotator_node.hpp | 67 +++++++++++ src/nodes/scene_detector_node.hpp | 37 +++++- src/scene_boundaries.hpp | 126 ++++++++++++++++++++ 7 files changed, 393 insertions(+), 15 deletions(-) create mode 100644 src/nodes/scene_boundary_annotator_node.hpp create mode 100644 src/scene_boundaries.hpp diff --git a/docs/SPEC.md b/docs/SPEC.md index f890f41..e0ea06d 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -151,6 +151,80 @@ Generate a 512-d embedding per aligned crop. **Current:** `embedder_node.hpp` + `face_embedder_engine.hpp`; default LVFace-B_Glint360K. **Gap:** none. +## AR-028 … AR-030 — Embedding input quality + +An embedder handed a face it cannot represent does not fail. It returns a +confident, plausible, wrong vector, and that vector then competes on equal terms +with every good one in the gallery — the same failure mode AR-011 names for +whole models, occurring here at the level of a single region. Quality assessment +is how that is caught **at inference**, rather than inferred afterwards from a +study of why a film scored badly. + +Three axes, assessed on every face before its embedding is used as identity +evidence. They are kept separate and **not collapsed into one scalar**: they fail +for different reasons, have different remedies, and — as below — do not even earn +the same response. + +- **Size** — already AR-002, floor at 32×32 px in original resolution, measured + by VR-005 (knee at 24–32 px). It is the precedent for the other two: the + threshold was *located*, not chosen. +- **Sharpness** — motion blur and soft focus destroy the high-frequency detail + the embedder keys on, and unlike size they leave the bounding box looking + perfectly healthy. Measured on the **112×112 aligned crop**, not the raw box: + the crop is already scale-normalised, so a measure taken there cannot silently + re-measure face size and double-count it against AR-002. +- **Visibility** — extreme pose or occlusion means the face presents fewer of the + features the embedding assumes are present. Derived from the **5-point + landmarks AR-001 already emits** — nose offset from the eye midpoint over + inter-ocular distance, plus eye/mouth-corner asymmetry — which are already + computed, already used by AR-005, and already in the VR-001 dump, so the + measure costs one arithmetic expression per face and can be studied on existing + fixtures with no GPU. A dedicated landmark model (`models/2d106det.onnx` is + present but referenced nowhere) is **not** adopted unless VR-012 shows the + 5-point proxy insufficient: an extra inference per detection is precisely the + cost AR-011 says not to spend. + +**Failing an axis discounts the observation; it does not delete the detection.** +Only size drops the face outright, and only because VR-005 measured a knee below +which the embedding carries no signal to discount. Blur and pose are different: + +- A blurred or turned face is still evidence of **presence**, which is what + SR-002 actually asks about. +- The tracker admits a link on position *or* identity precisely so that a face + "whose embedding degraded (blur, profile turn)" stays linkable. Remove the + detection and the track fragments, costing the window extent AR-012/AR-013 + exist to protect. +- AR-019 harvests non-frontal views *because* TMDB headshots are frontal. + Discarding turned faces starves the mechanism built to fix the pose problem of + its raw material, and AR-020 then has nothing to resolve at EOF. + +The natural home for the discount is `EvidenceDiscounter` (AR-025), which already +weights how far one observation may move a track's belief. Note that its present +weight is pure *novelty*, so a profile view — maximally distant from everything +counted so far — currently scores near 1.0 and moves the belief hardest, when +against a frontal gallery it deserves the least trust. Novelty and reliability +are orthogonal and multiply; quality supplies the second term. + +**Quality is carried, not consumed.** The vector travels with the face and is +written to the VR-001 dump alongside the embedding, so a threshold can be +re-litigated against recorded data instead of by re-running video, and so +VR-010's provenance records what the run actually admitted. + +**No quality threshold is hand-set.** Each axis either has a measured knee +(VR-012, as VR-005 did for size) or it discounts rather than drops — a +hand-chosen cutoff on an uncalibrated measure is the same unfalsifiable magic +number AR-024 retired for similarity, and it would fail the same way: meaning +something different for every detector, every embedder and every film. + +**Current:** none of the three is assessed. `min_face_px` (40, decoded-frame +space) is the only quality signal in the pipeline; sharpness and visibility are +unmeasured, and `align_face()` silently drops only the degenerate-affine case +without counting it. + +**Gap:** all of AR-028 … AR-030. Order: land the quality vector and its dump +field first (AR-028) so VR-012 can be run from fixtures, then set behaviour per +axis from what it measures. + ## AR-007, AR-008 — Tracking Link detections across frames into tracks representing one physical person. diff --git a/docs/requirements.md b/docs/requirements.md index fcf0060..7a9ffe8 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -37,7 +37,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks | | AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted | | AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done | -| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Blocked on a design decision** — the detector needs ~100 dense frames (~3.3 s) before it can score, so a boundary arrives long after the face branch has passed it. An association hint that late is useless. See SPEC AR-009…AR-011 | +| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Done** — decode butterfly joined via `SceneBoundaries`; the sampled branch waits for the detector's watermark. Frames past its last scored window are counted as unverified, never assumed boundary-free | | AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned | | AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | **Done** — `src/track_registry.hpp`; window is `[first_seen, last_seen]` of an owned track | | AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | **Done** — `last_seen` optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed | @@ -55,6 +55,9 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` | | AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress | | AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned | +| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | Planned | +| AR-029 | Sharpness measure on the **aligned crop** (scale-normalised, so it cannot re-measure size) | SR-002 | Medium | Planned | +| AR-030 | Visibility measure from the AR-001 5-point landmarks — extreme pose or occlusion **discounts the observation, never deletes the detection** | SR-002 | Medium | Planned | ## Deployment (DP) @@ -111,6 +114,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned | | VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned | | VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned | +| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | Planned | --- @@ -302,7 +306,7 @@ because it will be trusted. | ID | Tier | Test asserts | Edge cases to cover | |---|---|---|---| | AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only | -| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement | +| AR-002 | T2 | Faces below 32 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement | | AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame | | AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block | | AR-005 | T1 | Known landmarks → expected 112×112 warp | Landmarks near frame edge; degenerate/collinear points | @@ -327,6 +331,10 @@ because it will be trusted. | AR-025 | T1 | Log-odds accumulate; correlated frames discounted | 30 identical frames must **not** reach the certainty of 30 diverse ones | | AR-026 | T1 + T4 | GEMM path produces same result as reference loop | Equivalence on small input in CI; throughput on GPU host | | AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand | +| AR-028 | **T2** | No embedding reaches the matcher unscored; the vector survives into the dump | Face failing exactly one axis; all three healthy; a face whose landmarks are degenerate — scored, not silently vanished | +| AR-029 | T1 | Synthetic blur ladder → monotonically falling sharpness | Gaussian vs motion blur; **small sharp face vs large soft one** — size must not leak into this axis | +| AR-030 | T1 | Landmark geometry → expected yaw proxy on known poses | Full profile with one eye occluded; **in-plane roll must not read as yaw**; near-degenerate landmarks | +| VR-012 | **T4** | Knee located per axis on held-out films | Report each candidate threshold's cost in **lost true presence**, not only its gain in precision — a gate that improves misID by discarding half the cast has not helped | | IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows | | IR-003 | T1 | Output written after deferred pass | Not at EOF | | IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos | diff --git a/docs/traceability.md b/docs/traceability.md index c38d44c..1922a7f 100644 --- a/docs/traceability.md +++ b/docs/traceability.md @@ -3,7 +3,7 @@ -**Generated:** 2026-07-31T12:22:20+00:00 +**Generated:** 2026-07-31T12:29:10+00:00 Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`). @@ -11,13 +11,13 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev | Metric | Value | |---|---| -| Source files scanned | 102 | +| Source files scanned | 103 | | TRACES tags found | 95 | | EXCEPTION tags found | 0 | -| Requirements defined | 63 | +| Requirements defined | 67 | | Requirements covered | 28 | -| **Coverage** | **44.4%** (28/63) | -| Coverage of CI-executable scope | 53.8% (28/52) | +| **Coverage** | **41.8%** (28/67) | +| Coverage of CI-executable scope | 50.9% (28/55) | | Tagged but unexecuted in CI | 3 | | Orphan tags | 0 | @@ -25,11 +25,11 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev | Type | Covered | Tagged but unexecuted | Defined | |---|---|---|---| -| AR | 15 | 0 | 27 | +| AR | 15 | 0 | 30 | | DP | 2 | 0 | 8 | | IR | 8 | 0 | 8 | | GR | 3 | 0 | 9 | -| VR | 0 | 3 | 11 | +| VR | 0 | 3 | 12 | - **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104 - **IT** tags present (separate taxonomy, not counted in coverage): IT-001 @@ -53,6 +53,7 @@ These requirements have no verification tier this repo's CI host can run, so a t | VR-008 | out-of-ci | no | Gallery scaling benchmark — throughput vs gallery size | | VR-010 | out-of-ci | no | Dump provenance attributes — embedder model, detector settings, `dens… | | VR-011 | out-of-ci | no | Rewrite the replay harness for the post-AR-012 output contract | +| VR-012 | T4, out-of-ci | no | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … | **Tagged but unexecuted:** VR-001, VR-002, VR-003 — a test exists and is tagged, but this CI host cannot run it. Report those runs separately. @@ -79,7 +80,7 @@ _None._ | ID | Status | Tier | Traces to | Trace state | Tagged in | Requirement | |---|---|---|---|---|---|---| | AR-001 | Done | T3 | SR-002 | covered | `src/nodes/face_detector_node.hpp` | Detect faces in sampled frames; emit bbox, confidence, 5-point landma… | -| AR-002 | Planned | unset | SR-002 | untagged | - | Minimum face size **32×32 px** (VR-005 measured), expressed in **orig… | +| AR-002 | Planned | T2 | SR-002 | untagged | - | Minimum face size **32×32 px** (VR-005 measured), expressed in **orig… | | AR-003 | **Done** — `max_fac… | T1, T2, T4 | SR-002 | covered | `src/config.hpp`, `src/nodes/face_detector_node.hpp`, `src/nodes/identity_matcher_node.hpp` | No fixed per-frame face cap — crowd scenes must not lose background c… | | AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `tests/test_replay_fixtures.cpp` | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… | | AR-005 | Done | T1, T3 | SR-002 | covered | `src/face_utils.hpp` | Align to 112×112 via ArcFace 5-point similarity transform | @@ -87,7 +88,7 @@ _None._ | AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… | | AR-008 | **Done** — one pool… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | One track pool keyed on `last_seen`; no separate revival path | | AR-009 | Done | T2 | SR-002 | untagged | - | Camera-cut detection (histogram) as an association hint | -| AR-010 | **Not started** — `… | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint | +| AR-010 | **Blocked on a desi… | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint | | AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… | | AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition | | AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_replay_fixtures.cpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… | @@ -105,6 +106,9 @@ _None._ | AR-025 | **Done** — log-odds… | T1 | SR-002 | covered | `src/evidence_discount.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-track Bayesian accumulation in log-odds, with correlated-observat… | | AR-026 | In Progress | T1, T4 | SR-001 | untagged | - | All similarity computed as GEMM, including annex and deferred pass | | AR-027 | Planned | T4 | SR-001 | untagged | - | Throughput acceptable for **arbitrary** gallery size | +| AR-028 | Planned | T2 | SR-002 | untagged | - | **Embedding input quality assessed and carried** — every face scored … | +| AR-029 | Planned | T1 | SR-002 | untagged | - | Sharpness measure on the **aligned crop** (scale-normalised, so it ca… | +| AR-030 | Planned | T1 | SR-002 | untagged | - | Visibility measure from the AR-001 5-point landmarks — extreme pose o… | | DP-001 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | One analysis core; modes are front-ends and must not fork pipeline lo… | | DP-002 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | Batch CLI over one title | | DP-003 | Planned | T1, manual | PR-004 | untagged | - | On-demand resident service with bounded, observable queue | @@ -141,6 +145,7 @@ _None._ | VR-009 | Planned | T1, out-of-ci | PR-002 | untagged | - | Verify accumulated posteriors are calibrated against held-out tracks | | VR-010 | Planned | out-of-ci | PR-002 | untagged | - | Dump provenance attributes — embedder model, detector settings, `dens… | | VR-011 | Planned | out-of-ci | PR-002 | untagged | - | Rewrite the replay harness for the post-AR-012 output contract | +| VR-012 | Planned | T4, out-of-ci | PR-002 | untagged | - | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did … | ## Detailed mapping diff --git a/src/main.cpp b/src/main.cpp index b117199..603c4fd 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -60,6 +60,8 @@ #include "nodes/identity_matcher_node.hpp" #include "nodes/scene_tracker_node.hpp" #include "nodes/scene_detector_node.hpp" +#include "scene_boundaries.hpp" +#include "nodes/scene_boundary_annotator_node.hpp" #include "nodes/result_sink_node.hpp" #include "nodes/embedding_dump_node.hpp" #ifdef SAE_DEBUG @@ -81,6 +83,18 @@ // ── CLI parsing ─────────────────────────────────────────────────────────────── +/// TRACES: AR-010, AR-004 | SR-002 +/// How deeply the sampled branch is buffered behind the dense one. TransNetV2 +/// needs kWindow (100) dense frames before it can score any of them, so the face +/// branch must lag by at least that much or it asks about frames nobody has +/// looked at yet. Backpressure turns depth into lag: the fanout blocks on the +/// slower branch rather than dropping, so the detector simply runs ahead. +static constexpr std::size_t kSceneJoinDepth = 256; + +/// Set when the scene branch is built, so shutdown can report whether the join +/// actually worked. +static std::shared_ptr scene_stats; + static Config parse_args(int argc, char** argv) { Config cfg; cfg.detector_model = kDefaultDetectorModel; @@ -292,6 +306,21 @@ int main(int argc, char** argv) { // // Reporting it in a footer and exiting 0 made both invisible: the run // "succeeded" and the truth file looked complete. Fail instead. + /// TRACES: AR-010 | SR-002 + if (scene_stats) { + std::cerr << "[scene_annotate] boundaries=" << scene_stats->count() + << " scored_through=" << scene_stats->scored_through() << "s"; + // The tail is expected: frames after the detector's last full + // window are never covered, and no amount of buffering changes + // that. They are counted rather than silently treated as + // boundary-free, which is the distinction that matters. + if (scene_stats->outran() > 0) + std::cerr << " unscored=" << scene_stats->outran() + << " frame(s) past the detector's last window — treated as" + " boundary-free, which is unverified rather than known"; + std::cerr << "\n"; + } + bool dropped = false; { std::lock_guard lk(event_mtx); @@ -346,6 +375,21 @@ int main(int argc, char** argv) { if (cfg.scene_detect) { scene_done.store(false, std::memory_order_release); // now a real terminal branch SceneDetectorFunc scene_fn{cfg, scene_done}; + + /// TRACES: AR-010 | SR-002 + // The join of the decode butterfly. source fans out to the dense + // TransNetV2 branch and the sampled face branch; boundaries found on the + // first have to reach the second, and cannot ride the frames because the + // branches run in parallel. + // + // TransNetV2 buffers kWindow frames before it can score any of them, so + // the face branch must lag by at least that much or it will ask about + // frames nobody has looked at yet. Channel depth is what creates the lag: + // with backpressure (AR-004) the fanout blocks on the slower branch, so + // a deep face-branch channel lets the detector run ahead by its window + // rather than dropping anything. + auto boundaries = std::make_shared(); + scene_fn.set_boundaries(boundaries); kpn::ObjectNode, kpn::out<>, "scene_detector", 0> scene_node(scene_fn, 128); @@ -362,13 +406,34 @@ int main(int argc, char** argv) { return true; } return false; - }, 32); + }, kSceneJoinDepth); + + /// TRACES: AR-010 | SR-002 + // Stamp is_scene_boundary from the detector's published verdict. tol is + // half a sample interval: the two branches sample at different rates, so + // a boundary found on a dense frame rarely lands exactly on a sampled + // one, and half an interval attributes it to the nearest sampled frame + // and no further. + // + // outran() counts frames that arrived before the detector had scored + // them. Nonzero means the join depth is too shallow for the window, and + // those frames were annotated from an incomplete verdict — which would + // otherwise look exactly like "no boundary here". + SceneBoundaryAnnotatorFunc annotate_fn{boundaries, 0.5 / cfg.sample_fps}; + kpn::ObjectNode, kpn::out<"frame">, + "scene_annotate", 0> annotate(annotate_fn, kSceneJoinDepth); + + // Reported at shutdown: without this the join is unverifiable, and an + // annotator that never fired looks identical to footage with no + // boundaries. + scene_stats = boundaries; auto net = kpn::make_network( kpn::edge(source.output<"raw">(), campos.input<"raw">()), kpn::edge(source.output<"raw">(), scene_node.input<"dense">()), kpn::edge(campos.output<"frame">(), decimate.input<0>()), - kpn::edge(decimate.output<0>(), detector.input<"frame">()), + kpn::edge(decimate.output<0>(), annotate.input<"frame">()), + kpn::edge(annotate.output<"frame">(), detector.input<"frame">()), kpn::edge(detector.output<"scene">(), aligner.input<"scene">()), kpn::edge(aligner.output<"aligned">(), embedder.input<"aligned">()), kpn::edge(embedder.output<"embedded">(), ftracker.input<"embedded">()), diff --git a/src/nodes/scene_boundary_annotator_node.hpp b/src/nodes/scene_boundary_annotator_node.hpp new file mode 100644 index 0000000..664aa05 --- /dev/null +++ b/src/nodes/scene_boundary_annotator_node.hpp @@ -0,0 +1,67 @@ +#pragma once +/// TRACES: AR-010 | SR-002 +/// +/// SceneBoundaryAnnotatorFunc — the join of the decode butterfly. +/// +/// `source` fans out to two branches: dense frames to TransNetV2, sampled frames +/// to face detection. A boundary found on the first has to reach the second, and +/// cannot ride along in the frame because the branches run in parallel. +/// +/// This node sits on the sampled branch and stamps `Frame::is_scene_boundary` +/// from the detector's published verdict. +/// +/// **It only works because the sampled branch lags.** TransNetV2 buffers +/// `kWindow` frames before it can score any of them, so this node must not reach +/// a frame before the detector has an opinion about it. Channel depth creates +/// that lag: with backpressure (AR-004) the fanout blocks on the slower branch, +/// so a deep channel here lets the detector run ahead by its window instead of +/// anything being dropped. +/// +/// When the lag is insufficient the node **counts it** rather than guessing. +/// Annotating an unscored frame as boundary-free is indistinguishable from a +/// genuine "no boundary here", and that is the failure that makes a downstream +/// test pass while verifying nothing. + +#include "scene_boundaries.hpp" +#include "types.hpp" + +#include +#include +#include + +struct SceneBoundaryAnnotatorFunc { + static constexpr std::string_view label() { return "scene_annotate"; } + + /// `tol` is half a sample interval. The branches sample at different rates, + /// so a boundary found on a dense frame rarely lands exactly on a sampled + /// one; half an interval attributes it to the nearest sampled frame and no + /// further. + SceneBoundaryAnnotatorFunc(std::shared_ptr b, double tol) + : bounds_(std::move(b)), tol_(tol) {} + + Frame operator()(Frame f) { + if (f.eof || !bounds_) return f; + + // Wait for the detector's verdict to cover this frame. Channel depth + // alone cannot provide the lag: it holds frames back only when the + // consumer is slower, and this branch is orders of magnitude faster per + // frame than TransNetV2. Blocking here is what makes the join real. + // + // Safe under backpressure because the branches are independent: this + // node stalling does not stop the detector consuming dense frames, and + // the fanout keeps feeding it. + if (!bounds_->wait_until_scored(f.timestamp_sec)) { + // The detector finished without covering this frame — the tail after + // its last full window. Unknown, not negative; counted so it cannot + // pass for "no boundary here". + bounds_->note_outran(); + return f; + } + f.is_scene_boundary = bounds_->is_boundary(f.timestamp_sec, tol_); + return f; + } + +private: + std::shared_ptr bounds_; + double tol_{0.0}; +}; diff --git a/src/nodes/scene_detector_node.hpp b/src/nodes/scene_detector_node.hpp index 8ec954a..d694e67 100644 --- a/src/nodes/scene_detector_node.hpp +++ b/src/nodes/scene_detector_node.hpp @@ -1,6 +1,9 @@ #pragma once #include "types.hpp" #include "config.hpp" +#include "scene_boundaries.hpp" + +#include #include "inference/scene_detector.hpp" #include @@ -30,6 +33,11 @@ struct SceneDetectorFunc { static constexpr std::string_view label() { return "scene_detector"; } + /// TRACES: AR-010 | SR-002 + /// Publish each window's verdict as it is scored, so the face branch — held + /// back by channel depth — can consult it for frames it has not reached yet. + void set_boundaries(std::shared_ptr b) { shared_ = std::move(b); } + SceneDetectorFunc(const Config& cfg, std::atomic& done) : detector_(make_scene_detector(cfg)) , threshold_(cfg.scene_threshold) @@ -51,6 +59,9 @@ struct SceneDetectorFunc { void operator()(Frame f) { if (f.eof) { flush_remaining(); + // Release anyone waiting on the join: the tail frames after the last + // full window will never be covered, so waiting for them would hang. + if (shared_) shared_->finish(); write_output(); done_.store(true, std::memory_order_release); return; @@ -82,6 +93,7 @@ private: // otherwise skip the leading guard already covered by the previous window. const int lo = (window_base_ == 0) ? 0 : guard_; const int hi = ISceneDetector::kWindow - guard_; + std::vector fresh; for (int i = lo; i < hi; ++i) { if (probs[i] <= threshold_) continue; // Local maximum → the boundary frame (avoid a run of high scores @@ -89,9 +101,19 @@ private: const bool peak = (i == 0 || probs[i] >= probs[i-1]) && (i == kLast_() || probs[i] >= probs[i+1]); - if (peak) + if (peak) { boundaries_.push_back({times_[i], probs[i]}); + fresh.push_back(times_[i]); + } } + + /// TRACES: AR-010 | SR-002 + // Publish with a watermark: everything up to times_[hi-1] now has a + // final verdict. The face branch consults this for frames it has not + // reached yet, and the watermark is what lets it tell "no boundary + // here" from "not scored yet". + if (shared_ && hi > lo) + shared_->publish(fresh, times_[hi - 1]); } // At EOF the tail (< kWindow frames) never formed a full window. Pad it out @@ -107,14 +129,24 @@ private: std::vector probs = detector_->detect_window(win); const int lo = (window_base_ == 0) ? 0 : guard_; + std::vector fresh; for (int i = lo; i < n; ++i) { // only real (non-padded) frames if (probs[i] <= threshold_) continue; const bool peak = (i == 0 || probs[i] >= probs[i-1]) && (i == n - 1 || probs[i] >= probs[i+1]); - if (peak) + if (peak) { boundaries_.push_back({times_[i], probs[i]}); + fresh.push_back(times_[i]); + } } + + /// TRACES: AR-010 | SR-002 + // Publish the tail too. Without this the final frames — everything after + // the last full window — reach the join with no verdict and are treated + // as boundary-free without evidence, which is precisely the ambiguity + // the watermark exists to prevent. + if (shared_ && n > 0) shared_->publish(fresh, times_[n - 1]); } void write_output() { @@ -176,4 +208,5 @@ private: int64_t window_base_{0}; // frame index of images_.front() std::vector boundaries_; bool written_{false}; + std::shared_ptr shared_; ///< AR-010 join point }; diff --git a/src/scene_boundaries.hpp b/src/scene_boundaries.hpp new file mode 100644 index 0000000..c2f8002 --- /dev/null +++ b/src/scene_boundaries.hpp @@ -0,0 +1,126 @@ +#pragma once +/// TRACES: AR-010 | SR-002 +/// +/// SceneBoundaries — the join point of the decode butterfly. +/// +/// The topology forks after decode: one branch runs TransNetV2 over dense +/// frames, the other runs face detection over the sampled cadence. Boundaries +/// found on the first branch have to reach the second, and they cannot be +/// carried in the frames themselves because the branches are parallel. +/// +/// **Why this needs a watermark.** TransNetV2 buffers `kWindow` frames before it +/// can score any of them, so at any instant the detector has an opinion about +/// everything up to some time T and nothing after it. Without recording T, a +/// consumer asking "is there a boundary at t?" cannot distinguish *no* from +/// *not yet* — and those demand opposite behaviour. Silently treating unscored +/// frames as boundary-free is exactly the class of failure that makes a +/// verification pass vacuously. +/// +/// The consumer is held back by channel depth (see main.cpp) so that by the time +/// it pulls a frame, the detector has already scored past it. `scored_through()` +/// is what lets that assumption be *checked* rather than assumed. + +#include +#include +#include +#include + +class SceneBoundaries { +public: + /// Peaks closer than this are one boundary. Matches the dedup scenes.json + /// applies, so the two views agree. + static constexpr double kMergeSec = 0.04; + + /// Called by the scene detector as each window is scored. `through` is the + /// timestamp up to which its verdict is now final. + void publish(const std::vector& ts, double through) { + { + std::lock_guard g(mu_); + // Dedup on insert, matching what scenes.json does at write time. A run + // of adjacent high-scoring frames is one boundary, not several, and + // leaving them raw made this view report 357 where the file said 13 — + // the same event counted many times. Harmless for is_boundary(), which + // absorbs them in its tolerance, but a count nobody can reconcile with + // the output file is a bad diagnostic. + bounds_.insert(bounds_.end(), ts.begin(), ts.end()); + std::sort(bounds_.begin(), bounds_.end()); + bounds_.erase(std::unique(bounds_.begin(), bounds_.end(), + [](double a, double b) { return b - a < kMergeSec; }), + bounds_.end()); + scored_through_ = std::max(scored_through_, through); + } + cv_.notify_all(); + } + + /// True if a boundary falls within `tol` of `t`. + /// + /// `tol` exists because the two branches sample at different rates: a + /// boundary found on a dense frame rarely lands exactly on a sampled one. + /// Half a sample interval is the natural width — it attributes the boundary + /// to the nearest sampled frame and no further. + bool is_boundary(double t, double tol) const { + std::lock_guard g(mu_); + auto it = std::lower_bound(bounds_.begin(), bounds_.end(), t - tol); + return it != bounds_.end() && *it <= t + tol; + } + + /// The timestamp through which the detector's verdict is final. A consumer + /// past this point is asking about frames nobody has looked at yet. + double scored_through() const { + std::lock_guard g(mu_); + return scored_through_; + } + + /// Block until the detector's verdict covers `t`, or it finishes. + /// + /// Channel depth alone does NOT create the required lag: it only holds + /// frames back when the consumer is slower, and the face branch is roughly + /// four orders of magnitude faster per frame than TransNetV2. So the join + /// has to wait explicitly. + /// + /// Returns false if the detector finished without ever covering `t`, which + /// happens for the tail frames after its last full window. The caller must + /// distinguish that from a genuine "no boundary" rather than assuming. + bool wait_until_scored(double t) const { + std::unique_lock lk(mu_); + cv_.wait(lk, [&] { return finished_ || scored_through_ >= t; }); + return scored_through_ >= t; + } + + /// Called when the detector will publish nothing further. Without this the + /// join would deadlock on the tail: those frames are never covered by a full + /// window, so waiting for them would wait forever. + void finish() { + { + std::lock_guard g(mu_); + finished_ = true; + } + cv_.notify_all(); + } + + std::size_t count() const { + std::lock_guard g(mu_); + return bounds_.size(); + } + + /// Consumers that outran the detector. Nonzero means the face branch is not + /// buffered deeply enough for the detector's window, so some frames were + /// annotated from an incomplete verdict — a real misconfiguration, and one + /// that would otherwise be invisible. + void note_outran() const { + std::lock_guard g(mu_); + ++outran_; + } + std::size_t outran() const { + std::lock_guard g(mu_); + return outran_; + } + +private: + mutable std::mutex mu_; + mutable std::condition_variable cv_; + bool finished_{false}; + std::vector bounds_; + double scored_through_{-1.0}; + mutable std::size_t outran_{0}; +};