feat(scene): feed TransNetV2 at native rate, derive the dedup window from it
Closes both violations SPEC.md named under "Every model gets the input it was trained for". They are one bug, not two. The dense stream defaulted to 12 fps, so a 100-frame TransNetV2 window spanned ~8.3 s against the ~4 s it was trained on: half-speed motion over twice its temporal context. Boundary timestamps stayed correct throughout, which is exactly why the degradation was invisible and why the compressed separation it produced (~0.50 baseline against ~0.7+ peaks) was read as a property of the ONNX export rather than of the input. Dedup then merged boundaries closer than a literal 0.04 s — one frame at 25 fps, and wider than a frame at 30, so two cuts on consecutive frames became one. Nothing in scenes.json showed it; the file simply had fewer boundaries. Native rate is where that constant did the most damage, which is why fixing the decode rate without fixing the dedup would have made things worse. dedup_window_sec() now takes the median interval the detector was actually fed and halves it. Half a frame rather than a whole one: the only thing being merged is one frame scored by two overlapping windows, and two distinct frames are a full interval apart. Cost is real — dense decode is the pipeline's cost driver. It is accepted; dense_scale and scene_stride remain the reductions that do not run the model off-distribution. scene_threshold 0.60 was fitted against the 12 fps input and is now stale, so VR-006 goes from Low to Medium: it is no longer a refinement, it is a constant that no longer describes the input. AR-002 rides along because it was already implemented, just untagged and unverified — the register said Planned while the code was correct. The size filter becomes FaceDetectorFunc::drop_undersized(), tested at the threshold and at dense_scale 0.5, and checked end to end against the superhero dump, whose smallest face is exactly its recorded 32 px minimum, so the fixture check cannot pass vacuously. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: AR-002, AR-011 | SR-002 | UT-002, UT-003, IT-001
This commit is contained in:
+20
-9
@@ -91,17 +91,28 @@ struct Config {
|
||||
// at ~0.50; real boundaries spike to ~0.7+)
|
||||
int scene_stride{50}; // frames advanced between windows (≤ kWindow)
|
||||
|
||||
// Dense-decode throughput knobs (only active with scene_detect). Dense decode
|
||||
// of every native-rate frame is the pipeline's cost driver; these trade a
|
||||
// little boundary precision for a large speedup.
|
||||
// scene_decode_fps: rate the source decodes at in dense mode. Lower =
|
||||
// fewer frames decoded. TransNetV2 tolerates ~12fps; boundary timestamps
|
||||
// stay correct (keyed off each frame's real timestamp). 0 = native fps.
|
||||
// Dense-decode knobs (only active with scene_detect). Dense decode of every
|
||||
// native-rate frame is the pipeline's cost driver, which is what made the
|
||||
// temporal shortcut below tempting.
|
||||
/// TRACES: AR-011 | SR-002
|
||||
// scene_decode_fps: rate the source decodes at in dense mode.
|
||||
// **0 = native, and native is the only correct setting.** kWindow is 100
|
||||
// frames: at native 25 fps that window spans ~4 s, which is what
|
||||
// TransNetV2 was trained on; at the 12 fps this used to default to it
|
||||
// spans ~8.3 s, so the model saw half-speed motion over twice its
|
||||
// temporal context. Boundary *timestamps* stay right either way — which
|
||||
// is exactly why the degradation was invisible, and why the compressed
|
||||
// separation it produced (~0.50 baseline against ~0.7+ peaks) was read
|
||||
// as a property of the export rather than of the input. Lowering this
|
||||
// buys decode time by running the model off-distribution; reach for
|
||||
// dense_scale or scene_stride instead, which do not.
|
||||
// dense_scale: downscale factor applied to decoded frames in dense mode
|
||||
// (0<f≤1; e.g. 0.5 = half size). Cheaper sws_scale + smaller frames
|
||||
// through the fanout. NOTE: also shrinks what the face detector sees —
|
||||
// keep ≥0.5 on 1080p sources so SCRFD still resolves small faces. 1 = off.
|
||||
float scene_decode_fps{12.0f}; // dense decode rate (0 = native)
|
||||
// through the fanout. A spatial reduction, and TransNetV2 downsamples to
|
||||
// 48×27 regardless, so unlike the above it is a documented, understood
|
||||
// degradation. NOTE: also shrinks what the face detector sees — keep
|
||||
// ≥0.5 on 1080p sources so SCRFD still resolves small faces. 1 = off.
|
||||
float scene_decode_fps{0.f}; // dense decode rate (0 = native)
|
||||
float dense_scale{1.0f}; // dense-mode frame downscale (1 = off)
|
||||
|
||||
// ── Face tracking (frame-to-frame) ───────────────────────────────────────
|
||||
|
||||
+3
-2
@@ -32,8 +32,9 @@
|
||||
// --scene-detector-engine <path> pre-built TransNetV2 TRT engine (TRT backend)
|
||||
// --scene-threshold <f> boundary sigmoid prob above this → cut (default: 0.60)
|
||||
// --scene-stride <N> frames between TransNetV2 windows (default: 50, ≤100)
|
||||
// --scene-decode-fps <f> dense decode rate in scene-detect mode (default: 12;
|
||||
// 0 = native fps). Lower = faster, coarser boundaries.
|
||||
// --scene-decode-fps <f> dense decode rate in scene-detect mode (default: 0 =
|
||||
// native, the only rate TransNetV2 is calibrated for;
|
||||
// AR-011). Lowering it runs the model off-distribution.
|
||||
// --dense-scale <f> downscale decoded frames in scene-detect mode (0<f≤1,
|
||||
// default 1=off). Speeds decode; keep ≥0.5 on 1080p.
|
||||
// --max-faces <N> max faces kept per frame (default: 10)
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
#include <algorithm>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
// ── FaceDetectorFunc ──────────────────────────────────────────────────────────
|
||||
// KPN node: runs SCRFD-500MF to detect ALL faces in a frame.
|
||||
@@ -22,22 +23,38 @@ struct FaceDetectorFunc {
|
||||
, min_face_px_(cfg.min_face_px)
|
||||
{}
|
||||
|
||||
SceneFrame operator()(Frame f) {
|
||||
if (f.eof) return {std::move(f), {}};
|
||||
|
||||
auto faces = detector_->detect(f.image);
|
||||
|
||||
// Drop faces below minimum pixel size (too small for reliable ArcFace
|
||||
// alignment). Note: when dense_scale downscaled the frame, both the
|
||||
// detection coords and min_face_px are in downscaled space — so scale
|
||||
// the threshold down to match, keeping the physical size cutoff constant.
|
||||
const float min_px = (f.bbox_upscale != 1.f)
|
||||
? min_face_px_ / f.bbox_upscale : min_face_px_;
|
||||
/// TRACES: AR-002 | SR-002
|
||||
// Drop faces below the minimum size — too small for reliable ArcFace
|
||||
// alignment, and below the resolution where identification still holds
|
||||
// (VR-013 measured the knee end to end).
|
||||
//
|
||||
// The minimum is expressed in ORIGINAL video resolution, which is what makes
|
||||
// it a property of the footage rather than of a throughput knob. When
|
||||
// dense_scale downscaled the frame the detector's boxes are in downscaled
|
||||
// space, and `bbox_upscale` is what maps them back; dividing the threshold by
|
||||
// it rather than multiplying every box keeps the comparison on the detector's
|
||||
// own numbers and the physical cutoff constant across scales.
|
||||
//
|
||||
// Strictly less-than: a face exactly at the minimum is admissible, which is
|
||||
// what a "minimum of 40x40" means.
|
||||
static void drop_undersized(std::vector<DetectedFace>& faces,
|
||||
float min_face_px,
|
||||
float bbox_upscale) {
|
||||
const float min_px = (bbox_upscale > 0.f) ? min_face_px / bbox_upscale
|
||||
: min_face_px;
|
||||
faces.erase(
|
||||
std::remove_if(faces.begin(), faces.end(), [&](const DetectedFace& d) {
|
||||
return d.bbox.width < min_px || d.bbox.height < min_px;
|
||||
}),
|
||||
faces.end());
|
||||
}
|
||||
|
||||
SceneFrame operator()(Frame f) {
|
||||
if (f.eof) return {std::move(f), {}};
|
||||
|
||||
auto faces = detector_->detect(f.image);
|
||||
|
||||
drop_undersized(faces, min_face_px_, f.bbox_upscale);
|
||||
|
||||
// Sort largest-first so max_faces_ keeps the most informative detections
|
||||
std::sort(faces.begin(), faces.end(),
|
||||
|
||||
@@ -67,6 +67,15 @@ struct SceneDetectorFunc {
|
||||
return;
|
||||
}
|
||||
|
||||
/// TRACES: AR-011 | SR-002
|
||||
// Learn the cadence of the stream from the stream itself, rather than
|
||||
// assuming one. See dedup_window_sec().
|
||||
if (prev_ts_ >= 0.0 && intervals_.size() < kCadenceSamples) {
|
||||
const double dt = f.timestamp_sec - prev_ts_;
|
||||
if (dt > 0.0) intervals_.push_back(dt);
|
||||
}
|
||||
prev_ts_ = f.timestamp_sec;
|
||||
|
||||
images_.push_back(f.image);
|
||||
times_.push_back(f.timestamp_sec);
|
||||
|
||||
@@ -81,6 +90,34 @@ struct SceneDetectorFunc {
|
||||
}
|
||||
}
|
||||
|
||||
/// TRACES: AR-011 | SR-002
|
||||
// How close two boundaries have to be before they are the same boundary,
|
||||
// derived from the cadence the detector was actually fed.
|
||||
//
|
||||
// What this replaces is a literal 0.04 s — one frame at 25 fps, and silently
|
||||
// wrong at any other rate. On a 30 fps source it spans more than a frame, so
|
||||
// two cuts on consecutive frames merge into one and a real boundary is lost;
|
||||
// the output does not show this, it simply contains fewer cuts. Assuming a
|
||||
// frame rate is the same class of mistake as feeding a model the wrong rate,
|
||||
// which is why this belongs to AR-011 and not to a tidy-up.
|
||||
//
|
||||
// Half a frame, not a whole one, because the only thing being deduplicated is
|
||||
// one frame scored by two overlapping windows — a gap of zero. Two distinct
|
||||
// frames are a full interval apart and must both survive. Half an interval
|
||||
// separates those two cases without putting the decision on the knife-edge
|
||||
// where floating-point error settles it.
|
||||
//
|
||||
// Median, not mean: a seek, or a gap where the decoder dropped a frame,
|
||||
// contributes one long interval that would drag a mean and cannot move a
|
||||
// median.
|
||||
static double dedup_window_sec(std::vector<double> intervals) {
|
||||
if (intervals.empty()) return 0.0; // <2 frames: nothing to deduplicate
|
||||
const std::size_t mid = intervals.size() / 2;
|
||||
std::nth_element(intervals.begin(), intervals.begin() + mid,
|
||||
intervals.end());
|
||||
return intervals[mid] * 0.5;
|
||||
}
|
||||
|
||||
private:
|
||||
// Run TransNetV2 on the leading kWindow frames of the buffer and record any
|
||||
// boundaries found within the trusted centre region.
|
||||
@@ -154,6 +191,8 @@ private:
|
||||
written_ = true;
|
||||
|
||||
// Merge boundaries closer than one frame apart (dedup across window seams).
|
||||
const double dedup_sec = dedup_window_sec(intervals_);
|
||||
|
||||
std::sort(boundaries_.begin(), boundaries_.end(),
|
||||
[](const Boundary& a, const Boundary& b) {
|
||||
return a.t < b.t;
|
||||
@@ -167,7 +206,7 @@ private:
|
||||
nlohmann::json cuts = nlohmann::json::array();
|
||||
double last_t = -1e9;
|
||||
for (const auto& b : boundaries_) {
|
||||
if (b.t - last_t < 0.04) continue; // ~1 frame @25fps dedup
|
||||
if (b.t - last_t < dedup_sec) continue;
|
||||
cuts.push_back({{"t", b.t}, {"probability", b.prob}});
|
||||
last_t = b.t;
|
||||
}
|
||||
@@ -180,8 +219,12 @@ private:
|
||||
return;
|
||||
}
|
||||
f << root.dump(2) << "\n";
|
||||
// Report the derived cadence: VR-006 re-tunes scene_threshold against it,
|
||||
// and a rate that is not the source's is the first thing to suspect.
|
||||
std::cerr << "\n[scene_detector] wrote " << root["cuts"].size()
|
||||
<< " boundaries → " << output_path_ << "\n";
|
||||
<< " boundaries → " << output_path_
|
||||
<< " (dedup=" << dedup_sec << "s from "
|
||||
<< (dedup_sec > 0.0 ? 0.5 / dedup_sec : 0.0) << " fps)\n";
|
||||
}
|
||||
|
||||
static int kLast_() { return ISceneDetector::kWindow - 1; }
|
||||
@@ -195,6 +238,10 @@ private:
|
||||
|
||||
struct Boundary { double t; float prob; };
|
||||
|
||||
// Enough to establish a rate; bounded so a feature-length film does not
|
||||
// accumulate one double per frame for a number that stops moving early.
|
||||
static constexpr std::size_t kCadenceSamples = 512;
|
||||
|
||||
std::unique_ptr<ISceneDetector> detector_;
|
||||
float threshold_;
|
||||
int stride_;
|
||||
@@ -207,6 +254,8 @@ private:
|
||||
std::deque<double> times_;
|
||||
int64_t window_base_{0}; // frame index of images_.front()
|
||||
std::vector<Boundary> boundaries_;
|
||||
double prev_ts_{-1.0}; // AR-011: cadence, learned not assumed
|
||||
std::vector<double> intervals_;
|
||||
bool written_{false};
|
||||
std::shared_ptr<SceneBoundaries> shared_; ///< AR-010 join point
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user