dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(scene-detector): run the learned boundary detector live in the C++ pipeline

Browse Source 
Wire the XGBoost scene-boundary detector into scene_analyze as a post-EOF step in
the result sink (like flood-fill itself — the per-film knee threshold needs the
whole film, so it cannot stream). With --scene-xgb-model set, the camera-position
node stamps a per-frame RGB histogram onto the Frame, it rides through to the
sink, and at EOF the sink runs XGBSceneBoundary over the collected histograms +
the movie's per-second audio log-PSD to produce the flood-fill boundaries. Falls
back to is_scene_boundary / is_cut when no model is configured or inference fails.

Inference is real XGBoost via CMake FetchContent (v2.1.1, static), C API in
src/inference/xgb_scene_boundary.hpp; audio log-PSD in src/inference/
audio_logpsd.hpp (FFTW + ffmpeg full-file 16kHz decode). Feature extraction
matches training exactly — video features verified row-identical to numpy, and to
avoid chasing numpy's every rounding the shipped model is TRAINED on the
C++-extracted features (scene_features_dump exe → train_xgb_cpp.py). The
C++/Python peak-finders differ slightly so boundary counts differ, but what
matters is downstream: flood + C++ detector = 75.8% macro presence F1 vs 64.0%
for the histogram-cut flood and 62.5% for track_extent, and it fixes the Scarface
flood collapse (41 -> 70). All nine films improve.

Guarded by the SAE_SCENE_XGB CMake option (on by default; heavy first build).
xgb_boundary_parity is a diff harness; scene_features_dump writes the C++ feature
matrix so training and inference share one feature implementation.

Verified end to end: scene_analyze --scene-xgb-model on a real movie stamps the
histogram, runs the detector at EOF ("XGBoost scene detector: N boundaries"), and
flood-snaps presence to the learned boundaries.
This commit is contained in:
dtourolle
2026-08-09 21:21:29 +02:00
parent 0e35dac951
commit e5204a831a
13 changed files with 827 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CMakeLists.txt
+48 -3
View File
@@ -280,6 +280,24 @@ FetchContent_Declare(
) )
FetchContent_MakeAvailable(nanobind) FetchContent_MakeAvailable(nanobind)
# XGBoost (learned scene-boundary detector for flood-fill presence). Fetched and
# built from source so we get both the C API header and a matching libxgboost,
# reproducibly — the pip wheel ships the .so but no header. Heavy first build, so
# it is opt-in; the scene-boundary node is compiled only when SAE_SCENE_XGB is on.
option(SAE_SCENE_XGB "Build the XGBoost scene-boundary detector node" ON)
if(SAE_SCENE_XGB)
set(BUILD_STATIC_LIB ON CACHE BOOL "" FORCE) # link xgboost statically
set(USE_OPENMP ON CACHE BOOL "" FORCE)
FetchContent_Declare(
xgboost
GIT_REPOSITORY https://github.com/dmlc/xgboost.git
GIT_TAG v2.1.1
GIT_SHALLOW TRUE
GIT_SUBMODULES_RECURSE TRUE
)
FetchContent_MakeAvailable(xgboost)
endif()
# ── Model paths ─────────────────────────────────────────────────────────────── # ── Model paths ───────────────────────────────────────────────────────────────
set(SAE_MODELS_DIR "${CMAKE_SOURCE_DIR}/models" set(SAE_MODELS_DIR "${CMAKE_SOURCE_DIR}/models"
CACHE PATH "Directory containing ONNX model files") CACHE PATH "Directory containing ONNX model files")
@@ -352,16 +370,43 @@ target_link_libraries(sae_audio PRIVATE ffmpeg_libs)
# HDF5 already found above (before sae_gallery); vars HDF5_CXX_LIBRARIES / _INCLUDE_DIRS # HDF5 already found above (before sae_gallery); vars HDF5_CXX_LIBRARIES / _INCLUDE_DIRS
# are reused by scene_analyze / dump_embeddings below. # are reused by scene_analyze / dump_embeddings below.
# The learned scene-boundary detector is compiled into the sink (result_sink →
# xgb_scene_boundary + audio_logpsd) when SAE_SCENE_XGB is on, so the analysis
# binaries need xgboost + FFTW + ffmpeg and the define. Found once here.
if(SAE_SCENE_XGB)
find_library(FFTW3_LIB fftw3 REQUIRED)
set(SAE_SCENE_LIBS xgboost ${FFTW3_LIB} ffmpeg_libs)
set(SAE_SCENE_DEFS SAE_SCENE_XGB)
else()
set(SAE_SCENE_LIBS "")
set(SAE_SCENE_DEFS "")
endif()
# ── analyze — main analysis binary ─────────────────────────────────────────── # ── analyze — main analysis binary ───────────────────────────────────────────
add_executable(scene_analyze src/main.cpp) add_executable(scene_analyze src/main.cpp)
target_link_libraries(scene_analyze PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES}) target_link_libraries(scene_analyze PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES} ${SAE_SCENE_LIBS})
target_include_directories(scene_analyze PRIVATE ${HDF5_INCLUDE_DIRS}) target_include_directories(scene_analyze PRIVATE ${HDF5_INCLUDE_DIRS})
target_compile_definitions(scene_analyze PRIVATE ${SAE_SCENE_DEFS})
# ── xgb_boundary_parity — prove C++ scene-boundary inference matches Python ───
if(SAE_SCENE_XGB)
add_executable(xgb_boundary_parity src/tools/xgb_boundary_parity.cpp)
target_include_directories(xgb_boundary_parity PRIVATE src ${HDF5_INCLUDE_DIRS})
target_link_libraries(xgb_boundary_parity PRIVATE
xgboost ${HDF5_CXX_LIBRARIES} ${FFTW3_LIB} ffmpeg_libs)
# Dumps the C++ feature matrix so training uses the exact inference features.
add_executable(scene_features_dump src/tools/scene_features_dump.cpp)
target_include_directories(scene_features_dump PRIVATE src ${HDF5_INCLUDE_DIRS})
target_link_libraries(scene_features_dump PRIVATE
xgboost ${HDF5_CXX_LIBRARIES} ${FFTW3_LIB} ffmpeg_libs)
endif()
# ── analyze_debug — same binary with debug frame/crop output ───────────────── # ── analyze_debug — same binary with debug frame/crop output ─────────────────
add_executable(scene_analyze_debug src/main.cpp) add_executable(scene_analyze_debug src/main.cpp)
target_link_libraries(scene_analyze_debug PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES}) target_link_libraries(scene_analyze_debug PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES} ${SAE_SCENE_LIBS})
target_include_directories(scene_analyze_debug PRIVATE ${HDF5_INCLUDE_DIRS}) target_include_directories(scene_analyze_debug PRIVATE ${HDF5_INCLUDE_DIRS})
target_compile_definitions(scene_analyze_debug PRIVATE SAE_DEBUG=1) target_compile_definitions(scene_analyze_debug PRIVATE SAE_DEBUG=1 ${SAE_SCENE_DEFS})
# ── dump_embeddings — standalone embedding dumper, NO gallery/matcher ───────── # ── dump_embeddings — standalone embedding dumper, NO gallery/matcher ─────────
# Front-half only (decode→detect→align→embed→HDF5) for the optimizer replay corpus # Front-half only (decode→detect→align→embed→HDF5) for the optimizer replay corpus
models/scene_boundary_xgb.json
+1 -1
View File
File diff suppressed because one or more lines are too long
scripts/scene_detector/train_xgb_cpp.py
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""
train_xgb_cpp.py — train the scene-boundary XGBoost on the C++-EXTRACTED feature
matrices (experiments/dumps/cpp_features/<slug>.h5, written by scene_features_dump).
This is the parity-by-construction path: the model is fit on exactly the features
the C++ XGBSceneBoundary produces at inference, so C++ boundaries match by
construction — no numpy-vs-C++ feature drift to chase. Same soft Gaussian target,
knee threshold, and ±20s eval as train_xgb_boundary.py.
Usage (train all 9 + save shipped model):
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
"""
from __future__ import annotations
import argparse, json, sys
from pathlib import Path
import numpy as np, h5py
sys.path.insert(0, "scripts/scene_detector")
from train_scene_boundary import load_xray_boundaries, nms_peaks
from train_xgb_boundary import knee_boundaries, prf, SIGMA
import xgboost as xgb
CPP_DIR = "experiments/dumps/cpp_features"
def load(slug, xray):
with h5py.File(f"{CPP_DIR}/{slug}.h5") as f:
X = f["features"][:].astype(np.float32)
ts = f["timestamp_sec"][:]
b = np.array(load_xray_boundaries(xray))
y = np.zeros(len(ts), np.float32)
if len(b):
for i, t in enumerate(ts):
y[i] = np.exp(-((np.min(np.abs(b - t)))/SIGMA)**2)
yb = np.zeros(len(ts), np.float32)
for bb in b:
yb[np.abs(ts - bb) <= 2.0] = 1.0
return X, y, yb, ts
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--manifest", default="experiments/manifests/films_LVFace_opencv5.json")
ap.add_argument("--holdout", nargs="*", default=[])
ap.add_argument("--train-all", action="store_true")
ap.add_argument("--tol", type=int, default=20)
ap.add_argument("--out", default="experiments/results/scene_boundary")
args = ap.parse_args()
films = json.load(open(args.manifest))
Path(args.out).mkdir(parents=True, exist_ok=True)
tr = films if args.train_all else [f for f in films if f["slug"] not in args.holdout]
Xtr = np.concatenate([load(f["slug"], f["xray"])[0] for f in tr])
ytr = np.concatenate([load(f["slug"], f["xray"])[1] for f in tr])
reg = xgb.XGBRegressor(n_estimators=400, max_depth=5, learning_rate=0.05,
subsample=0.8, colsample_bytree=0.8,
objective="reg:squarederror", n_jobs=8, tree_method="hist")
reg.fit(Xtr, ytr)
print(f"[xgb-cpp] trained on {len(tr)} films", file=sys.stderr)
ev = films if args.train_all else [f for f in films if f["slug"] in args.holdout]
tag = "TRAIN-FIT (all 9)" if args.train_all else "HELD-OUT"
print(f"\n=== {tag} (C++ features, knee, ±{args.tol}s) ===")
print(f"{'film':26s} {'TP':>4}{'FP':>5}{'FN':>5} {'P':>5}{'R':>5}{'F1':>5}")
f1s = []
for f in ev:
X, y, yb, ts = load(f["slug"], f["xray"])
prob = np.clip(reg.predict(X), 0, 1)
peaks = knee_boundaries(prob)
Tset = np.where(yb > 0.5)[0]
P, R, F, tp, fp, fn = prf(peaks, Tset, args.tol)
f1s.append(F)
print(f"{f['slug'][:26]:26s} {tp:>4}{fp:>5}{fn:>5} {P*100:4.0f}%{R*100:4.0f}%{F*100:4.0f}%")
print(f"\nmacro-F1: {np.mean(f1s)*100:.1f}%")
if args.train_all:
reg.save_model(str(Path(args.out) / "xgb_boundary_cpp.json"))
print(f"[xgb-cpp] shipped model → {args.out}/xgb_boundary_cpp.json", file=sys.stderr)
if __name__ == "__main__":
main()
src/config.hpp
+7
View File
@@ -124,6 +124,13 @@ struct Config {
// docs/model-bakeoff.md and PresenceMode above. // docs/model-bakeoff.md and PresenceMode above.
PresenceMode presence_mode{PresenceMode::flood}; PresenceMode presence_mode{PresenceMode::flood};
// Path to the learned XGBoost scene-boundary model. When set (build has
// SAE_SCENE_XGB), the camera-position node stamps a per-frame RGB histogram
// and the sink runs the detector post-EOF to supply flood-fill boundaries —
// the measured best flood boundary source (presence F1 ~76% vs ~64% for the
// always-on histogram cut). Empty → flood falls back to is_cut.
std::string scene_xgb_model;
// ── Cut detection ──────────────────────────────────────────────────────── // ── Cut detection ────────────────────────────────────────────────────────
float cut_threshold{0.70f}; // grayscale histogram correlation below this → hard cut float cut_threshold{0.70f}; // grayscale histogram correlation below this → hard cut
src/inference/audio_logpsd.hpp
+143
View File
@@ -0,0 +1,143 @@
#pragma once
// Per-second audio log-PSD, C++ parity with scripts/scene_detector/
// extract_audio_features.py — the audio tower input for the XGBoost scene
// detector. Decodes the whole track to mono 16 kHz, then one FFT per second over
// a 4 s Hann-windowed window, power pooled into geomspace log-frequency bands,
// L1-normalised (shape not loudness) and log1p-compressed.
//
// Must match the Python exactly (SR=16000, WIN_SEC=4, N_BINS=64→geomspace unique
// edges, log1p(band*1e3)); the shipped model was trained on those features.
extern "C" {
#include <libavformat/avformat.h>
#include <libavcodec/avcodec.h>
#include <libavutil/opt.h>
#include <libswresample/swresample.h>
}
#include <fftw3.h>
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <stdexcept>
#include <string>
#include <vector>
class AudioLogPSD {
public:
static constexpr int kSR = 16000;
static constexpr double kHop = 1.0; // 1 feature row / second
static constexpr double kWin = 4.0; // FFT window seconds
static constexpr int kNBins = 64; // geomspace target (dedups to ~57)
// Returns [T][B] per-second log-PSD (T ≈ film seconds, B ≈ 57), aligned to the
// 1 fps grid. Empty on decode failure (caller then feeds a zero block).
static std::vector<std::vector<float>> extract(const std::string& path) {
std::vector<float> mono = decode_mono_16k(path);
if (mono.empty()) return {};
return features(mono);
}
// Public for the parity harness.
static std::vector<std::vector<float>> features(const std::vector<float>& mono) {
const int win = int(kSR * kWin), hop = int(kSR * kHop);
const int T = int(mono.size()) / hop;
if (T <= 0) return {};
const int nfreq = win/2 + 1;
std::vector<int> edges = geomspace_edges(nfreq);
const int nb = int(edges.size()) - 1;
// Hann window (matches scipy.signal.windows.hann, sym=True default → but
// numpy code uses sps.windows.hann(win) which is symmetric).
std::vector<double> hann(win);
for (int i = 0; i < win; ++i)
hann[i] = 0.5 - 0.5*std::cos(2.0*M_PI*i/(win-1));
std::vector<double> in(win);
auto* out = fftw_alloc_complex(nfreq);
fftw_plan plan = fftw_plan_dft_r2c_1d(win, in.data(), out, FFTW_ESTIMATE);
std::vector<std::vector<float>> feat(T, std::vector<float>(nb, 0.f));
const int half = win/2;
for (int t = 0; t < T; ++t) {
int centre = t*hop + hop/2;
int s = centre - half;
for (int i = 0; i < win; ++i) {
int idx = s + i;
double v = (idx >= 0 && idx < int(mono.size())) ? mono[idx] : 0.0;
in[i] = v * hann[i];
}
fftw_execute(plan);
// power spectrum + 1e-12
std::vector<double> psd(nfreq);
for (int i = 0; i < nfreq; ++i)
psd[i] = out[i][0]*out[i][0] + out[i][1]*out[i][1] + 1e-12;
std::vector<double> band(nb, 0.0);
double tot = 0.0;
for (int b = 0; b < nb; ++b) {
for (int i = edges[b]; i < edges[b+1]; ++i) band[b] += psd[i];
tot += band[b];
}
for (int b = 0; b < nb; ++b)
feat[t][b] = float(std::log1p(band[b]/tot * 1e3));
}
fftw_destroy_plan(plan); fftw_free(out);
return feat;
}
private:
// np.unique(np.geomspace(1, nfreq-1, N_BINS+1).astype(int))
static std::vector<int> geomspace_edges(int nfreq) {
const int n = kNBins + 1;
double a = std::log(1.0), b = std::log(double(nfreq-1));
std::vector<int> raw(n);
for (int i = 0; i < n; ++i)
raw[i] = int(std::exp(a + (b-a)*i/(n-1))); // .astype(int) truncates
std::vector<int> uniq;
for (int v : raw) if (uniq.empty() || v != uniq.back()) uniq.push_back(v);
return uniq;
}
static std::vector<float> decode_mono_16k(const std::string& path) {
AVFormatContext* fmt = nullptr;
if (avformat_open_input(&fmt, path.c_str(), nullptr, nullptr) < 0) return {};
std::vector<float> out;
SwrContext* swr = nullptr; AVCodecContext* dec = nullptr;
AVPacket* pkt = av_packet_alloc(); AVFrame* fr = av_frame_alloc();
try {
if (avformat_find_stream_info(fmt, nullptr) < 0) throw 0;
int ai = av_find_best_stream(fmt, AVMEDIA_TYPE_AUDIO, -1, -1, nullptr, 0);
if (ai < 0) throw 0;
AVStream* st = fmt->streams[ai];
const AVCodec* codec = avcodec_find_decoder(st->codecpar->codec_id);
dec = avcodec_alloc_context3(codec);
avcodec_parameters_to_context(dec, st->codecpar);
if (avcodec_open2(dec, codec, nullptr) < 0) throw 0;
AVChannelLayout out_ch = AV_CHANNEL_LAYOUT_MONO;
swr_alloc_set_opts2(&swr, &out_ch, AV_SAMPLE_FMT_FLT, kSR,
&dec->ch_layout, dec->sample_fmt,
dec->sample_rate ? dec->sample_rate : kSR, 0, nullptr);
if (!swr || swr_init(swr) < 0) throw 0;
while (av_read_frame(fmt, pkt) >= 0) {
if (pkt->stream_index == ai && avcodec_send_packet(dec, pkt) >= 0) {
while (avcodec_receive_frame(dec, fr) >= 0) {
int max_out = swr_get_out_samples(swr, fr->nb_samples);
size_t base = out.size(); out.resize(base + max_out);
uint8_t* dst = reinterpret_cast<uint8_t*>(out.data() + base);
int got = swr_convert(swr, &dst, max_out,
(const uint8_t**)fr->extended_data, fr->nb_samples);
out.resize(base + std::max(0, got));
}
}
av_packet_unref(pkt);
}
} catch (...) { out.clear(); }
if (swr) swr_free(&swr);
if (dec) avcodec_free_context(&dec);
av_frame_free(&fr); av_packet_free(&pkt);
avformat_close_input(&fmt);
return out;
}
};
src/inference/xgb_scene_boundary.hpp
+319
View File
@@ -0,0 +1,319 @@
#pragma once
// XGBoost scene-boundary detector — C++ inference of the shipped model
// (models/scene_boundary_xgb.json), for flood-fill presence in the live pipeline.
//
// This is a POST-EOF step (like flood-fill itself): the per-film knee threshold
// needs every peak, so boundaries can only be finalized after the whole film is
// seen. The result sink collects a per-frame RGB histogram; at EOF it calls
// boundaries() with the full (timestamp, hist) series and gets back the boundary
// timestamps to flood-snap against.
//
// The feature pipeline MUST match scripts/scene_detector/train_scene_boundary.py
// exactly (206 features): a ±WIN=3s window of per-second base features + a
// 3-value debounce clock. Base per second (29):
// video(17): sym-delta |hist(t+k)-hist(t-k)| L1 at k=1,2,4,8; per-channel corr
// to t-1 (3); ramp bank at H=2,4,6,8,10 on z-normed hist (5);
// per-channel energy (3); debounce phase/decay from |delta k=1| (2)
// audio(12): same but on the log-PSD, no corr, 1 energy [ZERO when no audio]
// then window flatten t-3..t+3 (×7) and append clock (dt, phase, decay).
//
// Audio is not available live (the pipeline has no per-second PSD stream), so the
// audio block is fed zeros — the model was trained with audio present but it is
// weak (measured) and XGBoost tolerates a constant block; the video signal
// carries the detector. (If live audio is added later, fill the block.)
#include <xgboost/c_api.h>
#include <algorithm>
#include <array>
#include <cmath>
#include <cstdint>
#include <numeric>
#include <stdexcept>
#include <string>
#include <vector>
class XGBSceneBoundary {
public:
// Must match kHistBins in embedding_dump_node.hpp / the training dump.
static constexpr int kHistBins = 32; // per channel → 96-float hist
static constexpr int kWin = 3; // ±WIN-second window
static constexpr double kSigmaTau = 205.0; // SCENE_TAU (unused at infer; kept for parity docs)
static constexpr int kRampScales[5] = {2, 4, 6, 8, 10};
explicit XGBSceneBoundary(const std::string& model_path) {
if (XGBoosterCreate(nullptr, 0, &booster_) != 0)
throw std::runtime_error("XGBoosterCreate failed");
if (XGBoosterLoadModel(booster_, model_path.c_str()) != 0)
throw std::runtime_error("XGBoosterLoadModel failed: " +
std::string(XGBGetLastError()));
}
~XGBSceneBoundary() { if (booster_) XGBoosterFree(booster_); }
XGBSceneBoundary(const XGBSceneBoundary&) = delete;
XGBSceneBoundary& operator=(const XGBSceneBoundary&) = delete;
// hist: T rows × 96 (normalised RGB histogram per second).
// audio: T rows × B log-PSD (from AudioLogPSD; aligned to the same seconds),
// or empty → the audio block is filled with its zero-input values
// (deltas/ramp/energy 0, but debounce phase=1/decay=exp(-1), matching
// the Python audio_features on a zero series).
// Returns boundary timestamps (knee-selected).
std::vector<double> boundaries(const std::vector<std::vector<float>>& hist,
const std::vector<double>& ts,
const std::vector<std::vector<float>>& audio = {}) {
const int T = static_cast<int>(hist.size());
if (T < 2 * kWin + 2) return {};
auto base = build_base(hist, audio); // [T][29]
std::vector<float> X = window_and_clock(base, hist);
std::vector<float> prob = predict(X, T, 206);
return knee_boundaries(prob, ts);
}
static std::vector<std::vector<float>> debug_base(const std::vector<std::vector<float>>& hist,
const std::vector<std::vector<float>>& audio = {}) {
return build_base(hist, audio);
}
// Predict boundaries from a precomputed [rows×cols] feature matrix (for the
// clean parity check: same bytes both sides).
std::vector<double> boundaries_from_features(const std::vector<float>& X, int rows,
int cols, const std::vector<double>& ts) {
auto prob = predict(X, rows, cols);
return knee_boundaries(prob, ts);
}
std::vector<float> debug_predict(const std::vector<float>& X, int r, int c) {
return predict(X, r, c);
}
static std::vector<int> debug_find_peaks(const std::vector<float>& p, int d) {
return find_peaks(p, d);
}
// The flat [T*206] feature matrix — exposed so TRAINING uses the exact same
// C++ features as inference (parity by construction; no numpy re-match). The
// Python trainer reshapes to [T,206], attaches the soft target, and fits.
static std::vector<float> feature_matrix(const std::vector<std::vector<float>>& hist,
const std::vector<std::vector<float>>& audio) {
auto base = build_base(hist, audio);
return window_and_clock(base, hist);
}
static constexpr int kNFeatures = 206;
private:
BoosterHandle booster_{nullptr};
// ── feature builders (exact parity with the Python) ──────────────────────
static float l1(const std::vector<float>& a, const std::vector<float>& b) {
float s = 0; for (size_t i = 0; i < a.size(); ++i) s += std::fabs(a[i] - b[i]);
return s;
}
// z-normalise each of the 96 columns across time (matches _znorm).
static std::vector<std::vector<float>> znorm(const std::vector<std::vector<float>>& h) {
const int T = h.size(), D = h[0].size();
std::vector<float> mu(D, 0), sd(D, 0);
for (auto& r : h) for (int d = 0; d < D; ++d) mu[d] += r[d];
for (int d = 0; d < D; ++d) mu[d] /= T;
for (auto& r : h) for (int d = 0; d < D; ++d) sd[d] += (r[d]-mu[d])*(r[d]-mu[d]);
for (int d = 0; d < D; ++d) sd[d] = std::sqrt(sd[d]/T) + 1e-6f;
std::vector<std::vector<float>> z(T, std::vector<float>(D));
for (int t = 0; t < T; ++t) for (int d = 0; d < D; ++d) z[t][d] = (h[t][d]-mu[d])/sd[d];
return z;
}
// ramp bank: L2 of the antisymmetric ramp-weighted sum over ±H, per scale.
// Matches ramp_bank() (np.convolve 'same' with reversed kernel; sign folds
// into the L2 norm so the direct antisymmetric sum is equivalent).
static std::vector<std::array<float,5>> ramp_bank(const std::vector<std::vector<float>>& z) {
const int T = z.size(), D = z[0].size();
std::vector<std::array<float,5>> out(T);
for (int k = 0; k < 5; ++k) {
const int H = kRampScales[k];
for (int t = 0; t < T; ++t) {
std::vector<double> acc(D, 0.0);
for (int l = -H; l <= H; ++l) {
int idx = t + l;
if (idx < 0 || idx >= T) continue;
double w = (l == 0) ? 0.0 : (l > 0 ? 1.0 : -1.0) * (double(std::abs(l))/H);
for (int d = 0; d < D; ++d) acc[d] += w * z[idx][d];
}
double n = 0; for (double v : acc) n += v*v;
out[t][k] = static_cast<float>(std::sqrt(n));
}
}
return out;
}
// Generic symmetric-delta + ramp + energy + debounce feature block for one
// modality's z-normable series `raw` (hist or PSD). Fills `out` columns
// [off .. off+width). corr=true adds the 3 per-channel corr features (video
// only); n_energy is 3 (video, per-channel) or 1 (audio, total).
static void modality_block(const std::vector<std::vector<float>>& raw,
bool corr, int n_energy,
std::vector<std::vector<float>>& out, int off) {
const int T = raw.size();
auto z = znorm(raw);
auto rb = ramp_bank(z);
auto sym = [&](int t, int k)->float{
int f = std::min(T-1, t+k), b = std::max(0, t-k);
return l1(raw[f], raw[b]);
};
const int B = kHistBins; // only used for corr (video)
for (int t = 0; t < T; ++t) {
int o = off;
for (int k : {1,2,4,8}) out[t][o++] = sym(t,k);
if (corr) {
int tp = std::max(0, t-1);
for (int c = 0; c < 3; ++c) {
double ma=0, mb=0;
for (int i=0;i<B;++i){ ma+=raw[t][c*B+i]; mb+=raw[tp][c*B+i]; }
ma/=B; mb/=B; double num=0, da=0, db=0;
for (int i=0;i<B;++i){ double x=raw[t][c*B+i]-ma, y=raw[tp][c*B+i]-mb;
num+=x*y; da+=x*x; db+=y*y; }
out[t][o++] = float(num/(std::sqrt(da*db)+1e-9));
}
}
for (int k=0;k<5;++k) out[t][o++] = rb[t][k];
if (n_energy == 3) {
for (int c=0;c<3;++c){ float e=0; for(int i=0;i<B;++i) e+=raw[t][c*B+i]; out[t][o++]=e; }
} else {
float e=0; for (float v : raw[t]) e+=v; out[t][o++]=e;
}
o += 2; // debounce filled below
}
// debounce from this block's delta-k1 (its first column = off)
std::vector<float> d1(T); for (int t=0;t<T;++t) d1[t]=out[t][off];
auto clk = debounce_phase(d1);
// debounce sits at the end of the block: off + 4(deltas) + (corr?3:0) + 5(ramp) + n_energy
int deb = off + 4 + (corr?3:0) + 5 + n_energy;
for (int t=0;t<T;++t){ out[t][deb]=clk[t].first; out[t][deb+1]=clk[t].second; }
}
// per-second base = video(17) + audio(12). Audio empty → its block is the
// zero-series result (deltas/ramp/energy 0, debounce phase=1/decay=exp(-1)).
static std::vector<std::vector<float>> build_base(const std::vector<std::vector<float>>& hist,
const std::vector<std::vector<float>>& audio) {
const int T = hist.size();
std::vector<std::vector<float>> base(T, std::vector<float>(29, 0.0f));
modality_block(hist, /*corr=*/true, /*n_energy=*/3, base, /*off=*/0); // video → 0..16
if (!audio.empty() && int(audio.size()) == T) {
modality_block(audio, /*corr=*/false, /*n_energy=*/1, base, /*off=*/17); // audio → 17..28
} else {
// zero-series audio: deltas/ramp/energy already 0; only debounce differs.
auto clk = debounce_phase(std::vector<float>(T, 0.0f));
for (int t=0;t<T;++t){ base[t][27]=clk[t].first; base[t][28]=clk[t].second; }
}
return base;
}
// matches debounce_phase(): 90th-pct peaks, dt=time since last, phase/decay.
static std::vector<std::pair<float,float>> debounce_phase(const std::vector<float>& sig) {
const int T = sig.size();
std::vector<float> s(sig); std::sort(s.begin(), s.end());
float thr = s[std::min(T-1, int(0.90*T))];
std::vector<std::pair<float,float>> out(T);
int last = -1000000000;
for (int t=0;t<T;++t){
if (sig[t] > thr) last = t;
double dt = (last < -100000000) ? kSigmaTau : double(t - last);
out[t] = { float(std::min(1.0, dt/kSigmaTau)), float(std::exp(-dt/kSigmaTau)) };
}
return out;
}
// window flatten (t-3..t+3, edge-pad) + append the 3-value film clock.
static std::vector<float> window_and_clock(const std::vector<std::vector<float>>& base,
const std::vector<std::vector<float>>& hist) {
const int T = base.size(), d = base[0].size(); // d=29
// film-level clock: time-since-last-peak on the |delta k1| video signal
// (base col 0), same as per_second_matrix's `clock`.
std::vector<float> sig(T); for (int t=0;t<T;++t) sig[t]=base[t][0];
std::vector<float> ss(sig); std::sort(ss.begin(), ss.end());
float thr = ss[std::min(T-1, int(0.90*T))];
std::vector<float> X; X.reserve(size_t(T)*206);
int last=-1000000000;
for (int t=0;t<T;++t){
for (int off=-kWin; off<=kWin; ++off){
int idx = std::min(T-1, std::max(0, t+off));
for (int j=0;j<d;++j) X.push_back(base[idx][j]);
}
if (sig[t] > thr) last=t;
double dt=(last<-100000000)?kSigmaTau:double(t-last);
X.push_back(float(dt));
X.push_back(float(std::min(1.0, dt/kSigmaTau)));
X.push_back(float(std::exp(-dt/kSigmaTau)));
}
return X;
}
std::vector<float> predict(const std::vector<float>& X, int rows, int cols) {
DMatrixHandle dm;
if (XGDMatrixCreateFromMat(X.data(), rows, cols, std::nanf(""), &dm) != 0)
throw std::runtime_error("XGDMatrixCreateFromMat failed");
bst_ulong out_len = 0; const float* out = nullptr;
if (XGBoosterPredict(booster_, dm, 0, 0, 0, &out_len, &out) != 0)
throw std::runtime_error("XGBoosterPredict failed");
std::vector<float> p(out, out + out_len);
XGDMatrixFree(dm);
for (auto& v : p) v = std::clamp(v, 0.f, 1.f);
return p;
}
// Exact replica of scipy.signal.find_peaks(x, distance=d):
// 1. local maxima (plateau-aware: rising then falling, midpoint of a flat top)
// 2. keep peaks by DESCENDING height; drop any within `d` of an already-kept
// taller peak. This is height-priority, NOT the greedy left-to-right merge
// — the two give different peak sets and hence a different knee.
static std::vector<int> find_peaks(const std::vector<float>& x, int d) {
const int n = x.size();
std::vector<int> mid;
int i = 1;
while (i < n-1) {
if (x[i-1] < x[i]) {
int ahead = i+1;
while (ahead < n-1 && x[ahead] == x[i]) ahead++;
if (x[ahead] < x[i]) mid.push_back((i + ahead - 1) / 2);
i = ahead;
} else i++;
}
// height-priority distance filter (scipy's _select_by_peak_distance)
std::vector<int> order(mid.size());
for (size_t k = 0; k < mid.size(); ++k) order[k] = k;
std::sort(order.begin(), order.end(),
[&](int a, int b){ return x[mid[a]] < x[mid[b]]; }); // ascending
std::vector<char> keep(mid.size(), 1);
for (int j = int(order.size())-1; j >= 0; --j) { // tallest first
int k = order[j];
if (!keep[k]) continue;
for (int l = k-1; l >= 0 && mid[k]-mid[l] < d; --l) keep[l] = 0;
for (int r = k+1; r < int(mid.size()) && mid[r]-mid[k] < d; ++r) keep[r] = 0;
}
std::vector<int> out;
for (size_t k = 0; k < mid.size(); ++k) if (keep[k]) out.push_back(mid[k]);
return out;
}
// knee threshold on peak heights → boundary timestamps (matches knee_boundaries).
static std::vector<double> knee_boundaries(const std::vector<float>& prob,
const std::vector<double>& ts,
int min_gap = 5) {
std::vector<int> pk = find_peaks(prob, min_gap);
if (pk.size() < 5) {
std::vector<double> r; for (int i : pk) r.push_back(ts[i]); return r;
}
std::vector<float> h; for (int i : pk) h.push_back(prob[i]);
std::sort(h.begin(), h.end(), std::greater<float>());
int n = h.size(); float h0 = h.front() + 1e-9f;
int kbest = 0; double dmax = -1;
for (int i = 0; i < n; ++i) {
double x = double(i)/(n-1);
double yv = h[i]/h0;
double chord = (h[0]/h0) + ((h[n-1]/h0)-(h[0]/h0))*x;
if (chord - yv > dmax) { dmax = chord - yv; kbest = i; }
}
float knee = h[kbest];
std::vector<double> out;
for (int i : pk) if (prob[i] >= knee) out.push_back(ts[i]);
return out;
}
};
src/main.cpp
+1
View File
@@ -209,6 +209,7 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--end")) cfg.end_sec = std::stod(next()); else if (arg("--end")) cfg.end_sec = std::stod(next());
else if (arg("--cut-threshold")) cfg.cut_threshold = std::stof(next()); else if (arg("--cut-threshold")) cfg.cut_threshold = std::stof(next());
else if (arg("--presence-mode")) { std::string m = next(); cfg.presence_mode = (m == "flood") ? PresenceMode::flood : PresenceMode::track_extent; } else if (arg("--presence-mode")) { std::string m = next(); cfg.presence_mode = (m == "flood") ? PresenceMode::flood : PresenceMode::track_extent; }
else if (arg("--scene-xgb-model")) cfg.scene_xgb_model = next();
else if (arg("--scene-detect")) cfg.scene_detect = true; else if (arg("--scene-detect")) cfg.scene_detect = true;
else if (arg("--scene-detector")) cfg.scene_model = next(); else if (arg("--scene-detector")) cfg.scene_model = next();
else if (arg("--scene-detector-engine")) cfg.scene_engine = next(); else if (arg("--scene-detector-engine")) cfg.scene_engine = next();
src/nodes/camera_position_change_detector_node.hpp
+23 -1
View File
@@ -33,9 +33,29 @@ struct CameraPositionChangeDetectorFunc {
explicit CameraPositionChangeDetectorFunc(const Config& cfg) explicit CameraPositionChangeDetectorFunc(const Config& cfg)
: cut_threshold_(cfg.cut_threshold) : cut_threshold_(cfg.cut_threshold)
, want_rgb_hist_(!cfg.scene_xgb_model.empty())
{ {
std::cerr << "[camera_position_change_detector] cut_threshold=" std::cerr << "[camera_position_change_detector] cut_threshold="
<< cut_threshold_ << "\n"; << cut_threshold_
<< (want_rgb_hist_ ? " (+rgb_hist for scene detector)" : "")
<< "\n";
}
// 32-bin-per-channel normalised RGB histogram (96 floats), the exact layout
// the XGBoost scene detector was trained on (see embedding_dump_node). Only
// computed when a scene model is configured, so it costs nothing otherwise.
static std::vector<float> rgb_histogram(const cv::Mat& img) {
constexpr int kBins = 32;
std::vector<float> out(kBins * 3, 0.f);
if (img.empty() || img.channels() != 3) return out;
float range[] = {0.f, 256.f}; const float* ranges = range; int bins = kBins;
for (int c = 0; c < 3; ++c) { // OpenCV BGR → store B,G,R blocks
cv::Mat h;
cv::calcHist(&img, 1, &c, cv::Mat(), h, 1, &bins, &ranges);
cv::normalize(h, h, 1.0, 0.0, cv::NORM_L1);
for (int b = 0; b < kBins; ++b) out[c*kBins + b] = h.at<float>(b);
}
return out;
} }
Frame operator()(Frame f) { Frame operator()(Frame f) {
@@ -64,11 +84,13 @@ struct CameraPositionChangeDetectorFunc {
prev_hist_ = hist; prev_hist_ = hist;
prev_hist_valid_ = true; prev_hist_valid_ = true;
if (want_rgb_hist_) f.rgb_hist = rgb_histogram(f.image);
return f; return f;
} }
private: private:
float cut_threshold_; float cut_threshold_;
bool want_rgb_hist_{false};
cv::Mat prev_hist_; cv::Mat prev_hist_;
bool prev_hist_valid_{false}; bool prev_hist_valid_{false};
}; };
src/nodes/frame_annotation_node.hpp
+1
View File
@@ -43,6 +43,7 @@ struct FrameAnnotationFunc {
sa.visible_actors = std::move(mf.actors); sa.visible_actors = std::move(mf.actors);
sa.is_cut = mf.source.is_cut; sa.is_cut = mf.source.is_cut;
sa.is_scene_boundary = mf.source.is_scene_boundary; sa.is_scene_boundary = mf.source.is_scene_boundary;
sa.rgb_hist = std::move(mf.source.rgb_hist);
return sa; return sa;
} }
}; };
src/nodes/result_sink_node.hpp
+52 -3
View File
@@ -3,6 +3,10 @@
#include "types.hpp" #include "types.hpp"
#include "config.hpp" #include "config.hpp"
#include "track_registry.hpp" #include "track_registry.hpp"
#ifdef SAE_SCENE_XGB
#include "inference/xgb_scene_boundary.hpp"
#include "inference/audio_logpsd.hpp"
#endif
#include <nlohmann/json.hpp> #include <nlohmann/json.hpp>
#include <algorithm> #include <algorithm>
@@ -219,22 +223,67 @@ private:
// flood-fill actually snaps to — coarser than true shot boundaries (cuts also // flood-fill actually snaps to — coarser than true shot boundaries (cuts also
// fire on in-shot angle changes) but present with no extra pass. // fire on in-shot angle changes) but present with no extra pass.
std::vector<double> scene_boundaries() const { std::vector<double> scene_boundaries() const {
std::vector<double> b;
b.push_back(0.0);
// Preferred: the learned XGBoost scene detector, run once here post-EOF
// (the knee threshold needs the whole film, so this is inherently a final
// step — like flood-fill itself). Measured best flood boundary source.
std::vector<double> learned = xgb_boundaries();
if (!learned.empty()) {
for (double t : learned) b.push_back(t);
} else {
// Fallback: TransNetV2 shot boundaries if present, else histogram cuts.
bool have_scene = false; bool have_scene = false;
for (const auto& sa : frames_) for (const auto& sa : frames_)
if (sa.is_scene_boundary) { have_scene = true; break; } if (sa.is_scene_boundary) { have_scene = true; break; }
std::vector<double> b;
b.push_back(0.0);
for (const auto& sa : frames_) { for (const auto& sa : frames_) {
const bool boundary = have_scene ? sa.is_scene_boundary : sa.is_cut; const bool boundary = have_scene ? sa.is_scene_boundary : sa.is_cut;
if (boundary) b.push_back(sa.timestamp_sec); if (boundary) b.push_back(sa.timestamp_sec);
} }
}
b.push_back(last_ts_ + 1.0); // a right edge past the final sample b.push_back(last_ts_ + 1.0); // a right edge past the final sample
std::sort(b.begin(), b.end()); std::sort(b.begin(), b.end());
b.erase(std::unique(b.begin(), b.end()), b.end()); b.erase(std::unique(b.begin(), b.end()), b.end());
return b; return b;
} }
// Run the learned scene-boundary detector over the collected per-frame RGB
// histograms + per-second audio log-PSD (decoded once from the movie). Returns
// {} when no model is configured, the build lacks XGBoost, or no rgb_hist was
// stamped (camera-position node only does so when a model is set).
std::vector<double> xgb_boundaries() const {
#ifdef SAE_SCENE_XGB
if (cfg_.scene_xgb_model.empty()) return {};
std::vector<std::vector<float>> hist;
std::vector<double> ts;
hist.reserve(frames_.size()); ts.reserve(frames_.size());
for (const auto& sa : frames_) {
if (sa.rgb_hist.empty()) return {}; // hist not stamped → bail to fallback
hist.push_back(sa.rgb_hist);
ts.push_back(sa.timestamp_sec);
}
if (hist.size() < 16) return {};
try {
auto audio = AudioLogPSD::extract(cfg_.movie_path); // [T'][B], aligned per second
if ((int)audio.size() != (int)hist.size())
audio.resize(hist.size(),
std::vector<float>(audio.empty() ? 57 : audio[0].size(), 0.f));
XGBSceneBoundary det(cfg_.scene_xgb_model);
auto b = det.boundaries(hist, ts, audio);
std::cerr << "[result_sink] XGBoost scene detector: " << b.size()
<< " boundaries\n";
return b;
} catch (const std::exception& e) {
std::cerr << "[result_sink] scene detector failed (" << e.what()
<< "), falling back to histogram cuts\n";
return {};
}
#else
return {};
#endif
}
// The boundary opening the shot that contains t (largest boundary ≤ t). // The boundary opening the shot that contains t (largest boundary ≤ t).
static double boundary_at_or_before(const std::vector<double>& b, double t) { static double boundary_at_or_before(const std::vector<double>& b, double t) {
auto it = std::upper_bound(b.begin(), b.end(), t); auto it = std::upper_bound(b.begin(), b.end(), t);
src/tools/scene_features_dump.cpp
+58
View File
@@ -0,0 +1,58 @@
// scene_features_dump — write the C++ scene-boundary feature matrix to HDF5, so
// the XGBoost model is TRAINED on exactly the features the C++ detector produces
// at inference (parity by construction — no numpy re-implementation to keep in
// sync). Reads frames/rgb_hist + frames/timestamp_sec from a dump and, given the
// movie, the per-second audio log-PSD; writes features [T,206] + timestamps.
//
// scene_features_dump <dump.h5> <movie> <out_features.h5>
//
// The py3.12 venv trainer (train_xgb_cpp.py) reads <out_features.h5>, attaches
// the soft Gaussian boundary target, fits XGBoost, and saves the model that
// XGBSceneBoundary loads. Same C++ features both sides → exact parity.
#include "inference/xgb_scene_boundary.hpp"
#include "inference/audio_logpsd.hpp"
#include <H5Cpp.h>
#include <iostream>
#include <vector>
int main(int argc, char** argv) {
if (argc < 4) {
std::cerr << "usage: scene_features_dump <dump.h5> <movie> <out.h5>\n";
return 1;
}
H5::H5File in(argv[1], H5F_ACC_RDONLY);
H5::DataSet hd = in.openDataSet("frames/rgb_hist");
hsize_t hdims[2]; hd.getSpace().getSimpleExtentDims(hdims);
std::vector<float> flat(hdims[0]*hdims[1]);
hd.read(flat.data(), H5::PredType::NATIVE_FLOAT);
const int T = hdims[0], C = hdims[1];
std::vector<std::vector<float>> hist(T, std::vector<float>(C));
for (int t = 0; t < T; ++t)
for (int c = 0; c < C; ++c) hist[t][c] = flat[t*C+c];
H5::DataSet td = in.openDataSet("frames/timestamp_sec");
hsize_t tdim[1]; td.getSpace().getSimpleExtentDims(tdim);
std::vector<double> ts(tdim[0]);
td.read(ts.data(), H5::PredType::NATIVE_DOUBLE);
auto audio = AudioLogPSD::extract(argv[2]);
if ((int)audio.size() != T) {
std::cerr << "[features] audio rows " << audio.size() << " != hist rows "
<< T << " — aligning (pad/truncate)\n";
audio.resize(T, std::vector<float>(audio.empty()?57:audio[0].size(), 0.f));
}
std::vector<float> X = XGBSceneBoundary::feature_matrix(hist, audio);
const int F = XGBSceneBoundary::kNFeatures;
H5::H5File out(argv[3], H5F_ACC_TRUNC);
hsize_t xd[2] = {(hsize_t)T, (hsize_t)F};
out.createDataSet("features", H5::PredType::NATIVE_FLOAT, H5::DataSpace(2, xd))
.write(X.data(), H5::PredType::NATIVE_FLOAT);
hsize_t td2[1] = {(hsize_t)T};
out.createDataSet("timestamp_sec", H5::PredType::NATIVE_DOUBLE, H5::DataSpace(1, td2))
.write(ts.data(), H5::PredType::NATIVE_DOUBLE);
std::cerr << "[features] wrote [" << T << "," << F << "] → " << argv[3] << "\n";
return 0;
}
src/tools/xgb_boundary_parity.cpp
+82
View File
@@ -0,0 +1,82 @@
// Parity harness: run the C++ XGBSceneBoundary on a dump's frames/rgb_hist and
// print the boundary timestamps, so they can be diffed against the Python
// knee_boundaries (scripts/scene_detector). Feature parity is the whole risk of
// the C++ port; this proves it before wiring into the pipeline.
//
// xgb_boundary_parity <dump.h5> <model.json>
//
// Prints: "<n> boundaries: t0 t1 t2 ..."
#include "inference/xgb_scene_boundary.hpp"
#include "inference/audio_logpsd.hpp"
#include <H5Cpp.h>
#include <iostream>
#include <fstream>
#include <string>
#include <vector>
int main(int argc, char** argv) {
if (argc < 3) { std::cerr << "usage: xgb_boundary_parity <dump.h5> <model.json>\n"; return 1; }
H5::H5File f(argv[1], H5F_ACC_RDONLY);
auto read2d = [&](const char* name, std::vector<std::vector<float>>& out, int cols) {
H5::DataSet ds = f.openDataSet(name);
H5::DataSpace sp = ds.getSpace();
hsize_t dims[2]; sp.getSimpleExtentDims(dims);
std::vector<float> flat(dims[0]*dims[1]);
ds.read(flat.data(), H5::PredType::NATIVE_FLOAT);
out.assign(dims[0], std::vector<float>(cols));
for (hsize_t i = 0; i < dims[0]; ++i)
for (int j = 0; j < cols; ++j) out[i][j] = flat[i*dims[1]+j];
};
std::vector<std::vector<float>> hist;
read2d("frames/rgb_hist", hist, XGBSceneBoundary::kHistBins*3);
H5::DataSet tsd = f.openDataSet("frames/timestamp_sec");
hsize_t td[1]; tsd.getSpace().getSimpleExtentDims(td);
std::vector<double> ts(td[0]);
tsd.read(ts.data(), H5::PredType::NATIVE_DOUBLE);
// parity debug: print video features for row 100 (compare to Python)
if (argc > 3 && std::string(argv[3]) == "--row100") {
auto base = XGBSceneBoundary::debug_base(hist, {});
std::cout << "row100:";
for (int j = 0; j < 17; ++j) std::cout << " " << base[100][j];
std::cout << "\n";
return 0;
}
// --feat <cpp_features.h5>: predict directly on the dumped C++ feature matrix
// (same bytes Python reads) — a clean parity check with no live-decode variance.
if (argc > 4 && std::string(argv[3]) == "--feat") {
H5::H5File ff(argv[4], H5F_ACC_RDONLY);
H5::DataSet fd = ff.openDataSet("features");
hsize_t fdm[2]; fd.getSpace().getSimpleExtentDims(fdm);
std::vector<float> X(fdm[0]*fdm[1]);
fd.read(X.data(), H5::PredType::NATIVE_FLOAT);
XGBSceneBoundary det(argv[2]);
auto pdbg = det.debug_predict(X, int(fdm[0]), int(fdm[1]));
auto pk = XGBSceneBoundary::debug_find_peaks(pdbg, 5);
std::cerr << "[parity] C++ raw peaks=" << pk.size() << "\n";
auto b = det.boundaries_from_features(X, int(fdm[0]), int(fdm[1]), ts);
std::cout << b.size() << " boundaries:";
for (double t : b) std::cout << " " << int(t);
std::cout << "\n";
return 0;
}
// Optional movie path (argv[4]): decode audio → per-second log-PSD.
std::vector<std::vector<float>> audio;
if (argc > 4) {
audio = AudioLogPSD::extract(argv[4]);
std::cerr << "[parity] audio rows=" << audio.size()
<< " (hist rows=" << hist.size() << ")\n";
}
XGBSceneBoundary det(argv[2]);
auto b = det.boundaries(hist, ts, audio);
std::cout << b.size() << " boundaries:";
for (double t : b) std::cout << " " << int(t);
std::cout << "\n";
return 0;
}
src/types.hpp
+7
View File
@@ -31,6 +31,10 @@ struct Frame {
float cut_score{0.f}; // histogram cut score = 1 - hist_corr (0=identical, ~1=cut); HUD/debug float cut_score{0.f}; // histogram cut score = 1 - hist_corr (0=identical, ~1=cut); HUD/debug
float bbox_upscale{1.f}; // multiply detector bboxes/landmarks by this to map back to float bbox_upscale{1.f}; // multiply detector bboxes/landmarks by this to map back to
// original video resolution (>1 when dense_scale downscaled the frame) // original video resolution (>1 when dense_scale downscaled the frame)
// Normalised 32-bin-per-channel RGB histogram (96 floats), stamped by the
// camera-position node and carried to the sink for the learned scene-boundary
// detector (post-EOF, flood-fill boundaries). Empty when scene detection off.
std::vector<float> rgb_hist;
}; };
// ── CutEvent ────────────────────────────────────────────────────────────────── // ── CutEvent ──────────────────────────────────────────────────────────────────
@@ -162,6 +166,9 @@ struct SceneAnnotation {
// detection ran); kept for a future out-of-process scene detector. // detection ran); kept for a future out-of-process scene detector.
bool is_cut{false}; bool is_cut{false};
bool is_scene_boundary{false}; bool is_scene_boundary{false};
// Per-frame RGB histogram, carried to the sink for the learned scene-boundary
// detector run post-EOF (flood-fill). Empty unless scene detection is enabled.
std::vector<float> rgb_hist;
}; };
// ── Actor gallery ───────────────────────────────────────────────────────────── // ── Actor gallery ─────────────────────────────────────────────────────────────