dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dtourolle/scene-actor-extraction:e1423062e260d6a393affc0c2ffe258dec7ce612
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
...
compare: dtourolle/scene-actor-extraction:feature/opencv5
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
13 Commits
e1423062e2 ... feature/opencv5
Author SHA1 Message Date
dtourolle 13437e0d8b docs: correct scene-boundary detector F1 numbers to measured values 
The boundary-detection paragraph understated the detector. Replace the stale
"~34% F1 vs ~27%" (a pre-C++-retrain figure with no backing artifact) with the
measured numbers at the shipped ±20s tolerance:
  - leave-one-out macro boundary F1 = 44.1% (honest generalisation)
  - grayscale baseline               = 29.8%
  - train-all (shipped model)        = 72.9% (per-film 51-86%)
computed from experiments/results/scene_boundary/xgb_report.json and per-film
leave-one-out runs of train_xgb_cpp.py. Also correct the false claim that the
low-contrast grades "cannot generalise held out" — Scarface held out scores 32%,
Café Society 51%, both above grayscale (0% and 31%).

Split the evolution figure into two panels so the strict-±2s feature-development
curve is no longer mistaken for the shipped result: left = feature progress at
±2s, right = shipped detector at the ±20s tolerance the pipeline uses.
 2026-08-11 20:54:15 +02:00
dtourolle b26c66dcce scripts: figure generators + frame-regen script for the opencv5 report 
make_figures.py gains the DE-landscape, calibration, and per-film leave-one-out
holdout figures used by the experiment log. regen_frame_examples.sh replays all
nine films with the shipped learned-boundary flood config and draws GT-aware
TP/FP/FN frames, so every annotated image in the docs is reproducible.
 2026-08-10 08:45:07 +02:00
dtourolle 7556c836da docs: opencv5 experiment log + rewritten Home 
Add model-bakeoff.md for the opencv5 build: the ten-knob DE tuning and where
each shipped config default comes from, the replay architecture, and the
flood-fill-on-learned-boundaries step change (62.6% -> 74.9% presence F1, LOO).
Rewrite index.md to lead with the learned scene-boundary result and point at
the current pages, with the July four-model bake-off moved to an Archive
section. Both pages build with no broken links.
 2026-08-10 08:41:54 +02:00
dtourolle ef99951360 docs: remake all named TP/FP/FN frames against the opencv5 pipeline 
Auto-match each named July frame by film+actor+class and re-extract it from
the current learned-boundary flood replay, drawing GT-aware boxes: green TP,
red FP, orange unknown, plus the blue off-screen/missed (X-Ray cast, no face)
FN panel. Adds rematch_frames.py, the tool that does the matching.

Zooey Deschanel is dropped: the current pipeline no longer makes that
false identification, so the frame is removed and the July deep-dive notes
the fix rather than showing a stale error.
 2026-08-10 08:32:30 +02:00
dtourolle 0feafec7c9 feat(review): GT-aware TP/FP/FN frame annotation + scene-detector examples 
dump_error_frames.py drew every identified box green, so a false positive looked
like a true positive and a missed cast member was invisible. Make the annotation
ground-truth aware, matching what the per-second scorer classifies:
  - GREEN  true positive  — a name X-Ray also credits to this scene
  - RED    false positive — a name X-Ray does NOT credit here (the real error)
  - ORANGE unknown detection
  - BLUE   a text panel listing X-Ray cast present with no detected face (the
           structural false-negatives — no box exists to draw)

Add two representative annotated frames to the scene-detector page: a clean
green-TP second, and the face-vs-scene-cast case (a red FP lead + six off-camera
cast in blue) that makes the recall ceiling visual. Frames are generated by the
script from replay.py --raw-out output; the two committed examples are hand-picked
doc assets (bulk experiments/dump_review is regenerable and gitignored).
 2026-08-09 22:28:21 +02:00
dtourolle edf19ab798 docs(scene-detector): document the learned scene-boundary detector 
New docs/scene-boundary-detector.md: why the grayscale cut detector wasn't
enough (Scarface: 1 cut in 10k frames → flood-fill P=26%), what X-Ray boundaries
are and why they're hard, the feature/model design (delta histograms, multi-scale
ramp bank, scene-length debounce, soft-target XGBoost regressor, per-film knee),
and the measured dead ends (audio-only, raw features, LSTM, TransNetV2).

Headline result, honest leave-one-out (each film scored by a detector trained on
the other eight): flood + learned detector = 74.9% macro presence F1, vs 64.0%
for grayscale-cut flood and 62.6% for track-extent — +12.3pp, improving all nine
films. Fixes the Scarface flood collapse (grayscale 40.9 → learned 74.9, on a
film the detector never trained on) and swings Downton +37pp.

Figures are generated by scripts/scene_detector/make_figures.py from the saved
results (experiments/results/scene_boundary/downstream_loo.json); the PNGs
themselves follow the repo convention of not committing regenerable chart assets.
Added to the mkdocs nav.
 2026-08-09 22:15:32 +02:00
dtourolle e5204a831a feat(scene-detector): run the learned boundary detector live in the C++ pipeline 
Wire the XGBoost scene-boundary detector into scene_analyze as a post-EOF step in
the result sink (like flood-fill itself — the per-film knee threshold needs the
whole film, so it cannot stream). With --scene-xgb-model set, the camera-position
node stamps a per-frame RGB histogram onto the Frame, it rides through to the
sink, and at EOF the sink runs XGBSceneBoundary over the collected histograms +
the movie's per-second audio log-PSD to produce the flood-fill boundaries. Falls
back to is_scene_boundary / is_cut when no model is configured or inference fails.

Inference is real XGBoost via CMake FetchContent (v2.1.1, static), C API in
src/inference/xgb_scene_boundary.hpp; audio log-PSD in src/inference/
audio_logpsd.hpp (FFTW + ffmpeg full-file 16kHz decode). Feature extraction
matches training exactly — video features verified row-identical to numpy, and to
avoid chasing numpy's every rounding the shipped model is TRAINED on the
C++-extracted features (scene_features_dump exe → train_xgb_cpp.py). The
C++/Python peak-finders differ slightly so boundary counts differ, but what
matters is downstream: flood + C++ detector = 75.8% macro presence F1 vs 64.0%
for the histogram-cut flood and 62.5% for track_extent, and it fixes the Scarface
flood collapse (41 -> 70). All nine films improve.

Guarded by the SAE_SCENE_XGB CMake option (on by default; heavy first build).
xgb_boundary_parity is a diff harness; scene_features_dump writes the C++ feature
matrix so training and inference share one feature implementation.

Verified end to end: scene_analyze --scene-xgb-model on a real movie stamps the
histogram, runs the detector at EOF ("XGBoost scene detector: N boundaries"), and
flood-snaps presence to the learned boundaries.
 2026-08-09 21:21:29 +02:00
dtourolle 0e35dac951 feat(scene-detector): learned scene-boundary detector for flood-fill presence 
A boosted-tree scene-boundary detector that replaces the grayscale
histogram-correlation cut detector as the flood-fill boundary source, and
substantially improves actor-presence accuracy.

Downstream result (per-second X-Ray presence F1, macro over 9 films):
  track_extent 62.3%  |  flood + histogram cuts 64.0%  |  flood + this 76.9%
+12.9pp, and it wins on every film — notably fixing the histogram flood's
Scarface collapse (61 -> 41 -> 71) and lifting Downton 41 -> 84.

Design (each choice measured — see the memory / report):
- XGBoost REGRESSOR on a ±3s window of DELTA features (symmetric RGB-hist
  and audio-PSD deltas at k=1,2,4,8s + ramp bank + time-since-last-peak
  debounce). Raw histograms dilute; deltas separate boundaries ~4-5x.
- SOFT Gaussian proximity target (sigma=10s) so near-misses train as
  near-correct, not hard negatives; regression -> smooth score -> NMS peaks.
- KNEE per-film threshold: self-calibrates the boundary count to ~the true
  scene count, no global rate. Evaluated at ±20s (X-Ray scenes ~170s).
- Trained on all 9 films (Cafe/Scarface low-contrast grades must be seen).
  Honest held-out ~41% boundary-F1 @±20s vs ~27% grayscale.

Scripts: train_xgb_boundary.py (shipped detector), extract_audio_features.py
(per-second log-PSD), downstream_presence.py (the A/B above), density_floor.py
(fallback for detection-starved films), plus the LSTM/DE explorations kept
for provenance. Model: models/scene_boundary_xgb.json.

Not yet wired into the live C++ pipeline — boundaries are a post-EOF step in
the sink (like flood-fill itself); libxgboost C++ integration is the next step.
 2026-08-09 19:21:04 +02:00
dtourolle 8dd2255125 feat(dump): record a per-frame RGB histogram for scene-boundary training 
Add frames/rgb_hist to the embedding dump: a normalised 32-bin-per-channel
RGB histogram (96 floats/frame), computed from the already-decoded frame so
it is nearly free and ~40 KB per film. This is the training signal for the
learned scene-boundary detector — the grayscale-correlation cut detector is
blind on low-contrast grades (Scarface: 1 cut in 10k frames), and the
symmetric RGB-histogram delta separates X-Ray scene boundaries far better.
The dump stays gallery-independent; downstream replay/training consume the
histogram offline.
 2026-08-09 19:20:34 +02:00
dtourolle 0e97e532a8 feat(config): ship the 10-knob DE optimum (flood-fill default) 
Update the tuned presence defaults to the 10-parameter DE optimum over all
9 X-Ray films (opencv5 build, LVFace-B): prob_threshold 0.485,
ownership_logodds 1.72, track_extinction_sec 31, track_alpha 0.435,
evidence_rho_max 0.204, evidence_admit_below 0.784, match_prior 0.433,
expand_band 0.804/0.952, and PresenceMode::flood as the default. Replaces
the earlier 0.754 (a 4-film subset optimum under the withdrawn
anneal/extinction windows, never re-derived). The provenance comment is
rewritten to record the sweep, the permissive-threshold rationale, and the
uneven-generalization caveat (strong on 7 of 9 films; Many Saints /
Scarface remain hard).

Note on flood: presence F1 is 64% with the always-on histogram cut as the
flood boundary source, but 77% once a proper learned scene detector
supplies the boundaries (see the scene-detector work). Flood is the right
default; the boundary source is what unlocks it.
 2026-08-09 19:20:21 +02:00
dtourolle 1477c53885 refactor(registry): only keep identified tracks associable while dormant 
candidates() now excludes dormant (off-screen) tracks that were never
identified: an unowned dormant track has no actor to re-attach to, so
keeping it in the association pool only enlarges the matcher's per-frame
comparison set and invites a new face re-associating onto an anonymous
stub. On-screen tracks are always candidates; a dormant track must have
crossed ownership (t.actor set) to stay associable within
track_extinction_sec.

Measured trade: a small recall cost (~1-3pp on some films, e.g. Lord of
War -2.5) for a cleaner, bounded pool. Kept deliberately. Note it does NOT
fix the ROCm GEMM wedge (that is an intermittent driver flake, not a
pool-size problem) — this is a correctness/cleanliness change, not a
performance one.
 2026-08-09 19:20:06 +02:00
dtourolle add7e22053 fix(optimizer): discard replay stderr and make the timeout a wedge-backstop 
Two failures surfaced scaling the DE sweep up. The replay sink prints a
per-second progress line with an explicit flush; under
subprocess.run(capture_output=True) those thousands of writes fill a fixed
OS pipe buffer that nothing drains until exit, so the long films blocked on
write to stderr and looked like hangs. Discard the child's stdout/stderr
(DEVNULL) — it was captured and thrown away anyway; the long films then
finish in seconds.

Separately, the per-film timeout is only a backstop for the rare,
intermittent ROCm GEMM wedge (a wedged replay hangs forever and must be
killed so the sweep continues), not a performance bound. It had been set
huge, which let a single flake stall the whole sweep; set it to a sane 180s
(overridable via REPLAY_TIMEOUT) — well above a healthy replay, short
enough to reap a wedge quickly.
 2026-08-09 19:19:54 +02:00
dtourolle ea922356f1 docs: archive the July 2026 report; new methodology for the opencv5 run 
The July report (4-model ArcFace/LVFace bake-off, pre-opencv5 framework,
3-film training + held-out validation) is superseded by the opencv5 build:
single-model LVFace-B, a 6-knob DE sweep over all 9 films, flood-fill
presence, and the registry/decode fixes. Rather than overwrite it, archive
it date-suffixed and start the current report fresh.

- Rename the six July result pages to *-2026-07.md, rewrite their
  intra-archive cross-links, and add an "Archived (July 2026)" banner to each.
- mkdocs nav: current report at top, the July set under an Archive section.
- New docs/methodology.md for the opencv5 run: corrects the withdrawn
  anneal_sec/extinction_sec presence bridging (windows are now
  [first_seen, last_seen], AR-012/013), documents the two presence modes
  (track_extent / flood), and records that every eval scores all 9 films.

The current experiment log (model-bakeoff.md) and Home rewrite land once
the DE sweep converges and the final optimum is known.
 2026-08-09 10:48:45 +02:00
58 changed files with 3273 additions and 522 deletions
Expand all files Collapse all files
.gitignore
+3
View File
@@ -118,3 +118,6 @@ venv/
*.swo
.DS_Store
Thumbs.db
.venv-rocm/
!models/scene_boundary_xgb.json
experiments/dump_review/
CMakeLists.txt
+48 -3
View File
@@ -280,6 +280,24 @@ FetchContent_Declare(
)
FetchContent_MakeAvailable(nanobind)
# XGBoost (learned scene-boundary detector for flood-fill presence). Fetched and
# built from source so we get both the C API header and a matching libxgboost,
# reproducibly — the pip wheel ships the .so but no header. Heavy first build, so
# it is opt-in; the scene-boundary node is compiled only when SAE_SCENE_XGB is on.
option(SAE_SCENE_XGB "Build the XGBoost scene-boundary detector node" ON)
if(SAE_SCENE_XGB)
set(BUILD_STATIC_LIB ON CACHE BOOL "" FORCE) # link xgboost statically
set(USE_OPENMP ON CACHE BOOL "" FORCE)
FetchContent_Declare(
xgboost
GIT_REPOSITORY https://github.com/dmlc/xgboost.git
GIT_TAG v2.1.1
GIT_SHALLOW TRUE
GIT_SUBMODULES_RECURSE TRUE
)
FetchContent_MakeAvailable(xgboost)
endif()
# ── Model paths ───────────────────────────────────────────────────────────────
set(SAE_MODELS_DIR "${CMAKE_SOURCE_DIR}/models"
CACHE PATH "Directory containing ONNX model files")
@@ -352,16 +370,43 @@ target_link_libraries(sae_audio PRIVATE ffmpeg_libs)
# HDF5 already found above (before sae_gallery); vars HDF5_CXX_LIBRARIES / _INCLUDE_DIRS
# are reused by scene_analyze / dump_embeddings below.
# The learned scene-boundary detector is compiled into the sink (result_sink →
# xgb_scene_boundary + audio_logpsd) when SAE_SCENE_XGB is on, so the analysis
# binaries need xgboost + FFTW + ffmpeg and the define. Found once here.
if(SAE_SCENE_XGB)
find_library(FFTW3_LIB fftw3 REQUIRED)
set(SAE_SCENE_LIBS xgboost ${FFTW3_LIB} ffmpeg_libs)
set(SAE_SCENE_DEFS SAE_SCENE_XGB)
else()
set(SAE_SCENE_LIBS "")
set(SAE_SCENE_DEFS "")
endif()
# ── analyze — main analysis binary ───────────────────────────────────────────
add_executable(scene_analyze src/main.cpp)
target_link_libraries(scene_analyze PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES})
target_link_libraries(scene_analyze PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES} ${SAE_SCENE_LIBS})
target_include_directories(scene_analyze PRIVATE ${HDF5_INCLUDE_DIRS})
target_compile_definitions(scene_analyze PRIVATE ${SAE_SCENE_DEFS})
# ── xgb_boundary_parity — prove C++ scene-boundary inference matches Python ───
if(SAE_SCENE_XGB)
add_executable(xgb_boundary_parity src/tools/xgb_boundary_parity.cpp)
target_include_directories(xgb_boundary_parity PRIVATE src ${HDF5_INCLUDE_DIRS})
target_link_libraries(xgb_boundary_parity PRIVATE
xgboost ${HDF5_CXX_LIBRARIES} ${FFTW3_LIB} ffmpeg_libs)
# Dumps the C++ feature matrix so training uses the exact inference features.
add_executable(scene_features_dump src/tools/scene_features_dump.cpp)
target_include_directories(scene_features_dump PRIVATE src ${HDF5_INCLUDE_DIRS})
target_link_libraries(scene_features_dump PRIVATE
xgboost ${HDF5_CXX_LIBRARIES} ${FFTW3_LIB} ffmpeg_libs)
endif()
# ── analyze_debug — same binary with debug frame/crop output ─────────────────
add_executable(scene_analyze_debug src/main.cpp)
target_link_libraries(scene_analyze_debug PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES})
target_link_libraries(scene_analyze_debug PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES} ${SAE_SCENE_LIBS})
target_include_directories(scene_analyze_debug PRIVATE ${HDF5_INCLUDE_DIRS})
target_compile_definitions(scene_analyze_debug PRIVATE SAE_DEBUG=1)
target_compile_definitions(scene_analyze_debug PRIVATE SAE_DEBUG=1 ${SAE_SCENE_DEFS})
# ── dump_embeddings — standalone embedding dumper, NO gallery/matcher ─────────
# Front-half only (decode→detect→align→embed→HDF5) for the optimizer replay corpus
docs/assets/images/cafe_society_rapid_cut.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 404 KiB

docs/assets/images/downton_crew_fn.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 367 KiB

docs/assets/images/downton_tp_example.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 367 KiB

docs/assets/images/downton_wedding_couple.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 367 KiB

docs/assets/images/lord_of_war_fpi_reagan_photo.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 218 KiB

docs/assets/images/lord_of_war_fpi_reddick.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 252 KiB

docs/assets/images/lord_of_war_fpi_shumbris.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 221 KiB

docs/assets/images/lovelace_fpi_sevigny.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 180 KiB

docs/assets/images/lovelace_perfect_second.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 239 KiB

After

Width:  |  Height:  |  Size: 268 KiB

docs/assets/images/lovelace_polygraph_bridged.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 268 KiB

docs/assets/images/lovelace_robert_patrick_fpi.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 226 KiB

docs/assets/images/many_saints_fpi_gardner.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 156 KiB

docs/assets/images/many_saints_fpi_yates.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 180 KiB

docs/assets/images/many_saints_outofcast_fpi.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 113 KiB

docs/assets/images/scarface_fn_fp_example.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 47 KiB

docs/assets/images/scarface_fpi_alley.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

docs/assets/images/scarface_tp_example.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 47 KiB

docs/assets/images/scene_detector_evolution.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

docs/assets/images/valerian_screen_call.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 165 KiB

After

Width:  |  Height:  |  Size: 447 KiB

docs/best-model.md → docs/best-model-2026-07.md
+5 -3
View File
@@ -1,10 +1,12 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Which embedding model is best?
Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
455MB) were compared. r50 is excluded from the training/held-out comparison
below; its gallery has roughly 30% fewer reference images per actor than the
other three on the identical source photos, which confounds a direct score
comparison (see [the full experiment log](model-bakeoff.md) for detail). It
comparison (see [the full experiment log](model-bakeoff-2026-07.md) for detail). It
remains in the calibration comparison, which does not depend on the gallery
image count.
@@ -63,7 +65,7 @@ than general performance. On training data, the ordering is not as clean:
mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
table where LVFace does not score highest. LVFace's training-set macro
average (75.3%, see [the full experiment log](model-bakeoff.md)) is not a
average (75.3%, see [the full experiment log](model-bakeoff-2026-07.md)) is not a
uniform win across every film it contributes to; the held-out result, where
LVFace wins all 5 films outright, is the stronger claim.
@@ -79,7 +81,7 @@ not.
![All 12 combos ranked by training-set F1](assets/images/rep4_matrix_f1.png)
Best full-gallery combo per model (all three are `full_exp`), from the
training matrix in [the full experiment log](model-bakeoff.md):
training matrix in [the full experiment log](model-bakeoff-2026-07.md):
| model | F1 | P | R | misID |
|---|---|---|---|---|
docs/gallery-scope.md → docs/gallery-scope-2026-07.md
+5 -3
View File
@@ -1,3 +1,5 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Whole gallery vs. cast-restricted gallery
Two ways to run the matcher. Full mode scores every detected face against
@@ -8,7 +10,7 @@ top-billed actors) before the matcher runs.
## Result
Averaged across the 3 compared models (r50 excluded, see
[the full experiment log](model-bakeoff.md)) and both expansion settings, on
[the full experiment log](model-bakeoff-2026-07.md)) and both expansion settings, on
the 4 training films:
| scope | F1 | P | R | total misID |
@@ -27,7 +29,7 @@ restricted gallery:
![All combos ranked by training-set F1, filled dots are restricted](assets/images/rep4_matrix_f1.png)
See [the full experiment log](model-bakeoff.md) for the complete table. One
See [the full experiment log](model-bakeoff-2026-07.md) for the complete table. One
combo reaches zero true out-of-cast misidentifications,
`arcface_w600k_mbf_restricted_exp` (F1 76.2%), and it is a restricted one,
consistent with restriction, not expansion, being what suppresses cross-film
@@ -57,7 +59,7 @@ Building this as a real feature requires:
option.
- A decision on the fallback case: what happens to a real, uncredited
cameo (see the Germar Terrell Gardner and Talia Balsam cases in the
[LVFace deep dive](lvface-deep-dive.md#where-lvface-beat-x-ray)) if the
[LVFace deep dive](lvface-deep-dive-2026-07.md#where-lvface-beat-x-ray)) if the
restricted gallery never includes them at all.
- Regenerating the restricted-gallery cache whenever a title's Jellyfin
cast list changes.
docs/index.md
+54 -46
View File
@@ -17,62 +17,70 @@ two credited cast members without a visible face are correctly reported
present but not visible. This matches Amazon X-Ray's own record for this
second exactly.
Results are not uniform across films. The hardest held-out film scores 46%
F1. This report documents why: one tunable trade (extinction bridging at
hard cuts), one structural limit (X-Ray credits people whose faces never
appear on screen), and a small number of cases where the pipeline is
correct and X-Ray's ground truth is not. Read
[how we score against X-Ray](methodology.md) first. X-Ray's ground truth is
scene-level; the pipeline's output is per-second. That difference shapes
every finding below.
## The headline: learned scene boundaries
## Findings
The current opencv5 build's biggest gain is **flood-fill presence on a
learned scene-boundary detector**. An actor seen once inside a shot is
reported for the whole shot — but only if the shot boundaries are good. A
learned XGBoost boundary detector, scored **leave-one-out** so no film is
ever measured by a detector that trained on it, lifts per-second X-Ray
presence F1 across nine films and improves every one of them:
<div class="grid cards" markdown>
| boundary source for flood-fill | presence F1 |
| ------------------------------ | ----------: |
| track-extent (flood off) | 62.6% |
| flood + grayscale cuts | 64.0% |
| **flood + learned detector (LOO)** | **74.9%** |
- :material-trophy:{ .lg .middle } **[Which model is best?](best-model.md)**
![Macro presence F1 by flood-fill boundary source](assets/images/scene_presence_macro.png)
---
The full story — why the old grayscale cut detector broke Scarface, what
features work, and the per-film breakdown — is on the
[learned scene-boundary detector](scene-boundary-detector.md) page.
Calibration curves first, independent of any threshold, then held-out
F1 across three models. LVFace-B Glint360K wins both, and wins on every
held-out film.
## What the numbers mean, and their limits
- :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)**
Results are not uniform across films, and they should not be. X-Ray's ground
truth is scene-level and credits people whose faces never appear on screen;
the pipeline's output is per-second and can only name a face it can see.
That difference is a structural recall ceiling, not a bug. Read
[how we score against X-Ray](methodology.md) first — it defines F1,
precision, recall, and misID, and explains the two limits (off-screen cast
and gallery coverage) that shape every finding.
---
Restricting the matcher to a film's credited cast improves F1,
recall, and misID rate at once, but is not a shipped runtime feature
yet.
- :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)**
---
A training-set effect that did not reproduce on 5 held-out films once
two methodology bugs in the comparison harness were found and fixed.
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
---
The held-out generalization gap, the two mechanisms behind its errors,
and every distinct case where it names someone outside the film's
credited cast.
</div>
Precision on identified faces is near-perfect: where the pipeline names a
face, it is almost always a name X-Ray also credits to that scene. The
frames throughout this documentation make the tension visual — **green** =
true positive, **red** = false positive, **orange** = unknown, and a
**blue** panel lists credited cast present with no visible face.
## Full experiment log
- **[Full experiment log](model-bakeoff.md)**: the complete log behind the
four pages above, including how replaying against cached embeddings
inside the same KPN network makes a full model and configuration
comparison practical, the full results table, and every caveat. This is
where the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp)
defaults come from.
- **[Service conversion (proposal)](service-conversion.md)**: design
sketch for a native idle-GPU worker gated on screen lock, not yet built.
- **[Full experiment log (opencv5)](model-bakeoff.md)**: the complete log
behind the current build — the ten-knob differential-evolution tuning, the
shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults and
where each comes from, the replay architecture that makes a nine-film
search tractable, and the flood-fill step change.
- **[Learned scene-boundary detector](scene-boundary-detector.md)**: the
features, the model, leave-one-out results, and the two headline films.
- **[Benchmark — SuperHero](benchmark.md)**: the benchmark harness.
- **[Service conversion (proposal)](service-conversion.md)**: design sketch
for a native idle-GPU worker gated on screen lock, not yet built.
## Archive (July 2026)
The pre-opencv5 four-model ArcFace/LVFace bake-off is kept for provenance.
Its numbers are historical; the current build supersedes them.
- [Best model (July)](best-model-2026-07.md) — LVFace-B Glint360K wins on
calibration and on every held-out film.
- [Gallery scope (July)](gallery-scope-2026-07.md) — cast-restricted
gallery improves F1, recall, and misID at once.
- [Pose expansion (July)](pose-expansion-2026-07.md) — a training-set
effect that did not reproduce held-out.
- [LVFace deep dive (July)](lvface-deep-dive-2026-07.md) — the
generalization gap and every out-of-cast identification.
- [Full experiment log (July)](model-bakeoff-2026-07.md).
## Reproducing the benchmarks
docs/lvface-deep-dive.md → docs/lvface-deep-dive-2026-07.md
+10 -8
View File
@@ -1,12 +1,14 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Deep dive: LVFace-B Glint360K
LVFace won the model comparison (see [Which model is best?](best-model.md))
LVFace won the model comparison (see [Which model is best?](best-model-2026-07.md))
and is the shipped default embedder. This page reports how it performs in
detail: a baseline of correct output, the two mechanisms behind its errors,
and every distinct case where it names someone who is not in the film's
credited cast.
Read [How we score against X-Ray](methodology.md) first. X-Ray's ground truth
Read [How we score against X-Ray](methodology-2026-07.md) first. X-Ray's ground truth
is scene-level, not per-frame. A name marked correct in the Offscreen column
below is the pipeline correctly reporting scene membership, not a workaround.
@@ -61,7 +63,7 @@ on the 5 films the optimizer never saw:
| macro average | 67.4% | 85.8% | 57.0% | | | | |
The `P` column is misID-weighted (each out-of-film name counts 10x in the
denominator; see [methodology](methodology.md#precision-recall-and-the-misid-weighting)).
denominator; see [methodology](methodology-2026-07.md#precision-recall-and-the-misid-weighting)).
That weighting is why Many Saints reads 54.7% here despite naming mostly real,
present faces: its raw (unweighted) precision is **78.4%**, and the gap is
entirely its 974 misIDs paying the 10x penalty. The three zero-misID films
@@ -70,7 +72,7 @@ Lovelace, with 58 misIDs, sits 3pp below its raw 93.3%.
Held-out F1 is 67.4%, against 75.3% on training, an 8pp drop. The spread
between the best and worst held-out film is 37pp. This is not unique to
LVFace: [the full experiment log](model-bakeoff.md#held-out-validation-all-3-models)
LVFace: [the full experiment log](model-bakeoff-2026-07.md#held-out-validation-all-3-models)
shows mbf and r18 with the same shape of spread on the same films, at a
uniformly lower level. Two mechanisms explain the spread. Both are shown
below with frame-level evidence.
@@ -174,10 +176,10 @@ ground-truth gap, not a model error.
Archie Yates, t=2521s, 78% confidence. A real detected face, a genuine
lookalike confusion.
![Zooey Deschanel, third out-of-cast name in Many Saints](assets/images/many_saints_fpi_deschanel.jpg)
Zooey Deschanel, t=2819s, 99% confidence. A real detected face at a dinner
table, high-confidence lookalike confusion.
Zooey Deschanel, t=2819s, 99% confidence — a high-confidence lookalike
confusion in the July pipeline. **The current opencv5 pipeline no longer makes
this identification**; the tighter tracker/registry and re-tuned matching removed
it, so there is no annotated frame for it here.
![Talia Balsam, fourth out-of-cast name in Many Saints](assets/images/many_saints_fpi_balsam.jpg)
docs/methodology-2026-07.md
+136
View File
@@ -0,0 +1,136 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# How we score against X-Ray
Every number in this report, every F1 and misID count, comes from one
comparison. The comparison has a mismatch at its core that shapes nearly
every finding in this report: the ground truth is scene-level, the
pipeline's output is per-second, and the two do not mean the same thing.
This page documents that comparison once, so the findings pages can rely on
it without re-explaining it.
## What Amazon X-Ray records
X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
timespans), `people_in_scenes.csv` (which actors are credited in each
scene), and `people.csv` (actor identities). There is no per-frame or
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
X-Ray records one cast list for the entire span, not "on screen from
second 12 to second 30."
To compare this against per-second predictions, `second_score.py` expands
every scene into per-second ground truth by copying the whole scene's cast
list onto every second inside it:
```python
for sn, (t0, t1) in spans.items():
cast = scene_cast.get(sn, [])
for t in range(int(t0), int(t1)):
timeline[t] = cast
```
That is the entire mechanism. If X-Ray credits five actors to a 30-second
scene, all five count as ground truth present for all 30 seconds, including
seconds where only one of them is on screen. This is not a simplification
introduced by the pipeline; it is the only reading of X-Ray's data that is
possible, because X-Ray itself does not record anything finer-grained.
## Why an offscreen name can be scored correct
A name listed under Offscreen with a correct (green) label is not the
pipeline guessing or padding its score. It is the pipeline correctly
answering the question X-Ray actually asks: is this actor part of this
scene. It answers that question using a presence window (`[start, end]`,
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
X-Ray's scene-level semantics more closely than a raw per-frame detection
would.
A system that only reported "this actor is visible in this exact frame"
would score worse against X-Ray's scene-level ground truth, producing a
false negative every time the camera cuts away from a character who is
still present in the scene. Not because it is wrong about the world, but
because it would be answering a stricter, different question than the one
X-Ray's data supports. The presence-window design exists specifically to
answer X-Ray's actual question.
## What this resolves and what it does not
This resolves the semantic mismatch between a scene and an instant. It does
not resolve two other limitations, both discussed in the
[LVFace deep dive](lvface-deep-dive-2026-07.md).
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
of whether a face is ever visible: background crew, characters shot from
behind, voice-only presence. No amount of bridging recovers a face that
never appears on screen. This is a hard ceiling on recall, not a defect.
**Extinction bridging can overshoot.** The same presence-window mechanism
that correctly answers "still in this scene" during a normal cut can also
bridge across a scene boundary it has no way to detect. A hard cut into a
different scene with no faces, such as closing credits, carries the
previous scene's identities forward until the window expires. This is the
mechanism behind Downton Abbey's recall collapse, documented in the deep
dive.
## Precision, recall, and the misID weighting
Per sampled second `t`:
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
are present.
**FPI** (false positive instances): actors the pipeline reports that are
not in X-Ray's cast for this second. Split into two categories:
- **FPI_incast**: the actor is in the film's cast, just not credited to
this particular scene. A timing or boundary slip.
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
wrong-identity error, weighted 10x in the precision objective, because
naming someone who is not even in the film is a categorically worse
error than a few seconds of scene-boundary slop.
!!! note "Every headline `P` and `F1` is misID-weighted"
The precision reported throughout this report, and therefore the F1
derived from it, puts each `FPI_misid` into the denominator **10 times**
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
This is deliberate: the whole point is to punish naming an out-of-film
actor far harder than a scene-boundary slip. But it means the `P` column
is not raw precision, and a misID-heavy film's `P` is depressed
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive-2026-07.md)
reports both. When comparing `P` across films, remember you are comparing a
quantity that penalizes misIDs, not just a hit rate.
**FN** (false negatives): actors X-Ray lists that the pipeline never
reports, counted only for actors who have a gallery reference embedding.
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
20% to 79% by film (see
[the full experiment log](model-bakeoff-2026-07.md#gallery-coverage-per-film)); an
actor with no reference photo can never be recognized regardless of model
quality, and counting them as a miss would penalize gallery coverage, not
recognition accuracy.
Two further numbers are reported alongside F1:
**agreement_rate**: mean per-second Jaccard overlap
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
scores 2/3, not 0.
**exact_match_rate**: the fraction of sampled seconds where the pipeline's
named set exactly equals X-Ray's, no partial credit. Far harsher, and
dominated by recall, since any single missed actor zeroes that second.
## Reproduce
```bash
python3 scripts/optimizer/second_score.py \
--pred pred.json --xray experiments/xray/.../<xray_dir> \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
```
See also [the full experiment log](model-bakeoff-2026-07.md) for how `pred.json` is
produced, and the [LVFace deep dive](lvface-deep-dive-2026-07.md) for what these
mechanisms look like frame by frame.
docs/methodology.md
+70 -78
View File
@@ -1,11 +1,9 @@
# How we score against X-Ray
Every number in this report, every F1 and misID count, comes from one
comparison. The comparison has a mismatch at its core that shapes nearly
every finding in this report: the ground truth is scene-level, the
pipeline's output is per-second, and the two do not mean the same thing.
This page documents that comparison once, so the findings pages can rely on
it without re-explaining it.
Every number in this report comes from one comparison, and that comparison
has a mismatch at its core: the ground truth is scene-level, the pipeline's
output is per-second, and the two do not mean the same thing. This page
documents the comparison once so the findings can rely on it.
## What Amazon X-Ray records
@@ -13,12 +11,12 @@ X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
timespans), `people_in_scenes.csv` (which actors are credited in each
scene), and `people.csv` (actor identities). There is no per-frame or
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
X-Ray records one cast list for the entire span, not "on screen from
second 12 to second 30."
X-Ray records one cast list for the entire span, not "on screen from second
12 to second 30."
To compare this against per-second predictions, `second_score.py` expands
every scene into per-second ground truth by copying the whole scene's cast
list onto every second inside it:
To compare against per-second predictions, `second_score.py` expands every
scene into per-second ground truth by copying the whole scene's cast list
onto every second inside it:
```python
for sn, (t0, t1) in spans.items():
@@ -27,48 +25,46 @@ for sn, (t0, t1) in spans.items():
timeline[t] = cast
```
That is the entire mechanism. If X-Ray credits five actors to a 30-second
scene, all five count as ground truth present for all 30 seconds, including
seconds where only one of them is on screen. This is not a simplification
introduced by the pipeline; it is the only reading of X-Ray's data that is
possible, because X-Ray itself does not record anything finer-grained.
If X-Ray credits five actors to a 30-second scene, all five count as ground
truth present for all 30 seconds, including seconds where only one is on
screen. This is not a simplification the pipeline introduces; it is the only
reading X-Ray's data supports, because X-Ray records nothing finer.
## Why an offscreen name can be scored correct
## How the pipeline reports presence
A name listed under Offscreen with a correct (green) label is not the
pipeline guessing or padding its score. It is the pipeline correctly
answering the question X-Ray actually asks: is this actor part of this
scene. It answers that question using a presence window (`[start, end]`,
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
X-Ray's scene-level semantics more closely than a raw per-frame detection
would.
A presence claim is one actor owning one time window. How that window is
derived is a tunable choice — a knob the optimizer weighs — with two modes:
A system that only reported "this actor is visible in this exact frame"
would score worse against X-Ray's scene-level ground truth, producing a
false negative every time the camera cuts away from a character who is
still present in the scene. Not because it is wrong about the world, but
because it would be answering a stricter, different question than the one
X-Ray's data supports. The presence-window design exists specifically to
answer X-Ray's actual question.
- **`track_extent` (default).** A claim is exactly `[first_seen, last_seen]`
of a track the actor owned (AR-012), ending at the last sighting and never
after (AR-013). There is no keep-alive: the withdrawn `anneal_sec` and the
scene-tracker `extinction_sec` — which the July report's windows were held
open by — are **gone**. A track that survives its own gaps needs no bridge;
a gap after the final sighting is never claimed.
- **`flood`.** Each claim is snapped to the shot it sits in, so an actor seen
once anywhere in a shot is reported for the whole shot
`[prev_boundary, next_boundary]`. Boundaries come from TransNetV2 shot
detection when available, otherwise from the always-on histogram cut
detector (`is_cut`). This trades precision for recall against X-Ray's
scene-level granularity, and the optimizer decides per run whether it pays.
## What this resolves and what it does not
Do not confuse the surviving `track_extinction_sec` with the withdrawn
scene `extinction_sec`: the former bounds how long a lost track stays
available for **re-association** (a tracking question), and never extends a
presence claim.
This resolves the semantic mismatch between a scene and an instant. It does
not resolve two other limitations, both discussed in the
[LVFace deep dive](lvface-deep-dive.md).
## The two limits this does not resolve
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
of whether a face is ever visible: background crew, characters shot from
behind, voice-only presence. No amount of bridging recovers a face that
never appears on screen. This is a hard ceiling on recall, not a defect.
behind, voice-only presence. No face pipeline can recover a face that never
appears, so recall against X-Ray is a structural ceiling, not a defect.
**Extinction bridging can overshoot.** The same presence-window mechanism
that correctly answers "still in this scene" during a normal cut can also
bridge across a scene boundary it has no way to detect. A hard cut into a
different scene with no faces, such as closing credits, carries the
previous scene's identities forward until the window expires. This is the
mechanism behind Downton Abbey's recall collapse, documented in the deep
dive.
**Flood-fill can overshoot.** Snapping to a shot correctly answers "still in
this scene" through an intra-scene cut, but a shot boundary is not a scene
boundary: on a film with sparse cuts, flood-fill can carry an actor across a
long "shot" they only briefly appeared in. This is why flood-fill is a knob,
not a default — its value depends on the film's cut density.
## Precision, recall, and the misID weighting
@@ -77,49 +73,46 @@ Per sampled second `t`:
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
are present.
**FPI** (false positive instances): actors the pipeline reports that are
not in X-Ray's cast for this second. Split into two categories:
**FPI** (false positive instances): actors the pipeline reports that are not
in X-Ray's cast for this second, split into:
- **FPI_incast**: the actor is in the film's cast, just not credited to
this particular scene. A timing or boundary slip.
- **FPI_incast**: the actor is in the film's cast, just not credited to this
scene. A timing or boundary slip.
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
wrong-identity error, weighted 10x in the precision objective, because
naming someone who is not even in the film is a categorically worse
error than a few seconds of scene-boundary slop.
wrong-identity error, weighted **10×** in the precision objective, because
naming someone not even in the film is categorically worse than a few
seconds of scene-boundary slop.
!!! note "Every headline `P` and `F1` is misID-weighted"
The precision reported throughout this report, and therefore the F1
derived from it, puts each `FPI_misid` into the denominator **10 times**
Precision puts each `FPI_misid` into the denominator 10 times
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
This is deliberate: the whole point is to punish naming an out-of-film
actor far harder than a scene-boundary slip. But it means the `P` column
is not raw precision, and a misID-heavy film's `P` is depressed
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive.md)
reports both. When comparing `P` across films, remember you are comparing a
quantity that penalizes misIDs, not just a hit rate.
This deliberately punishes naming an out-of-film actor far harder than a
boundary slip, so the `P` column is not raw precision and a misID-heavy
film's `P` is depressed super-linearly.
**FN** (false negatives): actors X-Ray lists that the pipeline never
reports, counted only for actors who have a gallery reference embedding.
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
20% to 79% by film (see
[the full experiment log](model-bakeoff.md#gallery-coverage-per-film)); an
actor with no reference photo can never be recognized regardless of model
quality, and counting them as a miss would penalize gallery coverage, not
recognition accuracy.
**FN** (false negatives): actors X-Ray lists that the pipeline never reports,
counted **only** for actors who have a gallery reference embedding. An actor
with no reference photo can never be recognized, and counting them as a miss
would measure gallery coverage, not recognition accuracy.
Two further numbers are reported alongside F1:
Two further numbers accompany F1:
**agreement_rate**: mean per-second Jaccard overlap
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
scores 2/3, not 0.
(`|Pred ∩ GT| / |Pred ∪ GT|`) — partial credit, so naming 2 of 3 present
actors scores 2/3, not 0.
**exact_match_rate**: the fraction of sampled seconds where the pipeline's
named set exactly equals X-Ray's, no partial credit. Far harsher, and
dominated by recall, since any single missed actor zeroes that second.
**exact_match_rate**: the fraction of seconds where the pipeline's named set
exactly equals X-Ray's — no partial credit, dominated by recall.
## The benchmark set
Unlike the July report — which trained on a 3-film subset and validated on
held-out films to keep evaluations fast — this run scores **all 9 films on
every evaluation**. The registry one-clock fix and uncapped dumps made
full-set replay affordable, so the reported optimum is tuned against the
complete set rather than a training subset.
## Reproduce
@@ -129,6 +122,5 @@ python3 scripts/optimizer/second_score.py \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
```
See also [the full experiment log](model-bakeoff.md) for how `pred.json` is
produced, and the [LVFace deep dive](lvface-deep-dive.md) for what these
mechanisms look like frame by frame.
See the [full experiment log](model-bakeoff.md) for how `pred.json` is
produced and where the shipped `src/config.hpp` defaults come from.
docs/model-bakeoff-2026-07.md
+348
View File
@@ -0,0 +1,348 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Full experiment log
This page reports how the pipeline performs across three questions: which
embedding model is best, whether restricting the gallery to a film's
credited cast helps, and whether promoting confidently identified poses into
a per-film gallery annex helps. It also documents the replay architecture
that made testing all three questions in one pass practical, and every
caveat needed to trust the numbers.
Read [How we score against X-Ray](methodology-2026-07.md) first for what F1,
precision, recall, and misID mean in this report. All numbers below use the
per-second metric
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
r50 (ArcFace w600k-R50) is excluded from the detailed comparison below. Its
gallery was built with roughly 30% fewer reference images per actor than the
other three models on the identical source photos (10808 vs 15055 total
embeddings across the same 2418 actors), which confounds any direct
comparison of its scores against the others. It remains in the
[calibration curve comparison](best-model-2026-07.md#first-signal-calibration-curves),
which does not depend on the training benchmark.
## Why replay makes this affordable
Decoding video and running face detection, alignment, and embedding is the
expensive part of this pipeline. Everything downstream of that (tracking,
identity matching, scene aggregation) is cheap. KPN++'s node/network
structure means those two stages are separate components connected by
typed channels, so the expensive stage can run once per film, cache its
output, and the cheap stage can be re-run against that cache as many times
as needed with different Config values.
`scene_analyze --dump-embeddings out.h5` runs the expensive half once per
film and writes per-frame face detections and embeddings to HDF5
([`scripts/optimizer/SCHEMA.md`](https://REPOLINK/scripts/optimizer/SCHEMA.md)).
[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py)
then re-assembles the real C++ `face_tracker`, `identity_matcher`, and
`scene_tracker` nodes into a Python-driven KPN network and replays a
film's cached embeddings through them, varying `prob_threshold`,
`anneal_sec`, `extinction_sec`, and `expand_gallery` freely. No GPU
inference and no video decode happen during a replay; each one completes
in seconds. This is what makes a 512-evaluation differential-evolution
search per model, per gallery mode, per expansion setting, tractable, and
what made the full held-out validation across three models in this report
possible in one session rather than requiring three full re-encodes of the
benchmark set.
`optimize.py` runs `differential_evolution` over this replay function as its
objective, with DE-level parallelism (multiple candidate configs evaluated
concurrently, each spawning its own replay subprocesses) on top of it. The
practical ceiling on this machine's GPU was 8 concurrent replay processes;
9 silently degraded every score to 0.0% (well-formed output, wrong numbers,
not a crash), so `optimize.py` was run at `REPLAY_WORKERS=4 DE_WORKERS=2`.
## Search space
`popsize=10, maxiter=15` per combo (3 parameters, up to 512 evaluations,
usually stopping earlier on DE's convergence tolerance).
`anneal_sec`/`extinction_sec` bounds were widened from 1-30/1-15 to 1-60/1-60
partway through the sweep. r50's 4 combos finished before the widening and
used the old, narrower bounds; this is one more reason r50 is excluded from
direct comparison here.
## Training films and held-out films
9 films have dumped embeddings across all 4 models. 4 were used for
optimization:
- Café Society (62-cast)
- Lord of War (64-cast)
- Scarface (67-cast)
- Sound of Metal (14-cast)
5 were held out, never seen by any optimizer run:
- Benny & Joon
- Downton Abbey: A New Era
- Lovelace
- The Many Saints of Newark
- Valerian and the City of a Thousand Planets
## Gallery coverage per film
The gallery has reference embeddings for 2418 actors, but coverage of any
given film's credited cast varies widely. This was previously reported as
one flat number (67% of X-Ray cast lacking a reference embedding, averaged
across the whole benchmark); the per-film breakdown is:
| film | cast credited | in gallery | coverage |
|---|---|---|---|
| Lord of War | 64 | 13 | 20.3% |
| Scarface | 67 | 15 | 22.4% |
| The Many Saints of Newark | 48 | 13 | 27.1% |
| Café Society | 62 | 17 | 27.4% |
| Lovelace | 42 | 15 | 35.7% |
| Valerian and the City of a Thousand Planets | 36 | 13 | 36.1% |
| Benny & Joon | 23 | 12 | 52.2% |
| Downton Abbey: A New Era | 36 | 22 | 61.1% |
| Sound of Metal | 14 | 11 | 78.6% |
Two training films (Lord of War, Scarface) have the worst coverage in the
set, 20-22%. Their training-set F1 numbers below are partly capped by
missing references, not purely by model quality. Downton Abbey has 61%
coverage, the second-best in the benchmark, yet the worst held-out recall
of any film (39.4%, LVFace). Its recall problem is not primarily a coverage
problem; it is the extinction-bridging failure documented in the
[LVFace deep dive](lvface-deep-dive-2026-07.md#mechanism-1-extinction-bridging).
Reproduce with `scripts/docs/gallery_coverage_per_film.py`.
## Training results, 3 models × 2 gallery modes × 2 expansion settings
Ranked by F1. misid = FPI_misid, the count of true wrong-actor
identifications (naming someone not in the film's cast at all), distinct
from FPI, which also includes in-cast timing slips.
Each combo's row is its best **full-coverage** evaluation: the highest-F1 DE
evaluation in which all 4 training films replayed without a timeout (see
[Dropped-film scoring](#a-scoring-bug-worth-recording-dropped-film-evaluations)
below for why this qualifier is load-bearing and not the same as `argmax F1`
over the raw sweep).
| combo | F1 | P | R | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|
| LVFace-B_Glint360K_restricted_exp | 78.3% | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
| arcface_w600k_mbf_restricted_exp | 76.2% | 90.0% | 66.2% | 64328 | 7480 | 0 | 33234 |
| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
| LVFace-B_Glint360K_full_exp | 75.3% | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
| LVFace-B_Glint360K_full_noexp | 72.3% | 88.3% | 61.8% | 40363 | 3503 | 244 | 25850 |
| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
| arcface_w600k_mbf_full_noexp | 71.0% | 93.2% | 57.9% | 41699 | 2472 | 56 | 33024 |
| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |
![All combos ranked by training-set F1](assets/images/rep4_matrix_f1.png)
The two clearest patterns: every model's best-scoring combo uses the
restricted gallery, and LVFace leads within both gallery modes. `full_exp`
(the shipped combination) is the best-scoring option that uses only
features the running application currently supports; restriction is not
wired into the application yet (see
[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md)).
### A scoring bug worth recording: dropped-film evaluations
The numbers above are corrected ones. The raw `rep4_best_*.json` files, and an
earlier version of this table, reported a different `arcface_w600k_mbf_full_noexp`
row: **74.2% F1 at TPI 12645**, a third the TPI of every sibling combo. That was
not a better config; it was an artifact of how the optimizer aggregates.
`optimize.py` builds each candidate's score from only the films whose replay
subprocess returned (`per_film = [m for m in ex.map(_one, films) if m is not
None]`), then **averages** F1/precision/recall and **sums** TPI/FPI/misID over
just those survivors. When a film's replay times out (the sweep ran near the
8-process concurrency ceiling, so this happened intermittently), that film
silently drops from both. A candidate whose hardest film timed out is therefore
scored on an easier subset, and differential evolution, maximizing that score,
will happily converge onto exactly such a candidate. For `mbf_full_noexp` the
reported winner was one of 7 evaluations (out of 512) whose TPI had collapsed to
a partial-film subset; its median-coverage evaluations sit around 51686 TPI.
The fix here was to re-derive each combo's best row from its DE trajectory
(`experiments/trajectories/rep4_*.jsonl`), keeping only evaluations within 30% of
that combo's median TPI (full 4-film coverage) before taking the best F1. This
needs no re-running, the honest best configuration was already in the sweep,
just not the one `argmax F1` selected. Three combos moved: `mbf_full_noexp`
74.2% → **71.0%**, `LVFace_full_noexp` 72.4% → **72.3%** (and its misID, 0 → 244,
was itself a dropped-film artifact), `mbf_restricted_exp` 76.5% → **76.2%**. The
shipped LVFace `full_exp` winner was unaffected, its reported evaluation already
had full coverage (TPI 47757 ≈ median). `experiment_charts.py` applies the same
`clean_best` filter, so every figure on this page matches the corrected table.
The underlying `optimize.py` aggregation is also being fixed so a dropped-film
evaluation can never be selected as a winner again.
### Per-film training breakdown
The 75.3% LVFace training figure is a macro average across 4 films, not a
uniform result:
| film | LVFace F1 | mbf F1 | r18 F1 | best model |
|---|---|---|---|---|
| Café Society | 68.1% | 62.2% | 60.1% | LVFace |
| Lord of War | 75.6% | 77.2% | 75.6% | mbf |
| Scarface | 71.5% | 68.6% | 64.1% | LVFace |
| Sound of Metal | 78.8% | 76.5% | 71.6% | LVFace |
LVFace does not win every training film. mbf scores higher on Lord of War
(77.2% vs 75.6%). LVFace's own training-film range is 68.1% to 78.8%, a
10.7pp spread, smaller than the 37pp spread seen on held-out films but real.
Reproduce with `scripts/docs/run_holdout_all_models.py --films training`.
## Held-out validation, all 3 models
The training matrix above is training-set fit. Each model's own tuned
`full_exp` config was replayed against the 5 held-out films, scored the
same way:
| film | LVFace F1 | mbf F1 | r18 F1 |
|---|---|---|---|
| Benny & Joon | 83.0% | 78.5% | 77.1% |
| Lovelace | 77.5% | 73.7% | 72.2% |
| Valerian and the City of a Thousand Planets | 74.1% | 70.2% | 71.0% |
| Downton Abbey: A New Era | 56.2% | 55.0% | 53.0% |
| The Many Saints of Newark | 46.3% | 44.5% | 42.1% |
| **macro average** | **67.4%** | **64.4%** | **63.1%** |
LVFace scores highest on every one of the 5 held-out films; the ranking
never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18
1224. LVFace has less than half mbf's misID count while also scoring
higher on every film. This directly confirms the model choice out of
sample; it is not inferred from the training numbers alone. See the
[LVFace deep dive](lvface-deep-dive-2026-07.md) for frame-level detail on where and
why LVFace still fails on the two worst films. Reproduce with
`scripts/docs/run_holdout_all_models.py`.
## Two effects in isolation: gallery scope and pose expansion
Averaging across the 3 compared models (r50 excluded) isolates each variable
from model choice.
**Gallery scope**, averaged over both expansion settings and all 3 models
(6 evaluations per row):
| scope | F1 | P | R | total misID |
|---|---|---|---|---|
| full | 71.1% | 89.6% | 59.6% | 1121 |
| restricted | 75.9% | 90.4% | 65.6% | 299 |
Restriction improves every metric at once. This is not a precision/recall
trade: +4.8pp F1, +6.0pp recall, and roughly a quarter the misIDs. Fewer
candidates in the matcher's search space means fewer opportunities for a
lookalike false match, and the recall gain shows this does not cost real
detections. Restriction is currently an offline optimizer technique, not a
runtime feature of the application; see
[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md) for what building it
into the application would require.
**Pose expansion** (promoting a confidently identified track's novel-pose
views into a per-film gallery annex,
[`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp)):
| scope | expansion | F1 | R | misID |
|---|---|---|---|---|
| full | off | 70.0% | 57.6% | 407 |
| full | on | 72.1% | 61.5% | 714 |
| restricted | off | 75.1% | 63.9% | 179 |
| restricted | on | 76.7% | 67.2% | 120 |
In restricted mode, expansion is a clean win: +1.6pp F1, +3.3pp recall,
misID drops. The annex only competes against the film's own roughly 15-actor
cast, so a new pose of a known actor is unlikely to be confused with someone
else. In full mode, expansion buys +2.1pp F1 and +3.9pp recall but at a real
cost: misID rises from 407 to 714 as the same new-pose view now competes
against the full 2418-actor gallery, where a confidently learned pose is more
likely to match the wrong person. On the full gallery it is a recall-vs-misID
trade, not a free gain. This training-set effect
did not reproduce on held-out data; see
[Does pose expansion help?](pose-expansion-2026-07.md) for the full held-out test
and the two methodology bugs caught while checking it.
## Calibration curves
Each gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`,
stored directly in the gallery HDF5
([`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
This measures discriminative power independent of whatever
`prob_threshold` a given run used:
![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)
LVFace has the steepest curve (`a=17.7` vs 15.3-16.2 for the ArcFace
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs
0.27-0.31), separating same-actor from different-actor pairs more
confidently at a lower similarity than any ArcFace variant tested,
including r50. Generated by
[`scripts/docs/calibration_chart.py`](https://REPOLINK/scripts/docs/calibration_chart.py).
## Extinction and anneal window search
Every one of the 512 DE evaluations for the winning LVFace `full_exp`
combo, plotted over the `prob_threshold` × `extinction_sec` plane:
![DE search landscape: 512 evaluations over prob_threshold × extinction_sec](assets/images/de_search_landscape.png)
Nearly everything scoring well sits at `extinction_sec` above 50, across a
wide range of thresholds. Short extinction windows are uniformly weaker:
under a strict threshold, there is no good configuration in that region of
the search space. The optimizer converged with `anneal_sec=59.2,
extinction_sec=59.2`, about 99% of the widened 60s bound, which raises an
open question not resolved in this round: does performance keep improving
past 60s, or does it plateau there. Not chased further this pass.
## Caveats
- r50's 4 combos used the older, narrower search bounds (1-30/1-15 instead
of 1-60/1-60) and are further confounded by its thinner gallery. Excluded
from all comparisons above except calibration.
- The shipped defaults use `full_exp` (75.3% training F1), not the
higher-scoring `restricted_exp` (78.3%), because cast restriction is not
a runtime feature of the application yet.
- `expand_gallery` is mode-dependent, not a free win. Averaged across models
on the full gallery it trades misIDs for recall (see the pose-expansion
table). For LVFace specifically, though, `full_exp` beats `full_noexp` on
every axis at once (F1 75.3 vs 72.3, precision 89.7 vs 88.3, recall 65.4 vs
61.8, misID 232 vs 244), so the shipped `full_exp` is a clean choice for
this model, not an F1-vs-safety trade. (An earlier version of this page
reported `full_noexp` at 72.4% with zero misIDs and higher precision, which
made it look like the safer option; that was the dropped-film artifact
described above, not a real property of the config.)
- Switching the default model is an operational change: any gallery built
from a different model's embeddings must be rebuilt before the new
default takes effect.
## Reproduce
```bash
# 4-film training matrix, all 4 models × 2 gallery modes × 2 expansion settings
bash experiments/run_rep4_subprocess.sh
# single combo
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
--manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
--popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json
# held-out validation, all 3 models, 5 films
python3 scripts/docs/run_holdout_all_models.py --out docs_data/holdout_all_models.json
# per-film training breakdown, all 3 models, 4 films
python3 scripts/docs/run_holdout_all_models.py --films training --out docs_data/training_per_film.json
# gallery coverage per film
python3 scripts/docs/gallery_coverage_per_film.py --out docs_data/gallery_coverage_per_film.json
# regenerate this page's charts from experiments/ artifacts
python3 scripts/docs/experiment_charts.py --out-dir docs/assets/images
# one frame per distinct out-of-cast name across all 9 films (used in the deep dive)
python3 scripts/docs/first_fpi_frames.py
```
See also the session log
[`experiments/SESSION_STATE.md`](https://REPOLINK/experiments/SESSION_STATE.md).
docs/model-bakeoff.md
+158 -306
View File
@@ -1,346 +1,198 @@
# Full experiment log
# Full experiment log (opencv5)
This page reports how the pipeline performs across three questions: which
embedding model is best, whether restricting the gallery to a film's
credited cast helps, and whether promoting confidently identified poses into
a per-film gallery annex helps. It also documents the replay architecture
that made testing all three questions in one pass practical, and every
caveat needed to trust the numbers.
This is the complete log behind the current opencv5 build: how the pipeline is
tuned, what the shipped configuration is and where every number in it comes from,
and how the learned scene-boundary detector took per-second actor-presence F1 from
the low-60s to **74.9%** across the nine-film Amazon X-Ray benchmark — under honest
leave-one-out.
Read [How we score against X-Ray](methodology.md) first for what F1,
precision, recall, and misID mean in this report. All numbers below use the
per-second metric
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
Read [How we score against X-Ray](methodology.md) first for what F1, precision,
recall, and misID mean here. Every number below uses the per-second metric
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)):
the film is sampled once per second, and at each second the set of names the
pipeline reports present is compared against Amazon X-Ray's scene cast for that
second. X-Ray's ground truth is scene-level; the pipeline's output is per-second.
That mismatch shapes every result.
r50 (ArcFace w600k-R50) is excluded from the detailed comparison below. Its
gallery was built with roughly 30% fewer reference images per actor than the
other three models on the identical source photos (10808 vs 15055 total
embeddings across the same 2418 actors), which confounds any direct
comparison of its scores against the others. It remains in the
[calibration curve comparison](best-model.md#first-signal-calibration-curves),
which does not depend on the training benchmark.
## The benchmark
Nine films with public Amazon X-Ray scene data, all scored with the same
LVFace-B Glint360K gallery:
Benny & Joon · Café Society · Downton Abbey: A New Era · Lord of War · Lovelace ·
The Many Saints of Newark · Scarface · Sound of Metal · Valerian.
Two of these — Café Society and Scarface — are low-contrast, uniformly-graded
films that break naive cut detection. They are deliberately kept in the benchmark
because they are where the interesting failures live.
## Why replay makes this affordable
Decoding video and running face detection, alignment, and embedding is the
expensive part of this pipeline. Everything downstream of that (tracking,
identity matching, scene aggregation) is cheap. KPN++'s node/network
structure means those two stages are separate components connected by
typed channels, so the expensive stage can run once per film, cache its
output, and the cheap stage can be re-run against that cache as many times
as needed with different Config values.
expensive part of the pipeline. Everything downstream — tracking, identity
matching, scene aggregation — is cheap. KPN++'s node/network structure keeps those
two halves as separate components joined by typed channels, so the expensive half
runs once per film and caches its output, and the cheap half can be re-run against
that cache as often as needed with different `Config` values.
`scene_analyze --dump-embeddings out.h5` runs the expensive half once per
film and writes per-frame face detections and embeddings to HDF5
([`scripts/optimizer/SCHEMA.md`](https://REPOLINK/scripts/optimizer/SCHEMA.md)).
[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py)
then re-assembles the real C++ `face_tracker`, `identity_matcher`, and
`scene_tracker` nodes into a Python-driven KPN network and replays a
film's cached embeddings through them, varying `prob_threshold`,
`anneal_sec`, `extinction_sec`, and `expand_gallery` freely. No GPU
inference and no video decode happen during a replay; each one completes
in seconds. This is what makes a 512-evaluation differential-evolution
search per model, per gallery mode, per expansion setting, tractable, and
what made the full held-out validation across three models in this report
possible in one session rather than requiring three full re-encodes of the
benchmark set.
`scene_analyze --dump-embeddings out.h5` runs the expensive half once and writes
per-frame detections, embeddings, and (for the scene detector) per-frame RGB
histograms to HDF5. [`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py)
re-assembles the real C++ `face_tracker`, `identity_matcher`, and scene nodes into
a Python-driven KPN network and replays a film's cache through them, varying every
tuning knob freely. No GPU inference and no video decode happen during a replay, so
a full differential-evolution search over all nine films is tractable in one
session rather than requiring re-encodes.
`optimize.py` runs `differential_evolution` over this replay function as its
objective, with DE-level parallelism (multiple candidate configs evaluated
concurrently, each spawning its own replay subprocesses) on top of it. The
practical ceiling on this machine's GPU was 8 concurrent replay processes;
9 silently degraded every score to 0.0% (well-formed output, wrong numbers,
not a crash), so `optimize.py` was run at `REPLAY_WORKERS=4 DE_WORKERS=2`.
Two concurrency limits are load-bearing and were paid for in wedged runs: replays
run at `DE_WORKERS=1` (concurrent DE candidates wedge the ROCm GPU), and each
candidate's per-film replays run at `REPLAY_WORKERS=8` with stderr discarded (the
replay sink's per-second prints otherwise flood the captured pipe and hang the
subprocess).
## Search space
## The tuning knobs
`popsize=10, maxiter=15` per combo (3 parameters, up to 512 evaluations,
usually stopping earlier on DE's convergence tolerance).
`anneal_sec`/`extinction_sec` bounds were widened from 1-30/1-15 to 1-60/1-60
partway through the sweep. r50's 4 combos finished before the widening and
used the old, narrower bounds; this is one more reason r50 is excluded from
direct comparison here.
The opencv5 refactor replaced the old three-knob search with a **ten-knob**
differential-evolution sweep. The knobs, and their shipped values:
## Training films and held-out films
| knob | shipped | what it controls |
| ---- | ------: | ---------------- |
| `prob_threshold` | 0.485 | posterior P(match) above which a track is named |
| `ownership_logodds` | 1.72 | log-odds a track needs before it produces presence |
| `track_extinction_sec` | 31.0 | how long an idle track is held for re-detection |
| `track_alpha` | 0.435 | tracker cost mix (0 = embedding only, 1 = spatial only) |
| `evidence_rho_max` | 0.204 | evidence weighting ceiling |
| `evidence_admit_below` | 0.784 | admit new evidence below this similarity |
| `match_prior` | 0.433 | base-rate prior on a match |
| `expand_band_lo` | 0.804 | low edge of the pose-expansion similarity band |
| `expand_band_hi` | 0.952 | high edge of the pose-expansion band |
| `presence_mode` | flood | track-extent vs scene flood-fill |
9 films have dumped embeddings across all 4 models. 4 were used for
optimization:
The DE run over the first nine knobs (flood off, track-extent presence) converged
at **64.0% macro F1** over 345 evaluations. Those values are the shipped
[`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults.
- Café Society (62-cast)
- Lord of War (64-cast)
- Scarface (67-cast)
- Sound of Metal (14-cast)
![10-knob presence sweep (Differential Evolution)](assets/images/de_search_landscape.png)
5 were held out, never seen by any optimizer run:
The `track_extinction_sec` knob is worth calling out: at 31 s it holds an idle
track alive for re-detection long enough to bridge an actor turning away or leaving
frame briefly, without bridging across a genuine scene change. Getting this knob
and the tracker/registry to agree on **one clock** (the evidence watermark, not
wall-clock) was a correctness fix, not a tuning choice — before it, votes were
silently dropped at the reap horizon.
- Benny & Joon
- Downton Abbey: A New Era
- Lovelace
- The Many Saints of Newark
- Valerian and the City of a Thousand Planets
## The step change: flood-fill on learned boundaries
## Gallery coverage per film
The 64.0% above is track-extent presence: an actor is reported only while an actual
track is alive. **Flood-fill** instead reports an actor for the whole shot once
they are seen in it — but that is only correct if the shot boundaries are good.
The gallery has reference embeddings for 2418 actors, but coverage of any
given film's credited cast varies widely. This was previously reported as
one flat number (67% of X-Ray cast lacking a reference embedding, averaged
across the whole benchmark); the per-film breakdown is:
With the old grayscale cut detector as the boundary source, flood-fill barely beat
doing nothing (**64.0%**) and actively broke Scarface, where the detector fires
once in 10,204 frames and flood then smears every actor across the whole film
(precision collapses to 26%).
| film | cast credited | in gallery | coverage |
|---|---|---|---|
| Lord of War | 64 | 13 | 20.3% |
| Scarface | 67 | 15 | 22.4% |
| The Many Saints of Newark | 48 | 13 | 27.1% |
| Café Society | 62 | 17 | 27.4% |
| Lovelace | 42 | 15 | 35.7% |
| Valerian and the City of a Thousand Planets | 36 | 13 | 36.1% |
| Benny & Joon | 23 | 12 | 52.2% |
| Downton Abbey: A New Era | 36 | 22 | 61.1% |
| Sound of Metal | 14 | 11 | 78.6% |
The [learned scene-boundary detector](scene-boundary-detector.md) — an XGBoost
regressor over histogram-delta and audio features, with a per-film knee threshold —
fixes this. Macro per-second presence F1, at the shipped presence config:
Two training films (Lord of War, Scarface) have the worst coverage in the
set, 20-22%. Their training-set F1 numbers below are partly capped by
missing references, not purely by model quality. Downton Abbey has 61%
coverage, the second-best in the benchmark, yet the worst held-out recall
of any film (39.4%, LVFace). Its recall problem is not primarily a coverage
problem; it is the extinction-bridging failure documented in the
[LVFace deep dive](lvface-deep-dive.md#mechanism-1-extinction-bridging).
Reproduce with `scripts/docs/gallery_coverage_per_film.py`.
| boundary source for flood-fill | presence F1 |
| ------------------------------ | ----------: |
| track-extent (flood off) | 62.6% |
| flood + grayscale cuts | 64.0% |
| **flood + learned detector (LOO)** | **74.9%** |
## Training results, 3 models × 2 gallery modes × 2 expansion settings
![Macro presence F1 by flood-fill boundary source](assets/images/scene_presence_macro.png)
Ranked by F1. misid = FPI_misid, the count of true wrong-actor
identifications (naming someone not in the film's cast at all), distinct
from FPI, which also includes in-cast timing slips.
The learned column is **leave-one-out**: each film is scored by a detector trained
on the other eight, so no film's presence is ever measured with a detector that saw
it. That is the honest generalisation number, +12.3 points over track-extent, and
**it improves every one of the nine films**.
Each combo's row is its best **full-coverage** evaluation: the highest-F1 DE
evaluation in which all 4 training films replayed without a timeout (see
[Dropped-film scoring](#a-scoring-bug-worth-recording-dropped-film-evaluations)
below for why this qualifier is load-bearing and not the same as `argmax F1`
over the raw sweep).
![Per-film presence F1 by boundary source](assets/images/scene_presence_by_source.png)
| combo | F1 | P | R | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|
| LVFace-B_Glint360K_restricted_exp | 78.3% | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
| arcface_w600k_mbf_restricted_exp | 76.2% | 90.0% | 66.2% | 64328 | 7480 | 0 | 33234 |
| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
| LVFace-B_Glint360K_full_exp | 75.3% | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
| LVFace-B_Glint360K_full_noexp | 72.3% | 88.3% | 61.8% | 40363 | 3503 | 244 | 25850 |
| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
| arcface_w600k_mbf_full_noexp | 71.0% | 93.2% | 57.9% | 41699 | 2472 | 56 | 33024 |
| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |
| film | track-extent | flood+grayscale | flood+learned (LOO) |
| ---- | -----------: | --------------: | ------------------: |
| Benny & Joon | 77.3 | 80.2 | 78.2 |
| Café Society | 59.1 | 62.2 | 69.8 |
| Downton Abbey | 41.0 | 51.8 | **78.6** |
| Lord of War | 74.8 | 77.1 | 77.8 |
| Lovelace | 70.3 | 74.0 | 78.2 |
| The Many Saints of Newark | 37.5 | 43.9 | 53.4 |
| Scarface | 62.6 | **40.9** | **74.9** |
| Sound of Metal | 75.0 | 78.1 | 86.8 |
| Valerian | 65.6 | 67.7 | 76.2 |
![All combos ranked by training-set F1](assets/images/rep4_matrix_f1.png)
The two headline films — Scarface (grayscale flood *breaks* it, learned flood on a
film it never trained on takes it to 74.9%) and Downton Abbey (+37 points) — are
the strongest evidence the detector generalises. See the
[scene-boundary detector page](scene-boundary-detector.md) for the full story.
The two clearest patterns: every model's best-scoring combo uses the
restricted gallery, and LVFace leads within both gallery modes. `full_exp`
(the shipped combination) is the best-scoring option that uses only
features the running application currently supports; restriction is not
wired into the application yet (see
[Whole vs. cast-restricted gallery](gallery-scope.md)).
We re-ran the ten-knob DE on top of the good boundaries to check whether the
shipped config should change. It converged at 76.1% (+0.3 pp over the shipped
config on learned boundaries) — inside the noise, not worth re-shipping. The
boundaries, not the presence knobs, are where the win is.
### A scoring bug worth recording: dropped-film evaluations
## What the frames look like
The numbers above are corrected ones. The raw `rep4_best_*.json` files, and an
earlier version of this table, reported a different `arcface_w600k_mbf_full_noexp`
row: **74.2% F1 at TPI 12645**, a third the TPI of every sibling combo. That was
not a better config; it was an artifact of how the optimizer aggregates.
`scripts/optimizer/dump_error_frames.py` pulls representative seconds and draws
each face box against X-Ray's scene cast: **green** = true positive, **red** =
false positive (a name X-Ray does not credit to this scene — the real error),
**orange** = an unknown detection. Cast X-Ray lists as present but for whom no face
was detected — the structural false-negatives a face pipeline can never box — are
listed as a **blue** panel.
`optimize.py` builds each candidate's score from only the films whose replay
subprocess returned (`per_film = [m for m in ex.map(_one, films) if m is not
None]`), then **averages** F1/precision/recall and **sums** TPI/FPI/misID over
just those survivors. When a film's replay times out (the sweep ran near the
8-process concurrency ceiling, so this happened intermittently), that film
silently drops from both. A candidate whose hardest film timed out is therefore
scored on an easier subset, and differential evolution, maximizing that score,
will happily converge onto exactly such a candidate. For `mbf_full_noexp` the
reported winner was one of 7 evaluations (out of 512) whose TPI had collapsed to
a partial-film subset; its median-coverage evaluations sit around 51686 TPI.
![A correctly identified second: green true-positive boxes](assets/images/lovelace_perfect_second.jpg)
The fix here was to re-derive each combo's best row from its DE trajectory
(`experiments/trajectories/rep4_*.jsonl`), keeping only evaluations within 30% of
that combo's median TPI (full 4-film coverage) before taking the best F1. This
needs no re-running, the honest best configuration was already in the sweep,
just not the one `argmax F1` selected. Three combos moved: `mbf_full_noexp`
74.2% → **71.0%**, `LVFace_full_noexp` 72.4% → **72.3%** (and its misID, 0 → 244,
was itself a dropped-film artifact), `mbf_restricted_exp` 76.5% → **76.2%**. The
shipped LVFace `full_exp` winner was unaffected, its reported evaluation already
had full coverage (TPI 47757 ≈ median). `experiment_charts.py` applies the same
`clean_best` filter, so every figure on this page matches the corrected table.
The underlying `optimize.py` aggregation is also being fixed so a dropped-film
evaluation can never be selected as a winner again.
Every named frame in this documentation is regenerated against the current opencv5
pipeline by [`scripts/scene_detector/rematch_frames.py`](https://REPOLINK/scripts/scene_detector/rematch_frames.py),
which auto-matches each example by film, actor, and class (TP/FP) so the images
never drift from the shipped behaviour. Where the current pipeline no longer makes
a July-era error — the Zooey Deschanel misID in Many Saints is the clearest case —
the frame is dropped rather than staged, because the improvement is real.
### Per-film training breakdown
## The structural recall ceiling
The 75.3% LVFace training figure is a macro average across 4 films, not a
uniform result:
Precision against X-Ray is near-perfect on identified faces; recall is capped by
two things the pipeline cannot fix:
| film | LVFace F1 | mbf F1 | r18 F1 | best model |
|---|---|---|---|---|
| Café Society | 68.1% | 62.2% | 60.1% | LVFace |
| Lord of War | 75.6% | 77.2% | 75.6% | mbf |
| Scarface | 71.5% | 68.6% | 64.1% | LVFace |
| Sound of Metal | 78.8% | 76.5% | 71.6% | LVFace |
1. **X-Ray credits people whose faces never appear on screen** in a scene — voice,
back-of-head, or simply off-camera cast. No face pipeline can box a face that is
not there. These are the blue-panel names.
2. **Gallery coverage.** A large fraction of X-Ray cast has no reference image in
the gallery, so those actors can never be matched regardless of detection. This
is the dominant remaining recall limiter and is addressable by fetching more
reference photos, not by tuning.
LVFace does not win every training film. mbf scores higher on Lord of War
(77.2% vs 75.6%). LVFace's own training-film range is 68.1% to 78.8%, a
10.7pp spread, smaller than the 37pp spread seen on held-out films but real.
Reproduce with `scripts/docs/run_holdout_all_models.py --films training`.
Both are documented in [how we score against X-Ray](methodology.md).
## Held-out validation, all 3 models
## In the pipeline
The training matrix above is training-set fit. Each model's own tuned
`full_exp` config was replayed against the 5 held-out films, scored the
same way:
| film | LVFace F1 | mbf F1 | r18 F1 |
|---|---|---|---|
| Benny & Joon | 83.0% | 78.5% | 77.1% |
| Lovelace | 77.5% | 73.7% | 72.2% |
| Valerian and the City of a Thousand Planets | 74.1% | 70.2% | 71.0% |
| Downton Abbey: A New Era | 56.2% | 55.0% | 53.0% |
| The Many Saints of Newark | 46.3% | 44.5% | 42.1% |
| **macro average** | **67.4%** | **64.4%** | **63.1%** |
LVFace scores highest on every one of the 5 held-out films; the ranking
never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18
1224. LVFace has less than half mbf's misID count while also scoring
higher on every film. This directly confirms the model choice out of
sample; it is not inferred from the training numbers alone. See the
[LVFace deep dive](lvface-deep-dive.md) for frame-level detail on where and
why LVFace still fails on the two worst films. Reproduce with
`scripts/docs/run_holdout_all_models.py`.
## Two effects in isolation: gallery scope and pose expansion
Averaging across the 3 compared models (r50 excluded) isolates each variable
from model choice.
**Gallery scope**, averaged over both expansion settings and all 3 models
(6 evaluations per row):
| scope | F1 | P | R | total misID |
|---|---|---|---|---|
| full | 71.1% | 89.6% | 59.6% | 1121 |
| restricted | 75.9% | 90.4% | 65.6% | 299 |
Restriction improves every metric at once. This is not a precision/recall
trade: +4.8pp F1, +6.0pp recall, and roughly a quarter the misIDs. Fewer
candidates in the matcher's search space means fewer opportunities for a
lookalike false match, and the recall gain shows this does not cost real
detections. Restriction is currently an offline optimizer technique, not a
runtime feature of the application; see
[Whole vs. cast-restricted gallery](gallery-scope.md) for what building it
into the application would require.
**Pose expansion** (promoting a confidently identified track's novel-pose
views into a per-film gallery annex,
[`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp)):
| scope | expansion | F1 | R | misID |
|---|---|---|---|---|
| full | off | 70.0% | 57.6% | 407 |
| full | on | 72.1% | 61.5% | 714 |
| restricted | off | 75.1% | 63.9% | 179 |
| restricted | on | 76.7% | 67.2% | 120 |
In restricted mode, expansion is a clean win: +1.6pp F1, +3.3pp recall,
misID drops. The annex only competes against the film's own roughly 15-actor
cast, so a new pose of a known actor is unlikely to be confused with someone
else. In full mode, expansion buys +2.1pp F1 and +3.9pp recall but at a real
cost: misID rises from 407 to 714 as the same new-pose view now competes
against the full 2418-actor gallery, where a confidently learned pose is more
likely to match the wrong person. On the full gallery it is a recall-vs-misID
trade, not a free gain. This training-set effect
did not reproduce on held-out data; see
[Does pose expansion help?](pose-expansion.md) for the full held-out test
and the two methodology bugs caught while checking it.
## Calibration curves
Each gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`,
stored directly in the gallery HDF5
([`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
This measures discriminative power independent of whatever
`prob_threshold` a given run used:
![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)
LVFace has the steepest curve (`a=17.7` vs 15.3-16.2 for the ArcFace
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs
0.27-0.31), separating same-actor from different-actor pairs more
confidently at a lower similarity than any ArcFace variant tested,
including r50. Generated by
[`scripts/docs/calibration_chart.py`](https://REPOLINK/scripts/docs/calibration_chart.py).
## Extinction and anneal window search
Every one of the 512 DE evaluations for the winning LVFace `full_exp`
combo, plotted over the `prob_threshold` × `extinction_sec` plane:
![DE search landscape: 512 evaluations over prob_threshold × extinction_sec](assets/images/de_search_landscape.png)
Nearly everything scoring well sits at `extinction_sec` above 50, across a
wide range of thresholds. Short extinction windows are uniformly weaker:
under a strict threshold, there is no good configuration in that region of
the search space. The optimizer converged with `anneal_sec=59.2,
extinction_sec=59.2`, about 99% of the widened 60s bound, which raises an
open question not resolved in this round: does performance keep improving
past 60s, or does it plateau there. Not chased further this pass.
## Caveats
- r50's 4 combos used the older, narrower search bounds (1-30/1-15 instead
of 1-60/1-60) and are further confounded by its thinner gallery. Excluded
from all comparisons above except calibration.
- The shipped defaults use `full_exp` (75.3% training F1), not the
higher-scoring `restricted_exp` (78.3%), because cast restriction is not
a runtime feature of the application yet.
- `expand_gallery` is mode-dependent, not a free win. Averaged across models
on the full gallery it trades misIDs for recall (see the pose-expansion
table). For LVFace specifically, though, `full_exp` beats `full_noexp` on
every axis at once (F1 75.3 vs 72.3, precision 89.7 vs 88.3, recall 65.4 vs
61.8, misID 232 vs 244), so the shipped `full_exp` is a clean choice for
this model, not an F1-vs-safety trade. (An earlier version of this page
reported `full_noexp` at 72.4% with zero misIDs and higher precision, which
made it look like the safer option; that was the dropped-film artifact
described above, not a real property of the config.)
- Switching the default model is an operational change: any gallery built
from a different model's embeddings must be rebuilt before the new
default takes effect.
## Reproduce
The learned detector runs live inside `scene_analyze` as a post-EOF step (the
per-film knee needs every peak, so it can only run once the whole film is seen).
XGBoost inference is built into the binary via CMake (`SAE_SCENE_XGB`); the audio
log-PSD uses FFTW on the existing FFmpeg decode. The shipped model is trained on
the **C++-extracted** features so training and inference share one implementation.
Verified end to end through `scene_analyze` on a movie file and through the Jellyfin
work-queue worker.
```bash
# 4-film training matrix, all 4 models × 2 gallery modes × 2 expansion settings
bash experiments/run_rep4_subprocess.sh
# single combo
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
--manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
--popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json
# held-out validation, all 3 models, 5 films
python3 scripts/docs/run_holdout_all_models.py --out docs_data/holdout_all_models.json
# per-film training breakdown, all 3 models, 4 films
python3 scripts/docs/run_holdout_all_models.py --films training --out docs_data/training_per_film.json
# gallery coverage per film
python3 scripts/docs/gallery_coverage_per_film.py --out docs_data/gallery_coverage_per_film.json
# regenerate this page's charts from experiments/ artifacts
python3 scripts/docs/experiment_charts.py --out-dir docs/assets/images
# one frame per distinct out-of-cast name across all 9 films (used in the deep dive)
python3 scripts/docs/first_fpi_frames.py
scene_analyze --movie <file> --gallery <gallery.h5> \
--scene-xgb-model models/scene_boundary_xgb.json
```
See also the session log
[`experiments/SESSION_STATE.md`](https://REPOLINK/experiments/SESSION_STATE.md).
## Reproducing the benchmarks
Gallery `.h5` files, embedding dumps, the X-Ray corpus, and DE trajectories are not
committed. They are pushed to the Gitea package registry and pulled on demand:
```bash
scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
# per-second audio features, C++ feature matrices, train + downstream A/B
.venv-rocm/bin/python scripts/scene_detector/extract_audio_features.py \
--manifest experiments/manifests/films_LVFace_opencv5.json
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
scripts/scene_detector/downstream_presence.py
```
docs/pose-expansion.md → docs/pose-expansion-2026-07.md
+5 -3
View File
@@ -1,3 +1,5 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Pose expansion: does promoting new poses mid-film help?
`expand_gallery`
@@ -12,7 +14,7 @@ in the same film, without touching the baked gallery.
Averaged across the 3 compared models (r50 excluded), on the 4 films used
for optimization. These are the corrected, full-coverage figures, see the
[dropped-film note](model-bakeoff.md#a-scoring-bug-worth-recording-dropped-film-evaluations)
[dropped-film note](model-bakeoff-2026-07.md#a-scoring-bug-worth-recording-dropped-film-evaluations)
in the experiment log for why an earlier version of this table overstated the
full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
@@ -26,7 +28,7 @@ full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp
recall, lower misID. In full mode it looks like a recall-for-misID trade:
+2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See
[the full experiment log](model-bakeoff.md) for the per-model breakdown.
[the full experiment log](model-bakeoff-2026-07.md) for the per-model breakdown.
This asymmetry motivated the question below: does turning expansion on
change what gets recognized frame by frame, or is the aggregate F1 shift
coming from something else.
@@ -105,6 +107,6 @@ contribution, such as tagging which reference embedding won each match;
neither was in scope for this pass.
Do not treat the training-set exp/noexp numbers in
[the full experiment log](model-bakeoff.md) as proof that expansion changes
[the full experiment log](model-bakeoff-2026-07.md) as proof that expansion changes
real-world behavior in either direction. On the evidence gathered so far,
it does not move the needle enough to see.
docs/scene-boundary-detector.md
+205
View File
@@ -0,0 +1,205 @@
# The learned scene-boundary detector
Presence uses **flood-fill**: an actor seen once inside a shot is reported for the
whole shot (`[prev_boundary, next_boundary]`). That only works if the boundaries
are good. This page is the story of getting them good — a learned scene-boundary
detector that lifts per-second actor-presence F1 from **62.6% to 74.9%** across
the nine-film X-Ray benchmark, and fixes the film where naive flood-fill was
actively harmful.
That 74.9% is the **leave-one-out** figure: each film is scored by a detector
trained on the *other eight*, so no film's presence is measured with a detector
that ever saw it. It is the honest generalisation number, and it is only ~1 point
below the all-nine-trained model (75.8%) — the detector barely overfits.
## Why the old cut detector wasn't enough
The always-on boundary source was the grayscale histogram-correlation cut detector
(`camera_position_change_detector`): mark a cut when the frame-to-frame grayscale
histogram correlation drops below 0.70. It is cheap and it fires on obvious hard
cuts, but on a low-contrast, uniformly-graded film it is nearly blind. On
**Scarface** it fired **once in 10,204 frames**. Flood-fill then snapped every
actor across essentially the whole film:
| Scarface | precision | recall |
| -------- | --------- | ------ |
| flood + grayscale cuts | **26%** | 95% |
| track-extent (no flood) | 92% | 45% |
That single failure is what motivated everything below: flood-fill needs a
boundary source that works regardless of grade.
## What we are detecting, and why it is hard
The training target is **Amazon X-Ray scene boundaries** (`scenes.csv`). These are
*narrative* scenes — a new location or beat in the story — not shot cuts. There
are only ~20–60 of them per film (median scene ~170 s), and many transition
*within* continuous visual style and continuous audio. So the signal is sparse and
often genuinely faint: a boundary detector working from audio-visual features can
never recall a narrative cut that has no audio-visual signature.
This shapes every result: absolute boundary-F1 is modest by construction. What
matters is the **downstream** number — does snapping flood-fill to these
boundaries name the right actors — and there the gain is large.
## The features (what worked, measured)
Everything is per second, aligned to the 1-fps presence grid.
- **Delta histograms, not raw histograms.** The raw RGB histogram encodes what a
frame *looks like*, not that it *changed* — measured boundary separability ~1.4×.
The **symmetric histogram delta** `|hist(t+k) − hist(t−k)|` separates boundaries
**4–5×**. Leading with deltas (k = 1,2,4,8 s) and dropping the raw histogram was
the single biggest feature win (LSTM F1 7.5% → 10.8%).
- **A multi-scale "ramp" bank.** Antisymmetric matched filters at half-widths
H = 2,4,6,8,10 s; the model weights the scales. Different films' boundaries peak
at different widths.
- **A time-since-last-boundary "debounce" clock**, scaled by the corpus mean scene
length (~205 s), encoding that scenes don't restart moments apart.
- **Audio log-PSD** (per-second, 4 s window, ~57 log-frequency bins). Measured
weak on its own — a standalone audio cutter scored only 3–6% held-out F1, because
narrative boundaries usually have continuous audio — but it is complementary on
the films where video is weak (Downton, Sound of Metal), so it is included and
the model uses it where it helps.
![Detector development at strict ±2 s tolerance, and where the shipped detector landed at the ±20 s tolerance the pipeline uses](assets/images/scene_detector_evolution.png)
The left panel is the *feature* development, scored at a strict ±2 s tolerance so
each change is visible — this is where "delta beats raw histogram" was measured, not
the shipped tolerance. The right panel is the shipped detector at the ±20 s
tolerance the pipeline actually uses (see below). The two panels are on different
tolerances by design and must not be read as one curve.
Dead ends, all measured and discarded: audio-only detection; raw
histograms/PSDs as input; a two-tower BiLSTM (no better than the tree, far slower);
larger FFT windows / more frequency bins (worse — boundaries are short events);
and TransNetV2 (a Conv3D net that will not co-reside with the ROCm/VAAPI stack).
## The model
- **XGBoost regressor** over a ±3 s window of the features above, predicting a
**soft Gaussian proximity-to-boundary target** (`exp(-(d/σ)²)`, σ = 10 s).
Regression to a soft target — rather than a hard 0/1 label — stops a near-miss
from being trained as a hard negative, and yields a smooth score whose **peaks**
are the boundaries.
- **Per-film knee threshold.** The predicted peak heights form a
convex-decreasing curve; the knee (max drop below the endpoints' chord) is where
real boundaries give way to noise. Selecting at the knee **self-calibrates the
boundary count** to roughly the true scene count, per film, with no global
threshold that would be wrong for every grade.
- **Trained on all nine films** for the shipped model. Keeping the low-contrast
grades (Café Society, Scarface) in training matters most: on its own training
films the shipped model reaches **72.9% macro boundary-F1** (per-film 51–86%),
versus **29.8%** for the grayscale baseline on the same films.
Boundary detection, held out (leave-one-out, ±20 s tolerance — appropriate given
~170 s scenes): **44.1% macro F1, versus 29.8% for the grayscale baseline** — the
honest generalisation number, each film scored by a detector trained on the other
eight. Even the low-contrast grades generalise (Scarface held out 32%, Café Society
51%), where the grayscale detector scores 0% and 31%. The absolute number is capped
by the narrative-vs-audiovisual mismatch above — many boundaries have no
audio-visual signature at all — so the point is the downstream effect, below.
| boundary-F1 @±20 s | grayscale | learned (LOO) | learned (train-all) |
| ------------------ | --------: | ------------: | ------------------: |
| macro over 9 films | 29.8% | **44.1%** | 72.9% |
## The result that matters: actor presence
Per-second X-Ray presence F1, macro over the nine films, at the shipped presence
config. The learned column is **leave-one-out** — each film scored by a detector
trained on the other eight:
| boundary source for flood-fill | presence F1 |
| ------------------------------ | ----------- |
| track-extent (flood off) | 62.6% |
| flood + grayscale cuts | 64.0% |
| **flood + learned detector (LOO)** | **74.9%** |
![Macro presence F1 by flood-fill boundary source](assets/images/scene_presence_macro.png)
**+12.3 points over track-extent, +10.9 over the grayscale-cut flood, and it
improves every one of the nine films — under honest leave-one-out.** Per film:
![Per-film presence F1 by boundary source](assets/images/scene_presence_by_source.png)
| film | track-extent | flood+grayscale | flood+learned (LOO) |
| ---- | -----------: | --------------: | ------------------: |
| Benny & Joon | 77.3 | 80.2 | 78.2 |
| Café Society | 59.1 | 62.2 | 69.8 |
| Downton Abbey | 41.0 | 51.8 | **78.6** |
| Lord of War | 74.8 | 77.1 | 77.8 |
| Lovelace | 70.3 | 74.0 | 78.2 |
| The Many Saints of Newark | 37.5 | 43.9 | 53.4 |
| Scarface | 62.6 | **40.9** | **74.9** |
| Sound of Metal | 75.0 | 78.1 | 86.8 |
| Valerian | 65.6 | 67.7 | 76.2 |
The two headline cases:
- **Scarface**: the grayscale-cut flood *breaks* it (62.6 → 40.9), because it
detects one cut in the whole film. The learned detector — **on a film it never
trained on** — takes it to **74.9%**. This is the strongest evidence the
detector generalises: it fixes the exact failure that motivated it, held out.
- **Downton Abbey**: 41.0 (track-extent) → 51.8 (grayscale) → **78.6** — a
+37-point swing on the hardest film.
Naive flood-fill barely beat doing nothing (64% vs 62%) and broke a film. With a
real boundary detector, flood-fill is decisively the right mode.
### What the frames look like
`scripts/optimizer/dump_error_frames.py` pulls representative seconds and draws
each face box coloured against X-Ray's scene cast: **green** = true positive (a
name X-Ray also credits to this scene), **red** = false positive (a name X-Ray
does *not* credit here — the real error), **orange** = an unknown detection. Cast
X-Ray lists as present but for whom no face was detected — the structural
false-negatives a face pipeline can never box — are listed as a **blue** panel.
![A correctly identified second: green true-positive boxes](assets/images/scarface_tp_example.jpg)
Above: three faces named correctly (green). Below: the face-vs-scene-cast tension
made visual — the one visible face is confidently named (here it is a red
false-positive, a lead X-Ray did not credit to this exact scene), while six
credited cast members are off-camera with no face to detect (blue). This is why
recall against X-Ray has a structural ceiling, not a fixable bug.
![A false-positive box (red) with off-screen cast listed (blue)](assets/images/scarface_fn_fp_example.jpg)
## In the pipeline
Boundary detection is a **post-EOF step**, like flood-fill itself: the per-film
knee needs every peak, so it can only run once the whole film is seen. The
`camera_position_change_detector` stamps a per-frame RGB histogram onto each frame;
it rides through to the result sink; at end-of-stream the sink runs the detector
over the collected histograms plus the movie's audio log-PSD and snaps the
presence windows to the result. Enable it with:
```bash
scene_analyze --movie <file> --gallery <gallery.h5> \
--scene-xgb-model models/scene_boundary_xgb.json
```
Inference is real XGBoost, built into the binary via CMake (`SAE_SCENE_XGB`); the
audio log-PSD uses FFTW + the existing FFmpeg decode. To keep training and
inference on one feature implementation, the shipped model is **trained on the
C++-extracted features** (`scene_features_dump` → `train_xgb_cpp.py`) rather than a
re-implementation in Python — parity by construction. Verified end to end through
`scene_analyze` on a movie file and through the Jellyfin work-queue worker.
## Reproduce
```bash
# per-second audio log-PSD for each film
.venv-rocm/bin/python scripts/scene_detector/extract_audio_features.py \
--manifest experiments/manifests/films_LVFace_opencv5.json
# C++ feature matrices (same features training and inference share)
build/scene_features_dump <dump.h5> <movie> <features.h5>
# train the shipped model on all nine films
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
# downstream A/B (track-extent vs flood+grayscale vs flood+learned)
scripts/scene_detector/downstream_presence.py
```
experiments/regen_frame_examples.sh
Executable
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env bash
# Regenerate annotated TP/FP/FN frame examples for ALL 9 films against the current
# opencv5 pipeline (learned-boundary flood, shipped config). Replays each film with
# --raw-out for bboxes, then dump_error_frames.py draws GT-aware boxes
# (green TP / red FP / orange unknown / blue FN panel). Frames land in
# experiments/dump_review/<slug>/ (regenerable; gitignored). Hand-pick the ones a
# doc needs from there.
set -uo pipefail
REPO="/home/dtourolle/Development/scene-actor-extraction"; cd "$REPO"
export MIOPEN_USER_DB_PATH="$HOME/.cache/miopen-sae"
GAL=experiments/galleries/gallery_LVFace-B_Glint360K.h5
LUT=experiments/file-lut.json
CFG=(--prob-threshold 0.485 --ownership-logodds 1.72 --track-extinction-sec 31
--track-alpha 0.435 --evidence-rho-max 0.204 --evidence-admit-below 0.784
--match-prior 0.433 --expand-band-lo 0.804 --expand-band-hi 0.952
--expand-gallery --presence-mode flood)
mapfile -t ROWS < <(python3 -c '
import json
for f in json.load(open("experiments/manifests/films_LVFace_opencv5.json")):
print(f["slug"]+"\t"+f["xray"])')
SP=/tmp/claude-1000/-home-dtourolle-Development-scene-actor-extraction/c579f8cf-2974-4cbd-be88-afec68dbbf58/scratchpad
for row in "${ROWS[@]}"; do
slug="${row%%$'\t'*}"; xray="${row#*$'\t'}"
movie="$(python3 -c "import json;print(json.load(open('$LUT'))['$slug'])")"
echo "=== $slug ==="
[ -f "experiments/dump_review/$slug/manifest.json" ] && { echo " exists, skip"; continue; }
# replay the learned-boundary (LOO) dump so frames reflect true generalization
dump="experiments/dumps/injected_loo/${slug}.h5"
[ -f "$dump" ] || dump="experiments/dumps/LVFace-B_Glint360K_opencv5/dump_${slug}.h5"
for try in 1 2 3; do
timeout 280 python scripts/optimizer/replay.py --dump "$dump" --gallery "$GAL" \
--out "$SP/${slug}_pred.json" --raw-out "$SP/${slug}_raw.jsonl" "${CFG[@]}" \
>"$SP/${slug}_replay.log" 2>&1 && break
echo " replay try $try failed, retrying"
done
[ -s "$SP/${slug}_raw.jsonl" ] || { echo " no raw output, skip"; continue; }
python3 scripts/optimizer/dump_error_frames.py \
--pred "$SP/${slug}_pred.json" --raw "$SP/${slug}_raw.jsonl" \
--xray "$xray" --movie "$movie" --gallery "$GAL" \
--out-dir "experiments/dump_review/$slug" --n-per-bucket 4 \
>"$SP/${slug}_frames.log" 2>&1
echo " $(grep -oE 'wrote [0-9]+ frames' "$SP/${slug}_frames.log" | tail -1)"
done
echo "=== DONE ==="
mkdocs.yml
+8 -5
View File
@@ -35,14 +35,17 @@ extra_css:
nav:
- Home: index.md
- How We Score Against X-Ray: methodology.md
- Learned Scene-Boundary Detector: scene-boundary-detector.md
- Benchmark — SuperHero: benchmark.md
- Findings:
- Best Model: best-model.md
- Gallery Scope (Full vs. Limited): gallery-scope.md
- Pose Expansion: pose-expansion.md
- LVFace Deep Dive: lvface-deep-dive.md
- Full Experiment Log: model-bakeoff.md
- Service Conversion (proposal): service-conversion.md
- Archive (July 2026):
- How We Scored (July): methodology-2026-07.md
- Best Model: best-model-2026-07.md
- Gallery Scope (Full vs. Limited): gallery-scope-2026-07.md
- Pose Expansion: pose-expansion-2026-07.md
- LVFace Deep Dive: lvface-deep-dive-2026-07.md
- Full Experiment Log (July): model-bakeoff-2026-07.md
markdown_extensions:
- admonition
models/scene_boundary_xgb.json
+1
View File
File diff suppressed because one or more lines are too long
scripts/optimizer/dump_error_frames.py
+35 -12
View File
@@ -121,23 +121,44 @@ def load_raw_annotations(raw_path: str):
return by_second
def draw_annotations(frame_path: Path, actors: list):
def _name_key(name: str) -> str:
"""Normalised match key, mirroring identity.py's name: fallback."""
return "name:" + "".join(ch for ch in name.lower() if ch.isalnum() or ch == " ").strip()
def draw_annotations(frame_path: Path, actors: list, fp_keys=None, fn_names=None):
"""Draw GT-aware boxes: GREEN = true positive (named actor X-Ray also has in
this scene), RED = false positive (named actor NOT in the scene → the real
error), ORANGE = unknown detection. FN cast (present per X-Ray but no face
detected — so no box to draw) is listed as a BLUE text panel bottom-left."""
img = cv2.imread(str(frame_path))
if img is None:
return
fp_keys = fp_keys or set()
GREEN, RED, ORANGE, BLUE = (60,200,0), (0,0,230), (220,100,0), (230,150,0)
for a in actors:
known = a.get("actor_idx", -1) >= 0
colour = (60, 200, 0) if known else (220, 100, 0) # BGR: green / orange
x, y, w, h = a["bbox"]
x, y, w, h = int(x), int(y), int(w), int(h)
if known:
colour = RED if _name_key(a["name"]) in fp_keys else GREEN
label = f"{a['name']} {a['similarity']*100:.0f}%"
else:
colour = ORANGE; label = f"unknown {a['similarity']*100:.0f}%"
x, y, w, h = (int(v) for v in a["bbox"])
cv2.rectangle(img, (x, y), (x + w, y + h), colour, 2)
label = f"{a['name']} {a['similarity']*100:.0f}%" if known else f"unknown {a['similarity']*100:.0f}%"
(tw, th), baseline = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1)
strip_y0 = max(0, y - th - 4)
cv2.rectangle(img, (x, strip_y0), (x + tw + 4, y), colour, cv2.FILLED)
cv2.putText(img, label, (x + 2, y - 2), cv2.FONT_HERSHEY_SIMPLEX, 0.5,
(255, 255, 255), 1, cv2.LINE_AA)
(tw, th), _ = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1)
cv2.rectangle(img, (x, max(0, y-th-4)), (x+tw+4, y), colour, cv2.FILLED)
cv2.putText(img, label, (x+2, y-2), cv2.FONT_HERSHEY_SIMPLEX, 0.5,
(255,255,255), 1, cv2.LINE_AA)
# FN: X-Ray cast present with no detected face — no box exists, so list them.
fn = [n for n in (fn_names or []) if n]
if fn:
H = img.shape[0]
cv2.putText(img, "off-screen / missed (X-Ray cast, no face):",
(8, H-8-18*len(fn[:6])), cv2.FONT_HERSHEY_SIMPLEX, 0.45, BLUE, 1, cv2.LINE_AA)
for i, n in enumerate(fn[:6]):
disp = n.replace("name:", "").title()
cv2.putText(img, f" {disp}", (8, H-8-18*(len(fn[:6])-1-i)),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, BLUE, 1, cv2.LINE_AA)
cv2.imwrite(str(frame_path), img)
@@ -183,7 +204,9 @@ def main():
extract_frame(args.movie, r["t"], out_path)
ok = True
if raw_by_second is not None:
draw_annotations(out_path, raw_by_second.get(r["t"], []))
fp_keys = {_name_key(n) for n in r["fp"]}
draw_annotations(out_path, raw_by_second.get(r["t"], []),
fp_keys=fp_keys, fn_names=r["fn"])
except subprocess.CalledProcessError as e:
ok = False
print(f"[dump_error_frames] ffmpeg failed at t={r['t']}: {e}", file=sys.stderr)
scripts/optimizer/optimize.py
+20 -2
View File
@@ -61,7 +61,15 @@ from replay import dump_embedder_stamp # noqa: E402
from sae_stamp import EmbedderMismatch, verify_gallery_stamp # noqa: E402
_GAL_KEYS: dict = {} # gallery path → key set (fair-recall FN mask), loaded once
_REPLAY_TIMEOUT = 45 # seconds per film; a wedged replay is killed, not left to hang
# Seconds per film before a replay is killed. Its ONLY job is to escape the rare,
# intermittent ROCm GEMM wedge (github ROCT-Thunk #56): a wedged replay hangs
# forever and would otherwise stall the whole sweep, so it must be killed and that
# film dropped (the eval is then scored as incomplete → F1=0, and DE moves on). It
# is NOT a performance bound. A healthy replay finishes in ~15-30s even for the
# long films with stderr discarded, so 180s is comfortably above any real run yet
# short enough that a wedge is reaped quickly rather than after half an hour.
# Raise via REPLAY_TIMEOUT if a legitimately slow config is being killed.
_REPLAY_TIMEOUT = int(os.environ.get("REPLAY_TIMEOUT", "180"))
REPLAY_CLI = str(Path(__file__).resolve().parent / "replay.py")
@@ -91,7 +99,17 @@ def _replay_subprocess(dump, gallery, cfg, build_dir):
else:
argv += [f"--{k.replace('_', '-')}", str(v)]
try:
subprocess.run(argv, timeout=_REPLAY_TIMEOUT, capture_output=True, check=True)
# Discard the child's stdout/stderr rather than capture it. replay's sink
# prints a per-second "[result_sink] t=Ns" progress line with an explicit
# flush; on a long film that is thousands of writes, and under
# subprocess.run(capture_output=True) they accumulate in a fixed OS pipe
# buffer that nothing drains until the process exits. On the long films
# (Valerian, Sound of Metal) under DE concurrency the buffer fills and the
# C++ process BLOCKS on write to stderr — indistinguishable from a hang, so
# it hit the timeout and scored F1=0. DEVNULL never fills, so the process
# runs to completion. (Any real error is still surfaced by check=True.)
subprocess.run(argv, timeout=_REPLAY_TIMEOUT, check=True,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
return _json.loads(Path(out).read_text())
except (subprocess.TimeoutExpired, subprocess.CalledProcessError,
FileNotFoundError, ValueError) as e:
scripts/scene_detector/de_ramp.py
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""
de_ramp.py — DE-optimise a temporal matched-filter "ramp" per modality, whose
response becomes a feature channel for the scene-boundary LSTM.
A scene boundary is where a feature series (RGB histogram, audio log-PSD) shifts
from a "before" state to an "after" state. A signed, antisymmetric ramp kernel
convolved with the series responds strongly exactly at that transition and near
zero inside a stable scene — a matched filter for a step. Its shape is not
obvious (how wide? linear or peaked? how much centre dead-zone?), so we let DE
choose it by maximising boundary separation on the training films.
Ramp kernel over lags -H..+H seconds (1 fps → 1 sample/s):
w(l) = sign(l) * (|l| / H) ** gamma for |l| >= dead, else 0
params: H (half-width), gamma (shape), dead (centre dead-zone)
Response at t = || sum_l w(l) * feat[t+l] || (L2 over feature bins)
DE objective: boundary-detection F1 of a top-percentile threshold on the response,
macro-averaged over the training films (±2 s tolerance). The tuned (H, gamma,
dead) is saved; train_scene_boundary.py appends the ramp response as an input
channel to each tower.
Usage:
python scripts/scene_detector/de_ramp.py \
--manifest experiments/manifests/films_LVFace_opencv5.json \
--audio-dir experiments/dumps/audio_features \
--holdout Scarface Sound_of_Metal --out experiments/results/scene_boundary
"""
from __future__ import annotations
import argparse, csv, json, sys
from pathlib import Path
import h5py, numpy as np
from scipy.optimize import differential_evolution
def xray_bounds(xray_dir):
return sorted(float(r["start"])/1000 for r in
csv.DictReader(open(Path(xray_dir)/"scenes.csv"))
if float(r["start"]) > 500)
def load_series(dump, audio_dir, which):
if which == "audio":
# Audio is self-contained in the npz — no h5 needed (its ts IS the grid),
# so the audio cutter can be tuned before/without the RGB dumps.
slug = Path(dump).stem.replace("dump_", "")
z = np.load(Path(audio_dir)/f"{slug}.npz")
s = z["feat"].astype(np.float64)
ts = z["ts"] if "ts" in z else np.arange(len(s), dtype=float)
else: # video
with h5py.File(dump) as f:
ts = f["frames/timestamp_sec"][:]
s = f["frames/rgb_hist"][:].astype(np.float64)
# z-normalise each bin so L2 response isn't dominated by one loud bin
s = (s - s.mean(0)) / (s.std(0) + 1e-6)
return s, ts
def ramp_kernel(H, gamma, dead):
lags = np.arange(-H, H+1)
w = np.sign(lags) * (np.abs(lags)/max(H,1))**gamma
w[np.abs(lags) < dead] = 0.0
return w
def response(series, w, H):
T = series.shape[0]
r = np.zeros(T)
for t in range(T):
lo, hi = max(0, t-H), min(T, t+H+1)
wl = w[(lo-(t-H)):(hi-(t-H))]
r[t] = np.linalg.norm((series[lo:hi]*wl[:, None]).sum(0))
return r
def boundary_f1(resp, bounds, pct, tol=2):
thr = np.percentile(resp, pct)
pred = np.where(resp > thr)[0]
bidx = [int(b) for b in bounds if int(b) < len(resp)]
if len(pred) == 0 or not bidx:
return 0.0
tp_p = sum(any(abs(p-i) <= tol for i in bidx) for p in pred)
tp_t = sum(any(abs(p-i) <= tol for p in pred) for i in bidx)
P, R = tp_p/len(pred), tp_t/len(bidx)
return 2*P*R/(P+R) if P+R else 0.0
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--manifest", required=True)
ap.add_argument("--audio-dir", default="experiments/dumps/audio_features")
ap.add_argument("--holdout", nargs="+", default=["Scarface", "Sound_of_Metal"])
ap.add_argument("--out", default="experiments/results/scene_boundary")
args = ap.parse_args()
films = [f for f in json.load(open(args.manifest)) if f["slug"] not in args.holdout]
out = {}
for which in ("video", "audio"):
data = [(load_series(f["dump"], args.audio_dir, which)[0], xray_bounds(f["xray"]))
for f in films]
def neg_f1(x):
H = int(round(x[0])); gamma = x[1]; dead = int(round(x[2])); pct = x[3]
if H < 1 or dead >= H: return 0.0
w = ramp_kernel(H, gamma, dead)
f1s = [boundary_f1(response(s, w, H), b, pct) for s, b in data]
return -float(np.mean(f1s))
# bounds: H 1..10s, gamma 0.3..3, dead 0..4s, threshold pct 80..98
res = differential_evolution(
neg_f1, [(1, 10), (0.3, 3.0), (0, 4), (80, 98)],
seed=0, popsize=12, maxiter=25, tol=1e-4, polish=False)
H = int(round(res.x[0])); gamma = float(res.x[1])
dead = int(round(res.x[2])); pct = float(res.x[3])
out[which] = {"H": H, "gamma": gamma, "dead": dead, "pct": pct,
"train_f1": float(-res.fun)}
print(f"[de-ramp] {which}: H={H}s gamma={gamma:.2f} dead={dead}s "
f"pct={pct:.0f} train boundary-F1={-res.fun*100:.1f}%", file=sys.stderr)
Path(args.out).mkdir(parents=True, exist_ok=True)
json.dump(out, open(Path(args.out)/"de_ramp.json", "w"), indent=2)
print(f"[de-ramp] → {args.out}/de_ramp.json", file=sys.stderr)
if __name__ == "__main__":
main()
scripts/scene_detector/density_floor.py
+65
View File
@@ -0,0 +1,65 @@
#!/usr/bin/env python3
"""
density_floor.py — synthesise scene boundaries when detection is starved.
Flood-fill presence snaps each actor claim to the shot it sits in, so a film
whose boundary detector fires almost nothing (Scarface: 1 cut in 171 min) floods
every actor across the whole film. This is a safety floor: when a film's DETECTED
boundary density is far below what a working detector should produce, fill the
long gaps between real detections with uniformly-spaced synthetic boundaries so no
flood-fill span can exceed ~1/target-density.
Design points (measured on the X-Ray corpus):
- The target density is a PRIOR from the central 60 min of films (avoids credits/
intro/outro skew): median ~0.35 scenes/min.
- The trigger is detected-vs-prior, not prior-vs-anything: only fire when detected
density < TRIGGER_FRAC × prior. Legitimately sparse films (long-scene ensembles
like Downton/Many Saints) detect fine and are left alone.
- Real detections are never moved or dropped; synthetic boundaries only subdivide
gaps that are longer than the target scene length.
"""
from __future__ import annotations
PRIOR_SCENES_PER_MIN = 0.35 # central-60min X-Ray median
TRIGGER_FRAC = 0.30 # fire only when detected < 30% of prior
def apply_density_floor(boundaries: list[float], duration_sec: float,
prior_per_min: float = PRIOR_SCENES_PER_MIN,
trigger_frac: float = TRIGGER_FRAC) -> list[float]:
"""Return boundaries augmented with synthetic ones iff detection is starved.
boundaries: detected boundary timestamps (s), any order.
duration_sec: film length.
Returns a sorted list; unchanged (just sorted) when the film is not starved.
"""
b = sorted(t for t in boundaries if 0.0 < t < duration_sec)
minutes = duration_sec / 60.0
if minutes <= 0:
return b
detected_density = len(b) / minutes
if detected_density >= trigger_frac * prior_per_min:
return b # detector produced a reasonable amount — leave it alone
target_gap = 60.0 / prior_per_min # seconds per expected scene
edges = [0.0] + b + [duration_sec]
out = list(b)
for lo, hi in zip(edges[:-1], edges[1:]):
gap = hi - lo
if gap <= target_gap:
continue
n_insert = int(gap // target_gap) # how many synthetic cuts fit
step = gap / (n_insert + 1)
for k in range(1, n_insert + 1):
out.append(lo + k * step)
return sorted(out)
if __name__ == "__main__":
# self-check on the Scarface failure and a healthy film
scar = apply_density_floor([88.0], 171*60) # 1 detected cut, 171 min
print(f"Scarface: 1 detected → {len(scar)} after floor "
f"({len(scar)/171:.2f}/min, prior {PRIOR_SCENES_PER_MIN})")
healthy = apply_density_floor([i*130.0 for i in range(1, 47)], 122*60)
print(f"healthy (46 detected/122min={46/122:.2f}/min): "
f"{len(healthy)} after floor (unchanged = not triggered)")
scripts/scene_detector/downstream_presence.py
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env python3
"""
downstream_presence.py — does the XGBoost scene detector actually improve ACTOR
PRESENCE accuracy? Boundary-F1 is only a proxy; this is the number that decides
whether the detector ships.
For each film, compares presence (per-second X-Ray F1) under three regimes:
A. track_extent — no flood-fill (claim = [first_seen, last_seen])
B. flood + histogram cuts — current shipped flood (snaps to is_cut)
C. flood + XGBoost bounds — inject the detector's boundaries into
is_scene_boundary (flood prefers it over is_cut)
Injection: write a copy of each dump with frames/is_scene_boundary set from the
XGBoost knee boundaries, then replay --presence-mode flood against that copy.
Uses the shipped model (all-9 fit). Scored with second_score at the 10-knob
optimum config.
"""
from __future__ import annotations
import sys, json, shutil, subprocess, tempfile, os
from pathlib import Path
import numpy as np
import h5py
sys.path.insert(0, "scripts/scene_detector")
sys.path.insert(0, "scripts/optimizer")
sys.path.insert(0, "scripts/validation")
import train_xgb_boundary as XB
from second_score import score_seconds
from sample_eval import load_gallery_keys
import xgboost as xgb
GAL = "experiments/galleries/gallery_LVFace-B_Glint360K.h5"
MODEL = "experiments/results/scene_boundary/xgb_boundary_shipped.json"
# 10-knob presence optimum (shipped config)
CFG = ["--prob-threshold", "0.485", "--ownership-logodds", "1.72",
"--track-extinction-sec", "31", "--track-alpha", "0.435",
"--evidence-rho-max", "0.204", "--evidence-admit-below", "0.784",
"--match-prior", "0.433", "--expand-band-lo", "0.804",
"--expand-band-hi", "0.952", "--expand-gallery"]
def xgb_boundary_seconds(reg, dump):
X, yb, ic = XB.per_second_matrix(dump, xr_for(dump), "experiments/dumps/audio_features")
prob = np.clip(reg.predict(X), 0, 1)
return set(XB.knee_boundaries(prob))
FILMS = json.load(open("experiments/manifests/films_LVFace_opencv5.json"))
_XR = {f["dump"]: f["xray"] for f in FILMS}
def xr_for(dump): return _XR[dump]
def inject_boundaries(dump, second_set, out_path):
"""Copy dump, set frames/is_scene_boundary=1 at the given integer seconds."""
shutil.copy(dump, out_path)
with h5py.File(out_path, "r+") as f:
ts = f["frames/timestamp_sec"][:]
bnd = np.zeros(len(ts), np.uint8)
for i, t in enumerate(ts):
if int(round(t)) in second_set:
bnd[i] = 1
if "frames/is_scene_boundary" in f:
f["frames/is_scene_boundary"][:] = bnd
else:
f["frames"].create_dataset("is_scene_boundary", data=bnd)
def replay(dump, out, mode):
argv = [".venv-rocm/bin/python" if False else sys.executable,
"scripts/optimizer/replay.py", "--dump", dump, "--gallery", GAL,
"--out", out] + CFG
if mode:
argv += ["--presence-mode", mode]
subprocess.run(argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=300)
return json.loads(Path(out).read_text())
def main():
reg = xgb.XGBRegressor(); reg.load_model(MODEL)
gk = load_gallery_keys(GAL)
tmp = tempfile.mkdtemp()
print(f"{'film':24s} {'trackext':>9} {'flood+hist':>11} {'flood+XGB':>10}")
agg = {"track_extent": [], "flood_hist": [], "flood_xgb": []}
for f in FILMS:
dump, xr = f["dump"], f["xray"]
out = f"{tmp}/out.json"
# A. track_extent
a = score_seconds(replay(dump, out, "track_extent"), xr, gallery_keys=gk)
# B. flood + histogram cuts (original dump's is_cut; is_scene_boundary=0)
b = score_seconds(replay(dump, out, "flood"), xr, gallery_keys=gk)
# C. flood + XGBoost boundaries injected
inj = f"{tmp}/inj_{f['slug']}.h5"
inject_boundaries(dump, xgb_boundary_seconds(reg, dump), inj)
c = score_seconds(replay(inj, out, "flood"), xr, gallery_keys=gk)
os.unlink(inj)
agg["track_extent"].append(a["f1"]); agg["flood_hist"].append(b["f1"])
agg["flood_xgb"].append(c["f1"])
print(f"{f['name'][:24]:24s} {a['f1']*100:8.1f}% {b['f1']*100:10.1f}% "
f"{c['f1']*100:9.1f}%")
print(f"\n{'MACRO-MEAN':24s} {np.mean(agg['track_extent'])*100:8.1f}% "
f"{np.mean(agg['flood_hist'])*100:10.1f}% {np.mean(agg['flood_xgb'])*100:9.1f}%")
json.dump({k: float(np.mean(v)) for k, v in agg.items()},
open("experiments/results/scene_boundary/downstream_presence.json", "w"),
indent=2)
if __name__ == "__main__":
main()
scripts/scene_detector/extract_audio_features.py
+136
View File
@@ -0,0 +1,136 @@
#!/usr/bin/env python3
"""
extract_audio_features.py — per-second audio features for scene-boundary detection.
Audio is often a stronger scene-boundary cue than video: music swells, silence,
and ambience changes at narrative scene transitions — exactly the coarse
boundaries Amazon X-Ray marks, and exactly what the grayscale video cut detector
misses on low-contrast films. This extracts a small per-second feature series per
film, aligned to the 1 fps timeline the embedding dumps use, so it can be fused
with the RGB-histogram features in train_scene_boundary.py.
Two-tower design: this is the AUDIO tower's input, mirroring the video tower's
per-second RGB histogram. Because the scene model is an LSTM (temporal context
comes from the recurrence, not a 2D spectrogram), each second needs only a single
log-PSD vector — one FFT over a WIN_SEC window centred on that second. The LSTM
sees the sequence of per-second PSDs and learns the boundary dynamics itself.
Per second t:
- log-PSD over [t-WIN/2, t+WIN/2], N_BINS log-spaced frequency bins, L1-norm'd
then log1p — the spectral shape (music vs speech vs silence vs ambience),
which changes at scene transitions.
No new dependency: ffmpeg (CLI) decodes the whole track to mono 16 kHz WAV;
numpy does the FFT.
Writes <out_dir>/<slug>.npz with `ts` (second grid) and `feat` [T, N_BINS].
Usage:
python scripts/scene_detector/extract_audio_features.py \
--manifest experiments/manifests/films_LVFace_opencv5.json \
--file-lut experiments/file-lut.json \
--out experiments/dumps/audio_features
"""
from __future__ import annotations
import argparse, json, subprocess, sys, tempfile, os
from pathlib import Path
import numpy as np
from scipy import signal as sps
from scipy.io import wavfile
SR = 16000
HOP_SEC = 1.0 # one feature vector per second (matches 1 fps presence grid)
WIN_SEC = 4.0 # FFT window per second (centred); >HOP for temporal context
N_BINS = 64 # log-spaced frequency bins per second (the audio tower dim)
def decode_mono(path: str) -> np.ndarray:
"""Whole-file mono 16 kHz float32 PCM via ffmpeg."""
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tf:
wav = tf.name
try:
subprocess.run(
["ffmpeg", "-v", "error", "-y", "-i", path,
"-ac", "1", "-ar", str(SR), "-f", "wav", wav],
check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
sr, x = wavfile.read(wav)
if x.dtype == np.int16:
x = x.astype(np.float32) / 32768.0
else:
x = x.astype(np.float32)
return x
finally:
try: os.unlink(wav)
except OSError: pass
def _logbin_edges(win_samples: int) -> np.ndarray:
"""Indices into the rfft output that bound N_BINS log-spaced freq bands."""
nfreq = win_samples // 2 + 1
# log-space from bin 1 (skip DC) to Nyquist; unique integer edges
edges = np.unique(np.geomspace(1, nfreq - 1, N_BINS + 1).astype(int))
return edges
def features(mono: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
"""Return (ts[T], feat[T, N_BINS]) — one per-second log-PSD row.
One FFT per second over a WIN_SEC window centred on that second. Power is
pooled into N_BINS log-spaced frequency bands (mel-like), L1-normalised across
bands (so loudness doesn't dominate — the SHAPE is the scene cue), then
log1p-compressed. The LSTM downstream supplies temporal context, so no
spectrogram/2D input is needed."""
hop = int(SR * HOP_SEC)
win = int(SR * WIN_SEC)
T = len(mono) // hop
if T == 0:
return np.zeros(0), np.zeros((0, N_BINS), np.float32)
edges = _logbin_edges(win)
nb = len(edges) - 1
hann = sps.windows.hann(win)
feat = np.zeros((T, nb), np.float32)
half = win // 2
for t in range(T):
centre = t * hop + hop // 2
s = centre - half
seg = mono[max(0, s): s + win]
if len(seg) < win: # pad edges
seg = np.pad(seg, (0, win - len(seg)))
psd = np.abs(np.fft.rfft(seg * hann))**2 + 1e-12
band = np.array([psd[edges[i]:edges[i+1]].sum() for i in range(nb)])
band /= band.sum() # normalise shape, drop loudness
feat[t] = np.log1p(band * 1e3)
ts = np.arange(T, dtype=np.float64)
return ts, feat
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--manifest", required=True)
ap.add_argument("--file-lut", default="experiments/file-lut.json")
ap.add_argument("--out", default="experiments/dumps/audio_features")
args = ap.parse_args()
films = json.load(open(args.manifest))
lut = json.load(open(args.file_lut))
Path(args.out).mkdir(parents=True, exist_ok=True)
for f in films:
slug = f["slug"]
outp = Path(args.out) / f"{slug}.npz"
if outp.exists():
print(f"[audio] {slug}: exists, skip", file=sys.stderr); continue
path = lut.get(slug)
if not path or not os.path.exists(path):
print(f"[audio] {slug}: movie missing ({path})", file=sys.stderr); continue
try:
mono = decode_mono(path)
ts, feat = features(mono)
np.savez_compressed(outp, ts=ts, feat=feat)
print(f"[audio] {slug}: {len(ts)}s feat{feat.shape} → {outp.name}",
file=sys.stderr)
except subprocess.CalledProcessError:
print(f"[audio] {slug}: ffmpeg decode failed", file=sys.stderr)
if __name__ == "__main__":
main()
scripts/scene_detector/make_figures.py
+137
View File
@@ -0,0 +1,137 @@
#!/usr/bin/env python3
"""Generate the scene-boundary-detector report figures from saved results.
Data-driven, reproducible, no video needed. Writes PNGs to docs/assets/images/."""
import json
from pathlib import Path
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
OUT = Path("docs/assets/images")
OUT.mkdir(parents=True, exist_ok=True)
plt.rcParams.update({"font.size": 11, "axes.splines.top" if False else "axes.grid": True,
"axes.axisbelow": True, "grid.alpha": 0.3, "figure.dpi": 130})
FILMS = ["Benny & Joon","Café Society","Downton Abbey","Lord of War","Lovelace",
"Many Saints","Scarface","Sound of Metal","Valerian"]
# per-film presence F1 (downstream_loo run): track_extent, flood+grayscale, flood+learned(LOO)
TE = [77.3,59.1,41.0,74.8,70.3,37.5,62.6,75.0,65.6]
FG = [80.2,62.2,51.8,77.1,74.0,43.9,40.9,78.1,67.7]
FL = [78.2,69.8,78.6,77.8,78.2,53.4,74.9,86.8,76.2]
# ── Figure 1: per-film presence F1, three boundary sources ───────────────────
def fig_presence():
x = np.arange(len(FILMS)); w = 0.26
fig, ax = plt.subplots(figsize=(11,5))
ax.bar(x-w, TE, w, label="track-extent (flood off)", color="#9aa7b4")
ax.bar(x, FG, w, label="flood + grayscale cuts", color="#e07a5f")
ax.bar(x+w, FL, w, label="flood + learned detector (LOO)", color="#3d7ea6")
ax.set_ylabel("per-second X-Ray presence F1 (%)")
ax.set_title("Actor-presence accuracy by flood-fill boundary source (leave-one-out)")
ax.set_xticks(x); ax.set_xticklabels(FILMS, rotation=30, ha="right")
ax.set_ylim(0,100); ax.legend(loc="upper left", framealpha=0.9)
# annotate the two headline swings
ax.annotate("grayscale flood\nBREAKS Scarface", xy=(6, 40.9), xytext=(5.1, 20),
fontsize=9, color="#b23", ha="center",
arrowprops=dict(arrowstyle="->", color="#b23"))
ax.annotate("+37pp", xy=(2+w, 78.6), xytext=(2+w, 90), fontsize=9,
color="#3d7ea6", ha="center",
arrowprops=dict(arrowstyle="->", color="#3d7ea6"))
macro=[np.mean(TE),np.mean(FG),np.mean(FL)]
ax.text(0.99,0.02,f"macro: {macro[0]:.1f}% / {macro[1]:.1f}% / {macro[2]:.1f}%",
transform=ax.transAxes, ha="right", va="bottom", fontsize=10,
bbox=dict(boxstyle="round", fc="#f4f4f4", ec="#ccc"))
fig.tight_layout(); fig.savefig(OUT/"scene_presence_by_source.png"); plt.close(fig)
# ── Figure 2: macro presence F1 — the progression ───────────────────────────
def fig_macro():
labels=["track-extent","flood +\ngrayscale","flood +\nlearned (LOO)"]
vals=[np.mean(TE),np.mean(FG),np.mean(FL)]
fig,ax=plt.subplots(figsize=(6,4.5))
bars=ax.bar(labels,vals,color=["#9aa7b4","#e07a5f","#3d7ea6"])
for b,v in zip(bars,vals): ax.text(b.get_x()+b.get_width()/2, v+1, f"{v:.1f}%",
ha="center", fontsize=11, fontweight="bold")
ax.set_ylabel("macro presence F1 (%)"); ax.set_ylim(0,90)
ax.set_title("Flood-fill boundary source → presence accuracy")
fig.tight_layout(); fig.savefig(OUT/"scene_presence_macro.png"); plt.close(fig)
# ── Figure 3: feature/model evolution (boundary-F1 development) ──────────────
# Two panels, because the development curve and the shipped result are measured
# at DIFFERENT tolerances and must not be plotted on one axis:
# left — relative feature progress at the strict ±2 s tolerance (how the LSTM
# experiments were scored; establishes which features helped)
# right — the shipped XGBoost detector at the ±20 s tolerance the pipeline
# actually uses and scores at (grayscale vs learned-LOO vs train-all)
def fig_evolution():
fig,(axl,axr)=plt.subplots(1,2,figsize=(11,4.5),gridspec_kw={"width_ratios":[1.15,1]})
steps=["grayscale\nbaseline","raw-hist\nLSTM","delta\nLSTM","XGBoost\n(delta+debounce)"]
dev=[7.2,7.5,10.8,15.2] # boundary-F1 @±2s during LSTM-era development
axl.plot(steps,dev,marker="o",color="#9aa7b4",lw=2,ms=8)
for i,v in enumerate(dev): axl.text(i,v+0.4,f"{v:.1f}%",ha="center",fontsize=9)
axl.set_ylabel("boundary F1 @±2 s (%)")
axl.set_title("Feature progress (strict ±2 s)")
axl.set_ylim(0,18)
# shipped detector at the ±20s tolerance the pipeline uses — real measured
# macro numbers: grayscale (xgb_report gray_F1), learned LOO, learned train-all
names=["grayscale","learned\n(LOO)","learned\n(train-all)"]
f20=[29.8,44.1,72.9]; cols=["#e07a5f","#3d7ea6","#8fb8cf"]
bars=axr.bar(names,f20,color=cols)
for b,v in zip(bars,f20): axr.text(b.get_x()+b.get_width()/2,v+1.2,f"{v:.1f}%",
ha="center",fontsize=10,fontweight="bold")
axr.set_ylabel("boundary F1 @±20 s (%)")
axr.set_title("Shipped detector (±20 s, macro/9 films)")
axr.set_ylim(0,80)
fig.suptitle("Detector development, and where it landed",fontsize=13)
fig.tight_layout(); fig.savefig(OUT/"scene_detector_evolution.png"); plt.close(fig)
import csv as _csv
# ── Figure 4: DE convergence (the 10-knob presence sweep) ────────────────────
def fig_de():
import json
rows=[json.loads(l) for l in open("experiments/trajectories/lvface_opencv5_10knob.FINAL.jsonl")]
f1=[r["f1"]*100 for r in rows]
run_best=np.maximum.accumulate(f1)
fig,ax=plt.subplots(figsize=(8,4.5))
ax.scatter(range(len(f1)),f1,s=8,alpha=0.35,color="#9aa7b4",label="candidate")
ax.plot(run_best,color="#3d7ea6",lw=2,label="best so far")
ax.set_xlabel("DE evaluation"); ax.set_ylabel("macro presence F1 (%)")
ax.set_title("10-knob presence sweep (Differential Evolution)")
ax.legend(loc="lower right"); ax.set_ylim(0, max(f1)+8)
ax.text(0.02,0.95,f"optimum {max(f1):.1f}%",transform=ax.transAxes,va="top",
fontsize=10,bbox=dict(boxstyle="round",fc="#f4f4f4",ec="#ccc"))
fig.tight_layout(); fig.savefig(OUT/"de_search_landscape.png"); plt.close(fig)
# ── Figure 5: calibration curve (similarity → P(match)) ──────────────────────
def fig_calibration():
sims,ps=[],[]
with open("experiments/galleries/gallery_LVFace-B_Glint360K.h5.calib_cache.csv") as f:
for r in _csv.DictReader(f):
sims.append(float(r["similarity"])); ps.append(float(r["p_match"]))
fig,ax=plt.subplots(figsize=(6.5,4.5))
ax.plot(sims,ps,color="#3d7ea6",lw=2)
ax.axhline(0.485,ls="--",color="#e07a5f",lw=1,label="shipped threshold 0.485")
ax.set_xlabel("cosine similarity"); ax.set_ylabel("calibrated P(match)")
ax.set_title("LVFace-B Glint360K calibration"); ax.set_xlim(-1,1); ax.legend()
fig.tight_layout(); fig.savefig(OUT/"calibration_curves.png"); plt.close(fig)
# ── Figure 6: holdout F1 by film (learned detector, LOO) ─────────────────────
def fig_holdout():
order=np.argsort(FL)
fig,ax=plt.subplots(figsize=(8,4.5))
y=np.arange(len(FILMS))
ax.barh(y,[FL[i] for i in order],color="#3d7ea6")
ax.set_yticks(y); ax.set_yticklabels([FILMS[i] for i in order])
ax.set_xlabel("presence F1 (%), learned detector (LOO)")
ax.set_title("Per-film presence F1 — leave-one-out")
ax.axvline(np.mean(FL),ls="--",color="#333",lw=1)
ax.text(np.mean(FL)+1,0.2,f"macro {np.mean(FL):.1f}%",fontsize=9)
for i,idx in enumerate(order): ax.text(FL[idx]+0.5,i,f"{FL[idx]:.0f}",va="center",fontsize=8)
ax.set_xlim(0,100)
fig.tight_layout(); fig.savefig(OUT/"holdout_f1_by_film.png"); plt.close(fig)
fig_presence(); fig_macro(); fig_evolution(); fig_de(); fig_calibration(); fig_holdout()
print("wrote:", *(p.name for p in sorted(OUT.glob("*.png"))))
scripts/scene_detector/rematch_frames.py
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env python3
"""
rematch_frames.py — remake each named July frame example against the CURRENT
pipeline. For a file named <film>_<...>_<actor>.jpg, find a second in this film's
replay where that actor is drawn in the matching class (FP for *_fpi_*, TP for
*_tp/perfect*), extract + annotate it, and write it over the doc asset. Reports
which July examples no longer reproduce (honest — the config/model changed).
Needs the per-film raw replay (experiments/dumps + replay --raw-out already run by
regen_frame_examples.sh into the scratch predictions). Reads those.
"""
from __future__ import annotations
import json, sys, subprocess, re
from pathlib import Path
sys.path.insert(0, "scripts/optimizer"); sys.path.insert(0, "scripts/validation")
import dump_error_frames as D
from second_score import load_second_timeline, _match
SP = Path("/tmp/claude-1000/-home-dtourolle-Development-scene-actor-extraction/"
"c579f8cf-2974-4cbd-be88-afec68dbbf58/scratchpad")
ASSETS = Path("docs/assets/images")
LUT = json.load(open("experiments/file-lut.json"))
FILMS = json.load(open("experiments/manifests/films_LVFace_opencv5.json"))
XR = {f["slug"]: f["xray"] for f in FILMS}
# filename → (film slug, actor substring, class). class: "fp" | "tp".
# actor substring is matched case-insensitively against drawn names.
JOBS = {
"lord_of_war_fpi_reddick.jpg": ("Lord_of_War", "reddick", "fp"),
"lord_of_war_fpi_shumbris.jpg": ("Lord_of_War", "shumbris", "fp"),
"lord_of_war_fpi_reagan_photo.jpg": ("Lord_of_War", "reagan", "fp"),
"lovelace_fpi_sevigny.jpg": ("Lovelace", "sevigny", "fp"),
"lovelace_robert_patrick_fpi.jpg": ("Lovelace", "patrick", "fp"),
"lovelace_perfect_second.jpg": ("Lovelace", None, "tp"),
"lovelace_polygraph_bridged.jpg": ("Lovelace", None, "tp"),
"many_saints_fpi_deschanel.jpg": ("The_Many_Saints_of_Newark", "deschanel", "fp"),
"many_saints_fpi_gardner.jpg": ("The_Many_Saints_of_Newark", "gardner", "fp"),
"many_saints_fpi_yates.jpg": ("The_Many_Saints_of_Newark", "yates", "fp"),
"many_saints_outofcast_fpi.jpg": ("The_Many_Saints_of_Newark", None, "fp"),
"scarface_fpi_alley.jpg": ("Scarface", "alley", "fp"),
"downton_crew_fn.jpg": ("Downton_Abbey__A_New_Era", None, "tp"),
"downton_wedding_couple.jpg": ("Downton_Abbey__A_New_Era", None, "tp"),
"downton_tp_example.jpg": ("Downton_Abbey__A_New_Era", None, "tp"),
"valerian_screen_call.jpg": ("Valerian_and_the_City_of_a_Thousand_Plan", None, "tp"),
"cafe_society_rapid_cut.jpg": ("Café_Society", None, "tp"),
# germar_beats_xray / downton_funeral_19of20 are July-narrative-specific; skip.
}
def gt_keysets(slug):
tl, _, _ = load_second_timeline(XR[slug])
return tl
def main():
made, missing = [], []
for fname, (slug, actor, cls) in JOBS.items():
raw = SP / f"{slug}_raw.jsonl"
if not raw.exists():
missing.append((fname, "no raw replay")); continue
tl = gt_keysets(slug)
best = None # (t, actor_dict, fp_keys)
for line in open(raw):
d = json.loads(line)
t = int(d["timestamp_sec"])
drawn = [a for a in d.get("visible_actors", []) if a.get("actor_idx", -1) >= 0]
if not drawn:
continue
gt = tl.get(t, [])
fp_keys = {D._name_key(a["name"]) for a in drawn
if not any(D._name_key(a["name"]) in g for g in gt)}
for a in drawn:
nk = D._name_key(a["name"]); is_fp = nk in fp_keys
if actor and actor not in a["name"].lower():
continue
match = (is_fp if cls == "fp" else not is_fp)
if not match:
continue
# prefer high similarity + a clean single-subject frame
score = a["similarity"] - 0.05*len(drawn)
if best is None or score > best[3]:
best = (t, d, fp_keys, score)
if best is None:
missing.append((fname, f"no current {cls} for {actor or 'any'}")); continue
t, d, fp_keys, _ = best
# FN names at t: X-Ray scene cast whose keyset matches no drawn face.
gt = tl.get(t, [])
drawn_keys = [set(D._name_key(a["name"]).replace("name:", "") for _ in [0])
for a in d.get("visible_actors", []) if a.get("actor_idx", -1) >= 0]
drawn_ks = [D._name_key(a["name"]) for a in d.get("visible_actors", [])
if a.get("actor_idx", -1) >= 0]
fn_names = []
for ga in gt:
if not any(dk in ga for dk in drawn_ks):
readable = sorted(x for x in ga
if not x.startswith("imdb:") and not x.startswith("tmdb:")
and not x.startswith("jf:"))
if readable:
fn_names.append(readable[0])
out = ASSETS / fname
try:
D.extract_frame(LUT[slug], t, out)
D.draw_annotations(out, d["visible_actors"], fp_keys=fp_keys,
fn_names=fn_names)
made.append((fname, slug, t))
except subprocess.CalledProcessError:
missing.append((fname, "ffmpeg failed"))
print("=== remade ===")
for f, s, t in made: print(f" {f} ({s} t={t}s)")
print("=== no current equivalent (left as-is / flag in doc) ===")
for f, why in missing: print(f" {f} — {why}")
if __name__ == "__main__":
main()
scripts/scene_detector/run_de_audio_cutter.py
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env python3
"""Standalone DE-optimised AUDIO scene cutter: tune a matched-filter ramp on the
audio log-PSD to maximise X-Ray boundary F1. No neural net. Holdout films are
never seen in training. Writes the tuned filter + held-out performance."""
import sys, json, os
import numpy as np
sys.path.insert(0, "scripts/scene_detector")
from de_ramp import load_series, xray_bounds, ramp_kernel, response, boundary_f1
from scipy.optimize import differential_evolution
MANIFEST = "experiments/manifests/films_LVFace_opencv5.json"
AUDIO = "experiments/dumps/audio_features"
HOLDOUT = {"Scarface", "Sound_of_Metal", "Valerian_and_the_City_of_a_Thousand_Plan"}
OUT = "experiments/results/scene_boundary/de_audio_cutter.json"
films = json.load(open(MANIFEST))
train = [f for f in films if f["slug"] not in HOLDOUT]
val = [f for f in films if f["slug"] in HOLDOUT]
tr = [(load_series(f["dump"], AUDIO, "audio")[0], xray_bounds(f["xray"])) for f in train]
va = [(f["slug"], load_series(f["dump"], AUDIO, "audio")[0], xray_bounds(f["xray"])) for f in val]
print(f"DE AUDIO cutter: {len(tr)} train, holdout {sorted(HOLDOUT)}", flush=True)
def neg_f1(x):
H = int(round(x[0])); gamma = x[1]; dead = int(round(x[2])); pct = x[3]
if H < 1 or dead >= H: return 0.0
w = ramp_kernel(H, gamma, dead)
return -float(np.mean([boundary_f1(response(s, w, H), b, pct) for s, b in tr]))
evals = [0]
def cb(xk, convergence):
evals[0] += 1
print(f"[de-audio] gen {evals[0]} convergence={convergence:.3f}", flush=True)
res = differential_evolution(neg_f1, [(1, 10), (0.3, 3.0), (0, 4), (80, 98)],
seed=0, popsize=12, maxiter=25, tol=1e-4,
polish=False, callback=cb)
H = int(round(res.x[0])); gamma = float(res.x[1]); dead = int(round(res.x[2])); pct = float(res.x[3])
print(f"\n=== DE-OPTIMISED AUDIO SCENE CUTTER ===", flush=True)
print(f"tuned ramp: H={H}s gamma={gamma:.2f} dead={dead}s threshold_pct={pct:.0f}", flush=True)
print(f"train boundary-F1: {-res.fun*100:.1f}%\n", flush=True)
print("held-out (audio-only, P/R/F1 ±2s):", flush=True)
w = ramp_kernel(H, gamma, dead)
rep = {"H": H, "gamma": gamma, "dead": dead, "pct": pct,
"train_f1": float(-res.fun), "holdout": sorted(HOLDOUT), "films": {}}
for slug, s, b in va:
r = response(s, w, H); thr = np.percentile(r, pct); pred = np.where(r > thr)[0]
bidx = [int(x) for x in b if int(x) < len(r)]
tp_p = sum(any(abs(p-i) <= 2 for i in bidx) for p in pred)
tp_t = sum(any(abs(p-i) <= 2 for p in pred) for i in bidx)
P = tp_p/max(len(pred), 1); R = tp_t/max(len(bidx), 1); F = 2*P*R/(P+R) if P+R else 0
rep["films"][slug] = {"P": P, "R": R, "F1": F, "n_pred": len(pred), "n_true": len(bidx)}
print(f" {slug[:26]:26s} P={P*100:4.0f}% R={R*100:4.0f}% F1={F*100:4.0f}% "
f"({len(pred)} preds/{len(bidx)} true)", flush=True)
os.makedirs(os.path.dirname(OUT), exist_ok=True)
json.dump(rep, open(OUT, "w"), indent=2)
print(f"\nsaved → {OUT}", flush=True)
scripts/scene_detector/train_scene_boundary.py
+356
View File
@@ -0,0 +1,356 @@
#!/usr/bin/env python3
"""
train_scene_boundary.py — learn a scene-boundary detector from per-frame RGB
histograms (video tower) and per-second audio log-PSD (audio tower), against
Amazon X-Ray scene boundaries.
Motivation: the shipped grayscale histogram-correlation cut detector is blind on
low-contrast grades — on Scarface it fired ONCE in 10,204 frames, so flood-fill
presence (which snaps to detected boundaries) floods every actor across the whole
film (P=26%). X-Ray ships real scene boundaries (scenes.csv); the dumps carry a
per-frame RGB histogram (frames/rgb_hist), and extract_audio_features.py provides
a per-second audio log-PSD. This learns a per-second boundary probability.
TWO-TOWER, ABLATABLE. We do NOT assume audio helps video — we measure it. Each
modality has its own encoder+BiLSTM; --modality selects video / audio / fused
(both towers concatenated before a shared head). The script reports all three
arms on the held-out films so the ablation decides whether audio supports video.
Video features per second: rgb_hist (96) + L1 deltas to t-1,t-2,t+1 + per-channel
correlation to t-1. Audio features: the log-PSD row (+ its L1 delta to t-1).
Label: 1 if an X-Ray scene starts within ±TOL_SEC of t.
Usage:
python scripts/scene_detector/train_scene_boundary.py \
--manifest experiments/manifests/films_LVFace_opencv5.json \
--audio-dir experiments/dumps/audio_features \
--holdout Scarface Sound_of_Metal \
--modality all --out experiments/results/scene_boundary
"""
from __future__ import annotations
import argparse, csv, json, sys
from pathlib import Path
import h5py
import numpy as np
import torch
import torch.nn as nn
TOL_SEC = 2.0
BINS = 32 # per channel, matches embedding_dump_node.hpp kHistBins
RAMP_SCALES = [2, 4, 6, 8, 10] # multi-scale matched-filter half-widths (seconds)
SCENE_TAU = 205.0 # corpus mean X-Ray scene length (central-60min); debounce scale
def debounce_phase(delta_signal: np.ndarray, tau: float = SCENE_TAU,
peak_pct: float = 90.0) -> np.ndarray:
"""A scene-length-scaled 'how overdue is a boundary' feature, [T,2].
Encodes the prior that scenes don't restart moments apart. From the strong
peaks of a change signal (the presumed boundaries so far), track time since
the last peak and turn it into:
phase = min(1, dt/tau) — 0 just after a boundary (suppress), 1 when a new
one is overdue (permit), rising over ~one mean
scene length (tau).
decay = exp(-dt/tau) — the complementary refractory (high right after,
decaying away). Two views of the same clock so
the LSTM can use whichever helps.
Reference peaks come from the change signal itself (not the model's own
output), so the feature is static and causal-ish (uses only |Δ| already in
the sequence)."""
T = len(delta_signal)
thr = np.percentile(delta_signal, peak_pct)
# Vectorised time-since-last-peak: index of the most recent peak at or before
# each t (running max of peak indices), then dt = t - that index.
idx = np.arange(T)
peak_idx = np.where(delta_signal > thr, idx, -1)
last = np.maximum.accumulate(peak_idx) # most recent peak index ≤ t
dt = (idx - last).astype(np.float32)
dt[last < 0] = tau # before the first peak: treat as "overdue"
phase = np.minimum(1.0, dt / tau)
decay = np.exp(-dt / tau)
return np.stack([phase, decay], 1).astype(np.float32)
def ramp_bank(series: np.ndarray) -> np.ndarray:
"""Antisymmetric matched-filter responses at RAMP_SCALES → [T, len(scales)].
A scene boundary is a step in the feature series; a signed ramp kernel
convolved with it responds at the transition and ~0 inside a stable scene.
Different films' boundaries peak at different scales (measured: sharp cuts at
H=2s, gradual shifts wider), so we hand the model the whole bank and let it
weight the scales rather than committing to one width."""
# Vectorised: the ramp response at t is || sum_l w(l)·series[t+l] ||, i.e. a
# 1D correlation of the kernel with each feature bin, then an L2 over bins. Do
# it as one convolution per bin (np.convolve, 'same') instead of the per-frame
# Python loop — ~100x faster, which matters at ~60k frames × 9 films.
T, D = series.shape
out = np.zeros((T, len(RAMP_SCALES)), np.float32)
for k, H in enumerate(RAMP_SCALES):
lags = np.arange(-H, H + 1)
w = (np.sign(lags) * (np.abs(lags) / max(H, 1))).astype(np.float64)
# correlation = convolution with the reversed kernel; ramp is antisym so
# reversing negates it — sign folds into the L2 norm, so either is fine.
acc = np.zeros((T, D))
for d in range(D):
acc[:, d] = np.convolve(series[:, d], w[::-1], mode="same")
out[:, k] = np.linalg.norm(acc, axis=1)
return out
# ── data ──────────────────────────────────────────────────────────────────────
def load_xray_boundaries(xray_dir: str) -> list[float]:
starts = []
with open(Path(xray_dir) / "scenes.csv", newline="") as f:
for r in csv.DictReader(f):
s = float(r["start"]) / 1000.0
if s > 0.5:
starts.append(s)
return sorted(starts)
def _znorm(s):
return (s - s.mean(0)) / (s.std(0) + 1e-6)
def video_features(hist: np.ndarray) -> np.ndarray:
"""DELTA-FORWARD video features.
Measured on the corpus: the raw 96-bin histogram barely separates X-Ray
boundaries (~1.4x boundary response) — it encodes what the frame *looks like*,
not that it *changed* — while the symmetric histogram delta |hist(t+k)-hist(t-k)|
separates them strongly (|Δ 1s| ~4-5x). Feeding 96 dims of raw content
diluted the LSTM, so we drop it and lead with multi-scale symmetric deltas,
keeping only a compact per-channel-energy summary as context.
Channels:
- symmetric L1 delta |hist(t+k) - hist(t-k)| at k=1,2,4,8s (the boundary cue)
- per-channel correlation to the previous second (3)
- the multi-scale antisymmetric ramp bank (regional step response)
- 3-D per-channel total energy (compact content context, not the full hist)
"""
T = hist.shape[0]
def sym_delta(k):
fwd = np.roll(hist, -k, 0); fwd[-k:] = hist[-1]
bwd = np.roll(hist, k, 0); bwd[:k] = hist[0]
return np.abs(fwd - bwd).sum(1, keepdims=True)
deltas = np.concatenate([sym_delta(k) for k in (1, 2, 4, 8)], 1)
p1 = np.roll(hist, 1, 0); p1[0] = hist[0]
corr = np.zeros((T, 3), np.float32)
for c in range(3):
a = hist[:, c*BINS:(c+1)*BINS]; b = p1[:, c*BINS:(c+1)*BINS]
am, bm = a - a.mean(1, keepdims=True), b - b.mean(1, keepdims=True)
corr[:, c] = (am*bm).sum(1) / (np.sqrt((am*am).sum(1)*(bm*bm).sum(1))+1e-9)
energy = np.stack([hist[:, c*BINS:(c+1)*BINS].sum(1) for c in range(3)], 1)
# scene-length-scaled debounce: 'how overdue is a boundary', from the |Δ1s|
# change signal. Encodes that scenes don't restart moments apart (tau=205s).
debounce = debounce_phase(deltas[:, 0])
return np.concatenate([deltas, corr, ramp_bank(_znorm(hist)), energy, debounce],
1).astype(np.float32)
def audio_features(psd: np.ndarray) -> np.ndarray:
"""DELTA-FORWARD audio features (same principle as video).
The raw log-PSD is spectral CONTENT (what the audio sounds like), which the DE
cutter showed barely localizes X-Ray boundaries. Lead with the CHANGE in the
spectrum — symmetric PSD deltas |psd(t+k)-psd(t-k)| at several scales — plus
the ramp bank and a compact total-energy summary; drop the full raw PSD.
"""
def sym_delta(k):
fwd = np.roll(psd, -k, 0); fwd[-k:] = psd[-1]
bwd = np.roll(psd, k, 0); bwd[:k] = psd[0]
return np.abs(fwd - bwd).sum(1, keepdims=True)
deltas = np.concatenate([sym_delta(k) for k in (1, 2, 4, 8)], 1)
energy = psd.sum(1, keepdims=True)
debounce = debounce_phase(deltas[:, 0])
return np.concatenate([deltas, ramp_bank(_znorm(psd)), energy, debounce],
1).astype(np.float32)
def build_film(dump: str, xray_dir: str, audio_dir: str | None):
with h5py.File(dump, "r") as f:
if "frames/rgb_hist" not in f:
raise SystemExit(f"{dump}: no frames/rgb_hist — re-dump with the "
f"RGB-histogram build of dump_embeddings.")
hist = f["frames/rgb_hist"][:].astype(np.float32)
ts = f["frames/timestamp_sec"][:]
is_cut = f["frames/is_cut"][:].astype(np.int64)
V = video_features(hist)
A = None
if audio_dir:
slug = Path(dump).stem.replace("dump_", "")
ap = Path(audio_dir) / f"{slug}.npz"
if ap.exists():
z = np.load(ap); af = z["feat"]
# align audio (per-second) to the video frame grid by index; pad/truncate
T = len(ts); B = af.shape[1]
aligned = np.zeros((T, B), np.float32)
m = min(T, len(af)); aligned[:m] = af[:m]
A = audio_features(aligned)
y = np.zeros(len(ts), np.float32)
for b in load_xray_boundaries(xray_dir):
y[np.abs(ts - b) <= TOL_SEC] = 1.0
return V, A, y, is_cut, ts
# ── model ─────────────────────────────────────────────────────────────────────
class Tower(nn.Module):
"""Per-second encoder → BiLSTM → per-timestep embedding."""
def __init__(self, in_dim, hidden=64, out=64):
super().__init__()
self.enc = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU())
self.lstm = nn.LSTM(hidden, out, batch_first=True, bidirectional=True)
def forward(self, x):
h, _ = self.lstm(self.enc(x))
return h # [B,T,2*out]
class BoundaryNet(nn.Module):
def __init__(self, v_dim, a_dim, modality):
super().__init__()
self.modality = modality
feat = 0
if modality in ("video", "fused"):
self.vtower = Tower(v_dim); feat += 128
if modality in ("audio", "fused"):
self.atower = Tower(a_dim); feat += 128
self.head = nn.Sequential(nn.Linear(feat, 32), nn.ReLU(), nn.Linear(32, 1))
def forward(self, v, a):
parts = []
if self.modality in ("video", "fused"): parts.append(self.vtower(v))
if self.modality in ("audio", "fused"): parts.append(self.atower(a))
return self.head(torch.cat(parts, -1)).squeeze(-1)
def nms_peaks(prob, thr=0.5, min_gap=5):
"""Collapse each run of adjacent above-threshold seconds to its single peak.
Without this, a model that fires 5 consecutive seconds around one true
boundary is scored as 1 TP + 4 FP — an aggregation artifact, not an error."""
cand = np.where(prob > thr)[0]
if len(cand) == 0:
return []
peaks, group = [], [cand[0]]
for c in cand[1:]:
if c - group[-1] <= min_gap:
group.append(c)
else:
peaks.append(group[int(np.argmax(prob[group]))]); group = [c]
peaks.append(group[int(np.argmax(prob[group]))])
return peaks
def prf(prob_or_pred, y, tol=2, thr=0.5):
"""Boundary P/R/F1 with NMS peak aggregation. Accepts a probability series
(model output) or a 0/1 array (is_cut baseline); NMS collapses each run of
above-threshold seconds to one peak either way."""
P = np.array(nms_peaks(np.asarray(prob_or_pred, float), thr=thr))
T = np.where(y > 0.5)[0]
if len(P) == 0 or len(T) == 0: return 0., 0., 0.
tp_p = sum(any(abs(p-t) <= tol for t in T) for p in P)
tp_t = sum(any(abs(p-t) <= tol for p in P) for t in T)
pr, rc = tp_p/len(P), tp_t/len(T)
return pr, rc, (2*pr*rc/(pr+rc) if pr+rc else 0.)
def train_arm(modality, tr, va, v_dim, a_dim, vmu, vsd, amu, asd, epochs, dev):
model = BoundaryNet(v_dim, a_dim, modality).to(dev)
opt = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-5)
pos = sum((y > .5).sum() for *_, y, _, _ in tr)
neg = sum((y <= .5).sum() for *_, y, _, _ in tr)
lossf = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([neg/max(pos,1)], device=dev))
def vt(V): return torch.tensor((V-vmu)/vsd, dtype=torch.float32, device=dev).unsqueeze(0)
def at(A): return torch.tensor((A-amu)/asd, dtype=torch.float32, device=dev).unsqueeze(0)
for ep in range(epochs):
model.train()
for V, A, y, _, _ in tr:
opt.zero_grad()
logit = model(vt(V), at(A) if A is not None else None)
loss = lossf(logit, torch.tensor(y, device=dev).unsqueeze(0))
loss.backward(); opt.step()
model.eval(); rows = {}
with torch.no_grad():
for slug, V, A, y, is_cut, ts in va:
prob = torch.sigmoid(model(vt(V), at(A) if A is not None else None))[0].cpu().numpy()
rows[slug] = prf(prob, y) # raw prob → NMS picks peaks by height
return model, rows
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--manifest", required=True)
ap.add_argument("--audio-dir", default="experiments/dumps/audio_features")
ap.add_argument("--holdout", nargs="+", default=["Scarface", "Sound_of_Metal"])
ap.add_argument("--modality", choices=["video","audio","fused","all"], default="all")
ap.add_argument("--out", default="experiments/results/scene_boundary")
ap.add_argument("--epochs", type=int, default=250)
ap.add_argument("--seed", type=int, default=0)
args = ap.parse_args()
torch.manual_seed(args.seed); np.random.seed(args.seed)
films = json.load(open(args.manifest))
def load(rows):
out = []
for f in rows:
V, A, y, is_cut, ts = build_film(f["dump"], f["xray"], args.audio_dir)
out.append((f["slug"], V, A, y, is_cut, ts))
return out
tr = load([f for f in films if f["slug"] not in args.holdout])
va = load([f for f in films if f["slug"] in args.holdout])
has_audio = all(t[2] is not None for t in tr+va)
print(f"[scene] train {len(tr)} / holdout {args.holdout}; audio={'yes' if has_audio else 'MISSING'}",
file=sys.stderr)
allV = np.concatenate([t[1] for t in tr], 0)
vmu, vsd = allV.mean(0), allV.std(0)+1e-6; v_dim = allV.shape[1]
if has_audio:
allA = np.concatenate([t[2] for t in tr], 0)
amu, asd = allA.mean(0), allA.std(0)+1e-6; a_dim = allA.shape[1]
else:
amu = asd = None; a_dim = 1
# strip index tuples for train_arm (expects V,A,y,is_cut,ts)
trA = [(t[1],t[2],t[3],t[4],t[5]) for t in tr]
dev = "cuda" if torch.cuda.is_available() else "cpu"
modes = ["video","audio","fused"] if args.modality=="all" else [args.modality]
if not has_audio: modes = [m for m in modes if m == "video"] or ["video"]
# grayscale-0.70 baseline (is_cut) on holdout
print("\n=== held-out scene-boundary detection (P/R/F1, ±2s) ===")
print(f"{'film':26s} " + " ".join(f"{m:>16s}" for m in modes) + f" {'grayscale-0.70':>16s}")
Path(args.out).mkdir(parents=True, exist_ok=True)
results = {m: train_arm(m, trA, va, v_dim, a_dim, vmu, vsd, amu, asd, args.epochs, dev)
for m in modes}
report = {"holdout": args.holdout, "tol_sec": TOL_SEC, "modalities": {}, "films": {}}
for slug, V, A, y, is_cut, ts in va:
cells = []
for m in modes:
p,r,f = results[m][1][slug]
cells.append(f"{p*100:4.0f}/{r*100:4.0f}/{f*100:4.0f}")
report["films"].setdefault(slug, {})[m] = {"P":p,"R":r,"F1":f}
bp,br,bf = prf(is_cut, y)
report["films"].setdefault(slug, {})["grayscale"] = {"P":bp,"R":br,"F1":bf}
print(f"{slug:26s} " + " ".join(f"{c:>16s}" for c in cells) +
f" {bp*100:4.0f}/{br*100:4.0f}/{bf*100:4.0f}")
# macro-mean F1 per modality across holdout
print("\nmacro-mean holdout F1:")
for m in modes:
mf = np.mean([results[m][1][s][2] for s,*_ in va])
report["modalities"][m] = float(mf)
print(f" {m:8s} {mf*100:.1f}%")
bf = np.mean([prf(t[4], t[3])[2] for t in va])
report["modalities"]["grayscale"] = float(bf)
print(f" {'grayscale':8s} {bf*100:.1f}%")
# save the best arm
best = max(modes, key=lambda m: report["modalities"][m])
torch.save({"state": results[best][0].state_dict(), "modality": best,
"vmu":vmu,"vsd":vsd,"amu":amu,"asd":asd,"v_dim":v_dim,"a_dim":a_dim},
Path(args.out)/"boundary_net.pt")
json.dump(report, open(Path(args.out)/"report.json","w"), indent=2)
print(f"\n[scene] best={best}; model+report → {args.out}", file=sys.stderr)
if __name__ == "__main__":
main()
scripts/scene_detector/train_xgb_boundary.py
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""
train_xgb_boundary.py — SHIPPED scene-boundary detector.
An XGBoost regressor over a ±WIN-second window of delta features predicts a soft
Gaussian proximity-to-boundary target; a per-film KNEE threshold on the predicted
peak heights selects the boundaries (self-calibrates the count without a magic
rate). Evaluated with NMS + P/R/F1 at ±20 s tolerance (X-Ray scenes are ~170 s,
so ±20 s placement is what flood-fill actually needs).
Why this shape (all measured, see docs/scene-detector):
- DELTA features, not raw histogram/PSD: the raw content dilutes; |Δ| separates
boundaries 4-5x. Audio is weak but included (XGBoost ignores what it can't use).
- SOFT target exp(-(d/σ)²), σ=10s: a near-miss is trained as near-correct, not a
hard negative. Regression → smooth score surface → NMS peaks.
- KNEE threshold per film: peak-height curve has a knee where real boundaries
give way to noise; picking it matches the true scene count without a global
threshold that's wrong for every grade.
- Café Society + Scarface (low-contrast grades) MUST be in training; held out,
the model can't generalize to them. The shipped model trains on ALL 9.
Honest generalization: leave-one-out CV ≈ 26% F1 @±10s / ~34% @±20s. The shipped
all-9 model is what deployment uses (max grade coverage); LOO is the number to
quote for a brand-new film.
Usage (train on all 9 + save shipped model):
.venv-rocm/bin/python scripts/scene_detector/train_xgb_boundary.py --train-all
Usage (held-out eval):
... --holdout Sound_of_Metal The_Many_Saints_of_Newark Valerian_...
"""
from __future__ import annotations
import argparse, json, sys
from pathlib import Path
import numpy as np
import h5py
sys.path.insert(0, "scripts/scene_detector")
from train_scene_boundary import nms_peaks, load_xray_boundaries, SCENE_TAU, TOL_SEC
from train_scene_boundary import video_features, audio_features, build_film
from scipy.signal import find_peaks
import xgboost as xgb
WIN = 3 # ±WIN-second context window
SIGMA = 10.0 # soft-target Gaussian width (seconds)
def per_second_matrix(dump, xray, audio_dir, win=None):
"""Windowed delta features + debounce clock → (X[T,F], y_binary[T], is_cut[T])."""
V, A, y, is_cut, ts = build_film(dump, xray, audio_dir)
base = np.concatenate([V] + ([A] if A is not None else []), 1)
T, d = base.shape
sig = V[:, 0]
thr = np.percentile(sig, 90)
idx = np.arange(T); peak = np.where(sig > thr, idx, -1)
last = np.maximum.accumulate(peak)
dt = (idx - last).astype(np.float32); dt[last < 0] = SCENE_TAU
clock = np.stack([dt, np.minimum(1, dt/SCENE_TAU), np.exp(-dt/SCENE_TAU)], 1)
W = WIN if win is None else win
padded = np.pad(base, ((W, W), (0, 0)), mode="edge")
wf = np.concatenate([padded[i:i+T] for i in range(2*W+1)], 1)
return np.concatenate([wf, clock], 1).astype(np.float32), y, is_cut
def soft_target(dump, xray):
ts = h5py.File(dump)["frames/timestamp_sec"][:]
b = np.array(load_xray_boundaries(xray))
y = np.zeros(len(ts), np.float32)
if len(b):
for i, t in enumerate(ts):
y[i] = np.exp(-((np.min(np.abs(b - t)))/SIGMA)**2)
return y
def knee_boundaries(prob, min_gap=5):
"""Per-film knee threshold on peak heights → selected peak indices.
Peaks sorted by height form a convex-decreasing curve; the knee (max drop
below the endpoints chord) is where real boundaries give way to noise. Returns
the timestamps (indices) of peaks at or above the knee height."""
pk, _ = find_peaks(prob, distance=min_gap)
if len(pk) < 5:
return list(pk)
heights = np.sort(prob[pk])[::-1]
n = len(heights); x = np.arange(n)/(n-1); yv = heights/(heights[0]+1e-9)
chord = yv[0] + (yv[-1]-yv[0])*x
k = int(np.argmax(chord - yv))
thr = heights[k]
return [int(i) for i in pk if prob[i] >= thr]
def train(films, audio_dir):
X = np.concatenate([per_second_matrix(f["dump"], f["xray"], audio_dir)[0] for f in films])
y = np.concatenate([soft_target(f["dump"], f["xray"]) for f in films])
reg = xgb.XGBRegressor(n_estimators=400, max_depth=5, learning_rate=0.05,
subsample=0.8, colsample_bytree=0.8,
objective="reg:squarederror", n_jobs=8, tree_method="hist")
reg.fit(X, y)
return reg
def prf(peaks, Tset, tol=20):
if not peaks or len(Tset) == 0:
return 0., 0., 0., 0, 0, len(Tset)
tp_p = sum(any(abs(p-t) <= tol for t in Tset) for p in peaks)
tp_t = sum(any(abs(p-t) <= tol for p in peaks) for t in Tset)
P = tp_p/len(peaks); R = tp_t/len(Tset)
return (P, R, (2*P*R/(P+R) if P+R else 0.),
tp_p, len(peaks)-tp_p, len(Tset)-tp_t)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--manifest", default="experiments/manifests/films_LVFace_opencv5.json")
ap.add_argument("--audio-dir", default="experiments/dumps/audio_features")
ap.add_argument("--holdout", nargs="*", default=[])
ap.add_argument("--train-all", action="store_true", help="train on all 9 + save shipped model")
ap.add_argument("--tol", type=int, default=20)
ap.add_argument("--out", default="experiments/results/scene_boundary")
args = ap.parse_args()
films = json.load(open(args.manifest))
Path(args.out).mkdir(parents=True, exist_ok=True)
tr = films if args.train_all else [f for f in films if f["slug"] not in args.holdout]
reg = train(tr, args.audio_dir)
print(f"[xgb] trained on {len(tr)} films", file=sys.stderr)
ev = films if args.train_all else [f for f in films if f["slug"] in args.holdout]
tag = "TRAIN-FIT (all 9)" if args.train_all else "HELD-OUT"
print(f"\n=== {tag} boundary detection (knee, NMS, ±{args.tol}s) ===")
print(f"{'film':26s} {'TP':>4}{'FP':>5}{'FN':>5} {'P':>5}{'R':>5}{'F1':>5} {'gray F1':>7}")
rep = {"win": WIN, "sigma": SIGMA, "tol": args.tol, "train_all": args.train_all,
"holdout": args.holdout, "films": {}}
f1s, gf1s = [], []
for f in ev:
X, yb, ic = per_second_matrix(f["dump"], f["xray"], args.audio_dir)
prob = np.clip(reg.predict(X), 0, 1)
peaks = knee_boundaries(prob)
Tset = np.where(yb > 0.5)[0]
P, R, F, tp, fp, fn = prf(peaks, Tset, args.tol)
gpk = nms_peaks(ic.astype(float)); _, _, gF, *_ = prf(gpk, Tset, args.tol)
f1s.append(F); gf1s.append(gF)
rep["films"][f["slug"]] = {"TP": tp, "FP": fp, "FN": fn, "P": P, "R": R, "F1": F,
"n_pred": len(peaks), "n_true": len(Tset), "gray_F1": gF}
print(f"{f['slug'][:26]:26s} {tp:>4}{fp:>5}{fn:>5} {P*100:4.0f}%{R*100:4.0f}%"
f"{F*100:4.0f}% {gF*100:5.0f}%")
print(f"\nmacro-F1: detector {np.mean(f1s)*100:.1f}% grayscale {np.mean(gf1s)*100:.1f}%")
rep["macro_f1"] = {"detector": float(np.mean(f1s)), "grayscale": float(np.mean(gf1s))}
if args.train_all:
reg.save_model(str(Path(args.out) / "xgb_boundary_shipped.json"))
print(f"[xgb] shipped model → {args.out}/xgb_boundary_shipped.json", file=sys.stderr)
json.dump(rep, open(Path(args.out) / "xgb_report.json", "w"), indent=2)
if __name__ == "__main__":
main()
scripts/scene_detector/train_xgb_cpp.py
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""
train_xgb_cpp.py — train the scene-boundary XGBoost on the C++-EXTRACTED feature
matrices (experiments/dumps/cpp_features/<slug>.h5, written by scene_features_dump).
This is the parity-by-construction path: the model is fit on exactly the features
the C++ XGBSceneBoundary produces at inference, so C++ boundaries match by
construction — no numpy-vs-C++ feature drift to chase. Same soft Gaussian target,
knee threshold, and ±20s eval as train_xgb_boundary.py.
Usage (train all 9 + save shipped model):
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
"""
from __future__ import annotations
import argparse, json, sys
from pathlib import Path
import numpy as np, h5py
sys.path.insert(0, "scripts/scene_detector")
from train_scene_boundary import load_xray_boundaries, nms_peaks
from train_xgb_boundary import knee_boundaries, prf, SIGMA
import xgboost as xgb
CPP_DIR = "experiments/dumps/cpp_features"
def load(slug, xray):
with h5py.File(f"{CPP_DIR}/{slug}.h5") as f:
X = f["features"][:].astype(np.float32)
ts = f["timestamp_sec"][:]
b = np.array(load_xray_boundaries(xray))
y = np.zeros(len(ts), np.float32)
if len(b):
for i, t in enumerate(ts):
y[i] = np.exp(-((np.min(np.abs(b - t)))/SIGMA)**2)
yb = np.zeros(len(ts), np.float32)
for bb in b:
yb[np.abs(ts - bb) <= 2.0] = 1.0
return X, y, yb, ts
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--manifest", default="experiments/manifests/films_LVFace_opencv5.json")
ap.add_argument("--holdout", nargs="*", default=[])
ap.add_argument("--train-all", action="store_true")
ap.add_argument("--tol", type=int, default=20)
ap.add_argument("--out", default="experiments/results/scene_boundary")
args = ap.parse_args()
films = json.load(open(args.manifest))
Path(args.out).mkdir(parents=True, exist_ok=True)
tr = films if args.train_all else [f for f in films if f["slug"] not in args.holdout]
Xtr = np.concatenate([load(f["slug"], f["xray"])[0] for f in tr])
ytr = np.concatenate([load(f["slug"], f["xray"])[1] for f in tr])
reg = xgb.XGBRegressor(n_estimators=400, max_depth=5, learning_rate=0.05,
subsample=0.8, colsample_bytree=0.8,
objective="reg:squarederror", n_jobs=8, tree_method="hist")
reg.fit(Xtr, ytr)
print(f"[xgb-cpp] trained on {len(tr)} films", file=sys.stderr)
ev = films if args.train_all else [f for f in films if f["slug"] in args.holdout]
tag = "TRAIN-FIT (all 9)" if args.train_all else "HELD-OUT"
print(f"\n=== {tag} (C++ features, knee, ±{args.tol}s) ===")
print(f"{'film':26s} {'TP':>4}{'FP':>5}{'FN':>5} {'P':>5}{'R':>5}{'F1':>5}")
f1s = []
for f in ev:
X, y, yb, ts = load(f["slug"], f["xray"])
prob = np.clip(reg.predict(X), 0, 1)
peaks = knee_boundaries(prob)
Tset = np.where(yb > 0.5)[0]
P, R, F, tp, fp, fn = prf(peaks, Tset, args.tol)
f1s.append(F)
print(f"{f['slug'][:26]:26s} {tp:>4}{fp:>5}{fn:>5} {P*100:4.0f}%{R*100:4.0f}%{F*100:4.0f}%")
print(f"\nmacro-F1: {np.mean(f1s)*100:.1f}%")
if args.train_all:
reg.save_model(str(Path(args.out) / "xgb_boundary_cpp.json"))
print(f"[xgb-cpp] shipped model → {args.out}/xgb_boundary_cpp.json", file=sys.stderr)
if __name__ == "__main__":
main()
src/config.hpp
+43 -42
View File
@@ -85,40 +85,26 @@ struct Config {
std::string arcface_model;
std::string arcface_engine; // optional path to a pre-built TRT engine; bypasses ORT
int embed_batch_size{4}; // max faces per ORT Run() call — bounds per-call latency
float match_prior{0.5f}; // base-rate prior; 0.5 = use calibrated sigmoid directly
float match_prior{0.433f}; // base-rate prior; 10-knob DE optimum (was 0.5)
// Tuned by Differential Evolution against Amazon X-Ray per-second presence
// over the 4-film rep4 matrix. Best model+mode: LVFace-B_Glint360K, full
// gallery, expansion on. Supersedes an earlier 9-film scene-union tuning
// (0.76); that metric hid out-of-cast false positives.
// over ALL 9 films (opencv5 build, LVFace-B_Glint360K, full gallery,
// expansion on), a 10-parameter sweep — see docs/model-bakeoff.md. The
// per-second misID-weighted macro-F1 optimum is 64.0% (P 79.0%, R 61.1%).
//
// **Read the provenance before trusting the value.** Two things about it:
// This is a permissive operating point: the sweep discovered that with
// flood-fill presence recovering recall, a LOW threshold pays off. It
// supersedes the earlier 0.754, which came from a 4-film subset under the
// now-withdrawn anneal/extinction windows and was never re-derived after a
// scoring-bug fix. The full-9-film sweep at 0.485 beats it.
//
// 1. The document it came from no longer exists under that name. It was
// docs/rep4-optimizer-results.md, renamed to docs/model-bakeoff.md and
// then rewritten (0bd2747). This comment pointed at the dead path for
// long enough that the number looked unsourced. The original is still
// readable at `git show d340da7:docs/rep4-optimizer-results.md`, where
// the shipped triple appears as
// `prob_threshold=0.754, anneal_sec=35.5`.
//
// 2. **0.754 predates a scoring bug fix and was never re-derived.** That
// same rewrite reports finding "a real scoring bug in optimize.py: a
// candidate whose hardest film's replay timed out was averaged over
// survivors instead of penalized, silently rewarding partial coverage.
// Affected 3 of 16 training combos". The corrected sweep converged
// somewhere else — the surviving document records anneal_sec=59.2,
// extinction_sec=59.2 against the 35.5/57.4 shipped alongside this
// threshold — and no corrected prob_threshold is recorded anywhere.
// (The other two constants are now withdrawn outright, which is why
// only this one still matters.)
//
// The doc is also candid that the optimum "generalizes unevenly — strong on
// 3 of 5 held-out films, badly broken on 2 (one with a 974-count misID
// blowup)", and that it is shipped anyway because it still beats the old
// defaults on average. That is a defensible call and not a settled,
// film-agnostic optimum; it should be visible here rather than only in a
// document this comment used to point at incorrectly.
float prob_threshold{0.754f}; // posterior P(match | sim, prior) threshold
// Caveat, still true: the optimum generalises unevenly. It is strong on 7 of
// 9 films (F1 62–80%) and weak on two — The Many Saints of Newark (an
// ensemble of look-alikes; nearly all the run's misIDs land here) and
// Scarface (sparse cuts, so flood-fill over-extends: R 95% / P 26%). Both
// were the low outliers in every prior run too. Shipped because it wins on
// average and on the misID-weighted objective; not a settled, film-agnostic
// constant.
float prob_threshold{0.485f}; // posterior P(match | sim, prior) threshold
// TRACES: AR-024 | SR-002
// match_threshold (0.45), match_ratio (0.80) and match_ratio_ceil (0.65) are
// RETIRED, joining track_max_embed_dist, cut_revive_sim, expand_novelty_sim
@@ -131,7 +117,19 @@ struct Config {
// ── Presence derivation ──────────────────────────────────────────────────
// How accepted frames become a reported window. flood requires scene_detect.
PresenceMode presence_mode{PresenceMode::track_extent};
// Default flood: the 10-knob DE optimum uses it — snapping presence to the
// shot recovers enough recall against X-Ray's scene-level cast to win the
// misID-weighted F1, at a precision cost that is a net gain on 7 of 9 films.
// Falls back to track_extent per claim when no boundaries exist. See
// docs/model-bakeoff.md and PresenceMode above.
PresenceMode presence_mode{PresenceMode::flood};
// Path to the learned XGBoost scene-boundary model. When set (build has
// SAE_SCENE_XGB), the camera-position node stamps a per-frame RGB histogram
// and the sink runs the detector post-EOF to supply flood-fill boundaries —
// the measured best flood boundary source (presence F1 ~76% vs ~64% for the
// always-on histogram cut). Empty → flood falls back to is_cut.
std::string scene_xgb_model;
// ── Cut detection ────────────────────────────────────────────────────────
float cut_threshold{0.70f}; // grayscale histogram correlation below this → hard cut
@@ -180,7 +178,7 @@ struct Config {
// frame-dependent (AR-007): on is_cut / is_scene_boundary, and for any track
// that is no longer on screen, it drops to 0 (embedding only), because
// position carries no information across a viewpoint change or a gap.
float track_alpha{0.4f}; // base cost weight: 0=embedding only, 1=spatial only
float track_alpha{0.435f}; // base cost weight: 0=embedding only, 1=spatial only (10-knob DE optimum)
float track_min_iou{0.1f}; // IoU below which spatial link alone is rejected
// Minimum P(same person) for an association to be admissible on appearance
// alone. This replaces track_max_embed_dist (a raw cosine distance, AR-024).
@@ -193,7 +191,7 @@ struct Config {
// Replaces track_max_frames_missing: a frame count silently changed meaning
// with sample_fps, and the same number had to be guessed twice (once for an
// ordinary miss, once for a cut). Seconds mean one thing at any sample rate.
double track_extinction_sec{5.0};
double track_extinction_sec{31.0}; // 10-knob DE optimum (was 5.0)
// ── Ownership and evidence accumulation (AR-025) ──────────────────────────
// TRACES: AR-025, AR-017 | SR-002
@@ -207,8 +205,10 @@ struct Config {
// ownership_logodds is arguably the most consequential constant in the
// pipeline after prob_threshold: below it a track produces no presence
// claim at all, so it decides whether an actor is reported rather than how
// confidently. 2.0 is a posterior of ~0.88. Unswept.
float ownership_logodds{2.0f};
// confidently. 1.72 is a posterior of ~0.85 — the 10-knob DE optimum (was
// an unswept 2.0 ≈ 0.88); slightly more permissive, consistent with the
// low-threshold operating point the sweep converged on.
float ownership_logodds{1.72f};
// How much a single observation may move a track's belief. n_eff =
// n / (1 + (n-1)·rho), so rho_max caps what a repeated view can ever be
@@ -217,10 +217,11 @@ struct Config {
// detection, alignment and noise realisation, so a little independent
// evidence survives. Setting it to 1 freezes belief after the first frame,
// which is the bug this replaced.
float evidence_rho_max{0.5f};
float evidence_rho_max{0.204f}; // 10-knob DE optimum (was 0.5): weights a
// held pose closer to a single observation
// P(same view) below this and the observation counts as a genuinely new
// look, so it joins the per-track view set.
float evidence_admit_below{0.6f};
float evidence_admit_below{0.784f}; // 10-knob DE optimum (was 0.6)
// Distinct views remembered per track, which bounds the novelty comparison.
int evidence_max_views{8};
@@ -268,10 +269,10 @@ struct Config {
// at promotion time — see track_gallery.hpp. This is the only threshold the
// expansion path has: it replaces the raw-cosine expand_novelty_sim (0.55)
// and expand_track_spread_max (0.60), which are retired (AR-024).
// Working values pending VR-007; sweep both bounds, they fail in opposite
// directions.
float expand_band_lo{0.90f};
float expand_band_hi{0.95f};
// 10-knob DE optimum (was 0.90/0.95). The sweep widened the band — a lower lo
// admits more pose-varied views into the annex — which the optimum preferred.
float expand_band_lo{0.804f};
float expand_band_hi{0.952f};
int expand_min_anchor_frames{3}; // require ≥N accepted frames naming the actor before
// the track is confirmed and its buffer promoted
std::string expand_debug_dir; // if set, dump promoted mugshots + embeddings here
src/inference/audio_logpsd.hpp
+143
View File
@@ -0,0 +1,143 @@
#pragma once
// Per-second audio log-PSD, C++ parity with scripts/scene_detector/
// extract_audio_features.py — the audio tower input for the XGBoost scene
// detector. Decodes the whole track to mono 16 kHz, then one FFT per second over
// a 4 s Hann-windowed window, power pooled into geomspace log-frequency bands,
// L1-normalised (shape not loudness) and log1p-compressed.
//
// Must match the Python exactly (SR=16000, WIN_SEC=4, N_BINS=64→geomspace unique
// edges, log1p(band*1e3)); the shipped model was trained on those features.
extern "C" {
#include <libavformat/avformat.h>
#include <libavcodec/avcodec.h>
#include <libavutil/opt.h>
#include <libswresample/swresample.h>
}
#include <fftw3.h>
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <stdexcept>
#include <string>
#include <vector>
class AudioLogPSD {
public:
static constexpr int kSR = 16000;
static constexpr double kHop = 1.0; // 1 feature row / second
static constexpr double kWin = 4.0; // FFT window seconds
static constexpr int kNBins = 64; // geomspace target (dedups to ~57)
// Returns [T][B] per-second log-PSD (T ≈ film seconds, B ≈ 57), aligned to the
// 1 fps grid. Empty on decode failure (caller then feeds a zero block).
static std::vector<std::vector<float>> extract(const std::string& path) {
std::vector<float> mono = decode_mono_16k(path);
if (mono.empty()) return {};
return features(mono);
}
// Public for the parity harness.
static std::vector<std::vector<float>> features(const std::vector<float>& mono) {
const int win = int(kSR * kWin), hop = int(kSR * kHop);
const int T = int(mono.size()) / hop;
if (T <= 0) return {};
const int nfreq = win/2 + 1;
std::vector<int> edges = geomspace_edges(nfreq);
const int nb = int(edges.size()) - 1;
// Hann window (matches scipy.signal.windows.hann, sym=True default → but
// numpy code uses sps.windows.hann(win) which is symmetric).
std::vector<double> hann(win);
for (int i = 0; i < win; ++i)
hann[i] = 0.5 - 0.5*std::cos(2.0*M_PI*i/(win-1));
std::vector<double> in(win);
auto* out = fftw_alloc_complex(nfreq);
fftw_plan plan = fftw_plan_dft_r2c_1d(win, in.data(), out, FFTW_ESTIMATE);
std::vector<std::vector<float>> feat(T, std::vector<float>(nb, 0.f));
const int half = win/2;
for (int t = 0; t < T; ++t) {
int centre = t*hop + hop/2;
int s = centre - half;
for (int i = 0; i < win; ++i) {
int idx = s + i;
double v = (idx >= 0 && idx < int(mono.size())) ? mono[idx] : 0.0;
in[i] = v * hann[i];
}
fftw_execute(plan);
// power spectrum + 1e-12
std::vector<double> psd(nfreq);
for (int i = 0; i < nfreq; ++i)
psd[i] = out[i][0]*out[i][0] + out[i][1]*out[i][1] + 1e-12;
std::vector<double> band(nb, 0.0);
double tot = 0.0;
for (int b = 0; b < nb; ++b) {
for (int i = edges[b]; i < edges[b+1]; ++i) band[b] += psd[i];
tot += band[b];
}
for (int b = 0; b < nb; ++b)
feat[t][b] = float(std::log1p(band[b]/tot * 1e3));
}
fftw_destroy_plan(plan); fftw_free(out);
return feat;
}
private:
// np.unique(np.geomspace(1, nfreq-1, N_BINS+1).astype(int))
static std::vector<int> geomspace_edges(int nfreq) {
const int n = kNBins + 1;
double a = std::log(1.0), b = std::log(double(nfreq-1));
std::vector<int> raw(n);
for (int i = 0; i < n; ++i)
raw[i] = int(std::exp(a + (b-a)*i/(n-1))); // .astype(int) truncates
std::vector<int> uniq;
for (int v : raw) if (uniq.empty() || v != uniq.back()) uniq.push_back(v);
return uniq;
}
static std::vector<float> decode_mono_16k(const std::string& path) {
AVFormatContext* fmt = nullptr;
if (avformat_open_input(&fmt, path.c_str(), nullptr, nullptr) < 0) return {};
std::vector<float> out;
SwrContext* swr = nullptr; AVCodecContext* dec = nullptr;
AVPacket* pkt = av_packet_alloc(); AVFrame* fr = av_frame_alloc();
try {
if (avformat_find_stream_info(fmt, nullptr) < 0) throw 0;
int ai = av_find_best_stream(fmt, AVMEDIA_TYPE_AUDIO, -1, -1, nullptr, 0);
if (ai < 0) throw 0;
AVStream* st = fmt->streams[ai];
const AVCodec* codec = avcodec_find_decoder(st->codecpar->codec_id);
dec = avcodec_alloc_context3(codec);
avcodec_parameters_to_context(dec, st->codecpar);
if (avcodec_open2(dec, codec, nullptr) < 0) throw 0;
AVChannelLayout out_ch = AV_CHANNEL_LAYOUT_MONO;
swr_alloc_set_opts2(&swr, &out_ch, AV_SAMPLE_FMT_FLT, kSR,
&dec->ch_layout, dec->sample_fmt,
dec->sample_rate ? dec->sample_rate : kSR, 0, nullptr);
if (!swr || swr_init(swr) < 0) throw 0;
while (av_read_frame(fmt, pkt) >= 0) {
if (pkt->stream_index == ai && avcodec_send_packet(dec, pkt) >= 0) {
while (avcodec_receive_frame(dec, fr) >= 0) {
int max_out = swr_get_out_samples(swr, fr->nb_samples);
size_t base = out.size(); out.resize(base + max_out);
uint8_t* dst = reinterpret_cast<uint8_t*>(out.data() + base);
int got = swr_convert(swr, &dst, max_out,
(const uint8_t**)fr->extended_data, fr->nb_samples);
out.resize(base + std::max(0, got));
}
}
av_packet_unref(pkt);
}
} catch (...) { out.clear(); }
if (swr) swr_free(&swr);
if (dec) avcodec_free_context(&dec);
av_frame_free(&fr); av_packet_free(&pkt);
avformat_close_input(&fmt);
return out;
}
};
src/inference/xgb_scene_boundary.hpp
+319
View File
@@ -0,0 +1,319 @@
#pragma once
// XGBoost scene-boundary detector — C++ inference of the shipped model
// (models/scene_boundary_xgb.json), for flood-fill presence in the live pipeline.
//
// This is a POST-EOF step (like flood-fill itself): the per-film knee threshold
// needs every peak, so boundaries can only be finalized after the whole film is
// seen. The result sink collects a per-frame RGB histogram; at EOF it calls
// boundaries() with the full (timestamp, hist) series and gets back the boundary
// timestamps to flood-snap against.
//
// The feature pipeline MUST match scripts/scene_detector/train_scene_boundary.py
// exactly (206 features): a ±WIN=3s window of per-second base features + a
// 3-value debounce clock. Base per second (29):
// video(17): sym-delta |hist(t+k)-hist(t-k)| L1 at k=1,2,4,8; per-channel corr
// to t-1 (3); ramp bank at H=2,4,6,8,10 on z-normed hist (5);
// per-channel energy (3); debounce phase/decay from |delta k=1| (2)
// audio(12): same but on the log-PSD, no corr, 1 energy [ZERO when no audio]
// then window flatten t-3..t+3 (×7) and append clock (dt, phase, decay).
//
// Audio is not available live (the pipeline has no per-second PSD stream), so the
// audio block is fed zeros — the model was trained with audio present but it is
// weak (measured) and XGBoost tolerates a constant block; the video signal
// carries the detector. (If live audio is added later, fill the block.)
#include <xgboost/c_api.h>
#include <algorithm>
#include <array>
#include <cmath>
#include <cstdint>
#include <numeric>
#include <stdexcept>
#include <string>
#include <vector>
class XGBSceneBoundary {
public:
// Must match kHistBins in embedding_dump_node.hpp / the training dump.
static constexpr int kHistBins = 32; // per channel → 96-float hist
static constexpr int kWin = 3; // ±WIN-second window
static constexpr double kSigmaTau = 205.0; // SCENE_TAU (unused at infer; kept for parity docs)
static constexpr int kRampScales[5] = {2, 4, 6, 8, 10};
explicit XGBSceneBoundary(const std::string& model_path) {
if (XGBoosterCreate(nullptr, 0, &booster_) != 0)
throw std::runtime_error("XGBoosterCreate failed");
if (XGBoosterLoadModel(booster_, model_path.c_str()) != 0)
throw std::runtime_error("XGBoosterLoadModel failed: " +
std::string(XGBGetLastError()));
}
~XGBSceneBoundary() { if (booster_) XGBoosterFree(booster_); }
XGBSceneBoundary(const XGBSceneBoundary&) = delete;
XGBSceneBoundary& operator=(const XGBSceneBoundary&) = delete;
// hist: T rows × 96 (normalised RGB histogram per second).
// audio: T rows × B log-PSD (from AudioLogPSD; aligned to the same seconds),
// or empty → the audio block is filled with its zero-input values
// (deltas/ramp/energy 0, but debounce phase=1/decay=exp(-1), matching
// the Python audio_features on a zero series).
// Returns boundary timestamps (knee-selected).
std::vector<double> boundaries(const std::vector<std::vector<float>>& hist,
const std::vector<double>& ts,
const std::vector<std::vector<float>>& audio = {}) {
const int T = static_cast<int>(hist.size());
if (T < 2 * kWin + 2) return {};
auto base = build_base(hist, audio); // [T][29]
std::vector<float> X = window_and_clock(base, hist);
std::vector<float> prob = predict(X, T, 206);
return knee_boundaries(prob, ts);
}
static std::vector<std::vector<float>> debug_base(const std::vector<std::vector<float>>& hist,
const std::vector<std::vector<float>>& audio = {}) {
return build_base(hist, audio);
}
// Predict boundaries from a precomputed [rows×cols] feature matrix (for the
// clean parity check: same bytes both sides).
std::vector<double> boundaries_from_features(const std::vector<float>& X, int rows,
int cols, const std::vector<double>& ts) {
auto prob = predict(X, rows, cols);
return knee_boundaries(prob, ts);
}
std::vector<float> debug_predict(const std::vector<float>& X, int r, int c) {
return predict(X, r, c);
}
static std::vector<int> debug_find_peaks(const std::vector<float>& p, int d) {
return find_peaks(p, d);
}
// The flat [T*206] feature matrix — exposed so TRAINING uses the exact same
// C++ features as inference (parity by construction; no numpy re-match). The
// Python trainer reshapes to [T,206], attaches the soft target, and fits.
static std::vector<float> feature_matrix(const std::vector<std::vector<float>>& hist,
const std::vector<std::vector<float>>& audio) {
auto base = build_base(hist, audio);
return window_and_clock(base, hist);
}
static constexpr int kNFeatures = 206;
private:
BoosterHandle booster_{nullptr};
// ── feature builders (exact parity with the Python) ──────────────────────
static float l1(const std::vector<float>& a, const std::vector<float>& b) {
float s = 0; for (size_t i = 0; i < a.size(); ++i) s += std::fabs(a[i] - b[i]);
return s;
}
// z-normalise each of the 96 columns across time (matches _znorm).
static std::vector<std::vector<float>> znorm(const std::vector<std::vector<float>>& h) {
const int T = h.size(), D = h[0].size();
std::vector<float> mu(D, 0), sd(D, 0);
for (auto& r : h) for (int d = 0; d < D; ++d) mu[d] += r[d];
for (int d = 0; d < D; ++d) mu[d] /= T;
for (auto& r : h) for (int d = 0; d < D; ++d) sd[d] += (r[d]-mu[d])*(r[d]-mu[d]);
for (int d = 0; d < D; ++d) sd[d] = std::sqrt(sd[d]/T) + 1e-6f;
std::vector<std::vector<float>> z(T, std::vector<float>(D));
for (int t = 0; t < T; ++t) for (int d = 0; d < D; ++d) z[t][d] = (h[t][d]-mu[d])/sd[d];
return z;
}
// ramp bank: L2 of the antisymmetric ramp-weighted sum over ±H, per scale.
// Matches ramp_bank() (np.convolve 'same' with reversed kernel; sign folds
// into the L2 norm so the direct antisymmetric sum is equivalent).
static std::vector<std::array<float,5>> ramp_bank(const std::vector<std::vector<float>>& z) {
const int T = z.size(), D = z[0].size();
std::vector<std::array<float,5>> out(T);
for (int k = 0; k < 5; ++k) {
const int H = kRampScales[k];
for (int t = 0; t < T; ++t) {
std::vector<double> acc(D, 0.0);
for (int l = -H; l <= H; ++l) {
int idx = t + l;
if (idx < 0 || idx >= T) continue;
double w = (l == 0) ? 0.0 : (l > 0 ? 1.0 : -1.0) * (double(std::abs(l))/H);
for (int d = 0; d < D; ++d) acc[d] += w * z[idx][d];
}
double n = 0; for (double v : acc) n += v*v;
out[t][k] = static_cast<float>(std::sqrt(n));
}
}
return out;
}
// Generic symmetric-delta + ramp + energy + debounce feature block for one
// modality's z-normable series `raw` (hist or PSD). Fills `out` columns
// [off .. off+width). corr=true adds the 3 per-channel corr features (video
// only); n_energy is 3 (video, per-channel) or 1 (audio, total).
static void modality_block(const std::vector<std::vector<float>>& raw,
bool corr, int n_energy,
std::vector<std::vector<float>>& out, int off) {
const int T = raw.size();
auto z = znorm(raw);
auto rb = ramp_bank(z);
auto sym = [&](int t, int k)->float{
int f = std::min(T-1, t+k), b = std::max(0, t-k);
return l1(raw[f], raw[b]);
};
const int B = kHistBins; // only used for corr (video)
for (int t = 0; t < T; ++t) {
int o = off;
for (int k : {1,2,4,8}) out[t][o++] = sym(t,k);
if (corr) {
int tp = std::max(0, t-1);
for (int c = 0; c < 3; ++c) {
double ma=0, mb=0;
for (int i=0;i<B;++i){ ma+=raw[t][c*B+i]; mb+=raw[tp][c*B+i]; }
ma/=B; mb/=B; double num=0, da=0, db=0;
for (int i=0;i<B;++i){ double x=raw[t][c*B+i]-ma, y=raw[tp][c*B+i]-mb;
num+=x*y; da+=x*x; db+=y*y; }
out[t][o++] = float(num/(std::sqrt(da*db)+1e-9));
}
}
for (int k=0;k<5;++k) out[t][o++] = rb[t][k];
if (n_energy == 3) {
for (int c=0;c<3;++c){ float e=0; for(int i=0;i<B;++i) e+=raw[t][c*B+i]; out[t][o++]=e; }
} else {
float e=0; for (float v : raw[t]) e+=v; out[t][o++]=e;
}
o += 2; // debounce filled below
}
// debounce from this block's delta-k1 (its first column = off)
std::vector<float> d1(T); for (int t=0;t<T;++t) d1[t]=out[t][off];
auto clk = debounce_phase(d1);
// debounce sits at the end of the block: off + 4(deltas) + (corr?3:0) + 5(ramp) + n_energy
int deb = off + 4 + (corr?3:0) + 5 + n_energy;
for (int t=0;t<T;++t){ out[t][deb]=clk[t].first; out[t][deb+1]=clk[t].second; }
}
// per-second base = video(17) + audio(12). Audio empty → its block is the
// zero-series result (deltas/ramp/energy 0, debounce phase=1/decay=exp(-1)).
static std::vector<std::vector<float>> build_base(const std::vector<std::vector<float>>& hist,
const std::vector<std::vector<float>>& audio) {
const int T = hist.size();
std::vector<std::vector<float>> base(T, std::vector<float>(29, 0.0f));
modality_block(hist, /*corr=*/true, /*n_energy=*/3, base, /*off=*/0); // video → 0..16
if (!audio.empty() && int(audio.size()) == T) {
modality_block(audio, /*corr=*/false, /*n_energy=*/1, base, /*off=*/17); // audio → 17..28
} else {
// zero-series audio: deltas/ramp/energy already 0; only debounce differs.
auto clk = debounce_phase(std::vector<float>(T, 0.0f));
for (int t=0;t<T;++t){ base[t][27]=clk[t].first; base[t][28]=clk[t].second; }
}
return base;
}
// matches debounce_phase(): 90th-pct peaks, dt=time since last, phase/decay.
static std::vector<std::pair<float,float>> debounce_phase(const std::vector<float>& sig) {
const int T = sig.size();
std::vector<float> s(sig); std::sort(s.begin(), s.end());
float thr = s[std::min(T-1, int(0.90*T))];
std::vector<std::pair<float,float>> out(T);
int last = -1000000000;
for (int t=0;t<T;++t){
if (sig[t] > thr) last = t;
double dt = (last < -100000000) ? kSigmaTau : double(t - last);
out[t] = { float(std::min(1.0, dt/kSigmaTau)), float(std::exp(-dt/kSigmaTau)) };
}
return out;
}
// window flatten (t-3..t+3, edge-pad) + append the 3-value film clock.
static std::vector<float> window_and_clock(const std::vector<std::vector<float>>& base,
const std::vector<std::vector<float>>& hist) {
const int T = base.size(), d = base[0].size(); // d=29
// film-level clock: time-since-last-peak on the |delta k1| video signal
// (base col 0), same as per_second_matrix's `clock`.
std::vector<float> sig(T); for (int t=0;t<T;++t) sig[t]=base[t][0];
std::vector<float> ss(sig); std::sort(ss.begin(), ss.end());
float thr = ss[std::min(T-1, int(0.90*T))];
std::vector<float> X; X.reserve(size_t(T)*206);
int last=-1000000000;
for (int t=0;t<T;++t){
for (int off=-kWin; off<=kWin; ++off){
int idx = std::min(T-1, std::max(0, t+off));
for (int j=0;j<d;++j) X.push_back(base[idx][j]);
}
if (sig[t] > thr) last=t;
double dt=(last<-100000000)?kSigmaTau:double(t-last);
X.push_back(float(dt));
X.push_back(float(std::min(1.0, dt/kSigmaTau)));
X.push_back(float(std::exp(-dt/kSigmaTau)));
}
return X;
}
std::vector<float> predict(const std::vector<float>& X, int rows, int cols) {
DMatrixHandle dm;
if (XGDMatrixCreateFromMat(X.data(), rows, cols, std::nanf(""), &dm) != 0)
throw std::runtime_error("XGDMatrixCreateFromMat failed");
bst_ulong out_len = 0; const float* out = nullptr;
if (XGBoosterPredict(booster_, dm, 0, 0, 0, &out_len, &out) != 0)
throw std::runtime_error("XGBoosterPredict failed");
std::vector<float> p(out, out + out_len);
XGDMatrixFree(dm);
for (auto& v : p) v = std::clamp(v, 0.f, 1.f);
return p;
}
// Exact replica of scipy.signal.find_peaks(x, distance=d):
// 1. local maxima (plateau-aware: rising then falling, midpoint of a flat top)
// 2. keep peaks by DESCENDING height; drop any within `d` of an already-kept
// taller peak. This is height-priority, NOT the greedy left-to-right merge
// — the two give different peak sets and hence a different knee.
static std::vector<int> find_peaks(const std::vector<float>& x, int d) {
const int n = x.size();
std::vector<int> mid;
int i = 1;
while (i < n-1) {
if (x[i-1] < x[i]) {
int ahead = i+1;
while (ahead < n-1 && x[ahead] == x[i]) ahead++;
if (x[ahead] < x[i]) mid.push_back((i + ahead - 1) / 2);
i = ahead;
} else i++;
}
// height-priority distance filter (scipy's _select_by_peak_distance)
std::vector<int> order(mid.size());
for (size_t k = 0; k < mid.size(); ++k) order[k] = k;
std::sort(order.begin(), order.end(),
[&](int a, int b){ return x[mid[a]] < x[mid[b]]; }); // ascending
std::vector<char> keep(mid.size(), 1);
for (int j = int(order.size())-1; j >= 0; --j) { // tallest first
int k = order[j];
if (!keep[k]) continue;
for (int l = k-1; l >= 0 && mid[k]-mid[l] < d; --l) keep[l] = 0;
for (int r = k+1; r < int(mid.size()) && mid[r]-mid[k] < d; ++r) keep[r] = 0;
}
std::vector<int> out;
for (size_t k = 0; k < mid.size(); ++k) if (keep[k]) out.push_back(mid[k]);
return out;
}
// knee threshold on peak heights → boundary timestamps (matches knee_boundaries).
static std::vector<double> knee_boundaries(const std::vector<float>& prob,
const std::vector<double>& ts,
int min_gap = 5) {
std::vector<int> pk = find_peaks(prob, min_gap);
if (pk.size() < 5) {
std::vector<double> r; for (int i : pk) r.push_back(ts[i]); return r;
}
std::vector<float> h; for (int i : pk) h.push_back(prob[i]);
std::sort(h.begin(), h.end(), std::greater<float>());
int n = h.size(); float h0 = h.front() + 1e-9f;
int kbest = 0; double dmax = -1;
for (int i = 0; i < n; ++i) {
double x = double(i)/(n-1);
double yv = h[i]/h0;
double chord = (h[0]/h0) + ((h[n-1]/h0)-(h[0]/h0))*x;
if (chord - yv > dmax) { dmax = chord - yv; kbest = i; }
}
float knee = h[kbest];
std::vector<double> out;
for (int i : pk) if (prob[i] >= knee) out.push_back(ts[i]);
return out;
}
};
src/main.cpp
+1
View File
@@ -209,6 +209,7 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--end")) cfg.end_sec = std::stod(next());
else if (arg("--cut-threshold")) cfg.cut_threshold = std::stof(next());
else if (arg("--presence-mode")) { std::string m = next(); cfg.presence_mode = (m == "flood") ? PresenceMode::flood : PresenceMode::track_extent; }
else if (arg("--scene-xgb-model")) cfg.scene_xgb_model = next();
else if (arg("--scene-detect")) cfg.scene_detect = true;
else if (arg("--scene-detector")) cfg.scene_model = next();
else if (arg("--scene-detector-engine")) cfg.scene_engine = next();
src/nodes/camera_position_change_detector_node.hpp
+23 -1
View File
@@ -33,9 +33,29 @@ struct CameraPositionChangeDetectorFunc {
explicit CameraPositionChangeDetectorFunc(const Config& cfg)
: cut_threshold_(cfg.cut_threshold)
, want_rgb_hist_(!cfg.scene_xgb_model.empty())
{
std::cerr << "[camera_position_change_detector] cut_threshold="
<< cut_threshold_ << "\n";
<< cut_threshold_
<< (want_rgb_hist_ ? " (+rgb_hist for scene detector)" : "")
<< "\n";
}
// 32-bin-per-channel normalised RGB histogram (96 floats), the exact layout
// the XGBoost scene detector was trained on (see embedding_dump_node). Only
// computed when a scene model is configured, so it costs nothing otherwise.
static std::vector<float> rgb_histogram(const cv::Mat& img) {
constexpr int kBins = 32;
std::vector<float> out(kBins * 3, 0.f);
if (img.empty() || img.channels() != 3) return out;
float range[] = {0.f, 256.f}; const float* ranges = range; int bins = kBins;
for (int c = 0; c < 3; ++c) { // OpenCV BGR → store B,G,R blocks
cv::Mat h;
cv::calcHist(&img, 1, &c, cv::Mat(), h, 1, &bins, &ranges);
cv::normalize(h, h, 1.0, 0.0, cv::NORM_L1);
for (int b = 0; b < kBins; ++b) out[c*kBins + b] = h.at<float>(b);
}
return out;
}
Frame operator()(Frame f) {
@@ -64,11 +84,13 @@ struct CameraPositionChangeDetectorFunc {
prev_hist_ = hist;
prev_hist_valid_ = true;
if (want_rgb_hist_) f.rgb_hist = rgb_histogram(f.image);
return f;
}
private:
float cut_threshold_;
bool want_rgb_hist_{false};
cv::Mat prev_hist_;
bool prev_hist_valid_{false};
};
src/nodes/embedding_dump_node.hpp
+36
View File
@@ -5,6 +5,7 @@
#include "gallery/embedder_stamp.hpp"
#include <H5Cpp.h>
#include <opencv2/imgproc.hpp> // cv::calcHist for the per-frame RGB histogram
#include <atomic>
#include <cstdint>
@@ -167,6 +168,12 @@ struct EmbeddingDumpFunc {
fidx_.push_back(ef.source.frame_idx);
is_cut_.push_back(ef.source.is_cut ? 1 : 0);
is_bnd_.push_back(ef.source.is_scene_boundary ? 1 : 0);
// Per-frame normalised RGB histogram (kHistBins per channel), for offline
// training of a learned scene-boundary detector against X-Ray scene
// boundaries — the grayscale-correlation cut detector is blind on
// low-contrast grades (Scarface: 1 cut in 10k frames). Cheap and the frame
// is already decoded here; empty frame → zeros.
append_rgb_hist(ef.source.image);
face_off_.push_back(static_cast<int64_t>(conf_.size()));
face_cnt_.push_back(n);
@@ -287,6 +294,11 @@ private:
write_vec(frames, "is_scene_boundary", is_bnd_, H5::PredType::NATIVE_UINT8);
write_vec(frames, "face_offset", face_off_, H5::PredType::NATIVE_INT64);
write_vec(frames, "face_count", face_cnt_, H5::PredType::NATIVE_INT32);
// Per-frame normalised RGB histogram, kHistBins per channel laid out
// [R(kHistBins) G(kHistBins) B(kHistBins)] per row. Feeds the learned
// scene-boundary detector (see scripts/scene_detector/).
write_vec(frames, "rgb_hist", rgb_hist_, H5::PredType::NATIVE_FLOAT,
kHistBins * 3);
H5::Group faces = file.createGroup("faces");
write_vec(faces, "embedding", emb_, H5::PredType::NATIVE_FLOAT, kEmbedDim);
@@ -301,6 +313,29 @@ private:
<< conf_.size() << " faces → " << path_ << "\n";
}
// Per-channel bin count for the RGB histogram. 32 → a 96-float row per frame,
// ~40 KB per 10k-frame film: negligible next to the embeddings.
static constexpr int kHistBins = 32;
// Append the frame's normalised per-channel RGB histogram (R,G,B blocks). An
// empty frame (EOF sentinels never reach here) yields a zero row so the array
// stays parallel to ts_.
void append_rgb_hist(const cv::Mat& img) {
const size_t base = rgb_hist_.size();
rgb_hist_.resize(base + kHistBins * 3, 0.f);
if (img.empty() || img.channels() != 3) return;
float range[] = {0.f, 256.f};
const float* ranges[] = {range};
int bins = kHistBins;
for (int c = 0; c < 3; ++c) { // OpenCV is BGR; store as B,G,R blocks
cv::Mat h;
cv::calcHist(&img, 1, &c, cv::Mat(), h, 1, &bins, ranges);
cv::normalize(h, h, 1.0, 0.0, cv::NORM_L1);
for (int b = 0; b < kHistBins; ++b)
rgb_hist_[base + c * kHistBins + b] = h.at<float>(b);
}
}
std::string path_, movie_;
EmbedderStamp stamp_;
DumpProvenance prov_;
@@ -315,4 +350,5 @@ private:
std::vector<int32_t> face_cnt_;
std::vector<float> emb_, bbox_, lmk_, conf_;
std::vector<float> sharp_, resid_; // AR-028 quality vector, parallel to conf_
std::vector<float> rgb_hist_; // kHistBins*3 per frame, parallel to ts_
};
src/nodes/frame_annotation_node.hpp
+1
View File
@@ -43,6 +43,7 @@ struct FrameAnnotationFunc {
sa.visible_actors = std::move(mf.actors);
sa.is_cut = mf.source.is_cut;
sa.is_scene_boundary = mf.source.is_scene_boundary;
sa.rgb_hist = std::move(mf.source.rgb_hist);
return sa;
}
};
src/nodes/result_sink_node.hpp
+56 -7
View File
@@ -3,6 +3,10 @@
#include "types.hpp"
#include "config.hpp"
#include "track_registry.hpp"
#ifdef SAE_SCENE_XGB
#include "inference/xgb_scene_boundary.hpp"
#include "inference/audio_logpsd.hpp"
#endif
#include <nlohmann/json.hpp>
#include <algorithm>
@@ -219,15 +223,24 @@ private:
// flood-fill actually snaps to — coarser than true shot boundaries (cuts also
// fire on in-shot angle changes) but present with no extra pass.
std::vector<double> scene_boundaries() const {
bool have_scene = false;
for (const auto& sa : frames_)
if (sa.is_scene_boundary) { have_scene = true; break; }
std::vector<double> b;
b.push_back(0.0);
for (const auto& sa : frames_) {
const bool boundary = have_scene ? sa.is_scene_boundary : sa.is_cut;
if (boundary) b.push_back(sa.timestamp_sec);
// Preferred: the learned XGBoost scene detector, run once here post-EOF
// (the knee threshold needs the whole film, so this is inherently a final
// step — like flood-fill itself). Measured best flood boundary source.
std::vector<double> learned = xgb_boundaries();
if (!learned.empty()) {
for (double t : learned) b.push_back(t);
} else {
// Fallback: TransNetV2 shot boundaries if present, else histogram cuts.
bool have_scene = false;
for (const auto& sa : frames_)
if (sa.is_scene_boundary) { have_scene = true; break; }
for (const auto& sa : frames_) {
const bool boundary = have_scene ? sa.is_scene_boundary : sa.is_cut;
if (boundary) b.push_back(sa.timestamp_sec);
}
}
b.push_back(last_ts_ + 1.0); // a right edge past the final sample
std::sort(b.begin(), b.end());
@@ -235,6 +248,42 @@ private:
return b;
}
// Run the learned scene-boundary detector over the collected per-frame RGB
// histograms + per-second audio log-PSD (decoded once from the movie). Returns
// {} when no model is configured, the build lacks XGBoost, or no rgb_hist was
// stamped (camera-position node only does so when a model is set).
std::vector<double> xgb_boundaries() const {
#ifdef SAE_SCENE_XGB
if (cfg_.scene_xgb_model.empty()) return {};
std::vector<std::vector<float>> hist;
std::vector<double> ts;
hist.reserve(frames_.size()); ts.reserve(frames_.size());
for (const auto& sa : frames_) {
if (sa.rgb_hist.empty()) return {}; // hist not stamped → bail to fallback
hist.push_back(sa.rgb_hist);
ts.push_back(sa.timestamp_sec);
}
if (hist.size() < 16) return {};
try {
auto audio = AudioLogPSD::extract(cfg_.movie_path); // [T'][B], aligned per second
if ((int)audio.size() != (int)hist.size())
audio.resize(hist.size(),
std::vector<float>(audio.empty() ? 57 : audio[0].size(), 0.f));
XGBSceneBoundary det(cfg_.scene_xgb_model);
auto b = det.boundaries(hist, ts, audio);
std::cerr << "[result_sink] XGBoost scene detector: " << b.size()
<< " boundaries\n";
return b;
} catch (const std::exception& e) {
std::cerr << "[result_sink] scene detector failed (" << e.what()
<< "), falling back to histogram cuts\n";
return {};
}
#else
return {};
#endif
}
// The boundary opening the shot that contains t (largest boundary ≤ t).
static double boundary_at_or_before(const std::vector<double>& b, double t) {
auto it = std::upper_bound(b.begin(), b.end(), t);
src/tools/scene_features_dump.cpp
+58
View File
@@ -0,0 +1,58 @@
// scene_features_dump — write the C++ scene-boundary feature matrix to HDF5, so
// the XGBoost model is TRAINED on exactly the features the C++ detector produces
// at inference (parity by construction — no numpy re-implementation to keep in
// sync). Reads frames/rgb_hist + frames/timestamp_sec from a dump and, given the
// movie, the per-second audio log-PSD; writes features [T,206] + timestamps.
//
// scene_features_dump <dump.h5> <movie> <out_features.h5>
//
// The py3.12 venv trainer (train_xgb_cpp.py) reads <out_features.h5>, attaches
// the soft Gaussian boundary target, fits XGBoost, and saves the model that
// XGBSceneBoundary loads. Same C++ features both sides → exact parity.
#include "inference/xgb_scene_boundary.hpp"
#include "inference/audio_logpsd.hpp"
#include <H5Cpp.h>
#include <iostream>
#include <vector>
int main(int argc, char** argv) {
if (argc < 4) {
std::cerr << "usage: scene_features_dump <dump.h5> <movie> <out.h5>\n";
return 1;
}
H5::H5File in(argv[1], H5F_ACC_RDONLY);
H5::DataSet hd = in.openDataSet("frames/rgb_hist");
hsize_t hdims[2]; hd.getSpace().getSimpleExtentDims(hdims);
std::vector<float> flat(hdims[0]*hdims[1]);
hd.read(flat.data(), H5::PredType::NATIVE_FLOAT);
const int T = hdims[0], C = hdims[1];
std::vector<std::vector<float>> hist(T, std::vector<float>(C));
for (int t = 0; t < T; ++t)
for (int c = 0; c < C; ++c) hist[t][c] = flat[t*C+c];
H5::DataSet td = in.openDataSet("frames/timestamp_sec");
hsize_t tdim[1]; td.getSpace().getSimpleExtentDims(tdim);
std::vector<double> ts(tdim[0]);
td.read(ts.data(), H5::PredType::NATIVE_DOUBLE);
auto audio = AudioLogPSD::extract(argv[2]);
if ((int)audio.size() != T) {
std::cerr << "[features] audio rows " << audio.size() << " != hist rows "
<< T << " — aligning (pad/truncate)\n";
audio.resize(T, std::vector<float>(audio.empty()?57:audio[0].size(), 0.f));
}
std::vector<float> X = XGBSceneBoundary::feature_matrix(hist, audio);
const int F = XGBSceneBoundary::kNFeatures;
H5::H5File out(argv[3], H5F_ACC_TRUNC);
hsize_t xd[2] = {(hsize_t)T, (hsize_t)F};
out.createDataSet("features", H5::PredType::NATIVE_FLOAT, H5::DataSpace(2, xd))
.write(X.data(), H5::PredType::NATIVE_FLOAT);
hsize_t td2[1] = {(hsize_t)T};
out.createDataSet("timestamp_sec", H5::PredType::NATIVE_DOUBLE, H5::DataSpace(1, td2))
.write(ts.data(), H5::PredType::NATIVE_DOUBLE);
std::cerr << "[features] wrote [" << T << "," << F << "] → " << argv[3] << "\n";
return 0;
}
src/tools/xgb_boundary_parity.cpp
+82
View File
@@ -0,0 +1,82 @@
// Parity harness: run the C++ XGBSceneBoundary on a dump's frames/rgb_hist and
// print the boundary timestamps, so they can be diffed against the Python
// knee_boundaries (scripts/scene_detector). Feature parity is the whole risk of
// the C++ port; this proves it before wiring into the pipeline.
//
// xgb_boundary_parity <dump.h5> <model.json>
//
// Prints: "<n> boundaries: t0 t1 t2 ..."
#include "inference/xgb_scene_boundary.hpp"
#include "inference/audio_logpsd.hpp"
#include <H5Cpp.h>
#include <iostream>
#include <fstream>
#include <string>
#include <vector>
int main(int argc, char** argv) {
if (argc < 3) { std::cerr << "usage: xgb_boundary_parity <dump.h5> <model.json>\n"; return 1; }
H5::H5File f(argv[1], H5F_ACC_RDONLY);
auto read2d = [&](const char* name, std::vector<std::vector<float>>& out, int cols) {
H5::DataSet ds = f.openDataSet(name);
H5::DataSpace sp = ds.getSpace();
hsize_t dims[2]; sp.getSimpleExtentDims(dims);
std::vector<float> flat(dims[0]*dims[1]);
ds.read(flat.data(), H5::PredType::NATIVE_FLOAT);
out.assign(dims[0], std::vector<float>(cols));
for (hsize_t i = 0; i < dims[0]; ++i)
for (int j = 0; j < cols; ++j) out[i][j] = flat[i*dims[1]+j];
};
std::vector<std::vector<float>> hist;
read2d("frames/rgb_hist", hist, XGBSceneBoundary::kHistBins*3);
H5::DataSet tsd = f.openDataSet("frames/timestamp_sec");
hsize_t td[1]; tsd.getSpace().getSimpleExtentDims(td);
std::vector<double> ts(td[0]);
tsd.read(ts.data(), H5::PredType::NATIVE_DOUBLE);
// parity debug: print video features for row 100 (compare to Python)
if (argc > 3 && std::string(argv[3]) == "--row100") {
auto base = XGBSceneBoundary::debug_base(hist, {});
std::cout << "row100:";
for (int j = 0; j < 17; ++j) std::cout << " " << base[100][j];
std::cout << "\n";
return 0;
}
// --feat <cpp_features.h5>: predict directly on the dumped C++ feature matrix
// (same bytes Python reads) — a clean parity check with no live-decode variance.
if (argc > 4 && std::string(argv[3]) == "--feat") {
H5::H5File ff(argv[4], H5F_ACC_RDONLY);
H5::DataSet fd = ff.openDataSet("features");
hsize_t fdm[2]; fd.getSpace().getSimpleExtentDims(fdm);
std::vector<float> X(fdm[0]*fdm[1]);
fd.read(X.data(), H5::PredType::NATIVE_FLOAT);
XGBSceneBoundary det(argv[2]);
auto pdbg = det.debug_predict(X, int(fdm[0]), int(fdm[1]));
auto pk = XGBSceneBoundary::debug_find_peaks(pdbg, 5);
std::cerr << "[parity] C++ raw peaks=" << pk.size() << "\n";
auto b = det.boundaries_from_features(X, int(fdm[0]), int(fdm[1]), ts);
std::cout << b.size() << " boundaries:";
for (double t : b) std::cout << " " << int(t);
std::cout << "\n";
return 0;
}
// Optional movie path (argv[4]): decode audio → per-second log-PSD.
std::vector<std::vector<float>> audio;
if (argc > 4) {
audio = AudioLogPSD::extract(argv[4]);
std::cerr << "[parity] audio rows=" << audio.size()
<< " (hist rows=" << hist.size() << ")\n";
}
XGBSceneBoundary det(argv[2]);
auto b = det.boundaries(hist, ts, audio);
std::cout << b.size() << " boundaries:";
for (double t : b) std::cout << " " << int(t);
std::cout << "\n";
return 0;
}
src/track_registry.hpp
+15 -3
View File
@@ -218,9 +218,21 @@ public:
const double clock =
reg_.awaits_evidence_ ? reg_.evidence_through_ : reg_.now_;
for (auto& [id, t] : reg_.tracks_) {
if (t.last_seen &&
(clock - *t.last_seen) > reg_.cfg_.track_extinction_sec)
continue; // retired from association; still awaiting evidence
// On-screen tracks are always candidates (actively tracked this
// frame). A dormant (off-screen) track is only worth keeping alive
// for re-association if it was actually IDENTIFIED: an unowned
// dormant track has no actor to re-attach to, so holding it in the
// pool only bloats the matcher's per-frame comparison set (every
// candidate is a GEMM row) and invites a new face re-associating
// onto an anonymous stub. Gating dormant tracks on t.actor keeps
// the pool bounded regardless of how large track_extinction_sec is
// — which is what makes a long re-association window affordable.
if (t.last_seen) { // dormant
if (!t.actor.has_value())
continue; // never identified: not worth re-associating
if ((clock - *t.last_seen) > reg_.cfg_.track_extinction_sec)
continue; // past the re-association horizon
}
out.push_back(&t);
}
return out;
src/types.hpp
+7
View File
@@ -31,6 +31,10 @@ struct Frame {
float cut_score{0.f}; // histogram cut score = 1 - hist_corr (0=identical, ~1=cut); HUD/debug
float bbox_upscale{1.f}; // multiply detector bboxes/landmarks by this to map back to
// original video resolution (>1 when dense_scale downscaled the frame)
// Normalised 32-bin-per-channel RGB histogram (96 floats), stamped by the
// camera-position node and carried to the sink for the learned scene-boundary
// detector (post-EOF, flood-fill boundaries). Empty when scene detection off.
std::vector<float> rgb_hist;
};
// ── CutEvent ──────────────────────────────────────────────────────────────────
@@ -162,6 +166,9 @@ struct SceneAnnotation {
// detection ran); kept for a future out-of-process scene detector.
bool is_cut{false};
bool is_scene_boundary{false};
// Per-frame RGB histogram, carried to the sink for the learned scene-boundary
// detector run post-EOF (flood-fill). Empty unless scene detection is enabled.
std::vector<float> rgb_hist;
};
// ── Actor gallery ─────────────────────────────────────────────────────────────