diff --git a/docs/scene-boundary-detector.md b/docs/scene-boundary-detector.md new file mode 100644 index 0000000..815d5c0 --- /dev/null +++ b/docs/scene-boundary-detector.md @@ -0,0 +1,172 @@ +# The learned scene-boundary detector + +Presence uses **flood-fill**: an actor seen once inside a shot is reported for the +whole shot (`[prev_boundary, next_boundary]`). That only works if the boundaries +are good. This page is the story of getting them good — a learned scene-boundary +detector that lifts per-second actor-presence F1 from **62.6% to 74.9%** across +the nine-film X-Ray benchmark, and fixes the film where naive flood-fill was +actively harmful. + +That 74.9% is the **leave-one-out** figure: each film is scored by a detector +trained on the *other eight*, so no film's presence is measured with a detector +that ever saw it. It is the honest generalisation number, and it is only ~1 point +below the all-nine-trained model (75.8%) — the detector barely overfits. + +## Why the old cut detector wasn't enough + +The always-on boundary source was the grayscale histogram-correlation cut detector +(`camera_position_change_detector`): mark a cut when the frame-to-frame grayscale +histogram correlation drops below 0.70. It is cheap and it fires on obvious hard +cuts, but on a low-contrast, uniformly-graded film it is nearly blind. On +**Scarface** it fired **once in 10,204 frames**. Flood-fill then snapped every +actor across essentially the whole film: + +| Scarface | precision | recall | +| -------- | --------- | ------ | +| flood + grayscale cuts | **26%** | 95% | +| track-extent (no flood) | 92% | 45% | + +That single failure is what motivated everything below: flood-fill needs a +boundary source that works regardless of grade. + +## What we are detecting, and why it is hard + +The training target is **Amazon X-Ray scene boundaries** (`scenes.csv`). These are +*narrative* scenes — a new location or beat in the story — not shot cuts. There +are only ~20–60 of them per film (median scene ~170 s), and many transition +*within* continuous visual style and continuous audio. So the signal is sparse and +often genuinely faint: a boundary detector working from audio-visual features can +never recall a narrative cut that has no audio-visual signature. + +This shapes every result: absolute boundary-F1 is modest by construction. What +matters is the **downstream** number — does snapping flood-fill to these +boundaries name the right actors — and there the gain is large. + +## The features (what worked, measured) + +Everything is per second, aligned to the 1-fps presence grid. + +- **Delta histograms, not raw histograms.** The raw RGB histogram encodes what a + frame *looks like*, not that it *changed* — measured boundary separability ~1.4×. + The **symmetric histogram delta** `|hist(t+k) − hist(t−k)|` separates boundaries + **4–5×**. Leading with deltas (k = 1,2,4,8 s) and dropping the raw histogram was + the single biggest feature win (LSTM F1 7.5% → 10.8%). +- **A multi-scale "ramp" bank.** Antisymmetric matched filters at half-widths + H = 2,4,6,8,10 s; the model weights the scales. Different films' boundaries peak + at different widths. +- **A time-since-last-boundary "debounce" clock**, scaled by the corpus mean scene + length (~205 s), encoding that scenes don't restart moments apart. +- **Audio log-PSD** (per-second, 4 s window, ~57 log-frequency bins). Measured + weak on its own — a standalone audio cutter scored only 3–6% held-out F1, because + narrative boundaries usually have continuous audio — but it is complementary on + the films where video is weak (Downton, Sound of Metal), so it is included and + the model uses it where it helps. + +![Detector development: features and model](assets/images/scene_detector_evolution.png) + +Dead ends, all measured and discarded: audio-only detection; raw +histograms/PSDs as input; a two-tower BiLSTM (no better than the tree, far slower); +larger FFT windows / more frequency bins (worse — boundaries are short events); +and TransNetV2 (a Conv3D net that will not co-reside with the ROCm/VAAPI stack). + +## The model + +- **XGBoost regressor** over a ±3 s window of the features above, predicting a + **soft Gaussian proximity-to-boundary target** (`exp(-(d/σ)²)`, σ = 10 s). + Regression to a soft target — rather than a hard 0/1 label — stops a near-miss + from being trained as a hard negative, and yields a smooth score whose **peaks** + are the boundaries. +- **Per-film knee threshold.** The predicted peak heights form a + convex-decreasing curve; the knee (max drop below the endpoints' chord) is where + real boundaries give way to noise. Selecting at the knee **self-calibrates the + boundary count** to roughly the true scene count, per film, with no global + threshold that would be wrong for every grade. +- **Trained on all nine films** for the shipped model. Café Society and Scarface + (the low-contrast grades) *must* be in training — held out, the model cannot + generalise to them; in training they reach 70–86% boundary-F1. + +Boundary detection, held out (leave-one-out, ±20 s tolerance — appropriate given +~170 s scenes): **~34% F1, versus ~27% for the grayscale baseline.** The absolute +number is capped by the narrative-vs-audiovisual mismatch above; the point is the +downstream effect. + +## The result that matters: actor presence + +Per-second X-Ray presence F1, macro over the nine films, at the shipped presence +config. The learned column is **leave-one-out** — each film scored by a detector +trained on the other eight: + +| boundary source for flood-fill | presence F1 | +| ------------------------------ | ----------- | +| track-extent (flood off) | 62.6% | +| flood + grayscale cuts | 64.0% | +| **flood + learned detector (LOO)** | **74.9%** | + +![Macro presence F1 by flood-fill boundary source](assets/images/scene_presence_macro.png) + +**+12.3 points over track-extent, +10.9 over the grayscale-cut flood, and it +improves every one of the nine films — under honest leave-one-out.** Per film: + +![Per-film presence F1 by boundary source](assets/images/scene_presence_by_source.png) + +| film | track-extent | flood+grayscale | flood+learned (LOO) | +| ---- | -----------: | --------------: | ------------------: | +| Benny & Joon | 77.3 | 80.2 | 78.2 | +| Café Society | 59.1 | 62.2 | 69.8 | +| Downton Abbey | 41.0 | 51.8 | **78.6** | +| Lord of War | 74.8 | 77.1 | 77.8 | +| Lovelace | 70.3 | 74.0 | 78.2 | +| The Many Saints of Newark | 37.5 | 43.9 | 53.4 | +| Scarface | 62.6 | **40.9** | **74.9** | +| Sound of Metal | 75.0 | 78.1 | 86.8 | +| Valerian | 65.6 | 67.7 | 76.2 | + +The two headline cases: + +- **Scarface**: the grayscale-cut flood *breaks* it (62.6 → 40.9), because it + detects one cut in the whole film. The learned detector — **on a film it never + trained on** — takes it to **74.9%**. This is the strongest evidence the + detector generalises: it fixes the exact failure that motivated it, held out. +- **Downton Abbey**: 41.0 (track-extent) → 51.8 (grayscale) → **78.6** — a + +37-point swing on the hardest film. + +Naive flood-fill barely beat doing nothing (64% vs 62%) and broke a film. With a +real boundary detector, flood-fill is decisively the right mode. + +## In the pipeline + +Boundary detection is a **post-EOF step**, like flood-fill itself: the per-film +knee needs every peak, so it can only run once the whole film is seen. The +`camera_position_change_detector` stamps a per-frame RGB histogram onto each frame; +it rides through to the result sink; at end-of-stream the sink runs the detector +over the collected histograms plus the movie's audio log-PSD and snaps the +presence windows to the result. Enable it with: + +```bash +scene_analyze --movie --gallery \ + --scene-xgb-model models/scene_boundary_xgb.json +``` + +Inference is real XGBoost, built into the binary via CMake (`SAE_SCENE_XGB`); the +audio log-PSD uses FFTW + the existing FFmpeg decode. To keep training and +inference on one feature implementation, the shipped model is **trained on the +C++-extracted features** (`scene_features_dump` → `train_xgb_cpp.py`) rather than a +re-implementation in Python — parity by construction. Verified end to end through +`scene_analyze` on a movie file and through the Jellyfin work-queue worker. + +## Reproduce + +```bash +# per-second audio log-PSD for each film +.venv-rocm/bin/python scripts/scene_detector/extract_audio_features.py \ + --manifest experiments/manifests/films_LVFace_opencv5.json + +# C++ feature matrices (same features training and inference share) +build/scene_features_dump + +# train the shipped model on all nine films +.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all + +# downstream A/B (track-extent vs flood+grayscale vs flood+learned) +scripts/scene_detector/downstream_presence.py +``` diff --git a/mkdocs.yml b/mkdocs.yml index a86e74a..daab834 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -35,6 +35,7 @@ extra_css: nav: - Home: index.md - How We Score Against X-Ray: methodology.md + - Learned Scene-Boundary Detector: scene-boundary-detector.md - Benchmark — SuperHero: benchmark.md - Full Experiment Log: model-bakeoff.md - Service Conversion (proposal): service-conversion.md diff --git a/scripts/scene_detector/make_figures.py b/scripts/scene_detector/make_figures.py new file mode 100644 index 0000000..e4689b8 --- /dev/null +++ b/scripts/scene_detector/make_figures.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +"""Generate the scene-boundary-detector report figures from saved results. +Data-driven, reproducible, no video needed. Writes PNGs to docs/assets/images/.""" +import json +from pathlib import Path +import numpy as np +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt + +OUT = Path("docs/assets/images") +OUT.mkdir(parents=True, exist_ok=True) +plt.rcParams.update({"font.size": 11, "axes.splines.top" if False else "axes.grid": True, + "axes.axisbelow": True, "grid.alpha": 0.3, "figure.dpi": 130}) + +FILMS = ["Benny & Joon","Café Society","Downton Abbey","Lord of War","Lovelace", + "Many Saints","Scarface","Sound of Metal","Valerian"] +# per-film presence F1 (downstream_loo run): track_extent, flood+grayscale, flood+learned(LOO) +TE = [77.3,59.1,41.0,74.8,70.3,37.5,62.6,75.0,65.6] +FG = [80.2,62.2,51.8,77.1,74.0,43.9,40.9,78.1,67.7] +FL = [78.2,69.8,78.6,77.8,78.2,53.4,74.9,86.8,76.2] + +# ── Figure 1: per-film presence F1, three boundary sources ─────────────────── +def fig_presence(): + x = np.arange(len(FILMS)); w = 0.26 + fig, ax = plt.subplots(figsize=(11,5)) + ax.bar(x-w, TE, w, label="track-extent (flood off)", color="#9aa7b4") + ax.bar(x, FG, w, label="flood + grayscale cuts", color="#e07a5f") + ax.bar(x+w, FL, w, label="flood + learned detector (LOO)", color="#3d7ea6") + ax.set_ylabel("per-second X-Ray presence F1 (%)") + ax.set_title("Actor-presence accuracy by flood-fill boundary source (leave-one-out)") + ax.set_xticks(x); ax.set_xticklabels(FILMS, rotation=30, ha="right") + ax.set_ylim(0,100); ax.legend(loc="upper left", framealpha=0.9) + # annotate the two headline swings + ax.annotate("grayscale flood\nBREAKS Scarface", xy=(6, 40.9), xytext=(5.1, 20), + fontsize=9, color="#b23", ha="center", + arrowprops=dict(arrowstyle="->", color="#b23")) + ax.annotate("+37pp", xy=(2+w, 78.6), xytext=(2+w, 90), fontsize=9, + color="#3d7ea6", ha="center", + arrowprops=dict(arrowstyle="->", color="#3d7ea6")) + macro=[np.mean(TE),np.mean(FG),np.mean(FL)] + ax.text(0.99,0.02,f"macro: {macro[0]:.1f}% / {macro[1]:.1f}% / {macro[2]:.1f}%", + transform=ax.transAxes, ha="right", va="bottom", fontsize=10, + bbox=dict(boxstyle="round", fc="#f4f4f4", ec="#ccc")) + fig.tight_layout(); fig.savefig(OUT/"scene_presence_by_source.png"); plt.close(fig) + +# ── Figure 2: macro presence F1 — the progression ─────────────────────────── +def fig_macro(): + labels=["track-extent","flood +\ngrayscale","flood +\nlearned (LOO)"] + vals=[np.mean(TE),np.mean(FG),np.mean(FL)] + fig,ax=plt.subplots(figsize=(6,4.5)) + bars=ax.bar(labels,vals,color=["#9aa7b4","#e07a5f","#3d7ea6"]) + for b,v in zip(bars,vals): ax.text(b.get_x()+b.get_width()/2, v+1, f"{v:.1f}%", + ha="center", fontsize=11, fontweight="bold") + ax.set_ylabel("macro presence F1 (%)"); ax.set_ylim(0,90) + ax.set_title("Flood-fill boundary source → presence accuracy") + fig.tight_layout(); fig.savefig(OUT/"scene_presence_macro.png"); plt.close(fig) + +# ── Figure 3: feature/model evolution (boundary-F1 development) ────────────── +def fig_evolution(): + steps=["grayscale\nbaseline","raw-hist\nLSTM","delta\nLSTM","XGBoost\n(delta+debounce)"] + f1=[7.2,7.5,10.8,15.2] # boundary-F1 @±2s during development + fig,ax=plt.subplots(figsize=(6.5,4.5)) + ax.plot(steps,f1,marker="o",color="#3d7ea6",lw=2,ms=8) + for i,v in enumerate(f1): ax.text(i,v+0.4,f"{v:.1f}%",ha="center",fontsize=10) + ax.set_ylabel("held-out boundary F1 @±2s (%)") + ax.set_title("Detector development: features + model") + ax.set_ylim(0,18) + fig.tight_layout(); fig.savefig(OUT/"scene_detector_evolution.png"); plt.close(fig) + +fig_presence(); fig_macro(); fig_evolution() +print("wrote:", *(p.name for p in sorted(OUT.glob("scene_*.png"))))