Merge feature/opencv5: correct scene-boundary detector F1 numbers
Fold in the documentation accuracy fix: the boundary-detection F1 numbers now reflect the measured values at the shipped ±20s tolerance (44.1% leave-one-out / 29.8% grayscale / 72.9% train-all), replacing the stale pre-retrain '~34%' figure, and the evolution figure is split so the ±2s development curve is not mistaken for the shipped result.
This commit is contained in:
Binary file not shown.
|
After Width: | Height: | Size: 80 KiB |
@@ -62,7 +62,13 @@ Everything is per second, aligned to the 1-fps presence grid.
|
|||||||
the films where video is weak (Downton, Sound of Metal), so it is included and
|
the films where video is weak (Downton, Sound of Metal), so it is included and
|
||||||
the model uses it where it helps.
|
the model uses it where it helps.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
|
The left panel is the *feature* development, scored at a strict ±2 s tolerance so
|
||||||
|
each change is visible — this is where "delta beats raw histogram" was measured, not
|
||||||
|
the shipped tolerance. The right panel is the shipped detector at the ±20 s
|
||||||
|
tolerance the pipeline actually uses (see below). The two panels are on different
|
||||||
|
tolerances by design and must not be read as one curve.
|
||||||
|
|
||||||
Dead ends, all measured and discarded: audio-only detection; raw
|
Dead ends, all measured and discarded: audio-only detection; raw
|
||||||
histograms/PSDs as input; a two-tower BiLSTM (no better than the tree, far slower);
|
histograms/PSDs as input; a two-tower BiLSTM (no better than the tree, far slower);
|
||||||
@@ -81,14 +87,22 @@ and TransNetV2 (a Conv3D net that will not co-reside with the ROCm/VAAPI stack).
|
|||||||
real boundaries give way to noise. Selecting at the knee **self-calibrates the
|
real boundaries give way to noise. Selecting at the knee **self-calibrates the
|
||||||
boundary count** to roughly the true scene count, per film, with no global
|
boundary count** to roughly the true scene count, per film, with no global
|
||||||
threshold that would be wrong for every grade.
|
threshold that would be wrong for every grade.
|
||||||
- **Trained on all nine films** for the shipped model. Café Society and Scarface
|
- **Trained on all nine films** for the shipped model. Keeping the low-contrast
|
||||||
(the low-contrast grades) *must* be in training — held out, the model cannot
|
grades (Café Society, Scarface) in training matters most: on its own training
|
||||||
generalise to them; in training they reach 70–86% boundary-F1.
|
films the shipped model reaches **72.9% macro boundary-F1** (per-film 51–86%),
|
||||||
|
versus **29.8%** for the grayscale baseline on the same films.
|
||||||
|
|
||||||
Boundary detection, held out (leave-one-out, ±20 s tolerance — appropriate given
|
Boundary detection, held out (leave-one-out, ±20 s tolerance — appropriate given
|
||||||
~170 s scenes): **~34% F1, versus ~27% for the grayscale baseline.** The absolute
|
~170 s scenes): **44.1% macro F1, versus 29.8% for the grayscale baseline** — the
|
||||||
number is capped by the narrative-vs-audiovisual mismatch above; the point is the
|
honest generalisation number, each film scored by a detector trained on the other
|
||||||
downstream effect.
|
eight. Even the low-contrast grades generalise (Scarface held out 32%, Café Society
|
||||||
|
51%), where the grayscale detector scores 0% and 31%. The absolute number is capped
|
||||||
|
by the narrative-vs-audiovisual mismatch above — many boundaries have no
|
||||||
|
audio-visual signature at all — so the point is the downstream effect, below.
|
||||||
|
|
||||||
|
| boundary-F1 @±20 s | grayscale | learned (LOO) | learned (train-all) |
|
||||||
|
| ------------------ | --------: | ------------: | ------------------: |
|
||||||
|
| macro over 9 films | 29.8% | **44.1%** | 72.9% |
|
||||||
|
|
||||||
## The result that matters: actor presence
|
## The result that matters: actor presence
|
||||||
|
|
||||||
|
|||||||
@@ -57,15 +57,34 @@ def fig_macro():
|
|||||||
fig.tight_layout(); fig.savefig(OUT/"scene_presence_macro.png"); plt.close(fig)
|
fig.tight_layout(); fig.savefig(OUT/"scene_presence_macro.png"); plt.close(fig)
|
||||||
|
|
||||||
# ── Figure 3: feature/model evolution (boundary-F1 development) ──────────────
|
# ── Figure 3: feature/model evolution (boundary-F1 development) ──────────────
|
||||||
|
# Two panels, because the development curve and the shipped result are measured
|
||||||
|
# at DIFFERENT tolerances and must not be plotted on one axis:
|
||||||
|
# left — relative feature progress at the strict ±2 s tolerance (how the LSTM
|
||||||
|
# experiments were scored; establishes which features helped)
|
||||||
|
# right — the shipped XGBoost detector at the ±20 s tolerance the pipeline
|
||||||
|
# actually uses and scores at (grayscale vs learned-LOO vs train-all)
|
||||||
def fig_evolution():
|
def fig_evolution():
|
||||||
|
fig,(axl,axr)=plt.subplots(1,2,figsize=(11,4.5),gridspec_kw={"width_ratios":[1.15,1]})
|
||||||
|
|
||||||
steps=["grayscale\nbaseline","raw-hist\nLSTM","delta\nLSTM","XGBoost\n(delta+debounce)"]
|
steps=["grayscale\nbaseline","raw-hist\nLSTM","delta\nLSTM","XGBoost\n(delta+debounce)"]
|
||||||
f1=[7.2,7.5,10.8,15.2] # boundary-F1 @±2s during development
|
dev=[7.2,7.5,10.8,15.2] # boundary-F1 @±2s during LSTM-era development
|
||||||
fig,ax=plt.subplots(figsize=(6.5,4.5))
|
axl.plot(steps,dev,marker="o",color="#9aa7b4",lw=2,ms=8)
|
||||||
ax.plot(steps,f1,marker="o",color="#3d7ea6",lw=2,ms=8)
|
for i,v in enumerate(dev): axl.text(i,v+0.4,f"{v:.1f}%",ha="center",fontsize=9)
|
||||||
for i,v in enumerate(f1): ax.text(i,v+0.4,f"{v:.1f}%",ha="center",fontsize=10)
|
axl.set_ylabel("boundary F1 @±2 s (%)")
|
||||||
ax.set_ylabel("held-out boundary F1 @±2s (%)")
|
axl.set_title("Feature progress (strict ±2 s)")
|
||||||
ax.set_title("Detector development: features + model")
|
axl.set_ylim(0,18)
|
||||||
ax.set_ylim(0,18)
|
|
||||||
|
# shipped detector at the ±20s tolerance the pipeline uses — real measured
|
||||||
|
# macro numbers: grayscale (xgb_report gray_F1), learned LOO, learned train-all
|
||||||
|
names=["grayscale","learned\n(LOO)","learned\n(train-all)"]
|
||||||
|
f20=[29.8,44.1,72.9]; cols=["#e07a5f","#3d7ea6","#8fb8cf"]
|
||||||
|
bars=axr.bar(names,f20,color=cols)
|
||||||
|
for b,v in zip(bars,f20): axr.text(b.get_x()+b.get_width()/2,v+1.2,f"{v:.1f}%",
|
||||||
|
ha="center",fontsize=10,fontweight="bold")
|
||||||
|
axr.set_ylabel("boundary F1 @±20 s (%)")
|
||||||
|
axr.set_title("Shipped detector (±20 s, macro/9 films)")
|
||||||
|
axr.set_ylim(0,80)
|
||||||
|
fig.suptitle("Detector development, and where it landed",fontsize=13)
|
||||||
fig.tight_layout(); fig.savefig(OUT/"scene_detector_evolution.png"); plt.close(fig)
|
fig.tight_layout(); fig.savefig(OUT/"scene_detector_evolution.png"); plt.close(fig)
|
||||||
|
|
||||||
import csv as _csv
|
import csv as _csv
|
||||||
|
|||||||
Reference in New Issue
Block a user