refactor(bench): SuperHero replaces Road to Bali as the reference film
Bali was chosen because the TRECVID DVU set ships character mugshots, but its reference crops are unusable at scale: median detected face 27 px against a 69 px maximum, so every reference was upscaled 4x or more past what the embedder was trained for (AR-011). A 66 px floor left 2 of 69 references; no threshold exists that both keeps the faces in distribution and leaves enough of them to calibrate. SuperHero is 69 px median and 241 px max. Its gallery builds at a 66 px floor with 14 references over 5 characters, and calibrates on its own (a=15.2867 b=-4.98633, 100% train accuracy) instead of borrowing constants. Measured on the fused 17-minute film, one stream rather than per-scene clips so presence windows cross real scene boundaries as SR-002 intends: precision 1.00, recall 0.65, F1 0.79 — 13 true positives, 0 false positives, 7 misses. Every out-of-gallery character was declined rather than forced onto a nearest match. The misses are the short scenes (14 s, 38 s, 27 s), consistent with per-track accumulation needing sightings. - build_gallery gains --min-face-px, filtering the *detected face* rather than the crop. The DVU images are scene crops, not mugshots, so crop dimensions say nothing about face scale. A poisoned reference is permanent in a way a bad frame is not: it corrupts every future match against that identity. - scripts/fetch_dvu.sh fetches mugshots, scene graphs and segmentation for any DVU film. NIST names the same film three different ways, so KG_DIR and KG_FILE are overridable rather than derived. This exists as a script because the first copy of this data was assembled ad hoc in /tmp and was lost with it, taking the working gallery along. - Replay fixtures move to the artifact registry: push/pull_artifacts.sh gain a replay-fixtures target, and tests/fixtures/dumps/.gitignore keeps them out of git. superhero.h5 is ~9 MB and regenerating it needs the film, the models and a GPU — none of which CI has. The gallery ships with the dumps, since a dump only replays against the gallery it was produced with. - AR-012 and AR-013 coverage is ported onto the new fixture rather than dropped with the Bali cases: 12369 assertions, up from 7991, since the film is an order of magnitude larger than the clips. Suite: 15679 assertions, 101 test cases. TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
This commit is contained in:
BIN
Binary file not shown.
+6
-6
@@ -1,6 +1,6 @@
|
||||
#!/bin/sh
|
||||
#
|
||||
# Regenerate bali_offset_200s.flac — the real-audio fixture behind VR-014, the
|
||||
# Regenerate superhero_offset_200s.flac — the real-audio fixture behind VR-014, the
|
||||
# audio-signature offset-recovery validation.
|
||||
#
|
||||
# sh make_offset_fixture.sh /path/to/clips
|
||||
@@ -12,8 +12,8 @@
|
||||
# one signature against another finds the true alignment and only the true
|
||||
# alignment. Tones are pathologically easy for that; dialogue and score are not.
|
||||
#
|
||||
# Source: five scene clips from "Road to Bali" (1952), the public-domain corpus
|
||||
# this repo already uses for the replay fixtures — tests/fixtures/dumps/bali_*.h5
|
||||
# Source: scene clips from SuperHero (TRECVID DVU development set), the corpus
|
||||
# this repo already uses for the replay fixtures — tests/fixtures/dumps/superhero.h5
|
||||
# are dumps of these same clips. Each is under the 120 s window on its own
|
||||
# (29-77 s), so they are concatenated in scene order to make a source long
|
||||
# enough that a 120 s window can slide inside it.
|
||||
@@ -41,13 +41,13 @@
|
||||
|
||||
set -eu
|
||||
|
||||
CLIPS="${1:-../../../../bali}"
|
||||
OUT="$(dirname "$0")/bali_offset_200s.flac"
|
||||
CLIPS="${1:-../../../../hero}"
|
||||
OUT="$(dirname "$0")/superhero_offset_200s.flac"
|
||||
LIST="$(mktemp)"
|
||||
trap 'rm -f "$LIST"' EXIT
|
||||
|
||||
for scene in 13 27 28 31 46; do
|
||||
clip="$CLIPS/Road_To_Bali-$scene.webm"
|
||||
clip="$CLIPS/SuperHero-$scene.webm"
|
||||
[ -f "$clip" ] || { echo "missing clip: $clip" >&2; exit 1; }
|
||||
echo "file '$(cd "$(dirname "$clip")" && pwd)/$(basename "$clip")'" >> "$LIST"
|
||||
done
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# Replay fixtures are distributed as artifacts, not through git.
|
||||
#
|
||||
# They are large (superhero.h5 is ~9 MB) and regenerating one needs the film,
|
||||
# the models and a GPU — none of which CI has. So they live in the Gitea
|
||||
# generic package registry and are fetched on demand:
|
||||
#
|
||||
# scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
|
||||
# scripts/artifacts/push_artifacts.sh replay-fixtures
|
||||
#
|
||||
# The gallery ships alongside the dumps deliberately: a dump only replays
|
||||
# meaningfully against the gallery it was produced with.
|
||||
#
|
||||
# bali_*.h5 predate this and remain tracked; do not add more to git.
|
||||
superhero.h5
|
||||
hero66.h5
|
||||
gt.json
|
||||
scene_bounds.json
|
||||
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
Reference in New Issue
Block a user