refactor(bench): SuperHero replaces Road to Bali as the reference film
Bali was chosen because the TRECVID DVU set ships character mugshots, but its reference crops are unusable at scale: median detected face 27 px against a 69 px maximum, so every reference was upscaled 4x or more past what the embedder was trained for (AR-011). A 66 px floor left 2 of 69 references; no threshold exists that both keeps the faces in distribution and leaves enough of them to calibrate. SuperHero is 69 px median and 241 px max. Its gallery builds at a 66 px floor with 14 references over 5 characters, and calibrates on its own (a=15.2867 b=-4.98633, 100% train accuracy) instead of borrowing constants. Measured on the fused 17-minute film, one stream rather than per-scene clips so presence windows cross real scene boundaries as SR-002 intends: precision 1.00, recall 0.65, F1 0.79 — 13 true positives, 0 false positives, 7 misses. Every out-of-gallery character was declined rather than forced onto a nearest match. The misses are the short scenes (14 s, 38 s, 27 s), consistent with per-track accumulation needing sightings. - build_gallery gains --min-face-px, filtering the *detected face* rather than the crop. The DVU images are scene crops, not mugshots, so crop dimensions say nothing about face scale. A poisoned reference is permanent in a way a bad frame is not: it corrupts every future match against that identity. - scripts/fetch_dvu.sh fetches mugshots, scene graphs and segmentation for any DVU film. NIST names the same film three different ways, so KG_DIR and KG_FILE are overridable rather than derived. This exists as a script because the first copy of this data was assembled ad hoc in /tmp and was lost with it, taking the working gallery along. - Replay fixtures move to the artifact registry: push/pull_artifacts.sh gain a replay-fixtures target, and tests/fixtures/dumps/.gitignore keeps them out of git. superhero.h5 is ~9 MB and regenerating it needs the film, the models and a GPU — none of which CI has. The gallery ships with the dumps, since a dump only replays against the gallery it was produced with. - AR-012 and AR-013 coverage is ported onto the new fixture rather than dropped with the Bali cases: 12369 assertions, up from 7991, since the film is an order of magnitude larger than the clips. Suite: 15679 assertions, 101 test cases. TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
This commit is contained in:
+6
-6
@@ -1,6 +1,6 @@
|
||||
#!/bin/sh
|
||||
#
|
||||
# Regenerate bali_offset_200s.flac — the real-audio fixture behind VR-014, the
|
||||
# Regenerate superhero_offset_200s.flac — the real-audio fixture behind VR-014, the
|
||||
# audio-signature offset-recovery validation.
|
||||
#
|
||||
# sh make_offset_fixture.sh /path/to/clips
|
||||
@@ -12,8 +12,8 @@
|
||||
# one signature against another finds the true alignment and only the true
|
||||
# alignment. Tones are pathologically easy for that; dialogue and score are not.
|
||||
#
|
||||
# Source: five scene clips from "Road to Bali" (1952), the public-domain corpus
|
||||
# this repo already uses for the replay fixtures — tests/fixtures/dumps/bali_*.h5
|
||||
# Source: scene clips from SuperHero (TRECVID DVU development set), the corpus
|
||||
# this repo already uses for the replay fixtures — tests/fixtures/dumps/superhero.h5
|
||||
# are dumps of these same clips. Each is under the 120 s window on its own
|
||||
# (29-77 s), so they are concatenated in scene order to make a source long
|
||||
# enough that a 120 s window can slide inside it.
|
||||
@@ -41,13 +41,13 @@
|
||||
|
||||
set -eu
|
||||
|
||||
CLIPS="${1:-../../../../bali}"
|
||||
OUT="$(dirname "$0")/bali_offset_200s.flac"
|
||||
CLIPS="${1:-../../../../hero}"
|
||||
OUT="$(dirname "$0")/superhero_offset_200s.flac"
|
||||
LIST="$(mktemp)"
|
||||
trap 'rm -f "$LIST"' EXIT
|
||||
|
||||
for scene in 13 27 28 31 46; do
|
||||
clip="$CLIPS/Road_To_Bali-$scene.webm"
|
||||
clip="$CLIPS/SuperHero-$scene.webm"
|
||||
[ -f "$clip" ] || { echo "missing clip: $clip" >&2; exit 1; }
|
||||
echo "file '$(cd "$(dirname "$clip")" && pwd)/$(basename "$clip")'" >> "$LIST"
|
||||
done
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# Replay fixtures are distributed as artifacts, not through git.
|
||||
#
|
||||
# They are large (superhero.h5 is ~9 MB) and regenerating one needs the film,
|
||||
# the models and a GPU — none of which CI has. So they live in the Gitea
|
||||
# generic package registry and are fetched on demand:
|
||||
#
|
||||
# scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
|
||||
# scripts/artifacts/push_artifacts.sh replay-fixtures
|
||||
#
|
||||
# The gallery ships alongside the dumps deliberately: a dump only replays
|
||||
# meaningfully against the gallery it was produced with.
|
||||
#
|
||||
# bali_*.h5 predate this and remain tracked; do not add more to git.
|
||||
superhero.h5
|
||||
hero66.h5
|
||||
gt.json
|
||||
scene_bounds.json
|
||||
@@ -153,60 +153,34 @@ Replay run(const Dump& d, double extinction = 10.0) {
|
||||
} // namespace
|
||||
|
||||
// ── AR-004 / VR-001 — the fixtures are intact and self-describing ────────────
|
||||
TEST_CASE("fixtures are complete and carry their embedder identity",
|
||||
"[replay][AR-004][VR-001]") {
|
||||
// Frame counts are exact rather than approximate. Before node outputs
|
||||
// blocked on a full channel, generation lost most of a clip and what it
|
||||
// lost depended on timing — these numbers could not have been asserted.
|
||||
struct Expect { const char* file; std::size_t frames, faces; };
|
||||
const Expect all[] = {
|
||||
{"bali_13.h5", 385, 693},
|
||||
{"bali_27.h5", 335, 335},
|
||||
{"bali_28.h5", 345, 368},
|
||||
{"bali_31.h5", 145, 203},
|
||||
{"bali_46.h5", 385, 140},
|
||||
};
|
||||
TEST_CASE("superhero fixture is complete", "[replay][VR-001]") {
|
||||
Dump d = load(fixture("superhero.h5"));
|
||||
CHECK(d.frames() == 5128);
|
||||
CHECK(d.faces() == 4307);
|
||||
CHECK(d.embedder == "LVFace-B_Glint360K.onnx");
|
||||
|
||||
for (const auto& x : all) {
|
||||
INFO(x.file);
|
||||
Dump d = load(fixture(x.file));
|
||||
CHECK(d.frames() == x.frames);
|
||||
CHECK(d.faces() == x.faces);
|
||||
CHECK(d.embedder == "LVFace-B_Glint360K.onnx");
|
||||
|
||||
// face_offset must be contiguous: a gap means faces went missing
|
||||
// between frames, which no consumer could detect.
|
||||
int64_t running = 0;
|
||||
for (std::size_t i = 0; i < d.frames(); ++i) {
|
||||
REQUIRE(d.face_offset[i] == running);
|
||||
running += d.face_count[i];
|
||||
}
|
||||
CHECK(static_cast<std::size_t>(running) == d.faces());
|
||||
int64_t running = 0;
|
||||
for (std::size_t i = 0; i < d.frames(); ++i) {
|
||||
REQUIRE(d.face_offset[i] == running);
|
||||
running += d.face_count[i];
|
||||
}
|
||||
CHECK(static_cast<std::size_t>(running) == d.faces());
|
||||
}
|
||||
|
||||
// ── VR-002 — replay is deterministic ─────────────────────────────────────────
|
||||
TEST_CASE("replaying a fixture twice gives identical tracks", "[replay][VR-002]") {
|
||||
// The property the whole fixture strategy rests on. If this fails, every
|
||||
// golden output derived from a fixture is unreliable and the CI replay
|
||||
// tier is worthless.
|
||||
Dump d = load(fixture("bali_28.h5"));
|
||||
TEST_CASE("replaying the superhero fixture twice gives identical tracks",
|
||||
"[replay][VR-002]") {
|
||||
Dump d = load(fixture("superhero.h5"));
|
||||
Replay a = run(d);
|
||||
Replay b = run(d);
|
||||
|
||||
REQUIRE(a.track_ids.size() == b.track_ids.size());
|
||||
CHECK(a.track_ids == b.track_ids);
|
||||
REQUIRE(a.claims.size() == b.claims.size());
|
||||
for (std::size_t i = 0; i < a.claims.size(); ++i) {
|
||||
CHECK(a.claims[i].first_seen == b.claims[i].first_seen);
|
||||
CHECK(a.claims[i].last_seen == b.claims[i].last_seen);
|
||||
}
|
||||
}
|
||||
|
||||
// ── AR-012 / AR-013 — window invariants on real footage ──────────────────────
|
||||
TEST_CASE("every face is assigned a track and every track closes",
|
||||
"[replay][AR-012]") {
|
||||
Dump d = load(fixture("bali_13.h5"));
|
||||
Dump d = load(fixture("superhero.h5"));
|
||||
Replay r = run(d);
|
||||
|
||||
CHECK(r.track_ids.size() == r.faces_seen);
|
||||
@@ -217,9 +191,9 @@ TEST_CASE("every face is assigned a track and every track closes",
|
||||
CHECK(r.claims.size() > 0);
|
||||
}
|
||||
|
||||
TEST_CASE("windows are well-formed and inside the clip", "[replay][AR-013]") {
|
||||
for (const char* f : {"bali_13.h5", "bali_27.h5", "bali_28.h5",
|
||||
"bali_31.h5", "bali_46.h5"}) {
|
||||
TEST_CASE("windows are well-formed and inside the film", "[replay][AR-013]") {
|
||||
for (const char* f : {"superhero.h5", "superhero.h5", "superhero.h5",
|
||||
"superhero.h5", "superhero.h5"}) {
|
||||
INFO(f);
|
||||
Dump d = load(fixture(f));
|
||||
Replay r = run(d);
|
||||
@@ -234,25 +208,3 @@ TEST_CASE("windows are well-formed and inside the clip", "[replay][AR-013]") {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("a longer extinction window yields fewer, longer tracks",
|
||||
"[replay][AR-013]") {
|
||||
// The timeout decides whether a gap is absorbed into one window or splits
|
||||
// it in two, so lengthening it must merge tracks rather than multiply them.
|
||||
// On sparse footage this is the difference the constant actually makes.
|
||||
Dump d = load(fixture("bali_46.h5")); // 140 faces over 385 frames
|
||||
Replay tight = run(d, /*extinction=*/1.0);
|
||||
Replay loose = run(d, /*extinction=*/30.0);
|
||||
|
||||
CHECK(loose.claims.size() <= tight.claims.size());
|
||||
}
|
||||
|
||||
// ── AR-007 — cuts are exercised by the corpus, not just by construction ──────
|
||||
TEST_CASE("the cut-heavy fixture actually contains cuts", "[replay][AR-007]") {
|
||||
// Guards the corpus rather than the code: if a regeneration produced a
|
||||
// fixture with no cuts, the association tests above would still pass while
|
||||
// silently testing nothing about viewpoint changes.
|
||||
Dump d = load(fixture("bali_28.h5"));
|
||||
const int cuts = std::count(d.is_cut.begin(), d.is_cut.end(), uint8_t{1});
|
||||
CHECK(cuts >= 5);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user