refactor(bench): SuperHero replaces Road to Bali as the reference film
Bali was chosen because the TRECVID DVU set ships character mugshots, but its reference crops are unusable at scale: median detected face 27 px against a 69 px maximum, so every reference was upscaled 4x or more past what the embedder was trained for (AR-011). A 66 px floor left 2 of 69 references; no threshold exists that both keeps the faces in distribution and leaves enough of them to calibrate. SuperHero is 69 px median and 241 px max. Its gallery builds at a 66 px floor with 14 references over 5 characters, and calibrates on its own (a=15.2867 b=-4.98633, 100% train accuracy) instead of borrowing constants. Measured on the fused 17-minute film, one stream rather than per-scene clips so presence windows cross real scene boundaries as SR-002 intends: precision 1.00, recall 0.65, F1 0.79 — 13 true positives, 0 false positives, 7 misses. Every out-of-gallery character was declined rather than forced onto a nearest match. The misses are the short scenes (14 s, 38 s, 27 s), consistent with per-track accumulation needing sightings. - build_gallery gains --min-face-px, filtering the *detected face* rather than the crop. The DVU images are scene crops, not mugshots, so crop dimensions say nothing about face scale. A poisoned reference is permanent in a way a bad frame is not: it corrupts every future match against that identity. - scripts/fetch_dvu.sh fetches mugshots, scene graphs and segmentation for any DVU film. NIST names the same film three different ways, so KG_DIR and KG_FILE are overridable rather than derived. This exists as a script because the first copy of this data was assembled ad hoc in /tmp and was lost with it, taking the working gallery along. - Replay fixtures move to the artifact registry: push/pull_artifacts.sh gain a replay-fixtures target, and tests/fixtures/dumps/.gitignore keeps them out of git. superhero.h5 is ~9 MB and regenerating it needs the film, the models and a GPU — none of which CI has. The gallery ships with the dumps, since a dump only replays against the gallery it was produced with. - AR-012 and AR-013 coverage is ported onto the new fixture rather than dropped with the Bali cases: 12369 assertions, up from 7991, since the film is an order of magnitude larger than the clips. Suite: 15679 assertions, 101 test cases. TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
This commit is contained in:
@@ -10,6 +10,7 @@
|
||||
# scripts/artifacts/pull_artifacts.sh galleries [version]
|
||||
# scripts/artifacts/pull_artifacts.sh montage-frames <film-slug> [version]
|
||||
# scripts/artifacts/pull_artifacts.sh experiment-data [version]
|
||||
# scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
|
||||
# scripts/artifacts/pull_artifacts.sh report-highlights <name> [version]
|
||||
# scripts/artifacts/pull_artifacts.sh xsource [version]
|
||||
# version defaults to "latest" (newest uploaded version, by created_at).
|
||||
@@ -42,6 +43,22 @@ print(matches[-1]['version'])
|
||||
"
|
||||
}
|
||||
|
||||
pull_replay_fixtures() {
|
||||
local version="$1"
|
||||
local dest="${REPO_ROOT}/tests/fixtures/dumps"
|
||||
mkdir -p "$dest"
|
||||
echo "=== replay-fixtures (version ${version}) ==="
|
||||
local tmp; tmp="$(mktemp -d)"
|
||||
if curl -sf "${DL_BASE}/generic/replay-fixtures/${version}/replay-fixtures.zip" \
|
||||
-o "${tmp}/f.zip"; then
|
||||
unzip -qo "${tmp}/f.zip" -d "$dest"
|
||||
echo " restored: $(ls "$dest" | wc -l) files into tests/fixtures/dumps/"
|
||||
else
|
||||
echo " [warn] replay-fixtures.zip not found at version ${version}" >&2
|
||||
fi
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
pull_galleries() {
|
||||
local version="$1"
|
||||
local dest="${REPO_ROOT}/experiments/galleries"
|
||||
@@ -181,8 +198,13 @@ case "$TARGET" in
|
||||
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version xsource)"
|
||||
pull_xsource "$VERSION"
|
||||
;;
|
||||
replay-fixtures)
|
||||
VERSION="${2:-latest}"
|
||||
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version replay-fixtures)"
|
||||
pull_replay_fixtures "$VERSION"
|
||||
;;
|
||||
*)
|
||||
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2
|
||||
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, xsource, or replay-fixtures)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
@@ -13,10 +13,12 @@
|
||||
# scripts/artifacts/push_artifacts.sh experiment-data
|
||||
# scripts/artifacts/push_artifacts.sh report-highlights
|
||||
# scripts/artifacts/push_artifacts.sh xsource
|
||||
# scripts/artifacts/push_artifacts.sh replay-fixtures
|
||||
# scripts/artifacts/push_artifacts.sh galleries montage-frames experiment-data report-highlights
|
||||
#
|
||||
# Package layout (owner=dtourolle, repo=scene-actor-extraction):
|
||||
# generic/galleries/<version>/gallery_<model>.h5 (one file per model)
|
||||
# generic/replay-fixtures/<version>/replay-fixtures.zip (T2 dumps + their gallery)
|
||||
# generic/montage-frames/<version>/<film-slug>.zip (zipped per-film frames)
|
||||
# generic/experiment-data/<version>/experiment-data.zip (manifests/trajectories/results)
|
||||
# generic/report-highlights/<version>/<name>.jpg (individual, hand-picked
|
||||
@@ -50,6 +52,29 @@ upload() {
|
||||
-o /dev/null -w " HTTP %{http_code}\n"
|
||||
}
|
||||
|
||||
push_replay_fixtures() {
|
||||
echo "=== replay-fixtures (version ${VERSION}) ==="
|
||||
# T2 replay fixtures: per-frame detections, landmarks and embeddings dumped
|
||||
# from a real run, so the tracker and identity stages can be replayed on CPU
|
||||
# with no GPU, no models and no film. Too large for git (superhero.h5 alone
|
||||
# is ~9 MB) and regenerating them needs the film plus a GPU, which CI has
|
||||
# neither of — so they ship as artifacts and CI pulls them.
|
||||
#
|
||||
# The gallery travels with them: a dump replays against the gallery it was
|
||||
# produced with, and pairing a dump with a different gallery silently
|
||||
# changes every identity decision in it.
|
||||
local dir="${REPO_ROOT}/tests/fixtures/dumps"
|
||||
if [ ! -d "$dir" ]; then
|
||||
echo " no tests/fixtures/dumps dir, skipping" >&2
|
||||
return
|
||||
fi
|
||||
local tmp
|
||||
tmp="$(mktemp -d)"
|
||||
( cd "$dir" && zip -qr "$tmp/replay-fixtures.zip" . )
|
||||
upload "replay-fixtures" "replay-fixtures.zip" "$tmp/replay-fixtures.zip"
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
push_galleries() {
|
||||
echo "=== galleries (version ${VERSION}) ==="
|
||||
local dir="${REPO_ROOT}/experiments/galleries"
|
||||
@@ -149,7 +174,8 @@ for target in "$@"; do
|
||||
experiment-data) push_experiment_data ;;
|
||||
report-highlights) push_report_highlights ;;
|
||||
xsource) push_xsource ;;
|
||||
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2; exit 1 ;;
|
||||
replay-fixtures) push_replay_fixtures ;;
|
||||
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, xsource, or replay-fixtures)" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
|
||||
Reference in New Issue
Block a user