dtourolle
d98dc2855a
Bali was chosen because the TRECVID DVU set ships character mugshots, but its reference crops are unusable at scale: median detected face 27 px against a 69 px maximum, so every reference was upscaled 4x or more past what the embedder was trained for (AR-011). A 66 px floor left 2 of 69 references; no threshold exists that both keeps the faces in distribution and leaves enough of them to calibrate. SuperHero is 69 px median and 241 px max. Its gallery builds at a 66 px floor with 14 references over 5 characters, and calibrates on its own (a=15.2867 b=-4.98633, 100% train accuracy) instead of borrowing constants. Measured on the fused 17-minute film, one stream rather than per-scene clips so presence windows cross real scene boundaries as SR-002 intends: precision 1.00, recall 0.65, F1 0.79 — 13 true positives, 0 false positives, 7 misses. Every out-of-gallery character was declined rather than forced onto a nearest match. The misses are the short scenes (14 s, 38 s, 27 s), consistent with per-track accumulation needing sightings. - build_gallery gains --min-face-px, filtering the *detected face* rather than the crop. The DVU images are scene crops, not mugshots, so crop dimensions say nothing about face scale. A poisoned reference is permanent in a way a bad frame is not: it corrupts every future match against that identity. - scripts/fetch_dvu.sh fetches mugshots, scene graphs and segmentation for any DVU film. NIST names the same film three different ways, so KG_DIR and KG_FILE are overridable rather than derived. This exists as a script because the first copy of this data was assembled ad hoc in /tmp and was lost with it, taking the working gallery along. - Replay fixtures move to the artifact registry: push/pull_artifacts.sh gain a replay-fixtures target, and tests/fixtures/dumps/.gitignore keeps them out of git. superhero.h5 is ~9 MB and regenerating it needs the film, the models and a GPU — none of which CI has. The gallery ships with the dumps, since a dump only replays against the gallery it was produced with. - AR-012 and AR-013 coverage is ported onto the new fixture rather than dropped with the Bali cases: 12369 assertions, up from 7991, since the film is an order of magnitude larger than the clips. Suite: 15679 assertions, 101 test cases. TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
68 lines
3.3 KiB
Bash
Executable File
68 lines
3.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# make_fixtures.sh — regenerate the committed replay fixtures.
|
|
#
|
|
# TRACES: VR-001 | PR-002
|
|
#
|
|
# CI never calls a model (see docs/requirements.md, "CI never calls a model"):
|
|
# the embedder is impractical on the N100 CI host, so inference happens HERE, on
|
|
# a machine with a GPU, and CI consumes the HDF5 dumps as data. Everything
|
|
# downstream of embedding — tracking, presence windows, belief accumulation,
|
|
# expansion — is cheap CPU maths and replays from these files.
|
|
#
|
|
# Reproducibility is a requirement, not a nicety. A fixture whose provenance is
|
|
# unknown is worse than no fixture, because it will be trusted. Every parameter
|
|
# that affects the output is pinned below rather than left to a default, and the
|
|
# dumps carry the embedder identity and SHA-256 (GR-004) so a replay cannot be
|
|
# silently scored against the wrong gallery.
|
|
#
|
|
# These are byte-reproducible only because node outputs block rather than drop
|
|
# on a full channel (AR-004). Before that fix the same command produced
|
|
# different dumps run to run, since what got dropped depended on timing.
|
|
#
|
|
# Source: hero/ — SuperHero, from the TRECVID DVU development set. Chosen over
|
|
# SuperHero on face scale: Bali reference crops had a median detected face of
|
|
# 27 px against a 69 px maximum, so every reference was upscaled far past what
|
|
# the embedder was trained for. SuperHero is 69 px median, 241 px max. That matters: derived
|
|
# fixtures can be committed, where anything cut from a copyrighted title could
|
|
# not live in the repository at all.
|
|
set -euo pipefail
|
|
|
|
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
CLIPS="${CLIPS:-$REPO/../hero}"
|
|
GALLERY="${GALLERY:-$REPO/gallery_lvface.h5}"
|
|
BIN="${BIN:-$REPO/build/scene_analyze}"
|
|
OUT="$REPO/tests/fixtures/dumps"
|
|
|
|
# Pinned. Changing either invalidates every committed fixture.
|
|
# fps 5 — 1 fps over a 77 s clip is 77 frames, too thin to exercise an
|
|
# extinction window measured in tens of seconds.
|
|
# min-face — 32 px. This is a *fixture* setting, deliberately below AR-002's
|
|
# production floor of 40 px (VR-013, measured end to end): the
|
|
# corpus is 480x360, where faces run 40-80 px, so pinning at 40
|
|
# would thin the dumps for reasons unrelated to what they test.
|
|
# 32 px is where VR-005 still shows 98.1% TPI, so the faces kept
|
|
# are identifiable; it is not the threshold the pipeline ships.
|
|
FPS=5
|
|
MIN_FACE_PX=32
|
|
|
|
[[ -x "$BIN" ]] || { echo "no scene_analyze at $BIN (set BIN=)" >&2; exit 1; }
|
|
[[ -f "$GALLERY" ]] || { echo "no gallery at $GALLERY (set GALLERY=)" >&2; exit 1; }
|
|
[[ -d "$CLIPS" ]] || { echo "no clips at $CLIPS (set CLIPS=)" >&2; exit 1; }
|
|
|
|
mkdir -p "$OUT"
|
|
|
|
for clip in "$CLIPS"/SuperHero-*.webm; do
|
|
n="$(basename "$clip" .webm)"; n="${n##*-}"
|
|
echo "── superhero_$n"
|
|
"$BIN" --movie "$clip" --gallery "$GALLERY" \
|
|
--fps "$FPS" --min-face-px "$MIN_FACE_PX" \
|
|
--dump-embeddings "$OUT/superhero_$n.h5" \
|
|
--output /dev/null 2>&1 | grep -E "wrote|dropped" || true
|
|
done
|
|
|
|
echo
|
|
echo "Regenerated in $OUT — verify the diff is empty if nothing upstream changed."
|
|
echo "A non-empty diff means detection, alignment or embedding moved. That is"
|
|
echo "either a regression or a deliberate change, and either way the golden"
|
|
echo "outputs derived from these fixtures need reviewing."
|