refactor(bench): SuperHero replaces Road to Bali as the reference film
Bali was chosen because the TRECVID DVU set ships character mugshots, but its reference crops are unusable at scale: median detected face 27 px against a 69 px maximum, so every reference was upscaled 4x or more past what the embedder was trained for (AR-011). A 66 px floor left 2 of 69 references; no threshold exists that both keeps the faces in distribution and leaves enough of them to calibrate. SuperHero is 69 px median and 241 px max. Its gallery builds at a 66 px floor with 14 references over 5 characters, and calibrates on its own (a=15.2867 b=-4.98633, 100% train accuracy) instead of borrowing constants. Measured on the fused 17-minute film, one stream rather than per-scene clips so presence windows cross real scene boundaries as SR-002 intends: precision 1.00, recall 0.65, F1 0.79 — 13 true positives, 0 false positives, 7 misses. Every out-of-gallery character was declined rather than forced onto a nearest match. The misses are the short scenes (14 s, 38 s, 27 s), consistent with per-track accumulation needing sightings. - build_gallery gains --min-face-px, filtering the *detected face* rather than the crop. The DVU images are scene crops, not mugshots, so crop dimensions say nothing about face scale. A poisoned reference is permanent in a way a bad frame is not: it corrupts every future match against that identity. - scripts/fetch_dvu.sh fetches mugshots, scene graphs and segmentation for any DVU film. NIST names the same film three different ways, so KG_DIR and KG_FILE are overridable rather than derived. This exists as a script because the first copy of this data was assembled ad hoc in /tmp and was lost with it, taking the working gallery along. - Replay fixtures move to the artifact registry: push/pull_artifacts.sh gain a replay-fixtures target, and tests/fixtures/dumps/.gitignore keeps them out of git. superhero.h5 is ~9 MB and regenerating it needs the film, the models and a GPU — none of which CI has. The gallery ships with the dumps, since a dump only replays against the gallery it was produced with. - AR-012 and AR-013 coverage is ported onto the new fixture rather than dropped with the Bali cases: 12369 assertions, up from 7991, since the film is an order of magnitude larger than the clips. Suite: 15679 assertions, 101 test cases. TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
#!/bin/bash
|
||||
# fetch_dvu.sh — pull one film's character mugshots and presence annotations from
|
||||
# the NIST TRECVID Deep Video Understanding development set.
|
||||
#
|
||||
# The DVU dev set is the reason Road to Bali is our benchmark film: it ships
|
||||
# 5-7 face crops per *character*, cut from the film itself, alongside
|
||||
# scene-scoped presence annotations. That matches SR-002 directly — presence is
|
||||
# per scene, not per frame — and it keeps ground truth in character space, so
|
||||
# scoring needs no actor->character mapping.
|
||||
#
|
||||
# This exists as a script, rather than as ad hoc commands, because the first
|
||||
# copy of this data lived in a temp directory and was lost to a /tmp wipe,
|
||||
# taking the working gallery with it.
|
||||
#
|
||||
# 14 films are asserted Creative Commons and need no data agreement (only the
|
||||
# 5 KinoLorber test films are gated).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/fetch_dvu.sh [film] [dest]
|
||||
# film default Road_To_Bali
|
||||
# dest default ./dvu
|
||||
set -euo pipefail
|
||||
|
||||
BASE="https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset"
|
||||
FILM="${1:-Road_To_Bali}"
|
||||
DEST="${2:-dvu}"
|
||||
|
||||
mkdir -p "$DEST/images" "$DEST/scenes"
|
||||
|
||||
echo "[dvu] $FILM -> $DEST"
|
||||
|
||||
# Scene segmentation: start/end as HH:MM:SS. Note valkaama.csv line 38 carries a
|
||||
# shift-key typo (01:!4:00) — parse defensively if you extend this to that film.
|
||||
echo "[dvu] scene segmentation"
|
||||
curl -fsSL "$BASE/scene.segmentation.reference/${FILM}.csv" \
|
||||
-o "$DEST/${FILM}.csv" || echo " (missing: ${FILM}.csv)"
|
||||
|
||||
# Entity types: which entities are Person vs Location/Concept. Only Person rows
|
||||
# become gallery identities — the images/ directory also holds Location and
|
||||
# Concept crops (bedroom, boat, ...), which must not enter a face gallery.
|
||||
#
|
||||
# Directory and file naming are inconsistent with the film slug used elsewhere:
|
||||
# the folder is Road_to_Bali (lowercase "to") while the entity file is
|
||||
# RoadToBali.entity.types.txt. Both are derived here rather than assumed.
|
||||
# NIST is inconsistent across all three axes, and not by a rule worth deriving:
|
||||
# Road to Bali is Road_To_Bali.csv / Road_to_Bali/ / RoadToBali.entity.types.txt,
|
||||
# while SuperHero is SuperHero.csv / superHero/ / superhero.entity.types.txt.
|
||||
# Defaults cover the Bali shape; override per film rather than guessing.
|
||||
# KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero dvu-hero
|
||||
KG_DIR="${KG_DIR:-${FILM//_To_/_to_}}"
|
||||
KG_FILE="${KG_FILE:-$(echo "$FILM" | sed -E 's/_([a-z])/\U\1/g; s/_//g')}"
|
||||
|
||||
echo "[dvu] entity types ($KG_DIR/$KG_FILE)"
|
||||
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/${KG_FILE}.entity.types.txt" \
|
||||
-o "$DEST/${FILM}.entity.types.txt" || echo " (missing: entity types)"
|
||||
|
||||
# Character face crops. Names are discovered from the directory listing rather
|
||||
# than probed as <Character>_N, since the crop count varies per character and
|
||||
# the listing is authoritative.
|
||||
echo "[dvu] character mugshots"
|
||||
PERSONS="$DEST/persons.txt"
|
||||
if [ -f "$DEST/${FILM}.entity.types.txt" ]; then
|
||||
grep -iE "person" "$DEST/${FILM}.entity.types.txt" \
|
||||
| sed -E 's/[[:space:]]*[:,].*$//' | tr -d '\r' \
|
||||
| awk '{print tolower($1)}' | sort -u > "$PERSONS"
|
||||
fi
|
||||
|
||||
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/images/" 2>/dev/null \
|
||||
| grep -oE 'href="[^"?/][^"]*\.png"' | sed -E 's/href="//; s/"//' | sort -u \
|
||||
> "$DEST/all_images.txt"
|
||||
|
||||
while read -r img; do
|
||||
[ -z "$img" ] && continue
|
||||
# Strip the trailing _N to recover the entity name.
|
||||
who="$(echo "$img" | sed -E 's/_[0-9]+\.png$//' | awk '{print tolower($0)}')"
|
||||
if [ -s "$PERSONS" ] && ! grep -qx "$who" "$PERSONS"; then
|
||||
continue # Location/Concept crop, not a face
|
||||
fi
|
||||
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/images/${img}" \
|
||||
-o "$DEST/images/${img}" 2>/dev/null || rm -f "$DEST/images/${img}"
|
||||
done < "$DEST/all_images.txt"
|
||||
|
||||
# Per-scene knowledge graphs. A Person->Location edge means that person was
|
||||
# present for the whole scene. Some of these contain a stray ", ," that breaks
|
||||
# strict JSON parsers.
|
||||
echo "[dvu] scene graphs"
|
||||
for n in $(seq 1 60); do
|
||||
curl -fsSL "$BASE/scenes_knowledge_graphs/${FILM//_/ }-${n}.json" \
|
||||
-o "$DEST/scenes/${FILM}-${n}.json" 2>/dev/null \
|
||||
|| curl -fsSL "$BASE/scenes_knowledge_graphs/${FILM}-${n}.json" \
|
||||
-o "$DEST/scenes/${FILM}-${n}.json" 2>/dev/null \
|
||||
|| rm -f "$DEST/scenes/${FILM}-${n}.json"
|
||||
done
|
||||
|
||||
echo "[dvu] done:"
|
||||
echo " mugshots: $(ls "$DEST/images" 2>/dev/null | wc -l)"
|
||||
echo " scenes: $(ls "$DEST/scenes" 2>/dev/null | wc -l)"
|
||||
echo " csv: $([ -f "$DEST/${FILM}.csv" ] && echo yes || echo no)"
|
||||
Reference in New Issue
Block a user