dtourolleandClaude Opus 5
d3ab598434
The cross-source study needs two 4K recordings and a hand-sorted set of face crops, neither of which belongs in git. Adds an xsource target to both artifact scripts. Push uploads the clips as-is (already compressed) and zips labelling/. Pull fetches both and regenerates frames with ffmpeg rather than downloading them: ~320 MB of PNG that is deterministic from the clips. The extraction settings are pinned in the script, not left to the caller, because the manifests key on frame filenames and on detection order within each frame — verify_labels.py runs afterwards and fails loudly if they drift. Pull refuses to overwrite an existing labelling/. It is human ground truth: somebody looked at 167 crops and placed each one, and silently replacing that with a remote copy would destroy the expensive half of the study. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: VR-013
93 lines
4.9 KiB
Markdown
93 lines
4.9 KiB
Markdown
# xsource — cross-source identification probe (VR-013)
|
||
|
||
Gallery from **one** recording, probes from **another**, swept over the probe's
|
||
input resolution. Complements VR-005, which asked the same question over gallery
|
||
mugshots: that one degrades an already-aligned 112×112 crop, holding alignment
|
||
perfect, so it isolates the embedder. This one downscales the **whole frame**
|
||
before the detector, so detection and landmark regression degrade with it.
|
||
|
||
Corpus: two Pexels clips of one shoot (4096×2160, 25 fps), four people, all four
|
||
present in both.
|
||
|
||
## Getting the data
|
||
|
||
Clips, frames and hand-sorted crops are gitignored; they live in the artifact
|
||
registry.
|
||
|
||
scripts/artifacts/pull_artifacts.sh xsource # clips + labelling, frames regenerated
|
||
scripts/artifacts/push_artifacts.sh xsource # after correcting labels
|
||
|
||
Pulling fetches the two clips and the hand-sorted crops, then regenerates the
|
||
frames with ffmpeg — ~320 MB of PNG that is deterministic from the clips, so it
|
||
is not worth shipping. Extraction settings are pinned in the pull script because
|
||
the manifests key on frame filenames *and* on detection order within each frame;
|
||
`verify_labels.py` runs at the end and will fail loudly if they drift.
|
||
|
||
Pull never overwrites an existing `labelling/`. That directory is human ground
|
||
truth — somebody looked at all 167 crops and put each one in a folder — and it
|
||
is the expensive part of this study, so push it once corrected.
|
||
|
||
Clips are Pexels-licensed: free to use, no attribution required, but not
|
||
CC or MIT. Fine as a frozen CI artifact on private infrastructure; do not
|
||
redistribute them as stock content.
|
||
|
||
## Scripts
|
||
|
||
| script | does |
|
||
|---|---|
|
||
| `dump_faces.py` | detect every face, write a context crop per detection + a manifest |
|
||
| `redraw_boxes.py` | redraw those crops with the detection boxed, in place |
|
||
| `propose_labels.py` | propose labels for one clip from another clip's hand-sorted folders |
|
||
| `make_review_site.py` | local `review.html` — current label, crop, better match, correct and export |
|
||
| `apply_corrections.py` | apply the exported `corrections.json` |
|
||
| `verify_labels.py` | integrity gate: index consistency, duplicates, separation. Exits non-zero on failure |
|
||
| `resolution_sweep.py` | the VR-013 measurement |
|
||
| `failure_analysis.py` | what explains the misses — pose, size, blur, detector confidence |
|
||
| `landmark_voting.py` | average SCRFD's overlapping detections instead of discarding them |
|
||
| `pose_label.py` | mesh-estimated head pose, for hand correction (feeds VR-012) |
|
||
|
||
Everything drives the shipped C++ through `sae_embed`; nothing reimplements
|
||
detection, alignment, the embedder or the calibration. Scoring goes through the
|
||
production gallery sigmoid — never a raw cosine (AR-024).
|
||
|
||
LD_PRELOAD=/usr/lib/libcudnn_cnn.so.9 python3 resolution_sweep.py
|
||
|
||
The preload is needed while ORT's CUDA provider looks for
|
||
`cudnnGetConvolutionBackwardDataAlgorithm_v7`, which cuDNN 9 moved into
|
||
`libcudnn_cnn.so.9` behind a dispatch stub. Without it everything silently falls
|
||
back to CPU.
|
||
|
||
## What it found
|
||
|
||
**Resolution is not the binding constraint here.** TPI holds ~41–47% from 4096×2160
|
||
down to ~45 px faces, then falls: 23 px → 26%, 18 px → 12%, 14 px → 1.5%. Holding
|
||
90% of the plateau needs roughly 50 px end to end, against VR-005's ~22 px — the
|
||
gap is detection and landmark error, which VR-005 excludes by construction.
|
||
|
||
**FPI is 0.0% at every scale.** Resolution loss goes entirely to TBI: the pipeline
|
||
stops naming people rather than naming the wrong one.
|
||
|
||
**The ceiling is cross-view, not resolution.** Every person matches themselves
|
||
strongly *within* a recording (sim 0.55–0.85) and collapses *across* the two
|
||
(0.14–0.45, threshold 0.335). Only the person with frontal **gallery** references
|
||
identified reliably, whatever their probe pose — so the lever is gallery pose
|
||
coverage (`docs/pose-expansion.md`), not a better landmark model.
|
||
|
||
**Landmark voting helps.** SCRFD predicts each face from several anchors and NMS
|
||
discards all but one, throwing away a median of 3 landmark estimates per face.
|
||
Averaging them, weighted by confidence, lifts cross-clip TPI 41% → 49% for one
|
||
forward pass and no extra model. A MediaPipe mesh as landmark source went the
|
||
other way (41% → 16%): more stable within a recording, but a ring centroid is not
|
||
the annotated landmark ArcFace was trained on, and the embedder punishes the
|
||
off-distribution crop.
|
||
|
||
## Reading these numbers
|
||
|
||
Four identities, 70 probes, one shoot. The ~47% plateau is pose, not resolution —
|
||
half these faces are turned away and never clear threshold at any scale, so the
|
||
absolute rates say little and the *shape* is the result. Both clips contain all
|
||
four people, so there is no out-of-gallery class and the 10×-weighted out-of-cast
|
||
misID is **untested** here; holding one identity out of the gallery would fix
|
||
that. And the resolution curve is dominated by the single subject whose gallery
|
||
references are frontal.
|