dtourolleandClaude Opus 5
01d7ead1e7
Gallery from one recording, probes from another, sweeping the probe's input resolution end to end. VR-005 asked the same question over gallery mugshots but degraded an already-aligned 112x112 crop with alignment held perfect, so it isolates the embedder. Here the whole frame is downscaled before the detector, so detection and landmark regression degrade with it — which is most of the difference. Corpus is two 4096x2160 clips of one shoot, four people, hand-sorted. Ground truth is sorted by hand and gated by verify_labels.py; labels carried down the scales geometrically by box position, never by embedding similarity, which would keep only the faces the embedder already gets right and drop the ones the sweep exists to find. Findings, all scored through the production gallery sigmoid at prob_threshold 0.754 — never a raw cosine: - Holding 90% of the plateau needs ~50 px end to end, against VR-005's ~22 px. min_face_px at 40 looks right; 32 would admit faces in the falling region. - FPI is 0.0% at every scale. Resolution loss goes entirely to TBI. - The ceiling is cross-view, not resolution: everyone matches themselves within a recording (0.55-0.85) and collapses across two (0.14-0.45, threshold 0.335). Only the subject with frontal *gallery* references identified reliably, whatever their probe pose — so the lever is gallery pose coverage, not a better landmark source. - Averaging SCRFD's overlapping detections instead of discarding them at NMS lifts cross-recording TPI 41% -> 49%, for one forward pass and no extra model. Four identities and one shoot, so the shape is the result and the absolute rates are not. Both clips contain all four people, so there is no out-of-gallery class and the 10x-weighted out-of-cast misID is untested here. Clips, frames, hand-sorted crops and results are gitignored and belong in the artifact registry — the sorting is human ground truth and expensive to redo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: VR-013 | AR-002, AR-005, AR-024
73 lines
4.0 KiB
Markdown
73 lines
4.0 KiB
Markdown
# xsource — cross-source identification probe (VR-013)
|
||
|
||
Gallery from **one** recording, probes from **another**, swept over the probe's
|
||
input resolution. Complements VR-005, which asked the same question over gallery
|
||
mugshots: that one degrades an already-aligned 112×112 crop, holding alignment
|
||
perfect, so it isolates the embedder. This one downscales the **whole frame**
|
||
before the detector, so detection and landmark regression degrade with it.
|
||
|
||
Corpus: two Pexels clips of one shoot (4096×2160, 25 fps), four people, all four
|
||
present in both. Clips and hand-sorted crops are gitignored — push them with
|
||
`scripts/artifacts/push_artifacts.sh`, because the sorting is human ground truth
|
||
and expensive to redo.
|
||
|
||
## Scripts
|
||
|
||
| script | does |
|
||
|---|---|
|
||
| `dump_faces.py` | detect every face, write a context crop per detection + a manifest |
|
||
| `redraw_boxes.py` | redraw those crops with the detection boxed, in place |
|
||
| `propose_labels.py` | propose labels for one clip from another clip's hand-sorted folders |
|
||
| `make_review_site.py` | local `review.html` — current label, crop, better match, correct and export |
|
||
| `apply_corrections.py` | apply the exported `corrections.json` |
|
||
| `verify_labels.py` | integrity gate: index consistency, duplicates, separation. Exits non-zero on failure |
|
||
| `resolution_sweep.py` | the VR-013 measurement |
|
||
| `failure_analysis.py` | what explains the misses — pose, size, blur, detector confidence |
|
||
| `landmark_voting.py` | average SCRFD's overlapping detections instead of discarding them |
|
||
| `pose_label.py` | mesh-estimated head pose, for hand correction (feeds VR-012) |
|
||
|
||
Everything drives the shipped C++ through `sae_embed`; nothing reimplements
|
||
detection, alignment, the embedder or the calibration. Scoring goes through the
|
||
production gallery sigmoid — never a raw cosine (AR-024).
|
||
|
||
LD_PRELOAD=/usr/lib/libcudnn_cnn.so.9 python3 resolution_sweep.py
|
||
|
||
The preload is needed while ORT's CUDA provider looks for
|
||
`cudnnGetConvolutionBackwardDataAlgorithm_v7`, which cuDNN 9 moved into
|
||
`libcudnn_cnn.so.9` behind a dispatch stub. Without it everything silently falls
|
||
back to CPU.
|
||
|
||
## What it found
|
||
|
||
**Resolution is not the binding constraint here.** TPI holds ~41–47% from 4096×2160
|
||
down to ~45 px faces, then falls: 23 px → 26%, 18 px → 12%, 14 px → 1.5%. Holding
|
||
90% of the plateau needs roughly 50 px end to end, against VR-005's ~22 px — the
|
||
gap is detection and landmark error, which VR-005 excludes by construction.
|
||
|
||
**FPI is 0.0% at every scale.** Resolution loss goes entirely to TBI: the pipeline
|
||
stops naming people rather than naming the wrong one.
|
||
|
||
**The ceiling is cross-view, not resolution.** Every person matches themselves
|
||
strongly *within* a recording (sim 0.55–0.85) and collapses *across* the two
|
||
(0.14–0.45, threshold 0.335). Only the person with frontal **gallery** references
|
||
identified reliably, whatever their probe pose — so the lever is gallery pose
|
||
coverage (`docs/pose-expansion.md`), not a better landmark model.
|
||
|
||
**Landmark voting helps.** SCRFD predicts each face from several anchors and NMS
|
||
discards all but one, throwing away a median of 3 landmark estimates per face.
|
||
Averaging them, weighted by confidence, lifts cross-clip TPI 41% → 49% for one
|
||
forward pass and no extra model. A MediaPipe mesh as landmark source went the
|
||
other way (41% → 16%): more stable within a recording, but a ring centroid is not
|
||
the annotated landmark ArcFace was trained on, and the embedder punishes the
|
||
off-distribution crop.
|
||
|
||
## Reading these numbers
|
||
|
||
Four identities, 70 probes, one shoot. The ~47% plateau is pose, not resolution —
|
||
half these faces are turned away and never clear threshold at any scale, so the
|
||
absolute rates say little and the *shape* is the result. Both clips contain all
|
||
four people, so there is no out-of-gallery class and the 10×-weighted out-of-cast
|
||
misID is **untested** here; holding one identity out of the gallery would fix
|
||
that. And the resolution curve is dominated by the single subject whose gallery
|
||
references are frontal.
|