scene-actor-extraction/experiments/xsource

xsource — cross-source identification probe (VR-013)

Gallery from one recording, probes from another, swept over the probe's input resolution. Complements VR-005, which asked the same question over gallery mugshots: that one degrades an already-aligned 112×112 crop, holding alignment perfect, so it isolates the embedder. This one downscales the whole frame before the detector, so detection and landmark regression degrade with it.

Corpus: two Pexels clips of one shoot (4096×2160, 25 fps), four people, all four present in both. Clips and hand-sorted crops are gitignored — push them with scripts/artifacts/push_artifacts.sh, because the sorting is human ground truth and expensive to redo.

Scripts

script	does
dump_faces.py	detect every face, write a context crop per detection + a manifest
redraw_boxes.py	redraw those crops with the detection boxed, in place
propose_labels.py	propose labels for one clip from another clip's hand-sorted folders
make_review_site.py	local review.html — current label, crop, better match, correct and export
apply_corrections.py	apply the exported corrections.json
verify_labels.py	integrity gate: index consistency, duplicates, separation. Exits non-zero on failure
resolution_sweep.py	the VR-013 measurement
failure_analysis.py	what explains the misses — pose, size, blur, detector confidence
landmark_voting.py	average SCRFD's overlapping detections instead of discarding them
pose_label.py	mesh-estimated head pose, for hand correction (feeds VR-012)

Everything drives the shipped C++ through sae_embed; nothing reimplements detection, alignment, the embedder or the calibration. Scoring goes through the production gallery sigmoid — never a raw cosine (AR-024).

LD_PRELOAD=/usr/lib/libcudnn_cnn.so.9 python3 resolution_sweep.py

The preload is needed while ORT's CUDA provider looks for cudnnGetConvolutionBackwardDataAlgorithm_v7, which cuDNN 9 moved into libcudnn_cnn.so.9 behind a dispatch stub. Without it everything silently falls back to CPU.

What it found

Resolution is not the binding constraint here. TPI holds ~41–47% from 4096×2160 down to ~45 px faces, then falls: 23 px → 26%, 18 px → 12%, 14 px → 1.5%. Holding 90% of the plateau needs roughly 50 px end to end, against VR-005's ~22 px — the gap is detection and landmark error, which VR-005 excludes by construction.

FPI is 0.0% at every scale. Resolution loss goes entirely to TBI: the pipeline stops naming people rather than naming the wrong one.

The ceiling is cross-view, not resolution. Every person matches themselves strongly within a recording (sim 0.55–0.85) and collapses across the two (0.14–0.45, threshold 0.335). Only the person with frontal gallery references identified reliably, whatever their probe pose — so the lever is gallery pose coverage (docs/pose-expansion.md), not a better landmark model.

Landmark voting helps. SCRFD predicts each face from several anchors and NMS discards all but one, throwing away a median of 3 landmark estimates per face. Averaging them, weighted by confidence, lifts cross-clip TPI 41% → 49% for one forward pass and no extra model. A MediaPipe mesh as landmark source went the other way (41% → 16%): more stable within a recording, but a ring centroid is not the annotated landmark ArcFace was trained on, and the embedder punishes the off-distribution crop.

Reading these numbers

Four identities, 70 probes, one shoot. The ~47% plateau is pose, not resolution — half these faces are turned away and never clear threshold at any scale, so the absolute rates say little and the shape is the result. Both clips contain all four people, so there is no out-of-gallery class and the 10×-weighted out-of-cast misID is untested here; holding one identity out of the gallery would fix that. And the resolution curve is dominated by the single subject whose gallery references are frontal.