scene-actor-extraction/experiments/xsource

xsource — cross-source identification probe (VR-013)

Gallery from one recording, probes from another, swept over the probe's input resolution. Complements VR-005, which asked the same question over gallery mugshots: that one degrades an already-aligned 112×112 crop, holding alignment perfect, so it isolates the embedder. This one downscales the whole frame before the detector, so detection and landmark regression degrade with it.

Corpus: two Pexels clips of one shoot (4096×2160, 25 fps), four people, all four present in both.

Getting the data

Clips, frames and hand-sorted crops are gitignored; they live in the artifact registry.

scripts/artifacts/pull_artifacts.sh xsource     # clips + labelling, frames regenerated
scripts/artifacts/push_artifacts.sh xsource     # after correcting labels

Pulling fetches the two clips and the hand-sorted crops, then regenerates the frames with ffmpeg — ~320 MB of PNG that is deterministic from the clips, so it is not worth shipping. Extraction settings are pinned in the pull script because the manifests key on frame filenames and on detection order within each frame; verify_labels.py runs at the end and will fail loudly if they drift.

Pull never overwrites an existing labelling/. That directory is human ground truth — somebody looked at all 167 crops and put each one in a folder — and it is the expensive part of this study, so push it once corrected.

Clips are Pexels-licensed: free to use, no attribution required, but not CC or MIT. Fine as a frozen CI artifact on private infrastructure; do not redistribute them as stock content.

Scripts

script	does
dump_faces.py	detect every face, write a context crop per detection + a manifest
redraw_boxes.py	redraw those crops with the detection boxed, in place
propose_labels.py	propose labels for one clip from another clip's hand-sorted folders
make_review_site.py	local review.html — current label, crop, better match, correct and export
apply_corrections.py	apply the exported corrections.json
verify_labels.py	integrity gate: index consistency, duplicates, separation. Exits non-zero on failure
resolution_sweep.py	the VR-013 measurement
failure_analysis.py	what explains the misses — pose, size, blur, detector confidence
landmark_voting.py	average SCRFD's overlapping detections instead of discarding them
pose_label.py	mesh-estimated head pose, for hand correction (feeds VR-012)

Everything drives the shipped C++ through sae_embed; nothing reimplements detection, alignment, the embedder or the calibration. Scoring goes through the production gallery sigmoid — never a raw cosine (AR-024).

LD_PRELOAD=/usr/lib/libcudnn_cnn.so.9 python3 resolution_sweep.py

The preload is needed while ORT's CUDA provider looks for cudnnGetConvolutionBackwardDataAlgorithm_v7, which cuDNN 9 moved into libcudnn_cnn.so.9 behind a dispatch stub. Without it everything silently falls back to CPU.

What it found

Resolution is not the binding constraint here. TPI holds ~41–47% from 4096×2160 down to ~45 px faces, then falls: 23 px → 26%, 18 px → 12%, 14 px → 1.5%. Holding 90% of the plateau needs roughly 50 px end to end, against VR-005's ~22 px — the gap is detection and landmark error, which VR-005 excludes by construction.

FPI is 0.0% at every scale. Resolution loss goes entirely to TBI: the pipeline stops naming people rather than naming the wrong one.

The ceiling is cross-view, not resolution. Every person matches themselves strongly within a recording (sim 0.55–0.85) and collapses across the two (0.14–0.45, threshold 0.335). Only the person with frontal gallery references identified reliably, whatever their probe pose — so the lever is gallery pose coverage (docs/pose-expansion.md), not a better landmark model.

Landmark voting helps. SCRFD predicts each face from several anchors and NMS discards all but one, throwing away a median of 3 landmark estimates per face. Averaging them, weighted by confidence, lifts cross-clip TPI 41% → 49% for one forward pass and no extra model. A MediaPipe mesh as landmark source went the other way (41% → 16%): more stable within a recording, but a ring centroid is not the annotated landmark ArcFace was trained on, and the embedder punishes the off-distribution crop.

Reading these numbers

Four identities, 70 probes, one shoot. The ~47% plateau is pose, not resolution — half these faces are turned away and never clear threshold at any scale, so the absolute rates say little and the shape is the result. Both clips contain all four people, so there is no out-of-gallery class and the 10×-weighted out-of-cast misID is untested here; holding one identity out of the gallery would fix that. And the resolution curve is dominated by the single subject whose gallery references are frontal.