diff --git a/docs/SPEC.md b/docs/SPEC.md index b692d28..a972b41 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -149,14 +149,55 @@ Produce the exact input ArcFace expects. **Current:** `align_face()` in `src/face_utils.hpp`, Umeyama fit via `umeyama_similarity()`, `cv::warpAffine` to `{112, 112}`. **Gap:** none. -> **Migration note.** Until this landed the fit was -> `cv::estimateAffinePartial2D(…, cv::RANSAC, 3.0)`. Where RANSAC kept all five -> points its final refit is the same least-squares optimum, so the two agree; -> they diverge exactly where a landmark fell outside the 3 px band — i.e. on the -> non-frontal faces. Galleries baked before this change therefore carry embeddings -> from a marginally different warp, concentrated on the hardest views. Rebuilding -> is cheap and removes the question; GR-004's embedder stamp does **not** catch an -> aligner change, only a model change. +> **Migration note — this was a defect, not a refinement.** Until this landed the +> fit was `cv::estimateAffinePartial2D(…, cv::RANSAC, 3.0)`. The expectation was +> that the two agree wherever RANSAC keeps all five points, leaving a small +> divergence on non-frontal faces. **Measured, that is wrong.** On 400 random +> gallery headshots, one model held fixed and only the estimator varied: +> +> | | median | p90 | max | +> |---|---|---|---| +> | Crop disagreement (source px, over the crop corners) | 16.97 | 75.91 | 223.31 | +> | `cos(umeyama, ransac)` for the resulting embedding | 0.791 | — | — | +> +> 83.5 % of crops embed to a cosine below 0.99 of their Umeyama counterpart — +> they are not the same face crop. The mechanism is that a 4-DoF similarity is +> exactly determined by **two** points, so every minimal RANSAC sample fits its +> own pair perfectly and is then scored on the other three. Real landmarks sit a +> median 2.74 canonical px from any similarity fit to the template (see AR-030 +> below), so images with a landmark outside the 3 px band are the common case, +> not the exception; RANSAC then keeps two or three inliers and returns a wildly +> under-determined transform. +> +> **Every gallery baked before this change must be rebuilt** — GR-004's embedder +> stamp catches a model change, not an aligner change, so nothing else would say +> so. +> +> **How much this cost in accuracy is a separate question, and the answer appears +> to be: less than the crop numbers suggest.** Rebuilding the full gallery +> (2456 actors) moved the intra/inter separation the AR-023 calibration is fitted +> from only slightly: +> +> | | intra-actor | inter-actor | separation | +> |---|---|---|---| +> | RANSAC | 0.6234 | 0.0407 | 0.5827 | +> | Umeyama | 0.6340 | 0.0440 | 0.5900 | +> +> The reconciliation is that the old warp was *wrong but self-consistent*: it +> produced a differently-framed face rather than a scrambled one, gallery and +> probe went through the same estimator, and the embedder tolerates framing +> variation. So the figures in `model-bakeoff.md`, `best-model.md` and +> `pose-expansion.md` were all produced through the broken warp on both sides and +> should be re-run, but there is no measured basis for expecting them to move far. +> +> The sharper evidence of the old instability is duplicate detection: rebuilding +> with an unchanged `dedup_tol` dropped **1614** near-duplicate images, where the +> original build dropped on the order of a hundred. Near-identical source images +> used to embed to visibly different vectors — RANSAC fitting two-point subsets is +> unstable under small landmark perturbations, and being RNG-driven it was not +> reproducible either. That instability is what a tracker accumulating evidence +> across frames pays for, and it is the strongest reason the fix is worth having +> independently of any accuracy delta. ## AR-006 — Embedding @@ -237,6 +278,15 @@ the same response. Only size drops the face outright, and only because VR-005 measured a knee below which the embedding carries no signal to discount. Blur and pose are different: + **The synthetic ladder is noise-free and therefore optimistic about the low + end.** Measured on 400 real TMDB/Jellyfin headshots — the most frontal, most + cooperative population the pipeline ever sees — the residual runs p5 1.11, + median 2.74, p90 4.82, max 6.35 canonical px. So landmark noise alone occupies + roughly the first 3 px, and the synthetic sweep's "26° yaw ≈ 1.2 px" sits + *below* the noise floor on real data. VR-012 must set any threshold against + this measured distribution, and a discount curve has to treat the first few + pixels as uninformative rather than as mild pose. + - A blurred or turned face is still evidence of **presence**, which is what SR-002 actually asks about. - The tracker admits a link on position *or* identity precisely so that a face diff --git a/docs/requirements.md b/docs/requirements.md index 0409f4f..9f15a43 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -32,7 +32,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned | | AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing | | AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs | -| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`; the RANSAC fit it replaces was RNG-driven and point-dropping. Galleries baked before the change want a rebuild | +| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far | | AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done | | AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks | | AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted | diff --git a/tests/test_face_utils.cpp b/tests/test_face_utils.cpp index 6cb1e75..08e24bc 100644 --- a/tests/test_face_utils.cpp +++ b/tests/test_face_utils.cpp @@ -1,6 +1,8 @@ +// TRACES: AR-005, AR-030 | SR-002 +// // Unit tests for the geometric/numeric helpers in types.hpp and face_utils.hpp: -// cosine_similarity and the ArcFace 5-point alignment transform. GPU-free, -// model-free. +// cosine_similarity, the ArcFace 5-point alignment transform, and the alignment +// residual that AR-030 reads as its visibility measure. GPU-free, model-free. #include #include