docs: VR-013 and VR-014 results; AR-002 raised to 40px

Records study results and the requirement change that follows from them.

VR-013 measures minimum face size end to end — gallery from one recording,
probes from another — rather than by degrading an already-aligned crop. Holding
90% of the plateau needs ~50 px that way against VR-005's ~22 px, the gap being
detection and landmark error rather than the embedder. AR-002 therefore takes
40 px, not 32: VR-005 isolates the embedder and is an upper bound, and 32 admits
faces in the falling region. FPI stayed 0.0% at every scale, and the ceiling is
cross-view rather than resolution.

VR-014 exercises audio-signature offset recovery on real film audio instead of
the synthetic golden tone. Forty random in-cap offsets, every one recovered to
the nearest frame, worst error 46 ms against a 500 ms budget — and 46 ms is the
quantisation floor rather than a result, since offsets land on whole 92.88 ms
frames. The runtime/2 anchor is confirmed through head-trimmed files.

The soft spot VR-014 found is tier labelling, not accuracy: the score drops with
sub-frame misalignment, so 27 of 40 correct alignments were demoted to `loose`.
One frame of slack in the score restores all forty to `audio` with false matches
unmoved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: AR-002, VR-005, VR-013, VR-014 | SR-002, SR-003
This commit is contained in:
2026-07-31 16:53:58 +02:00
co-authored by Claude Opus 5
parent d57e489e91
commit afca0524c9
4 changed files with 307 additions and 95 deletions
+82 -16
View File
@@ -40,12 +40,28 @@ Detect faces in sampled video frames.
presence (SR-002) a lower rate still answers the question, but it lengthens the
interval between samples and so weakens IoU-based association; sweep the two
together (VR-002).
- **Minimum face size is 66×66 px**, expressed in **original video resolution**,
- **Minimum face size is 40×40 px**, expressed in **original video resolution**,
not decoded-frame pixels. Stating it in original space decouples it from
`dense_scale`: otherwise a 0.5 downscale silently doubles the effective
threshold, and dense mode is exactly what scene detection uses.
66 is a working estimate of where ArcFace embeddings stop being reliable, not a
measured value — it should be replaced by the result of VR-005.
40 is **measured, not estimated** — it replaces an earlier 66 px guess. Two
studies bracket it, and the difference between them is the whole reason the
number is 40 rather than 32:
- **VR-005** degrades an already-aligned 112×112 crop and matches it against
a native-resolution gallery. Alignment is held perfect, so it isolates the
*embedder*: the knee sits at 2432 px, and 32 px still returns 98.1% TPI.
- **VR-013** downscales the **whole frame before the detector**, so detection
and landmark regression degrade along with it. End to end, holding 90% of
the plateau needs roughly **50 px**, against VR-005's ~22 px.
The gap is detection and landmark error, which VR-005 excludes by construction
— so VR-005 is an **upper bound on quality**, not a threshold, and reading a
floor off it would admit faces in the falling region. **AR-002 therefore takes
VR-013's number.** 40 sits below the 50 px plateau deliberately: FPI is 0.0% at
every scale in both studies, so resolution loss costs recall and never
precision, and an over-tight floor discards presence that SR-002 requires.
- Emits bounding box, detector confidence, and 5-point landmarks.
- Bounding boxes must be reported in **original video pixel space**. When
`dense_scale < 1` downscales the decoded frame, coordinates are rescaled by
@@ -59,7 +75,9 @@ Detect faces in sampled video frames.
**Current:** SCRFD-500MF via `face_detector_node.hpp`, thresholds in `config.hpp`
(`detector_conf` 0.5, `detector_nms` 0.4), `min_face_px` 40, `max_faces` 10.
**Gap:** `min_face_px` → 66 and re-expressed in original resolution; `max_faces`
**Gap:** `min_face_px` re-expressed in original resolution — the value 40 is
already correct after VR-013, so what remains is the space it is measured in, not
the number; `max_faces`
removed, gated on backpressure (AR-004).
## AR-004 — Backpressure
@@ -227,8 +245,8 @@ evidence. They are kept separate and **not collapsed into one scalar**: they fai
for different reasons, have different remedies, and — as below — do not even earn
the same response.
- **Size** — already AR-002, floor at 32×32 px in original resolution, measured
by VR-005 (knee at 2432 px). It is the precedent for the other two: the
- **Size** — already AR-002, floor at 40×40 px in original resolution, measured
end to end by VR-013. It is the precedent for the other two: the
threshold was *located*, not chosen.
- **Sharpness** — motion blur and soft focus destroy the high-frequency detail
the embedder keys on, and unlike size they leave the bounding box looking
@@ -260,6 +278,15 @@ the same response.
canonical px. Smooth and monotone with a usable range; the mapping onto real
faces is VR-012's to establish, and no threshold is set from these numbers.
**The synthetic ladder is noise-free and therefore optimistic about the low
end.** Measured on 400 real TMDB/Jellyfin headshots — the most frontal, most
cooperative population the pipeline ever sees — the residual runs p5 1.11,
median 2.74, p90 4.82, max 6.35 canonical px. So landmark noise alone occupies
roughly the first 3 px, and the synthetic sweep's "26° yaw ≈ 1.2 px" sits
*below* the noise floor on real data. VR-012 must set any threshold against
this measured distribution, and a discount curve has to treat the first few
pixels as uninformative rather than as mild pose.
Neither a dedicated landmark model (`models/2d106det.onnx` is present but
referenced nowhere — and it emits points, not pose) nor a direct pose CNN is
adopted unless VR-012 shows the residual insufficient. If one is needed the
@@ -278,15 +305,6 @@ the same response.
Only size drops the face outright, and only because VR-005 measured a knee below
which the embedding carries no signal to discount. Blur and pose are different:
**The synthetic ladder is noise-free and therefore optimistic about the low
end.** Measured on 400 real TMDB/Jellyfin headshots — the most frontal, most
cooperative population the pipeline ever sees — the residual runs p5 1.11,
median 2.74, p90 4.82, max 6.35 canonical px. So landmark noise alone occupies
roughly the first 3 px, and the synthetic sweep's "26° yaw ≈ 1.2 px" sits
*below* the noise floor on real data. VR-012 must set any threshold against
this measured distribution, and a discount curve has to treat the first few
pixels as uninformative rather than as mild pose.
- A blurred or turned face is still evidence of **presence**, which is what
SR-002 actually asks about.
- The tracker admits a link on position *or* identity precisely so that a face
@@ -1277,7 +1295,47 @@ plugin. Consequences to carry through:
- Files never processed by this pipeline still get a signature from the plugin;
the two paths coexist deliberately.
**Gap:** entire requirement — no audio path exists in the pipeline today.
**Current:** `src/audio_signature.*` implements the construction, and
`tests/fixtures/audio/` holds the golden vector shared verbatim with the plugin
repo, which now matches it byte for byte from C# (jRay `JR-042`/`JR-043`).
`sae_audio` (nanobind, as `sae_embed` and `sae_kpn` are) exposes the same C++ to
Python so a study drives the shipped code rather than a numpy port.
**VR-014 measures what the golden vector cannot** — that the signature actually
aligns a differently trimmed release, on real film audio rather than a synthetic
tone. It does, with an order of magnitude to spare.
**The accuracy question is settled and is not close.** What the offset is *for*
is shifting scene windows, which are seconds long, so half a second of error is
invisible; the budget is 500 ms. Over 40 random offsets inside the ±600-frame cap
the recovered offset was the nearest frame every time — **worst error 46 ms**.
That figure is the quantisation floor rather than a measurement of quality: the
offset is expressed in whole 92.88 ms frames, so no correct answer can ever be
worse than half a frame. The `runtime/2` anchor behaves as specified through real
head-trimmed files (cutting `delta` from the head moves the window by
`delta/2`), and both an out-of-cap offset and unrelated content are declined
outright (0.10 and 0.07).
**Where it is soft is tier labelling, not alignment.** The *score* at the correct
offset falls with sub-frame misalignment — 0.940.99 when the true offset lands
within 0.1 of a frame boundary, 0.690.73 at half a frame — because the two
windows' frame grids no longer coincide. The offset stays right, but only 13 of
40 cleared the server's 0.85 `audio` threshold and the other 27 were demoted to
`loose`, a tier that means "possibly the same cut, degraded audio". The threshold
was calibrated on a re-encode at *zero* offset, where the score is 1.00.
The remedy is measured, not proposed (UT-108): counting a frame as agreeing if
its peak bin matches **within ±1 frame** returns all 40 to `audio` (worst 0.906)
while unrelated content and out-of-cap offsets stay at 0.12 and 0.16 — the gap
that makes the threshold mean anything is untouched. It costs 81 ms of offset
accuracy, of a 500 ms budget, because the flattened peak lets the argmax pick an
adjacent frame. ±2 frames buys nothing further. Adopting it is a
[server spec](../../JRay-public-server/SPEC.md) §3 change — the score is
normative and shared by three repos — so this repo measures it and leaves the
decision there.
**Gap:** the signature is computed but **not yet emitted** into the truth file —
that is the `IR-002` field and the coordinated `schema_version` bump.
## IR-006 — Jellyfin round-trip
@@ -1524,6 +1582,14 @@ Verify `landmarks`/`bbox`/`is_cut` suffice, and bump `schema_version` if not.
Quantify where ArcFace degrades, replacing the 66×66 estimate in A1 with a
measurement.
> **Result, and its limit.** Knee at 2432 px; 32 px returns 98.1% TPI at 0.0
> FPI. But the probe is an already-aligned 112×112 crop, so alignment is held
> perfect and this measures the **embedder alone** — an upper bound, not a
> threshold. **VR-013** re-asks the question end to end, downscaling the whole
> frame before the detector, and lands near 50 px. AR-002's floor of 40 px comes
> from VR-013; this study is what shows how much of the gap is detection and
> landmark error rather than embedding.
**Method.**
1. Select ~100 gallery actors having more than one mugshot.