docs: VR-013 and VR-014 results; AR-002 raised to 40px
Records study results and the requirement change that follows from them. VR-013 measures minimum face size end to end — gallery from one recording, probes from another — rather than by degrading an already-aligned crop. Holding 90% of the plateau needs ~50 px that way against VR-005's ~22 px, the gap being detection and landmark error rather than the embedder. AR-002 therefore takes 40 px, not 32: VR-005 isolates the embedder and is an upper bound, and 32 admits faces in the falling region. FPI stayed 0.0% at every scale, and the ceiling is cross-view rather than resolution. VR-014 exercises audio-signature offset recovery on real film audio instead of the synthetic golden tone. Forty random in-cap offsets, every one recovered to the nearest frame, worst error 46 ms against a 500 ms budget — and 46 ms is the quantisation floor rather than a result, since offsets land on whole 92.88 ms frames. The runtime/2 anchor is confirmed through head-trimmed files. The soft spot VR-014 found is tier labelling, not accuracy: the score drops with sub-frame misalignment, so 27 of 40 correct alignments were demoted to `loose`. One frame of slack in the score restores all forty to `audio` with false matches unmoved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: AR-002, VR-005, VR-013, VR-014 | SR-002, SR-003
This commit is contained in:
+82
-16
@@ -40,12 +40,28 @@ Detect faces in sampled video frames.
|
||||
presence (SR-002) a lower rate still answers the question, but it lengthens the
|
||||
interval between samples and so weakens IoU-based association; sweep the two
|
||||
together (VR-002).
|
||||
- **Minimum face size is 66×66 px**, expressed in **original video resolution**,
|
||||
- **Minimum face size is 40×40 px**, expressed in **original video resolution**,
|
||||
not decoded-frame pixels. Stating it in original space decouples it from
|
||||
`dense_scale`: otherwise a 0.5 downscale silently doubles the effective
|
||||
threshold, and dense mode is exactly what scene detection uses.
|
||||
66 is a working estimate of where ArcFace embeddings stop being reliable, not a
|
||||
measured value — it should be replaced by the result of VR-005.
|
||||
|
||||
40 is **measured, not estimated** — it replaces an earlier 66 px guess. Two
|
||||
studies bracket it, and the difference between them is the whole reason the
|
||||
number is 40 rather than 32:
|
||||
|
||||
- **VR-005** degrades an already-aligned 112×112 crop and matches it against
|
||||
a native-resolution gallery. Alignment is held perfect, so it isolates the
|
||||
*embedder*: the knee sits at 24–32 px, and 32 px still returns 98.1% TPI.
|
||||
- **VR-013** downscales the **whole frame before the detector**, so detection
|
||||
and landmark regression degrade along with it. End to end, holding 90% of
|
||||
the plateau needs roughly **50 px**, against VR-005's ~22 px.
|
||||
|
||||
The gap is detection and landmark error, which VR-005 excludes by construction
|
||||
— so VR-005 is an **upper bound on quality**, not a threshold, and reading a
|
||||
floor off it would admit faces in the falling region. **AR-002 therefore takes
|
||||
VR-013's number.** 40 sits below the 50 px plateau deliberately: FPI is 0.0% at
|
||||
every scale in both studies, so resolution loss costs recall and never
|
||||
precision, and an over-tight floor discards presence that SR-002 requires.
|
||||
- Emits bounding box, detector confidence, and 5-point landmarks.
|
||||
- Bounding boxes must be reported in **original video pixel space**. When
|
||||
`dense_scale < 1` downscales the decoded frame, coordinates are rescaled by
|
||||
@@ -59,7 +75,9 @@ Detect faces in sampled video frames.
|
||||
**Current:** SCRFD-500MF via `face_detector_node.hpp`, thresholds in `config.hpp`
|
||||
(`detector_conf` 0.5, `detector_nms` 0.4), `min_face_px` 40, `max_faces` 10.
|
||||
|
||||
**Gap:** `min_face_px` → 66 and re-expressed in original resolution; `max_faces`
|
||||
**Gap:** `min_face_px` re-expressed in original resolution — the value 40 is
|
||||
already correct after VR-013, so what remains is the space it is measured in, not
|
||||
the number; `max_faces`
|
||||
removed, gated on backpressure (AR-004).
|
||||
|
||||
## AR-004 — Backpressure
|
||||
@@ -227,8 +245,8 @@ evidence. They are kept separate and **not collapsed into one scalar**: they fai
|
||||
for different reasons, have different remedies, and — as below — do not even earn
|
||||
the same response.
|
||||
|
||||
- **Size** — already AR-002, floor at 32×32 px in original resolution, measured
|
||||
by VR-005 (knee at 24–32 px). It is the precedent for the other two: the
|
||||
- **Size** — already AR-002, floor at 40×40 px in original resolution, measured
|
||||
end to end by VR-013. It is the precedent for the other two: the
|
||||
threshold was *located*, not chosen.
|
||||
- **Sharpness** — motion blur and soft focus destroy the high-frequency detail
|
||||
the embedder keys on, and unlike size they leave the bounding box looking
|
||||
@@ -260,6 +278,15 @@ the same response.
|
||||
canonical px. Smooth and monotone with a usable range; the mapping onto real
|
||||
faces is VR-012's to establish, and no threshold is set from these numbers.
|
||||
|
||||
**The synthetic ladder is noise-free and therefore optimistic about the low
|
||||
end.** Measured on 400 real TMDB/Jellyfin headshots — the most frontal, most
|
||||
cooperative population the pipeline ever sees — the residual runs p5 1.11,
|
||||
median 2.74, p90 4.82, max 6.35 canonical px. So landmark noise alone occupies
|
||||
roughly the first 3 px, and the synthetic sweep's "26° yaw ≈ 1.2 px" sits
|
||||
*below* the noise floor on real data. VR-012 must set any threshold against
|
||||
this measured distribution, and a discount curve has to treat the first few
|
||||
pixels as uninformative rather than as mild pose.
|
||||
|
||||
Neither a dedicated landmark model (`models/2d106det.onnx` is present but
|
||||
referenced nowhere — and it emits points, not pose) nor a direct pose CNN is
|
||||
adopted unless VR-012 shows the residual insufficient. If one is needed the
|
||||
@@ -278,15 +305,6 @@ the same response.
|
||||
Only size drops the face outright, and only because VR-005 measured a knee below
|
||||
which the embedding carries no signal to discount. Blur and pose are different:
|
||||
|
||||
**The synthetic ladder is noise-free and therefore optimistic about the low
|
||||
end.** Measured on 400 real TMDB/Jellyfin headshots — the most frontal, most
|
||||
cooperative population the pipeline ever sees — the residual runs p5 1.11,
|
||||
median 2.74, p90 4.82, max 6.35 canonical px. So landmark noise alone occupies
|
||||
roughly the first 3 px, and the synthetic sweep's "26° yaw ≈ 1.2 px" sits
|
||||
*below* the noise floor on real data. VR-012 must set any threshold against
|
||||
this measured distribution, and a discount curve has to treat the first few
|
||||
pixels as uninformative rather than as mild pose.
|
||||
|
||||
- A blurred or turned face is still evidence of **presence**, which is what
|
||||
SR-002 actually asks about.
|
||||
- The tracker admits a link on position *or* identity precisely so that a face
|
||||
@@ -1277,7 +1295,47 @@ plugin. Consequences to carry through:
|
||||
- Files never processed by this pipeline still get a signature from the plugin;
|
||||
the two paths coexist deliberately.
|
||||
|
||||
**Gap:** entire requirement — no audio path exists in the pipeline today.
|
||||
**Current:** `src/audio_signature.*` implements the construction, and
|
||||
`tests/fixtures/audio/` holds the golden vector shared verbatim with the plugin
|
||||
repo, which now matches it byte for byte from C# (jRay `JR-042`/`JR-043`).
|
||||
`sae_audio` (nanobind, as `sae_embed` and `sae_kpn` are) exposes the same C++ to
|
||||
Python so a study drives the shipped code rather than a numpy port.
|
||||
|
||||
**VR-014 measures what the golden vector cannot** — that the signature actually
|
||||
aligns a differently trimmed release, on real film audio rather than a synthetic
|
||||
tone. It does, with an order of magnitude to spare.
|
||||
|
||||
**The accuracy question is settled and is not close.** What the offset is *for*
|
||||
is shifting scene windows, which are seconds long, so half a second of error is
|
||||
invisible; the budget is 500 ms. Over 40 random offsets inside the ±600-frame cap
|
||||
the recovered offset was the nearest frame every time — **worst error 46 ms**.
|
||||
That figure is the quantisation floor rather than a measurement of quality: the
|
||||
offset is expressed in whole 92.88 ms frames, so no correct answer can ever be
|
||||
worse than half a frame. The `runtime/2` anchor behaves as specified through real
|
||||
head-trimmed files (cutting `delta` from the head moves the window by
|
||||
`delta/2`), and both an out-of-cap offset and unrelated content are declined
|
||||
outright (0.10 and 0.07).
|
||||
|
||||
**Where it is soft is tier labelling, not alignment.** The *score* at the correct
|
||||
offset falls with sub-frame misalignment — 0.94–0.99 when the true offset lands
|
||||
within 0.1 of a frame boundary, 0.69–0.73 at half a frame — because the two
|
||||
windows' frame grids no longer coincide. The offset stays right, but only 13 of
|
||||
40 cleared the server's 0.85 `audio` threshold and the other 27 were demoted to
|
||||
`loose`, a tier that means "possibly the same cut, degraded audio". The threshold
|
||||
was calibrated on a re-encode at *zero* offset, where the score is 1.00.
|
||||
|
||||
The remedy is measured, not proposed (UT-108): counting a frame as agreeing if
|
||||
its peak bin matches **within ±1 frame** returns all 40 to `audio` (worst 0.906)
|
||||
while unrelated content and out-of-cap offsets stay at 0.12 and 0.16 — the gap
|
||||
that makes the threshold mean anything is untouched. It costs 81 ms of offset
|
||||
accuracy, of a 500 ms budget, because the flattened peak lets the argmax pick an
|
||||
adjacent frame. ±2 frames buys nothing further. Adopting it is a
|
||||
[server spec](../../JRay-public-server/SPEC.md) §3 change — the score is
|
||||
normative and shared by three repos — so this repo measures it and leaves the
|
||||
decision there.
|
||||
|
||||
**Gap:** the signature is computed but **not yet emitted** into the truth file —
|
||||
that is the `IR-002` field and the coordinated `schema_version` bump.
|
||||
|
||||
## IR-006 — Jellyfin round-trip
|
||||
|
||||
@@ -1524,6 +1582,14 @@ Verify `landmarks`/`bbox`/`is_cut` suffice, and bump `schema_version` if not.
|
||||
Quantify where ArcFace degrades, replacing the 66×66 estimate in A1 with a
|
||||
measurement.
|
||||
|
||||
> **Result, and its limit.** Knee at 24–32 px; 32 px returns 98.1% TPI at 0.0
|
||||
> FPI. But the probe is an already-aligned 112×112 crop, so alignment is held
|
||||
> perfect and this measures the **embedder alone** — an upper bound, not a
|
||||
> threshold. **VR-013** re-asks the question end to end, downscaling the whole
|
||||
> frame before the detector, and lands near 50 px. AR-002's floor of 40 px comes
|
||||
> from VR-013; this study is what shows how much of the gap is detection and
|
||||
> landmark error rather than embedding.
|
||||
|
||||
**Method.**
|
||||
|
||||
1. Select ~100 gallery actors having more than one mugshot.
|
||||
|
||||
Reference in New Issue
Block a user