docs(register): record the quality vector, the benchmark, and what lossless fanout costs
Status for the two changes just landed, plus the consequence AR-004's fix has for the scene join. The annotator's old comment said blocking there was safe because the branches are independent. That stopped being true when the fanout became lossless: it now stops popping once one branch stops taking, so a starved detector and a waiting annotator would wedge. What actually makes it safe is join depth -- the fanout can run the dense branch ahead by the whole of the sampled branch's buffering, which at kSceneJoinDepth 256 and sample_fps 5 against a 25 fps source is ~1200 dense frames against TransNetV2's 100-frame window. Cutting kSceneJoinDepth below the window would reintroduce the wedge, so it is now a correctness precondition rather than a tuning knob. TRACES: AR-004, AR-010, AR-028, AR-029 | VR-015 | SR-002
This commit is contained in:
+115
-13
@@ -253,6 +253,26 @@ the same response.
|
||||
perfectly healthy. Measured on the **112×112 aligned crop**, not the raw box:
|
||||
the crop is already scale-normalised, so a measure taken there cannot silently
|
||||
re-measure face size and double-count it against AR-002.
|
||||
|
||||
The measure is the **variance of the Laplacian divided by the variance of the
|
||||
crop** — `crop_sharpness()`, dimensionless. The division is the part that
|
||||
earns its place: a raw Laplacian variance, the textbook measure, scales with
|
||||
the square of image contrast, so a dim scene reads as soft and a graded-up one
|
||||
as sharp, and VR-012 would locate a different knee in every film. That is
|
||||
AR-024's objection to the raw cosine in another metric. Normalised, the axis
|
||||
means the same thing everywhere, which is the precondition for a single knee
|
||||
existing at all.
|
||||
|
||||
Read spectrally it is `E[|ω|⁴]` under the crop's own energy distribution, so
|
||||
the blur ladder is monotone by construction rather than by fitting: Gaussian
|
||||
blur multiplies that distribution by `e^{-σ²|ω|²}`, which can only move mass
|
||||
downward. Two consequences follow from the same identity and are recorded on
|
||||
the function: it needs the low-frequency mass real images have (on a
|
||||
flat-spectrum synthetic an anisotropic smear makes it *rise*, because the
|
||||
surviving perpendicular detail really is as fine as before), and it conflates
|
||||
focus with intrinsic texture, so a bearded face outscores a smooth one at equal
|
||||
focus. Both are true of every no-reference sharpness measure, and both are
|
||||
reasons AR-028 carries the number rather than thresholding on it.
|
||||
- **Visibility** — extreme pose or occlusion means the face presents fewer of the
|
||||
features the embedding assumes are present. The measure is the **residual of
|
||||
the AR-005 alignment fit**: the RMS landmark error, in canonical 112×112
|
||||
@@ -333,18 +353,43 @@ hand-chosen cutoff on an uncalibrated measure is the same unfalsifiable magic
|
||||
number AR-024 retired for similarity, and it would fail the same way: meaning
|
||||
something different for every detector, every embedder and every film.
|
||||
|
||||
**Current:** visibility is measured and carried — `estimate_alignment()` in
|
||||
`src/face_utils.hpp` returns the residual alongside the transform, and
|
||||
`FaceAlignerFunc` writes it to `DetectedFace::alignment_residual`. Size is
|
||||
`min_face_px` (40, decoded-frame space — AR-002 still open). Sharpness is
|
||||
unmeasured. Nothing yet *consumes* any of it: no discount is applied, and
|
||||
`align_face()` still drops the degenerate-fit case without counting it.
|
||||
**Current:** all three axes are measured and carried, and the vector reaches the
|
||||
dump. `FaceAlignerFunc` is where it is filled in, because both measured axes fall
|
||||
out of work the warp already does: visibility is the residual from
|
||||
`estimate_alignment()`, and sharpness is `crop_sharpness()` on the 112×112 crop
|
||||
the node has just produced. Size stays `bbox` — deliberately not copied into a
|
||||
field of its own, since that would hold the same quantity in two coordinate
|
||||
spaces and the copy is the one that drifts. No face is admitted unscored, so a
|
||||
negative value downstream is a bug rather than a poor-quality face. The
|
||||
degenerate-fit case is still dropped — it has no crop and no fit to score — but
|
||||
is now **counted** and reported once at EOF instead of vanishing.
|
||||
|
||||
**Gap:** AR-029 entirely. For AR-030, the measure exists but the discount does
|
||||
not — it must reach `EvidenceDiscounter` as the reliability term. For AR-028, the
|
||||
residual does not yet reach the VR-001 dump, which is what VR-012 needs to run
|
||||
from fixtures; that is the next step, since it unblocks the study that sets
|
||||
every remaining behaviour.
|
||||
`sharpness` and `alignment_residual` are written to the VR-001 dump as per-face
|
||||
columns parallel to `confidence`, taking the dump to `schema_version` 2. The bump
|
||||
is not for readers — both sides check by name, and a v1 dump still replays — but
|
||||
so that a consumer of the vector can tell *never scored* from *scored zero*,
|
||||
which is a real reading on this axis. Nothing yet *consumes* any of it.
|
||||
|
||||
**Gap:** three, in the order they block each other.
|
||||
|
||||
1. **The fixtures do not carry the vector.** They are v1, and re-dumping needs a
|
||||
GPU host (`scripts/make_fixtures.sh`), so until that runs VR-012 has recorded
|
||||
data available in principle and none in hand.
|
||||
2. **AR-030's discount does not exist.** The measure must reach
|
||||
`EvidenceDiscounter` as the reliability term, multiplying the novelty weight
|
||||
rather than replacing it.
|
||||
3. **Two properties of the sharpness measure are recorded but unquantified on
|
||||
real faces**, and both distort the low end of the axis, which is where a knee
|
||||
would go. It is exactly contrast-invariant in the algebra, but the 8-bit
|
||||
quantisation floor lands in the numerator, so a crop that is *dim and soft*
|
||||
reads sharper than it is — on the synthetic ladder a half-contrast copy reads
|
||||
0.9% high when sharp and 148% high at σ 2.5. Separately, `align_face` warps
|
||||
with `BORDER_CONSTANT`, so a face crossing the frame edge brings a hard black
|
||||
step into the crop, and a step edge is high-frequency; the normalisation
|
||||
blunts this but does not remove it. Neither is corrected here. The candidate
|
||||
fixes are a validity mask or a different border mode, and the second changes
|
||||
what the embedder is fed (AR-011) — so VR-012 measures the size of each effect
|
||||
on the dumped distribution first, and no correction is chosen before that.
|
||||
|
||||
## AR-007, AR-008 — Tracking
|
||||
|
||||
@@ -1575,7 +1620,10 @@ Persist pipeline state at the point where the expensive work ends.
|
||||
per-frame index table (`face_offset`, `face_count`) pointing into them. Avoids
|
||||
variable-length HDF5 types and reads straight into numpy.
|
||||
- Stores per frame: `timestamp_sec`, `frame_idx`, `is_cut`, `is_scene_boundary`.
|
||||
Per face: `embedding` [N,512], `bbox` [N,4], `landmarks` [N,10], `confidence`.
|
||||
Per face: `embedding` [N,512], `bbox` [N,4], `landmarks` [N,10], `confidence`,
|
||||
and from v2 the AR-028 quality vector — `sharpness` [N] and
|
||||
`alignment_residual` [N]. Size, its third axis, is `bbox` and is not
|
||||
duplicated.
|
||||
- Invariants: embeddings unit-norm; `face_offset` contiguous; bboxes and
|
||||
landmarks in **decoded-frame** pixels with `bbox_upscale` recorded alongside
|
||||
(the dump is a faithful tap, so it does not transform what the tracker saw —
|
||||
@@ -1586,10 +1634,16 @@ Persist pipeline state at the point where the expensive work ends.
|
||||
Schema owned by [`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md).
|
||||
|
||||
**Current:** C++ dump sink (`embedding_dump_node.hpp`, `dump_embeddings.cpp`),
|
||||
read by `replay.py`. **Gap:** **AR-012 breaks the replay contract.** Track extents
|
||||
read by `replay.py`. At `schema_version` 2, which AR-028 took it to by adding the
|
||||
quality columns; readers on both sides check the datasets by name, so a v1 dump
|
||||
still replays and reports the vector as unknown rather than as zero.
|
||||
|
||||
**Gap:** **AR-012 breaks the replay contract.** Track extents
|
||||
are decided in the tracker, which is *downstream* of the dump — so a replay can
|
||||
reproduce them, but only if the dump preserves everything the tracker needs.
|
||||
Verify `landmarks`/`bbox`/`is_cut` suffice, and bump `schema_version` if not.
|
||||
The committed fixtures are still v1, so they carry no quality vector until
|
||||
`scripts/make_fixtures.sh` is re-run on a GPU host.
|
||||
|
||||
## VR-002 — Replay and sweep
|
||||
|
||||
@@ -1712,6 +1766,54 @@ round 1 seeding references that corrupt round 2.
|
||||
"expansion helps live matching" from "expansion helps the second pass", which the
|
||||
current all-or-nothing `expand_gallery` flag cannot distinguish.
|
||||
|
||||
## VR-015 — Per-node cost and bottleneck attribution
|
||||
|
||||
**Requirement: a run must be able to report where its time went, per node, and
|
||||
which node is setting the pace.** Without it, optimisation is guesswork, and
|
||||
worse than guesswork — the obvious number is wrong in a specific, repeatable
|
||||
direction, so acting on it makes the pipeline slower.
|
||||
|
||||
**Why the obvious number is wrong.** KPN times a node across `fire_once`, which
|
||||
wraps the functor *and* `push_outputs`. Under AR-004 a push parks on a full
|
||||
downstream channel, so a node that is merely waiting bills that wait to itself.
|
||||
On the SuperHero reference run (`docs/benchmark.md`) `frame_source` reported
|
||||
`ema=141.899ms` per frame while its own decoder logged 12-18 ms: it was
|
||||
backpressured, and the report named the *fastest* node in the graph as the most
|
||||
expensive one. A second trap sits behind the first — `ema_exec_ms` is an
|
||||
exponentially weighted average, so `frames × ema` is not a total; on a film whose
|
||||
per-frame cost swings between crowd scenes and landscapes the two differ
|
||||
substantially.
|
||||
|
||||
**Method.** Three measurements per node, none of which is sufficient alone:
|
||||
|
||||
| Measure | What it is | What it cannot tell you |
|
||||
|---|---|---|
|
||||
| `cpu_ms` | thread CPU time (`CLOCK_THREAD_CPUTIME_ID`) | GPU wait — a device-bound node looks idle |
|
||||
| `exec_ms` | cumulative wall time inside the node | work from waiting — backpressure inflates it |
|
||||
| `pressure` | mean input fill − mean output fill | how expensive the node is, only that it paces |
|
||||
|
||||
Queue occupancy has to be **sampled during the run**. `current_fill` is
|
||||
instantaneous and every channel has drained by shutdown, so a single read at the
|
||||
end describes an idle pipeline however congested it was.
|
||||
|
||||
**The number that matters** is `pressure`, because work piles up in front of the
|
||||
bottleneck and starves everything after it, and that ordering holds whether the
|
||||
node is waiting on a core, a GPU or a disk. `cpu_share` then selects the repair:
|
||||
a pacing node with a saturated thread is CPU-bound and the work must get cheaper,
|
||||
while a pacing node with an idle thread is device-bound, where batch size and
|
||||
engine precision are the knobs and the C++ is not.
|
||||
|
||||
**Current:** `--benchmark <path>` writes the JSON report and prints a table at
|
||||
shutdown; `src/benchmark.hpp`. Attribution is a pure function over KPN snapshots,
|
||||
so it is verified on CI's GPU-free N100 (UT-120…UT-124) rather than only by
|
||||
running the pipeline. The node graph is recovered from KPN's channel names, so a
|
||||
re-wired topology needs no change here. Required `NodeStats::total_exec_us` in
|
||||
the KPN submodule — the EMA could not be turned into a total.
|
||||
|
||||
**Gap:** GPU utilisation and memory are not sampled, so a device-bound verdict
|
||||
says *that* a node waits on the GPU, not whether the GPU is saturated or merely
|
||||
badly fed. That distinction needs NVML, and it is what VR-008 will want anyway.
|
||||
|
||||
## VR-008 — Gallery scaling benchmark
|
||||
|
||||
Establish the throughput-versus-gallery-size curve required by A10.
|
||||
|
||||
Reference in New Issue
Block a user