diff --git a/docs/SPEC.md b/docs/SPEC.md index 4e332d5..efbb284 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -128,10 +128,43 @@ Two consequences worth stating: depends on timing. The same command run twice can produce different dumps, and a golden fixture cannot be built on that. -**Current:** fixed in KPN — node data outputs use `push_blocking`, sentinels -remain out-of-band so EOF can always overtake a stalled data path. Verified on -the same clip: 385 of 385 sampled frames written, zero drops, and two -consecutive runs byte-identical where previously they were not. +**Current:** fixed in KPN. Node data outputs *park* on a full channel — the +value is held in a one-slot buffer, the worker is released, and the channel's +space callback resubmits the node once the consumer drains. That replaced +`push_blocking`, which slept inside the push and, with one thread per node, +stopped that node draining its own input. Sentinels remain out-of-band so EOF +can always overtake a stalled data path. Verified on the same clip: 385 of 385 +sampled frames written, zero drops, and two consecutive runs byte-identical +where previously they were not. + +A later audit found the losslessness was still incomplete in three places, all +now closed and each pinned by a regression case in the KPN suite: + +- **`FilterNode` and `RouterNode`** were the last data paths still using the + throwing `push()` with the exception swallowed. A full output discarded the + value, and that included the **EOF sentinel**. The decimator passes EOF by + predicate but its output is reliably full — the embedder is the slowest node + in the chain — so the token was discarded, nothing downstream shut down, and + the run had to be killed. This was the wedge. +- **The sentinel could arrive ahead of a value still queued behind it.** `pop()` + observed the ring empty and then took the sentinel; a producer can push a + value *and* publish the sentinel inside that window, so a consumer treating + EOF as a hard stop loses the tail. +- **Two firings of one node could overlap**, because the submit gate was + released before the firing had finished with the node's state. That breaks the + one-slot park itself: a parked value can be overwritten by the other firing, + with no drop recorded anywhere. + +**New constraint:** a channel carries at most one undelivered sentinel. A second +offered before the first is taken is refused and reported, never queued and +never overwritten — two control tokens on one channel means the stream ended +twice. Single-shot EOF is what everything does today; this becomes live the +moment a pipeline is reused for a second input. + +**Consequence:** a lossless decimator is a backpressure point, not a relief +valve. The source now throttles to the face branch rather than quietly thinning +it. That is what this requirement asks for, but it changes the shape of a loaded +run and has not yet been benchmarked. It also ran *faster* (29 s → 17 s). A dropped frame has already cost its decode, and the overflow exception cost more — so the lossy path was paying for work it diff --git a/docs/requirements.md b/docs/requirements.md index 0314869..fbf9c46 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -31,7 +31,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done | | AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | **Done** — `FaceDetectorFunc::drop_undersized()`. The threshold is divided by `bbox_upscale` rather than every box multiplied, which keeps the comparison on the detector's own numbers and means turning `dense_scale` on cannot silently raise the minimum face the pipeline accepts. Verified at the threshold and at `dense_scale` 0.5 (UT-002), and end to end on the fixture (IT-001) — the superhero dump's smallest side is *exactly* its recorded 32 px, so the filter is binding there rather than vacuously satisfied | | AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing | -| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Mostly** — node outputs *park* on a full channel: the value is held, the worker released, and a channel space-callback resumes the node. Replaces `push_blocking`, which parked a scheduler worker inside the push and, with one thread per node, stopped that node draining its own input. Verified: 385/385 frames, 0 drops. Two remaining holes now closed: **(a)** `FanoutNode` dropped on overflow rather than waiting, so the AR-010 scene join shed frames exactly when the dense branch fell behind — measured at **9 of 2192 items delivered** to the slower of two branches, now lossless with the fast branch throttled to within its buffering; **(b)** the residual hang, recorded as ~1 run in 20 at a 300 s timeout, was a **startup** lost wake, not a mid-stream one — `start()` enables a node's inputs several statements before it installs the push callback, and a producer firing into that gap is accepted by the ring while waking nobody, since `Channel::push` signals only the empty→non-empty edge. Signature is zero items delivered, never a partial stall. Reproduced 7 times in 24 under CPU contention and 0 in 10 without; `start()` now closes with the level-triggered `on_input_ready()`, giving 0 in 24 on the same harness. **Consequence to hold onto:** a lossless fanout makes join depth a correctness precondition — one branch can now run ahead of another only by the slower branch's buffering, so `kSceneJoinDepth` must exceed the TransNetV2 window. **Gap:** capacity is still counted in *items*, not bytes, so a crowd frame carrying 60 crops occupies one slot exactly as an empty one does — the memory ceiling the plan asks for is unenforced | +| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Mostly** — node outputs *park* on a full channel: the value is held, the worker released, and a channel space-callback resumes the node. Replaces `push_blocking`, which parked a scheduler worker inside the push and, with one thread per node, stopped that node draining its own input. Verified: 385/385 frames, 0 drops. Holes closed since, in the order they surfaced: **(a)** `FanoutNode` dropped on overflow rather than waiting, so the AR-010 scene join shed frames exactly when the dense branch fell behind — measured at **9 of 2192 items delivered** to the slower of two branches, now lossless with the fast branch throttled to within its buffering; **(b)** the residual hang, recorded as ~1 run in 20 at a 300 s timeout, was a **startup** lost wake, not a mid-stream one — `start()` enables a node's inputs several statements before it installs the push callback, and a producer firing into that gap is accepted by the ring while waking nobody, since `Channel::push` signals only the empty→non-empty edge. Signature is zero items delivered, never a partial stall. Reproduced 7 times in 24 under CPU contention and 0 in 10 without; `start()` now closes with the level-triggered `on_input_ready()`, giving 0 in 24 on the same harness — though the *cause* was narrower than recorded there and is fixed properly in **(e)**; **(c)** `FilterNode` and `RouterNode` were the last data paths still using the throwing `push()` with the exception swallowed, so a full output discarded the value — including the **EOF sentinel**. The decimator passes EOF by predicate (`if (f.eof) return true;`) but its output is reliably full, the embedder being the slowest node in the chain, so the token was discarded, nothing downstream ever shut down, and the run had to be killed. **This is the wedge.** Both now route sentinels out-of-band and retry data until taken; the regression case delivers 6 of 40 values and never sets `saw_eof` before, 40 and terminating after; **(d)** the sentinel could be delivered *ahead of* a value still queued behind it — `pop()` observed the ring empty and then took the sentinel, and a producer can push a value *and* publish the sentinel inside that window, so any consumer treating EOF as a hard stop loses the tail. `take_sentinel` now re-checks emptiness *after* observing `has_eof_`, which is sound because the sentinel is published with a release store after the ring pushes. ~1 run in 15 before, 0 in 25 after; **(e)** two `fire_once` invocations for one node could overlap, because the submit gate was released before the firing had finished touching node state. That breaks the one-slot park the whole scheme rests on — a parked value can be overwritten by the other firing, with no drop recorded anywhere. ThreadSanitizer caught it as a race on `pending_done_`; the release is now the last act of a firing. The same sweep found the callbacks themselves being written while a running neighbour read them (ten TSan races), which is the *actual* cause of the startup lost wake in **(b)** — callbacks are now installed in a `prepare()` pass before any node starts. **New constraint:** a channel carries at most **one undelivered sentinel**; a second offered before the first is taken is refused and reported, never queued and never overwritten, since two control tokens on one channel means the stream ended twice. Single-shot EOF today, live the moment a pipeline is reused for a second input. **Consequence to hold onto:** a lossless fanout makes join depth a correctness precondition — one branch can now run ahead of another only by the slower branch's buffering, so `kSceneJoinDepth` must exceed the TransNetV2 window. Making the decimator lossless also makes it a backpressure point rather than a relief valve: the source now throttles to the face branch instead of quietly thinning it. Correct under this requirement, but it changes the shape of a loaded run and is **not yet benchmarked**. **Gap:** capacity is still counted in *items*, not bytes, so a crowd frame carrying 60 crops occupies one slot exactly as an empty one does — the memory ceiling the plan asks for is unenforced | | AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far | | AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done | | AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks | @@ -314,7 +314,7 @@ because it will be trusted. | AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only | | AR-002 | T2 | Faces below 40 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement | | AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame | -| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block. Two cases the KPN suite now pins, both of which failed before being written: a fanout feeding an unequal pair loses nothing *and* throttles the fast branch (either assertion alone passes on a broken implementation); and a node started with data already in its input still fires — the startup lost wake, which needs no contention to reproduce once the state is constructed directly | +| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block. Cases the KPN suite now pins, each of which failed before being written: a fanout feeding an unequal pair loses nothing *and* throttles the fast branch (either assertion alone passes on a broken implementation); a filter delivers EOF into a saturated output; a sentinel is never delivered ahead of a queued value; a twice-parked value keeps its payload; and a node started with data already in its input still fires — the startup lost wake, which needs no contention to reproduce once the state is constructed directly | | AR-005 | T1 | Known landmarks → expected 112×112 warp; the fit never mirrors | Landmarks near frame edge; degenerate/collinear points; a mirrored set — SVD returns a reflection unless the determinant guard rejects it | | AR-006 | T3 | Embeddings are unit-norm | Batch smaller than, equal to, larger than `embed_batch_size` | | AR-007 | T2 | Association picks the right track | Two faces crossing paths; one leaving frame as another enters |