feat(scene): feed TransNetV2 at native rate, derive the dedup window from it
Closes both violations SPEC.md named under "Every model gets the input it was trained for". They are one bug, not two. The dense stream defaulted to 12 fps, so a 100-frame TransNetV2 window spanned ~8.3 s against the ~4 s it was trained on: half-speed motion over twice its temporal context. Boundary timestamps stayed correct throughout, which is exactly why the degradation was invisible and why the compressed separation it produced (~0.50 baseline against ~0.7+ peaks) was read as a property of the ONNX export rather than of the input. Dedup then merged boundaries closer than a literal 0.04 s — one frame at 25 fps, and wider than a frame at 30, so two cuts on consecutive frames became one. Nothing in scenes.json showed it; the file simply had fewer boundaries. Native rate is where that constant did the most damage, which is why fixing the decode rate without fixing the dedup would have made things worse. dedup_window_sec() now takes the median interval the detector was actually fed and halves it. Half a frame rather than a whole one: the only thing being merged is one frame scored by two overlapping windows, and two distinct frames are a full interval apart. Cost is real — dense decode is the pipeline's cost driver. It is accepted; dense_scale and scene_stride remain the reductions that do not run the model off-distribution. scene_threshold 0.60 was fitted against the 12 fps input and is now stale, so VR-006 goes from Low to Medium: it is no longer a refinement, it is a constant that no longer describes the input. AR-002 rides along because it was already implemented, just untagged and unverified — the register said Planned while the code was correct. The size filter becomes FaceDetectorFunc::drop_undersized(), tested at the threshold and at dense_scale 0.5, and checked end to end against the superhero dump, whose smallest face is exactly its recorded 32 px minimum, so the fixture check cannot pass vacuously. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: AR-002, AR-011 | SR-002 | UT-002, UT-003, IT-001
This commit is contained in:
+29
-16
@@ -438,7 +438,7 @@ Both feed AR-007 as **association hints**: they tell the tracker that spatial
|
||||
continuity is broken and that association should weight embedding over IoU.
|
||||
Neither ends a presence window (AR-012).
|
||||
|
||||
In dense mode the source decodes at `scene_decode_fps` (default 12) and a
|
||||
In dense mode the source decodes at `scene_decode_fps` (default 0 = native) and a
|
||||
decimator splits the stream: full-resolution sampled frames to the face pipeline,
|
||||
downscaled dense frames to the scene detector
|
||||
(`frame_source_node.hpp:63`). `sample_fps` is independent of this — the face
|
||||
@@ -459,31 +459,44 @@ degrading what a single inference sees. A model run off-distribution produces
|
||||
confident, plausible, wrong output, and the error is invisible without a study
|
||||
that should not have been necessary.
|
||||
|
||||
Two places this is currently violated:
|
||||
Two places this was violated, both now closed:
|
||||
|
||||
1. **`scene_decode_fps = 12` starves TransNetV2.** `kWindow` is 100 frames. At
|
||||
native 25 fps that window spans ~4 s; at 12 fps it spans ~8.3 s, so the model
|
||||
sees roughly half-speed motion over twice the temporal context it was trained
|
||||
on. **Requirement: feed TransNetV2 at the source's native frame rate**, so a
|
||||
100-frame window covers the duration the model expects. The
|
||||
"tolerates ~12fps" note in `config.hpp` describes a compromise, and the
|
||||
recorded margin is consistent with it — a non-boundary baseline at ~0.50 with
|
||||
1. **`scene_decode_fps = 12` starved TransNetV2.** `kWindow` is 100 frames. At
|
||||
native 25 fps that window spans ~4 s; at 12 fps it spanned ~8.3 s, so the
|
||||
model saw roughly half-speed motion over twice the temporal context it was
|
||||
trained on. **Requirement: feed TransNetV2 at the source's native frame
|
||||
rate**, so a 100-frame window covers the duration the model expects. The
|
||||
"tolerates ~12fps" note in `config.hpp` described a compromise, and the
|
||||
recorded margin was consistent with it — a non-boundary baseline at ~0.50 with
|
||||
real boundaries reaching only ~0.7+ is a compressed separation, not a healthy
|
||||
one.
|
||||
one. **Done:** `scene_decode_fps` defaults to 0.
|
||||
|
||||
2. **Hardcoded 25 fps in boundary dedup.** `scene_detector_node.hpp:138` merges
|
||||
boundaries closer than `0.04 s` — "~1 frame @25fps". **Requirement: derive
|
||||
this from the source's actual frame rate.**
|
||||
2. **Hardcoded 25 fps in boundary dedup.** The node merged boundaries closer than
|
||||
`0.04 s` — "~1 frame @25fps". **Requirement: derive this from the source's
|
||||
actual frame rate. Done:** `SceneDetectorFunc::dedup_window_sec()` takes the
|
||||
median of the frame intervals the detector was actually fed and halves it.
|
||||
Half a frame rather than a whole one, because the only thing being merged is
|
||||
one frame scored by two overlapping windows; two distinct frames are a full
|
||||
interval apart and both have to survive.
|
||||
|
||||
The two are one change, not two. A native-rate stream is where the old constant
|
||||
did the most damage — at 30 fps, 0.04 s is wider than a frame, so two cuts on
|
||||
consecutive frames merged into one and the loss showed up nowhere: the file
|
||||
simply had fewer boundaries.
|
||||
|
||||
Dense decode is the pipeline's cost driver, so (1) is not free. The cost is
|
||||
accepted: the alternative is a boundary signal that steers association (AR-007) while
|
||||
being quietly unreliable. `dense_scale` remains available as a spatial reduction,
|
||||
since downscaling is a documented, understood degradation rather than a temporal
|
||||
one the model has no defence against.
|
||||
one the model has no defence against — and TransNetV2 downsamples to 48×27
|
||||
regardless.
|
||||
|
||||
**Current:** histogram cut in the decoder; `scene_detector_node.hpp` for
|
||||
TransNetV2. **Gap:** native-rate dense decode; framerate-derived dedup;
|
||||
`--scene-detect` is default-off despite now feeding association.
|
||||
TransNetV2, fed at native rate with a framerate-derived dedup window.
|
||||
**Gap:** `scene_threshold` (0.60) is still the value picked against 12 fps input
|
||||
and is now certainly wrong — VR-006 re-fits it, and until it does, boundary
|
||||
recall at native rate is untuned rather than better. `--scene-detect` is
|
||||
default-off despite now feeding association.
|
||||
|
||||
## AR-012 … AR-017 — Track-level identity propagation — **CHANGED BEHAVIOUR**
|
||||
|
||||
|
||||
@@ -29,7 +29,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
| ID | Requirement | Traces to | Priority | Status |
|
||||
|---|---|---|---|---|
|
||||
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
|
||||
| AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
|
||||
| AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | **Done** — `FaceDetectorFunc::drop_undersized()`. The threshold is divided by `bbox_upscale` rather than every box multiplied, which keeps the comparison on the detector's own numbers and means turning `dense_scale` on cannot silently raise the minimum face the pipeline accepts. Verified at the threshold and at `dense_scale` 0.5 (UT-002), and end to end on the fixture (IT-001) — the superhero dump's smallest side is *exactly* its recorded 32 px, so the filter is binding there rather than vacuously satisfied |
|
||||
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing |
|
||||
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Mostly** — node outputs *park* on a full channel: the value is held, the worker released, and a channel space-callback resumes the node. Replaces `push_blocking`, which parked a scheduler worker inside the push and, with one thread per node, stopped that node draining its own input. Verified: 385/385 frames, 0 drops. **Gap:** a rare hang survives, ~1 run in 20 at a 300 s timeout (was: every run). `FanoutNode` still drops on overflow (`fanout.hpp:129`) rather than parking, so the AR-010 scene join sheds frames exactly when the dense branch falls behind |
|
||||
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far |
|
||||
@@ -38,7 +38,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted |
|
||||
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
|
||||
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Done** — decode butterfly joined via `SceneBoundaries`; the sampled branch waits for the detector's watermark. Frames past its last scored window are counted as unverified, never assumed boundary-free |
|
||||
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
|
||||
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | **Done** — both violations SPEC.md named are closed. (1) `scene_decode_fps` defaults to 0 (native): at 12 fps a 100-frame `kWindow` spanned ~8.3 s instead of the ~4 s TransNetV2 was trained on, half-speed motion over twice its temporal context. (2) The boundary dedup window is derived from the cadence the detector was actually fed (`SceneDetectorFunc::dedup_window_sec()`, median observed interval, halved) rather than the literal 0.04 s — one frame at 25 fps, and at 30 fps wider than a frame, so two cuts on consecutive frames merged into one and the loss was invisible: the file simply had fewer boundaries. Derivation checked at 24/25/30 fps and under a seek (UT-003). **Consequence, not a gap:** `scene_threshold` 0.60 was fitted against the 12 fps input and is now certainly wrong — VR-006 re-fits it, and until then boundary recall at native rate is untuned rather than better. Dense decode is the cost driver, so this is not free; `dense_scale` and `scene_stride` remain the reductions that do not run the model off-distribution |
|
||||
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | **Done** — `src/track_registry.hpp`; window is `[first_seen, last_seen]` of an owned track |
|
||||
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | **Done** — `last_seen` optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed |
|
||||
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | **Done** — swap closes at `last_seen` and opens a successor at the swap frame; counted |
|
||||
@@ -108,7 +108,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done |
|
||||
| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done |
|
||||
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | **Done** — knee at 24–32 px; 32 px gives 98.1% TPI, 0.0 FPI at every size. Degrades an already-aligned 112×112 crop, so it isolates the embedder and is an **upper bound**; VR-013 measures the same question end to end and AR-002 takes its number, not this one |
|
||||
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned |
|
||||
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | **Medium** | **Planned, now unblocked** — native-rate decode landed with AR-011, so the prerequisite is met and the current 0.60 is a value fitted against input the pipeline no longer produces. Raised from Low for that reason: it is no longer a refinement, it is a stale constant |
|
||||
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
|
||||
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
|
||||
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
|
||||
|
||||
Reference in New Issue
Block a user