dtourolle
6f0ad83a55
Optimizer (scripts/optimizer/): replay.py runs the real C++ tracker/matcher/ scene_tracker chain over a dumped-embeddings HDF5 via sae_kpn, so a threshold sweep never re-decodes video or re-embeds faces. optimize.py drives scipy's differential_evolution over the knob space, with DE-level parallelism (multiple population candidates evaluated concurrently via a ThreadPoolExecutor) on top of per-film replay parallelism. second_score.py is the per-second X-Ray scoring metric (TPI/FPI/FN, out-of-cast misID weighted 10x, fair recall masked to gallery-known cast) that superseded an earlier scene-union metric. dump_error_frames.py / dump_scene_montage.py extract annotated video frames (bounding boxes, TPI/FPI/FN captions, onscreen-vs-offscreen split) for visual review of a replay against ground truth. Gallery utilities: cast_restrict.py, gallery_membership.py, fetch_missing_actors.py, reembed_gallery.py. scripts/validation/: X-Ray ground-truth loading and provider-agnostic identity matching (identity.py's keys_for — an actor is the union of every id we can derive, since pipeline output and ground truth don't share one id space). scripts/artifacts/: push/pull scripts for the Gitea generic package registry — galleries, montage frames, and experiment data (manifests/trajectories/results) are pushed there instead of committed, since none are needed to run the app, only benchmarks. Versioned by git short-SHA. scripts/docs/: MkDocs site build (build_site.sh) and the calibration-curve comparison chart (calibration_chart.py, matplotlib, reads each gallery's embedded calibration). Gallery-building scripts (make_jellyfin_gallery.py, make_gallery.py, filter_gallery.py, run_from_jellyfin.py, movienet_eval.py, movienet_prep.py, sae_gallery.py) updated to read/write HDF5 galleries exclusively, matching the engine-side format switch. run_from_jellyfin.py and the optimizer no longer carry movie source paths in shared manifests (some source filenames include scene-release tags) — resolved locally via a gitignored file-lut.json instead.
51 lines
2.3 KiB
Markdown
51 lines
2.3 KiB
Markdown
# Embedding-dump HDF5 schema (v1)
|
|
|
|
One file per analysed title. Captures the pipeline state at the `EmbeddedSceneFrame`
|
|
channel — i.e. after decode → detect → align → embed, but **before** tracking and
|
|
identity matching. Everything downstream (face tracker, identity matcher, scene
|
|
tracker/anneal) is cheap CPU math, so replaying from this file lets a parameter
|
|
sweep re-run the whole downstream tail thousands of times with no GPU and no video.
|
|
|
|
Written by the C++ dump sink (`--dump-embeddings out.h5`); read by
|
|
`scripts/optimizer/replay.py`.
|
|
|
|
## Layout
|
|
|
|
The dump is **flat/ragged**: all faces across all frames are concatenated into
|
|
per-face arrays, with a per-frame index table pointing into them. This avoids
|
|
variable-length HDF5 types and reads straight into numpy.
|
|
|
|
```
|
|
/ (root)
|
|
attrs:
|
|
schema_version : int = 1
|
|
movie : str (source video path)
|
|
sample_fps : float
|
|
embed_dim : int = 512
|
|
|
|
frames/ group — one row per sampled frame
|
|
timestamp_sec : float64 [F]
|
|
frame_idx : int64 [F]
|
|
is_cut : uint8 [F] (histogram intra-scene cut)
|
|
is_scene_boundary : uint8 [F] (TransNetV2 boundary; 0 if scene_detect off)
|
|
face_offset : int64 [F] start index into faces/* for this frame
|
|
face_count : int32 [F] number of faces in this frame
|
|
|
|
faces/ group — one row per detected face, concatenated
|
|
embedding : float32 [N, 512] L2-normalised ArcFace embedding
|
|
bbox : float32 [N, 4] x, y, w, h in original video pixels
|
|
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order
|
|
confidence : float32 [N] detector confidence
|
|
```
|
|
|
|
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
|
|
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.
|
|
|
|
## Invariants
|
|
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
|
|
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
|
|
- `bbox` is already mapped to original resolution (bbox_upscale applied at dump time),
|
|
matching what the identity matcher would emit.
|
|
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
|
|
- EOF sentinel frames are NOT written.
|