Files

T

dtourolle 6f0ad83a55 feat(tooling): X-Ray threshold optimizer, gallery utilities, artifact registry, docs build

Optimizer (scripts/optimizer/): replay.py runs the real C++ tracker/matcher/
scene_tracker chain over a dumped-embeddings HDF5 via sae_kpn, so a threshold
sweep never re-decodes video or re-embeds faces. optimize.py drives scipy's
differential_evolution over the knob space, with DE-level parallelism
(multiple population candidates evaluated concurrently via a ThreadPoolExecutor)
on top of per-film replay parallelism. second_score.py is the per-second X-Ray
scoring metric (TPI/FPI/FN, out-of-cast misID weighted 10x, fair recall masked
to gallery-known cast) that superseded an earlier scene-union metric.
dump_error_frames.py / dump_scene_montage.py extract annotated video frames
(bounding boxes, TPI/FPI/FN captions, onscreen-vs-offscreen split) for visual
review of a replay against ground truth. Gallery utilities: cast_restrict.py,
gallery_membership.py, fetch_missing_actors.py, reembed_gallery.py.

scripts/validation/: X-Ray ground-truth loading and provider-agnostic identity
matching (identity.py's keys_for — an actor is the union of every id we can
derive, since pipeline output and ground truth don't share one id space).

scripts/artifacts/: push/pull scripts for the Gitea generic package registry —
galleries, montage frames, and experiment data (manifests/trajectories/results)
are pushed there instead of committed, since none are needed to run the app,
only benchmarks. Versioned by git short-SHA.

scripts/docs/: MkDocs site build (build_site.sh) and the calibration-curve
comparison chart (calibration_chart.py, matplotlib, reads each gallery's
embedded calibration).

Gallery-building scripts (make_jellyfin_gallery.py, make_gallery.py,
filter_gallery.py, run_from_jellyfin.py, movienet_eval.py, movienet_prep.py,
sae_gallery.py) updated to read/write HDF5 galleries exclusively, matching the
engine-side format switch. run_from_jellyfin.py and the optimizer no longer
carry movie source paths in shared manifests (some source filenames include
scene-release tags) — resolved locally via a gitignored file-lut.json instead.

2026-07-19 19:06:48 +02:00

2.3 KiB

Raw Blame History

Embedding-dump HDF5 schema (v1)

One file per analysed title. Captures the pipeline state at the EmbeddedSceneFrame channel — i.e. after decode → detect → align → embed, but before tracking and identity matching. Everything downstream (face tracker, identity matcher, scene tracker/anneal) is cheap CPU math, so replaying from this file lets a parameter sweep re-run the whole downstream tail thousands of times with no GPU and no video.

Written by the C++ dump sink (--dump-embeddings out.h5); read by scripts/optimizer/replay.py.

Layout

The dump is flat/ragged: all faces across all frames are concatenated into per-face arrays, with a per-frame index table pointing into them. This avoids variable-length HDF5 types and reads straight into numpy.

/                                     (root)
  attrs:
    schema_version : int   = 1
    movie          : str   (source video path)
    sample_fps     : float
    embed_dim      : int   = 512

  frames/                             group — one row per sampled frame
    timestamp_sec  : float64 [F]
    frame_idx      : int64   [F]
    is_cut         : uint8   [F]      (histogram intra-scene cut)
    is_scene_boundary : uint8 [F]     (TransNetV2 boundary; 0 if scene_detect off)
    face_offset    : int64   [F]      start index into faces/* for this frame
    face_count     : int32   [F]      number of faces in this frame

  faces/                              group — one row per detected face, concatenated
    embedding      : float32 [N, 512] L2-normalised ArcFace embedding
    bbox           : float32 [N, 4]   x, y, w, h in original video pixels
    landmarks      : float32 [N, 10]  5 (x,y) pairs, SCRFD/ArcFace order
    confidence     : float32 [N]      detector confidence

F = number of sampled frames, N = total faces (= sum of face_count). Frame i's faces are faces/*[ face_offset[i] : face_offset[i]+face_count[i] ].

Invariants

embedding rows are unit-norm (cosine == dot product against the gallery).
face_offset[0] == 0; face_offset[i+1] == face_offset[i] + face_count[i].
bbox is already mapped to original resolution (bbox_upscale applied at dump time), matching what the identity matcher would emit.
A frame with no faces has face_count == 0 (still gets a row, so timestamps stay dense).
EOF sentinel frames are NOT written.

2.3 KiB Raw Blame History

Embedding-dump HDF5 schema (v1)

Layout

Invariants

2.3 KiB

Raw Blame History