890f1946bf85a16e7b358d6651f121d484d99b6b
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  dtourolleandClaude Opus 5
|
a667caa313 |
feat: provenance attributes on the embedding dump
VR-010 — a dump made with one detector/embedder pair was byte-indistinguishable from one made with another, except for the two attributes GR-004 added. Replayed against a gallery from a different model, cosine similarities are meaningless but look entirely plausible. The register states the principle directly: a fixture whose provenance is unknown is worse than no fixture, because it will be trusted. Sixteen attributes now record everything that determines the dump's content: detector model and thresholds, min_face_px, max_faces, cut_threshold, dense_scale, bbox_upscale, start/end, track_assoc_min_prob, and scene_detect. scene_detect is the one that matters most. is_scene_boundary is all-zero both when the detector found nothing and when it never ran, and those mean completely different things to a consumer — without the flag they are indistinguishable. No schema_version bump: new root attributes are additive and replay.py already reads attributes with a default, so older dumps stay readable and the committed fixtures — which predate this — still load. Also corrects SCHEMA.md, which claimed bbox was already mapped to original resolution at dump time. It is not; the upscale is applied downstream in the matcher, after the dump tap. Harmless while dense_scale is 1 and silently wrong otherwise, so bbox_upscale is now recorded and the doc says what the code does. Verified end to end: all sixteen attributes present and correct on a freshly generated dump. Suite: 92 cases, 6136 assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: VR-010, VR-001 | PR-002 |
||
|
dtourolleandClaude Opus 5
|
b35d49c772 |
docs: tag the implemented core with its requirement IDs
Adds TRACES tags to code that already satisfies a Done requirement, so coverage reflects what exists rather than starting from zero: AR-001 face detection, AR-005 ArcFace alignment, AR-023 calibration fit, DP-001/DP-002 the single analysis core behind the CLI, IR-001 truth-file emission, IR-006 the Jellyfin round trip, GR-001/GR-002 gallery build and incremental merge, VR-001 the embedding dump, VR-002 replay through the real nodes, VR-003 per-second scoring. Only Done requirements are tagged. A tag on Planned work would inflate coverage with fiction that looks plausible — the same failure family as a gate that cannot fail, and harder to spot. GR-005 (gallery never leaves the instance) stays untagged deliberately: it is a prohibition satisfied by the absence of an egress path, so there is no unit that decides it. Same shape as PR-005 in the system spec, which has no software row for the same reason. A goal held only by prohibitions cannot be verified by pointing at code. Coverage 5/63 to 14/63. The three VR tags are reported as tagged-but-unexecuted and excluded from the numerator, since their tier cannot run on the CI host — tagging deliberately cannot raise the number on its own. Suite still 64 cases, 3199 assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: AR-001, AR-005, AR-023, DP-001, DP-002, IR-001, IR-006, GR-001, GR-002, VR-001, VR-002, VR-003 |
||
|
Claude
|
7db40f430d |
GR-004: bind galleries to the embedder that built them
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.
The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.
Mismatch is a hard error in every mode, with no bypass, naming both sides.
Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.
Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.
Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.
Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.
Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
dtourolle
|
0bd2747069 |
docs: full data-grounded rewrite of the performance report
Replaces narrative claims with verified numbers across all report pages: - Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out films): LVFace wins every film outright, not just "consistent with" the training-set pick. r50 dropped from the detailed comparison (gallery has ~30% fewer reference images per actor than the other three models on identical source photos). - Per-film training breakdown: LVFace does not win every training film (mbf beats it on Lord of War); the 75.3% macro figure hides a 10.7pp spread. - Gallery coverage computed per film (20.3%-78.6%) instead of one flat 67%-missing average. - Found and fixed a real scoring bug in optimize.py: a candidate whose hardest film's replay timed out was averaged over survivors instead of penalized, silently rewarding partial coverage. Affected 3 of 16 training combos; corrected throughout, and optimize.py now scores an incomplete evaluation f1=0.0 instead of averaging over whichever films happened to finish. - Every FPI frame in the deep dive now comes from the proper montage renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes), never the bare-box debug overlay used earlier. - Every distinct out-of-cast name across all 9 films gets its own frame at its first appearance (9 names, 4 films), not a single-example spot check: 2 ground-truth gaps, 1 photograph misread as a person, 6 genuine lookalike confusions. - New methodology.md: the scene-level-vs-per-second scoring mismatch that the rest of the report assumes, written out once. - Cut the deadlock/gdb debugging narrative from the experiment log; kept the one fact that matters (KPN's node/network split lets the expensive GPU stage run once and the cheap stage replay against cached embeddings). - Plain declarative style throughout, no em dashes, no blog voice. |
||
|
dtourolle
|
b1efefac6f |
docs: richer report — data figures, success/failure frames, commit-pinned repo links
- experiment_charts.py generates 4 figures from experiments/ artifacts: held-out per-film F1, 16-combo ranking, DE search landscape, and the Downton detector-vs-tracker ghost timeline (replaces the blank title-card screenshot) - new frames: 19-correct wedding shot (success case), Many Saints ghost-vs-unknown frame (three error classes in one image) - rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only as the on-disk artifact prefix, explained once - repo file references are now links via https://REPOLINK/<path> placeholders; build_site.sh pins them to the HEAD commit's raw URLs and fails the build if a linked path doesn't exist at HEAD - drop references to removed scripts (scene_score.py, score_config.py) and to session-memory names; mark artifact-registry paths with their pull commands - commit readme_example.jpg + pipeline_topology.svg so README renders on the plain Gitea repo view - deploy_pages.sh: push built site/ to the gitea-pages branch |
||
|
dtourolle
|
6f0ad83a55 |
feat(tooling): X-Ray threshold optimizer, gallery utilities, artifact registry, docs build
Optimizer (scripts/optimizer/): replay.py runs the real C++ tracker/matcher/ scene_tracker chain over a dumped-embeddings HDF5 via sae_kpn, so a threshold sweep never re-decodes video or re-embeds faces. optimize.py drives scipy's differential_evolution over the knob space, with DE-level parallelism (multiple population candidates evaluated concurrently via a ThreadPoolExecutor) on top of per-film replay parallelism. second_score.py is the per-second X-Ray scoring metric (TPI/FPI/FN, out-of-cast misID weighted 10x, fair recall masked to gallery-known cast) that superseded an earlier scene-union metric. dump_error_frames.py / dump_scene_montage.py extract annotated video frames (bounding boxes, TPI/FPI/FN captions, onscreen-vs-offscreen split) for visual review of a replay against ground truth. Gallery utilities: cast_restrict.py, gallery_membership.py, fetch_missing_actors.py, reembed_gallery.py. scripts/validation/: X-Ray ground-truth loading and provider-agnostic identity matching (identity.py's keys_for — an actor is the union of every id we can derive, since pipeline output and ground truth don't share one id space). scripts/artifacts/: push/pull scripts for the Gitea generic package registry — galleries, montage frames, and experiment data (manifests/trajectories/results) are pushed there instead of committed, since none are needed to run the app, only benchmarks. Versioned by git short-SHA. scripts/docs/: MkDocs site build (build_site.sh) and the calibration-curve comparison chart (calibration_chart.py, matplotlib, reads each gallery's embedded calibration). Gallery-building scripts (make_jellyfin_gallery.py, make_gallery.py, filter_gallery.py, run_from_jellyfin.py, movienet_eval.py, movienet_prep.py, sae_gallery.py) updated to read/write HDF5 galleries exclusively, matching the engine-side format switch. run_from_jellyfin.py and the optimizer no longer carry movie source paths in shared manifests (some source filenames include scene-release tags) — resolved locally via a gitignored file-lut.json instead. |