dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f891e579c5f1286aa11a5612c741dfb9db808865
scene-actor-extraction/src/gallery/gallery_report.hpp
T
dtourolleandClaude Opus 5 1dfd6fea11 feat: gallery build report 
GR-003 — the calibration fit already computed per-actor dedup counts, how many
actors are eligible for positive pairs, and a 200-bin histogram of the intra and
inter distributions, then discarded all of it to stderr. Nothing persisted, so
nobody could audit whether a gallery was any good.

The report is written alongside the gallery at build time. That is the right
moment: the matcher fits the same sigmoid at analysis time, but by then the
answer is per-run and nobody is looking, whereas build time is when a gallery's
quality is actually decided.

What it surfaces, in order of usefulness:
- actors with no usable image — a silent recall ceiling, since the pipeline can
  never name them and nothing else says why
- actors below the positive-pair threshold — not broken, so nothing complains;
  they just quietly weaken every threshold downstream
- near-duplicate references removed, per actor and total
- the fitted calibration AND the two distributions behind it

That last one is the point. Every threshold in the pipeline is expressed in the
probability space this sigmoid defines, so if the distributions overlap heavily
the calibration is weak and every downstream decision inherits it — while the
gallery still looks fine from the outside.

The gallery-derived prior, intra/(intra+inter), is computed and reported but the
shipped default of 0.5 is deliberately left alone. The spec records these as
disagreeing; now the real value is visible, so the decision can be made on
evidence rather than argument.

Three tests: a zero-image actor is visible in the report, an under-referenced
actor is counted, and the report round-trips through JSON.

Suite: 95 cases, 6142 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

TRACES: GR-003 | SR-001
2026-07-31 15:30:46 +02:00

496 lines
23 KiB
C++
Raw Blame History

#pragma once
/// TRACES: GR-003 | SR-001
///
/// The gallery build report — what the gallery *is*, written next to it.
///
/// A gallery is a silent artefact: it loads, it scores, it never complains. The
/// two ways it fails are both invisible from the outside.
///
/// 1. **An actor with zero usable images can never be recognised.** They are
/// dropped at build time (`gallery_builder.cpp` skips a directory whose
/// images all fail detection or alignment), so afterwards nothing in the
/// file records that they were ever meant to be there. Every scene they
/// appear in is a guaranteed miss, and recall is capped at a number nobody
/// computed. This is the single most useful line in the report.
/// 2. **A gallery can be quietly bad and look fine.** The Platt sigmoid
/// (AR-023) is fitted from two distributions — intra-class (same actor,
/// different reference) and inter-class (different actors) similarity — and
/// *every* threshold in the pipeline is expressed in the probability space
/// that fit defines (AR-024): identity acceptance, track association,
/// expansion admission, cluster merging. If those two distributions overlap
/// heavily the fit is weak, and every downstream decision silently inherits
/// that weakness while still reporting confident-looking probabilities. The
/// fit already computes the distributions and throws them away; emitting
/// them is what makes the quality of the whole probability space auditable
/// instead of assumed.
///
/// The report is therefore a build artefact, not a debug aid: it is the only
/// place the recall ceiling and the calibration's conditioning are written down.
///
/// **On the histograms being in cosine space.** They bin raw similarity, and
/// that is not an AR-024 violation: no decision is taken here. These two
/// distributions are the *input* the calibration is fitted from — they cannot be
/// expressed in the probability space the calibration defines, because that
/// space is their output. GR-003 asks for exactly this ("the intra/inter
/// distributions behind it"), for the same reason GR-008 characterises an
/// actor's reference spread in the metric space: shape is a property of the
/// metric, decisions are a property of the probability.
#include "gallery/gallery_calibration.hpp"
#include "gallery/embedder_stamp.hpp"
#include "types.hpp"
#include <nlohmann/json.hpp>
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <fstream>
#include <stdexcept>
#include <string>
#include <vector>
/// TRACES: GR-003 | SR-001
///
/// Per-actor image accounting from the build pass, including the actors that
/// produced nothing and were therefore dropped from the gallery.
///
/// Filled by `build_gallery()`. It has to be collected there and cannot be
/// recovered later: by the time a gallery exists, an actor with no usable image
/// is indistinguishable from an actor who was never requested.
struct GalleryBuildAudit {
struct ActorImages {
std::string imdb_id;
std::string name;
int images_seen = 0; // candidate image files in the actor's directory
int images_used = 0; // ...that yielded an embedding
int unreadable = 0; // cv::imread failed
int no_face = 0; // detector found nothing
int align_failed = 0; // 5-point warp failed
};
std::vector<ActorImages> actors; // every directory seen, in build order
};
/// TRACES: GR-003 | SR-001
struct GalleryReport {
// One row per actor the build considered. Actors with references == 0 are
// the zero-usable-image case: present in the source tree, absent from the
// gallery, unrecognisable for the life of the file.
struct Actor {
std::string imdb_id;
std::string name;
int images_seen = -1; // -1 = unknown (report built without a build audit)
int references = 0; // embeddings stored in the gallery
int distinct_references = 0; // ...after near-duplicate removal
int duplicates_removed = 0;
bool eligible_for_positive_pairs = false;
};
// The two distributions the sigmoid is fitted from, as the fit itself saw
// them: counts per similarity bin over [sim_min, sim_max].
struct Distributions {
int bins = 0;
float sim_min = -1.f;
float sim_max = 1.f;
std::vector<double> intra; // same actor, different reference image
std::vector<double> inter; // different actors
double intra_pairs = 0.0;
double inter_pairs = 0.0;
double intra_mean = 0.0;
double inter_mean = 0.0;
// Normalised histogram intersection, Σ_b min(p_intra[b], p_inter[b]).
// 0 = perfectly separated, 1 = indistinguishable. This is the number
// that says whether the calibration — and so every threshold expressed
// in its probability space — rests on anything.
double overlap = 0.0;
};
// GR-003 / AR-023 open question, reported but NOT applied. The spec asks for
// a gallery-derived prior of intra/(intra+inter); the shipped default is
// 0.5. Persisting the distributions makes the real value computable, so the
// decision can be taken on evidence rather than left implicit. Behaviour is
// unchanged: `applied` is always false here.
struct Prior {
double derived = 0.0; // intra_pairs / (intra_pairs + inter_pairs)
double derived_log_odds = 0.0; // log(p/(1-p)), the term AR-023 would add
float configured_default = 0.5f;
bool applied = false;
std::string note;
};
std::string schema{"sae.gallery_report/1"};
std::string gallery_path;
EmbedderStamp embedder;
// ── Summary ──────────────────────────────────────────────────────────────
int actors_total = 0; // considered (gallery + zero-usable)
int actors_in_gallery = 0;
int actors_zero_usable = 0;
int actors_below_positive_threshold = 0;
int64_t embeddings_total = 0;
int64_t distinct_embeddings_total = 0;
int64_t duplicates_removed_total = 0;
double mean_embeddings_per_actor = 0.0; // over actors in the gallery
int min_embeddings_for_positive_pairs = 0;
float dedup_similarity_threshold = 0.f;
// ── Calibration ──────────────────────────────────────────────────────────
float calib_a = 10.f;
float calib_b = -5.f;
bool calib_valid = false;
uint64_t calib_hash = 0;
double calib_train_accuracy_pct = 0.0;
float calib_boundary_p50 = 0.f; // similarity at which P(match) = 0.5
Distributions distributions;
Prior prior;
std::vector<Actor> actors;
// Names duplicated out of `actors` so the two failure modes are greppable
// without a JSON query. These are the lines a human reads first.
std::vector<std::string> zero_usable;
std::vector<std::string> below_positive_threshold;
};
/// TRACES: GR-003 | SR-001
///
/// Assembles the report from the three things that know a piece of the answer:
/// the gallery itself (who is in it, with how many references), the calibration
/// stats (dedup, eligibility, the two distributions), and the build audit (who
/// was considered and produced nothing). The audit is optional — a report built
/// from a stored gallery simply cannot know about the actors that never made it.
///
/// `stats` is indexed by actor index, so the flat arrays handed to
/// `calibrate_gallery()` must have used the gallery's own actor ordering.
inline GalleryReport build_gallery_report(const ActorGallery& gallery,
const GalleryCalibration& cal,
const GalleryCalibrationStats& stats,
const GalleryBuildAudit* audit = nullptr,
const std::string& gallery_path = "",
float configured_prior = 0.5f)
{
GalleryReport r;
r.gallery_path = gallery_path;
r.embedder = gallery.embedder;
r.calib_a = cal.a;
r.calib_b = cal.b;
r.calib_valid = cal.valid;
r.calib_hash = gallery.calib_hash;
r.calib_train_accuracy_pct = stats.train_accuracy_pct;
r.calib_boundary_p50 = cal.boundary_at(0.5f);
r.min_embeddings_for_positive_pairs = stats.min_embeddings_for_positive;
r.dedup_similarity_threshold = stats.dedup_sim_threshold;
auto audit_for = [&](const ActorGallery::Actor& a) -> const GalleryBuildAudit::ActorImages* {
if (!audit) return nullptr;
for (const auto& e : audit->actors) {
if (!a.imdb_id.empty() && e.imdb_id == a.imdb_id) return &e;
if (a.imdb_id.empty() && e.name == a.name) return &e;
}
return nullptr;
};
for (size_t i = 0; i < gallery.actors.size(); ++i) {
const auto& ga = gallery.actors[i];
GalleryReport::Actor row;
row.imdb_id = ga.imdb_id;
row.name = ga.name;
row.references = static_cast<int>(ga.embeddings.size());
if (const auto* au = audit_for(ga)) row.images_seen = au->images_seen;
if (i < stats.distinct_per_actor.size()) {
row.distinct_references = stats.distinct_per_actor[i];
row.duplicates_removed = stats.duplicates_removed_per_actor[i];
row.eligible_for_positive_pairs = stats.eligible[i] != 0;
} else {
// No calibration stats for this actor (the fit never saw them).
// Report the raw count rather than a fabricated distinct count.
row.distinct_references = row.references;
}
r.embeddings_total += row.references;
r.distinct_embeddings_total += row.distinct_references;
r.duplicates_removed_total += row.duplicates_removed;
if (!row.eligible_for_positive_pairs) {
++r.actors_below_positive_threshold;
r.below_positive_threshold.push_back(row.name);
}
r.actors.push_back(std::move(row));
}
r.actors_in_gallery = static_cast<int>(gallery.actors.size());
// Actors the build considered and could not use at all. They are not in the
// gallery, so this is the only record that they exist.
if (audit) {
for (const auto& e : audit->actors) {
if (e.images_used > 0) continue;
GalleryReport::Actor row;
row.imdb_id = e.imdb_id;
row.name = e.name;
row.images_seen = e.images_seen;
row.references = 0;
r.zero_usable.push_back(e.name);
r.actors.push_back(std::move(row));
}
}
r.actors_zero_usable = static_cast<int>(r.zero_usable.size());
r.actors_total = r.actors_in_gallery + r.actors_zero_usable;
r.mean_embeddings_per_actor =
r.actors_in_gallery > 0
? static_cast<double>(r.embeddings_total) / r.actors_in_gallery
: 0.0;
// ── The two distributions, straight out of the fit ───────────────────────
auto& d = r.distributions;
d.bins = stats.hist_bins;
d.sim_min = -1.f;
d.sim_max = 1.f;
d.intra = stats.intra_hist;
d.inter = stats.inter_hist;
d.intra_pairs = stats.n_intra_pairs;
d.inter_pairs = stats.n_inter_pairs;
if (d.bins > 0) {
const double bin_w = (d.sim_max - d.sim_min) / d.bins;
double si = 0.0, se = 0.0;
for (int b = 0; b < d.bins; ++b) {
const double centre = d.sim_min + (b + 0.5) * bin_w;
si += d.intra[b] * centre;
se += d.inter[b] * centre;
}
if (d.intra_pairs > 0.0) d.intra_mean = si / d.intra_pairs;
if (d.inter_pairs > 0.0) d.inter_mean = se / d.inter_pairs;
if (d.intra_pairs > 0.0 && d.inter_pairs > 0.0) {
double ov = 0.0;
for (int b = 0; b < d.bins; ++b)
ov += std::min(d.intra[b] / d.intra_pairs, d.inter[b] / d.inter_pairs);
d.overlap = ov;
}
}
// ── The prior AR-023 leaves open — computed, reported, not applied ────────
r.prior.configured_default = configured_prior;
r.prior.applied = false;
const double pair_total = d.intra_pairs + d.inter_pairs;
if (pair_total > 0.0) {
r.prior.derived = d.intra_pairs / pair_total;
const double p = std::clamp(r.prior.derived, 1e-12, 1.0 - 1e-12);
r.prior.derived_log_odds = std::log(p / (1.0 - p));
}
r.prior.note =
"AR-023 specifies a gallery-derived prior of intra/(intra+inter); the shipped "
"match_prior default is 0.5 (calibrated sigmoid used directly). The derived value "
"is the base rate of same-actor pairs among ALL enumerated gallery pairs, so it "
"falls as the cast grows (roughly (k-1)/((k-1)+(A-1)k) for A actors with k "
"references each) — it is a property of gallery size as much as of the embedder. "
"Reported here as evidence; NOT applied. Behaviour is unchanged until the choice "
"is recorded in the spec.";
return r;
}
// ── JSON ─────────────────────────────────────────────────────────────────────
/// TRACES: GR-003 | SR-001
inline nlohmann::json gallery_report_to_json(const GalleryReport& r) {
nlohmann::json j;
j["schema"] = r.schema;
j["gallery_path"] = r.gallery_path;
j["embedder"] = {{"model_name", r.embedder.model_name},
{"model_sha256", r.embedder.model_sha256},
{"embed_dim", r.embedder.embed_dim}};
j["summary"] = {
{"actors_total", r.actors_total},
{"actors_in_gallery", r.actors_in_gallery},
{"actors_zero_usable", r.actors_zero_usable},
{"actors_below_positive_threshold", r.actors_below_positive_threshold},
{"embeddings_total", r.embeddings_total},
{"distinct_embeddings_total", r.distinct_embeddings_total},
{"duplicates_removed_total", r.duplicates_removed_total},
{"mean_embeddings_per_actor", r.mean_embeddings_per_actor},
{"min_embeddings_for_positive_pairs", r.min_embeddings_for_positive_pairs},
{"dedup_similarity_threshold", r.dedup_similarity_threshold}};
j["calibration"] = {
{"a", r.calib_a},
{"b", r.calib_b},
{"valid", r.calib_valid},
{"hash", r.calib_hash},
{"train_accuracy_pct", r.calib_train_accuracy_pct},
{"boundary_p50", r.calib_boundary_p50}};
const auto& d = r.distributions;
j["distributions"] = {
{"bins", d.bins},
{"sim_min", d.sim_min},
{"sim_max", d.sim_max},
{"intra", d.intra},
{"inter", d.inter},
{"intra_pairs", d.intra_pairs},
{"inter_pairs", d.inter_pairs},
{"intra_mean", d.intra_mean},
{"inter_mean", d.inter_mean},
{"overlap", d.overlap}};
j["prior"] = {
{"derived", r.prior.derived},
{"derived_log_odds", r.prior.derived_log_odds},
{"configured_default", r.prior.configured_default},
{"applied", r.prior.applied},
{"note", r.prior.note}};
j["zero_usable"] = r.zero_usable;
j["below_positive_threshold"] = r.below_positive_threshold;
j["actors"] = nlohmann::json::array();
for (const auto& a : r.actors) {
j["actors"].push_back({
{"imdb_id", a.imdb_id},
{"name", a.name},
{"images_seen", a.images_seen},
{"references", a.references},
{"distinct_references", a.distinct_references},
{"duplicates_removed", a.duplicates_removed},
{"eligible_for_positive_pairs", a.eligible_for_positive_pairs}});
}
return j;
}
/// TRACES: GR-003 | SR-001
inline GalleryReport gallery_report_from_json(const nlohmann::json& j) {
GalleryReport r;
r.schema = j.value("schema", std::string{});
r.gallery_path = j.value("gallery_path", std::string{});
if (j.contains("embedder")) {
const auto& je = j.at("embedder");
r.embedder.model_name = je.value("model_name", "");
r.embedder.model_sha256 = je.value("model_sha256", "");
r.embedder.embed_dim = je.value("embed_dim", 512);
}
if (j.contains("summary")) {
const auto& s = j.at("summary");
r.actors_total = s.value("actors_total", 0);
r.actors_in_gallery = s.value("actors_in_gallery", 0);
r.actors_zero_usable = s.value("actors_zero_usable", 0);
r.actors_below_positive_threshold = s.value("actors_below_positive_threshold", 0);
r.embeddings_total = s.value("embeddings_total", int64_t{0});
r.distinct_embeddings_total = s.value("distinct_embeddings_total", int64_t{0});
r.duplicates_removed_total = s.value("duplicates_removed_total", int64_t{0});
r.mean_embeddings_per_actor = s.value("mean_embeddings_per_actor", 0.0);
r.min_embeddings_for_positive_pairs = s.value("min_embeddings_for_positive_pairs", 0);
r.dedup_similarity_threshold = s.value("dedup_similarity_threshold", 0.f);
}
if (j.contains("calibration")) {
const auto& c = j.at("calibration");
r.calib_a = c.value("a", 10.f);
r.calib_b = c.value("b", -5.f);
r.calib_valid = c.value("valid", false);
r.calib_hash = c.value("hash", uint64_t{0});
r.calib_train_accuracy_pct = c.value("train_accuracy_pct", 0.0);
r.calib_boundary_p50 = c.value("boundary_p50", 0.f);
}
if (j.contains("distributions")) {
const auto& d = j.at("distributions");
r.distributions.bins = d.value("bins", 0);
r.distributions.sim_min = d.value("sim_min", -1.f);
r.distributions.sim_max = d.value("sim_max", 1.f);
r.distributions.intra = d.value("intra", std::vector<double>{});
r.distributions.inter = d.value("inter", std::vector<double>{});
r.distributions.intra_pairs = d.value("intra_pairs", 0.0);
r.distributions.inter_pairs = d.value("inter_pairs", 0.0);
r.distributions.intra_mean = d.value("intra_mean", 0.0);
r.distributions.inter_mean = d.value("inter_mean", 0.0);
r.distributions.overlap = d.value("overlap", 0.0);
}
if (j.contains("prior")) {
const auto& p = j.at("prior");
r.prior.derived = p.value("derived", 0.0);
r.prior.derived_log_odds = p.value("derived_log_odds", 0.0);
r.prior.configured_default = p.value("configured_default", 0.5f);
r.prior.applied = p.value("applied", false);
r.prior.note = p.value("note", "");
}
r.zero_usable = j.value("zero_usable", std::vector<std::string>{});
r.below_positive_threshold = j.value("below_positive_threshold", std::vector<std::string>{});
if (j.contains("actors")) {
for (const auto& ja : j.at("actors")) {
GalleryReport::Actor a;
a.imdb_id = ja.value("imdb_id", "");
a.name = ja.value("name", "");
a.images_seen = ja.value("images_seen", -1);
a.references = ja.value("references", 0);
a.distinct_references = ja.value("distinct_references", 0);
a.duplicates_removed = ja.value("duplicates_removed", 0);
a.eligible_for_positive_pairs = ja.value("eligible_for_positive_pairs", false);
r.actors.push_back(std::move(a));
}
}
return r;
}
// "<dir>/cast.h5" → "<dir>/cast.report.json". A known gallery extension is
// replaced rather than appended to, so the report sits beside the gallery under
// the same stem.
inline std::string gallery_report_path(const std::string& gallery_path) {
auto slash = gallery_path.find_last_of("/\\");
auto dot = gallery_path.find_last_of('.');
std::string stem =
(dot != std::string::npos && (slash == std::string::npos || dot > slash))
? gallery_path.substr(0, dot)
: gallery_path;
return stem + ".report.json";
}
/// TRACES: GR-003 | SR-001
inline void save_gallery_report(const std::string& path, const GalleryReport& r) {
std::ofstream out(path);
if (!out.is_open())
throw std::runtime_error("save_gallery_report: cannot write " + path);
out << gallery_report_to_json(r).dump(2) << "\n";
}
/// TRACES: GR-003 | SR-001
inline GalleryReport load_gallery_report(const std::string& path) {
std::ifstream in(path);
if (!in.is_open())
throw std::runtime_error("load_gallery_report: cannot open " + path);
nlohmann::json j;
in >> j;
return gallery_report_from_json(j);
}
/// TRACES: GR-003 | SR-001
///
/// The report's headline, on stderr, at build time. The file is the audit trail;
/// this is what stops a bad gallery from being shipped without anyone noticing.
inline void log_gallery_report(const GalleryReport& r) {
std::cerr << "[gallery-report] " << r.actors_in_gallery << " actors / "
<< r.embeddings_total << " embeddings"
<< " (mean " << r.mean_embeddings_per_actor << " per actor)\n";
if (r.actors_zero_usable > 0) {
std::cerr << "[gallery-report] WARNING: " << r.actors_zero_usable
<< " actor(s) have NO usable image — they can never be recognised:\n";
for (const auto& n : r.zero_usable) std::cerr << " - " << n << "\n";
}
if (r.actors_below_positive_threshold > 0) {
std::cerr << "[gallery-report] " << r.actors_below_positive_threshold
<< " actor(s) below " << r.min_embeddings_for_positive_pairs
<< " distinct references — they contribute no positive pairs and "
"weaken the calibration\n";
}
if (r.duplicates_removed_total > 0)
std::cerr << "[gallery-report] " << r.duplicates_removed_total
<< " near-duplicate reference(s) removed\n";
std::cerr << "[gallery-report] calibration valid=" << r.calib_valid
<< " a=" << r.calib_a << " b=" << r.calib_b
<< " intra/inter overlap=" << r.distributions.overlap
<< " (intra mean=" << r.distributions.intra_mean
<< ", inter mean=" << r.distributions.inter_mean << ")\n";
std::cerr << "[gallery-report] gallery-derived prior would be "
<< r.prior.derived << " (log-odds " << r.prior.derived_log_odds
<< "); shipped default " << r.prior.configured_default
<< " is in force — reported, not applied\n";
}
Reference in New Issue View Git Blame Copy Permalink