docs: richer report — data figures, success/failure frames, commit-pinned repo links
- experiment_charts.py generates 4 figures from experiments/ artifacts: held-out per-film F1, 16-combo ranking, DE search landscape, and the Downton detector-vs-tracker ghost timeline (replaces the blank title-card screenshot) - new frames: 19-correct wedding shot (success case), Many Saints ghost-vs-unknown frame (three error classes in one image) - rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only as the on-disk artifact prefix, explained once - repo file references are now links via https://REPOLINK/<path> placeholders; build_site.sh pins them to the HEAD commit's raw URLs and fails the build if a linked path doesn't exist at HEAD - drop references to removed scripts (scene_score.py, score_config.py) and to session-memory names; mark artifact-registry paths with their pull commands - commit readme_example.jpg + pipeline_topology.svg so README renders on the plain Gitea repo view - deploy_pages.sh: push built site/ to the gitea-pages branch
This commit is contained in:
@@ -0,0 +1,59 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 980 260" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,sans-serif">
|
||||
<style>
|
||||
.box { fill:#1f6feb; stroke:#0d4f9e; stroke-width:1.5; }
|
||||
.box-debug { fill:#57534e; stroke:#3a3733; stroke-width:1.5; }
|
||||
.lbl { fill:#ffffff; font-size:13px; text-anchor:middle; dominant-baseline:middle; }
|
||||
.arrow { stroke:#57534e; stroke-width:2; marker-end:url(#arrowhead); fill:none; }
|
||||
.arrow-debug { stroke:#a8a29e; stroke-width:2; stroke-dasharray:4,3; marker-end:url(#arrowhead-debug); fill:none; }
|
||||
.caption { fill:#57534e; font-size:11px; text-anchor:middle; }
|
||||
</style>
|
||||
<defs>
|
||||
<marker id="arrowhead" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto">
|
||||
<path d="M0,0 L8,4 L0,8 Z" fill="#57534e"/>
|
||||
</marker>
|
||||
<marker id="arrowhead-debug" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto">
|
||||
<path d="M0,0 L8,4 L0,8 Z" fill="#a8a29e"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- top row: main chain -->
|
||||
<rect class="box" x="10" y="40" width="120" height="50" rx="6"/>
|
||||
<text class="lbl" x="70" y="65">frame_source</text>
|
||||
|
||||
<rect class="box" x="160" y="40" width="120" height="50" rx="6"/>
|
||||
<text class="lbl" x="220" y="65">face_detector</text>
|
||||
|
||||
<rect class="box" x="310" y="40" width="120" height="50" rx="6"/>
|
||||
<text class="lbl" x="370" y="65">face_aligner</text>
|
||||
|
||||
<rect class="box" x="460" y="40" width="110" height="50" rx="6"/>
|
||||
<text class="lbl" x="515" y="65">embedder</text>
|
||||
|
||||
<rect class="box" x="600" y="40" width="120" height="50" rx="6"/>
|
||||
<text class="lbl" x="660" y="65">face_tracker</text>
|
||||
|
||||
<rect class="box" x="750" y="40" width="130" height="50" rx="6"/>
|
||||
<text class="lbl" x="815" y="65">identity_matcher</text>
|
||||
|
||||
<line class="arrow" x1="130" y1="65" x2="158" y2="65"/>
|
||||
<line class="arrow" x1="280" y1="65" x2="308" y2="65"/>
|
||||
<line class="arrow" x1="430" y1="65" x2="458" y2="65"/>
|
||||
<line class="arrow" x1="570" y1="65" x2="598" y2="65"/>
|
||||
<line class="arrow" x1="720" y1="65" x2="748" y2="65"/>
|
||||
|
||||
<!-- second row: scene_tracker + result_sink -->
|
||||
<line class="arrow" x1="815" y1="90" x2="815" y2="150"/>
|
||||
<rect class="box" x="720" y="150" width="130" height="50" rx="6"/>
|
||||
<text class="lbl" x="785" y="175">scene_tracker</text>
|
||||
<line class="arrow" x1="718" y1="175" x2="582" y2="175"/>
|
||||
<rect class="box" x="452" y="150" width="120" height="50" rx="6"/>
|
||||
<text class="lbl" x="512" y="175">result_sink</text>
|
||||
|
||||
<!-- debug fan-out -->
|
||||
<line class="arrow-debug" x1="855" y1="90" x2="920" y2="148"/>
|
||||
<rect class="box-debug" x="855" y="150" width="120" height="50" rx="6"/>
|
||||
<text class="lbl" x="915" y="168">debug_renderer /</text>
|
||||
<text class="lbl" x="915" y="184">preview (opt-in)</text>
|
||||
|
||||
<text class="caption" x="490" y="230">solid = always-on data path · dashed = optional debug/preview fan-out from identity_matcher's output</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
+16
-7
@@ -9,7 +9,7 @@ open question: is LVFace (455MB) actually better, or just the biggest?
|
||||
|
||||
Each gallery carries a fitted Platt sigmoid `P(match | cosine similarity) =
|
||||
σ(a·sim + b)`, embedded directly in the gallery's HDF5 file
|
||||
(`src/gallery/gallery_calibration.hpp`). This is a property of the embedding
|
||||
([`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)). This is a property of the embedding
|
||||
space alone — computed from intra/inter-actor reference-image pairs, no
|
||||
tracking or scene logic involved — so it's a clean first read on discriminative
|
||||
power before running a single benchmark.
|
||||
@@ -31,7 +31,7 @@ head start before the tracking/scoring pipeline is even involved.
|
||||
## Second signal: F1 on the actual benchmark
|
||||
|
||||
Best full-gallery (no cast-restriction) result per model, from the 16-combo
|
||||
rep4 matrix (`rep4-optimizer-results.md`):
|
||||
bake-off matrix ([full experiment log](model-bakeoff.md)):
|
||||
|
||||
| model | F1 | P | R | misID |
|
||||
|---|---|---|---|---|
|
||||
@@ -40,16 +40,24 @@ rep4 matrix (`rep4-optimizer-results.md`):
|
||||
| ArcFace R18 | 69.1% | 87.6% | 57.7% | 242 |
|
||||
| ArcFace w600k-R50 | 68.5% | 94.0% | 54.1% | 150 |
|
||||
|
||||
The full 16-combo picture makes the model ordering visible at a glance — LVFace
|
||||
(yellow) tops both the restricted and full columns, and R18 (green) props up
|
||||
the bottom of the full-gallery ranking:
|
||||
|
||||
|
||||
|
||||
LVFace wins outright, with the highest recall of any full-mode combo. This
|
||||
reverses an earlier conclusion from a prior (superseded) benchmarking pass
|
||||
using a scene-union metric, which found the three models statistically
|
||||
indistinguishable (~85% each) and concluded LVFace wasn't worth its size — that
|
||||
metric hid out-of-cast false positives behind a gallery∩cast recall mask (see
|
||||
`optimizer-experiments.md`); the per-second metric used here does not.
|
||||
[the prior optimizer round](optimizer-experiments.md)); the per-second metric
|
||||
used here does not.
|
||||
|
||||
Held-out validation (5 films never seen by the optimizer) confirms LVFace's
|
||||
lead holds up out of sample — see the deep-dive page for the full breakdown,
|
||||
including where it fails.
|
||||
lead holds up out of sample — see the
|
||||
[LVFace deep dive](lvface-deep-dive.md) for the full breakdown, including
|
||||
where it fails.
|
||||
|
||||
## Caveat: model choice is an operational change
|
||||
|
||||
@@ -57,5 +65,6 @@ Switching the default embedder isn't just flipping a config value — the
|
||||
gallery itself is model-specific (embeddings from different models aren't
|
||||
comparable), so any existing gallery built against ArcFace w600k-R50 needs to
|
||||
be rebuilt from source images against LVFace before the new default takes
|
||||
effect. `scripts/optimizer/reembed_gallery.py` does this from a reference
|
||||
gallery's cached source images without re-downloading anything.
|
||||
effect. [`scripts/optimizer/reembed_gallery.py`](https://REPOLINK/scripts/optimizer/reembed_gallery.py)
|
||||
does this from a reference gallery's cached source images without
|
||||
re-downloading anything.
|
||||
|
||||
+14
-5
@@ -7,7 +7,7 @@ pre-filters each film's gallery down to just its Jellyfin-credited cast (typical
|
||||
|
||||
## The result
|
||||
|
||||
Averaged across all 4 models and both expansion settings, on the 4 rep4 training
|
||||
Averaged across all 4 models and both expansion settings, on the 4 bake-off training
|
||||
films:
|
||||
|
||||
| scope | F1 | P | R | total misID (8 evals) |
|
||||
@@ -23,7 +23,13 @@ with someone in the film, but isn't actually in it), and the recall gain shows
|
||||
it isn't costing real detections to get there.
|
||||
|
||||
Per-model, every single model's best-scoring combo in the full 16-way matrix is
|
||||
a `restricted` variant — see the full table in `rep4-optimizer-results.md`. Two
|
||||
a `restricted` variant — visible directly in the ranking below (filled dots =
|
||||
restricted, open = full; the filled dots cluster at the top for every color):
|
||||
|
||||
|
||||
|
||||
See the full table in the
|
||||
[bake-off experiment log](model-bakeoff.md). Two
|
||||
combos hit **zero** true out-of-cast misidentifications:
|
||||
`arcface_w600k_mbf_restricted_exp` (F1 76.5%) and, in full mode,
|
||||
`LVFace-B_Glint360K_full_noexp` (F1 72.4%) — restriction isn't the only way to
|
||||
@@ -32,7 +38,8 @@ reach misid=0, but it's the more reliable one.
|
||||
## Why this isn't the shipped default
|
||||
|
||||
Cast-restriction is implemented today only as an **offline optimizer technique**
|
||||
(`scripts/optimizer/cast_restrict.py`): it pre-builds a filtered gallery file
|
||||
([`scripts/optimizer/cast_restrict.py`](https://REPOLINK/scripts/optimizer/cast_restrict.py)):
|
||||
it pre-builds a filtered gallery file
|
||||
per film, using Jellyfin's own cast list, before the benchmark ever calls the
|
||||
matcher. There's no runtime "restrict matching to this title's credited cast"
|
||||
switch in the shipped application — `scene_analyze` always matches against
|
||||
@@ -41,7 +48,8 @@ whatever single gallery file it's given.
|
||||
Building that as a real feature would need, at minimum:
|
||||
|
||||
- A live Jellyfin cast lookup at analysis time (the title is already known —
|
||||
`run_from_jellyfin.py` already does this same lookup for its own
|
||||
[`scripts/run_from_jellyfin.py`](https://REPOLINK/scripts/run_from_jellyfin.py)
|
||||
already does this same lookup for its own
|
||||
`filter_gallery`-based restriction path, just not wired into `scene_analyze`
|
||||
itself as a first-class option).
|
||||
- A decision on the *fallback*: what happens to a real, uncredited cameo
|
||||
@@ -50,7 +58,8 @@ Building that as a real feature would need, at minimum:
|
||||
- Regenerating the restricted-gallery cache whenever the title's Jellyfin cast
|
||||
list changes.
|
||||
|
||||
This is why the shipped `src/config.hpp` defaults use the `full`-mode winner
|
||||
This is why the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp)
|
||||
defaults use the `full`-mode winner
|
||||
(`LVFace-B_Glint360K_full_exp`, F1 75.3% training / 67.4% held-out macro) rather
|
||||
than the higher-scoring `restricted_exp` (78.3%) — the 78.3% number describes a
|
||||
capability the app doesn't have yet, not what actually ships.
|
||||
|
||||
+16
-6
@@ -5,6 +5,16 @@ film or TV episode — built on [KPN++](https://gitea.tourolle.paris/dtourolle/K
|
||||
(a C++20 Kahn Process Network library) for the detect → track → match → scene
|
||||
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
|
||||
|
||||
This is what a good second looks like — one sampled frame from a held-out film,
|
||||
19 faces named, all 19 correct, the rest honestly declared unknown:
|
||||
|
||||
|
||||
|
||||
And this is why the work isn't done: on this same film the same config misses
|
||||
6 in 10 of the actor-seconds X-Ray says are present, and on the worst held-out
|
||||
film it reports ghost actors over empty walls — at 100% confidence. Both
|
||||
stories, with the evidence, are in the pages below.
|
||||
|
||||
## Start here — four questions this bake-off answers
|
||||
|
||||
- **[Which model is best?](best-model.md)** — calibration curves first
|
||||
@@ -24,14 +34,14 @@ pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
|
||||
|
||||
## The full technical log
|
||||
|
||||
- **[Rep4 model bake-off + threshold re-tune](rep4-optimizer-results.md)** —
|
||||
- **[Model bake-off + threshold re-tune](model-bakeoff.md)** —
|
||||
the complete experiment log behind the four pages above: the ROCm teardown
|
||||
deadlock root cause and fix, DE concurrency tuning, the full 16-combo
|
||||
results table, and every caveat. This is where the shipped `src/config.hpp`
|
||||
defaults come from.
|
||||
results table, and every caveat. This is where the shipped
|
||||
[`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults come from.
|
||||
- **[Optimizer experiments (prior round)](optimizer-experiments.md)** — the
|
||||
earlier scene-union-metric tuning pass, superseded by the per-second metric
|
||||
used in rep4 but kept for the ground-truth/architecture background.
|
||||
used in the bake-off but kept for the ground-truth/architecture background.
|
||||
- **[Service conversion (proposal)](service-conversion.md)** — design sketch
|
||||
for an idle-GPU Docker worker, not yet built.
|
||||
|
||||
@@ -47,5 +57,5 @@ scripts/artifacts/pull_artifacts.sh experiment-data
|
||||
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
|
||||
```
|
||||
|
||||
See `scripts/artifacts/push_artifacts.sh` for the upload side (requires a
|
||||
`GITEA_TOKEN` with package write scope).
|
||||
See [`scripts/artifacts/push_artifacts.sh`](https://REPOLINK/scripts/artifacts/push_artifacts.sh)
|
||||
for the upload side (requires a `GITEA_TOKEN` with package write scope).
|
||||
|
||||
+82
-41
@@ -1,9 +1,28 @@
|
||||
# Deep dive: LVFace-B Glint360K
|
||||
|
||||
LVFace won the model bake-off (see `best-model.md`) and is the shipped default
|
||||
embedder. This page is the honest accounting of how it actually performs —
|
||||
including where it's wrong, and one case where the ground truth itself is
|
||||
wrong and LVFace is right.
|
||||
LVFace won the model bake-off (see [Which model is best?](best-model.md)) and is
|
||||
the shipped default embedder. This page is the honest accounting of how it
|
||||
actually performs — what a good second looks like, where it's wrong and *why*,
|
||||
and one case where the ground truth itself is wrong and LVFace is right.
|
||||
|
||||
## What good looks like
|
||||
|
||||
Before the failure analysis, the ceiling. This is a single sampled second from
|
||||
Downton Abbey's wedding scene — a packed, hat-heavy, period-costume group shot,
|
||||
about as hostile as ensemble framing gets:
|
||||
|
||||
|
||||
*Frame `downton_abbey/best/best_t00127.jpg` from the `montage-frames` artifact
|
||||
package (`scripts/artifacts/pull_artifacts.sh montage-frames
|
||||
Downton_Abbey__A_New_Era`) — green = identified, blue = detected but unknown.*
|
||||
|
||||
**Nineteen named faces in one frame, all nineteen correct** — Jim Carter half
|
||||
behind a flower arrangement, Penelope Wilton at a three-quarter turn, Lesley
|
||||
Nicol under a hat brim. The blue "unknown" boxes are the honest cases: faces the
|
||||
detector found but the matcher declined to name rather than guess. The one miss
|
||||
at this second is Maggie Smith — not on screen in this framing, but X-Ray marks
|
||||
her present for the scene. That distinction (on-screen face vs. scene-level
|
||||
ground truth) sets up everything below.
|
||||
|
||||
## Training vs. held-out: the generalization gap
|
||||
|
||||
@@ -11,6 +30,8 @@ The shipped config (`prob_threshold=0.754, anneal_sec=35.54,
|
||||
extinction_sec=57.43, expand_gallery=true`) was tuned against 4 films. Scored
|
||||
against the 5 films the optimizer never saw:
|
||||
|
||||
|
||||
|
||||
| film | F1 | P | R | TPI | FPI | misid | FN |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| Benny & Joon | 83.0% | 89.1% | 77.7% | 15125 | 1846 | 0 | 4337 |
|
||||
@@ -22,9 +43,10 @@ against the 5 films the optimizer never saw:
|
||||
|
||||
**67.4% held-out vs. 75.3% on training** — an ~8pp drop, and a **37pp spread
|
||||
between the best and worst held-out film**. The config does not generalize
|
||||
uniformly; two films are outright failure cases, for two different reasons.
|
||||
uniformly; two films are outright failure cases, for reasons that turn out to
|
||||
be one mechanism.
|
||||
|
||||
## Failure mode 1: frozen-bbox "ghost tracks"
|
||||
## The failure mode: frozen-bbox "ghost tracks"
|
||||
|
||||
Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst
|
||||
recall of the five) trace to the same root cause, verified directly against
|
||||
@@ -36,51 +58,69 @@ inferred from the score alone.
|
||||
At this second, three of the four labeled boxes ("Jon Bernthal", "Joey Diaz",
|
||||
"Billy Magnussen") sit over empty background — a blurred wall, hanging
|
||||
plates — with no face in them. The real face in frame carries a second,
|
||||
colliding label from another frozen box.
|
||||
colliding label from another frozen box. And it isn't an isolated second — the
|
||||
same signature recurs throughout the film:
|
||||
|
||||
|
||||
|
||||
*Frame `many_saints_intervals/w002_worst_t01382.jpg`, same artifact package —
|
||||
one frame, three distinct error classes.*
|
||||
|
||||
This is the starkest case: **15 actors named, all wrong, over a completely
|
||||
blank closing title card.** Confirmed against the dump directly: `face_count`
|
||||
is 0 from this point onward (no detector output at all), yet the same 15
|
||||
identities keep appearing with the *exact same bounding box, unchanged to the
|
||||
pixel*, for 57+ consecutive seconds.
|
||||
This frame is worth reading closely, because it separates three things that a
|
||||
single aggregate F1 number smears together. The two green labels ("Jon Bernthal
|
||||
100%", "Michela De Rossi 100%") float over a staircase and a policeman's back —
|
||||
frozen boxes from a previous shot, reported at full confidence. Meanwhile the
|
||||
two *real* frontal faces in frame get honest blue "unknown 0%" boxes (they're
|
||||
uncredited day-players with no gallery reference — the
|
||||
[gallery coverage gap](gallery-scope.md)), and three more people simply face
|
||||
away from camera, invisible to any face detector but still "present" in X-Ray's
|
||||
scene-level ground truth. Precision failure, gallery-coverage failure, and the
|
||||
face-vs-presence ceiling — one frame.
|
||||
|
||||
### The mechanism, measured
|
||||
|
||||
The starkest case is Downton Abbey's hard cut from a packed group shot into a
|
||||
long blank credits sequence. Plotting the detector's per-second `face_count`
|
||||
(from the dump HDF5, independent of the tracker) against what the tracker
|
||||
reports makes the failure legible at a glance:
|
||||
|
||||
|
||||
|
||||
From the cut onward the detector sees **zero faces for nearly a minute** — and
|
||||
the tracker keeps reporting the last group shot's 15 identities the entire
|
||||
time, each with the *exact same bounding box, unchanged to the pixel* (verified
|
||||
for Hugh Bonneville: `(1743.2, 0.0, 171.3, 317.8)` at every sampled second for
|
||||
57+ seconds). The staircase decay at the right edge is the extinction window
|
||||
finally expiring, actor by actor.
|
||||
|
||||
This is `SceneTrackerFunc::active_[actor_idx].last_bbox`
|
||||
(`src/nodes/scene_tracker_node.hpp`) being re-emitted unchanged — the
|
||||
extinction state machine working exactly as coded, not a bug. The film cuts
|
||||
from a packed group shot straight into 40+ seconds of blank titles/credits,
|
||||
and `extinction_sec=57.4` is comfortably long enough to bridge that entire gap
|
||||
without expiring, so the tracker faithfully reports "last known position" for
|
||||
a cast that is no longer on screen at all. `extinction_sec` was tuned toward
|
||||
long windows specifically because they bridge real gaps (occlusion, a turned
|
||||
face) in most training footage — this is the cost side of that trade,
|
||||
surfacing only when a film has a long enough faceless stretch to expose it.
|
||||
|
||||
## Failure mode 2: a genuine misID (for contrast)
|
||||
|
||||
Not every held-out failure is a ghost. This is a real face, correctly
|
||||
detected, confidently misidentified:
|
||||
|
||||
*(same many_saints_ghost_fpi.jpg frame above also shows Leslie Odom Jr.'s box
|
||||
carrying a second, colliding "Michael Gandolfini" label — two real tracks'
|
||||
frozen positions happening to overlap, not a detection error.)*
|
||||
([`src/nodes/scene_tracker_node.hpp`](https://REPOLINK/src/nodes/scene_tracker_node.hpp))
|
||||
being re-emitted unchanged — the
|
||||
extinction state machine working exactly as coded, not a bug. `extinction_sec`
|
||||
was tuned to 57.4s specifically because long windows bridge real gaps
|
||||
(occlusion, a turned face) in most footage; a hard cut into long faceless
|
||||
footage is the one case where that same bridging manufactures ghosts, and the
|
||||
training films never contained one long enough to punish it. The optimizer
|
||||
"discovered" the plateau at the top of its search range for a reason that only
|
||||
generalizes to films that never go faceless for a minute.
|
||||
|
||||
## Where LVFace beat X-Ray
|
||||
|
||||
Not every "misID" is actually wrong. `second_score.py` counts a name as a true
|
||||
out-of-cast misID whenever the named actor isn't in X-Ray's credited cast list
|
||||
for the film at all — but X-Ray's cast list is itself incomplete.
|
||||
Not every flagged "misID" is actually wrong.
|
||||
[`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)
|
||||
counts a name as a true out-of-cast misID whenever the named actor isn't in
|
||||
X-Ray's credited cast list for the film at all — but X-Ray's cast list is
|
||||
itself incomplete.
|
||||
|
||||
|
||||
|
||||
Germar Terrell Gardner — a real, clean, high-confidence detection — is counted
|
||||
as a misID here because he doesn't appear in X-Ray's `people.csv` for The Many
|
||||
Saints of Newark at all. But Jellyfin's independent cast metadata *does* credit
|
||||
him for this exact film (cross-checked via `experiments/manifests/
|
||||
jellyfin_casts.json`, a completely separate data source from X-Ray). This
|
||||
isn't a lookalike error or a gallery mixup — it's the pipeline correctly
|
||||
recognising a real cast member that one ground-truth source happened to omit.
|
||||
him for this exact film (cross-checked via
|
||||
`experiments/manifests/jellyfin_casts.json` from the `experiment-data` artifact
|
||||
package, a completely separate data source from X-Ray). This isn't a lookalike error or a gallery mixup — it's the
|
||||
pipeline correctly recognising a real cast member that one ground-truth source
|
||||
happened to omit.
|
||||
|
||||
This doesn't mean every flagged misID is secretly correct — Many Saints'
|
||||
974-count total is still overwhelmingly the frozen-bbox failure mode above,
|
||||
@@ -91,8 +131,9 @@ in the other direction too.
|
||||
|
||||
## Summary
|
||||
|
||||
LVFace is the right default: it wins the model comparison outright, and its
|
||||
failures are traceable, understood, and mostly attributable to one tunable
|
||||
knob (`extinction_sec`) rather than the embedder itself. The held-out
|
||||
LVFace is the right default: it wins the model comparison outright, it can name
|
||||
19 faces correctly in a single hostile group shot, and its failures are
|
||||
traceable, understood, and mostly attributable to one tunable knob
|
||||
(`extinction_sec`) rather than the embedder itself. The held-out
|
||||
generalization gap (75.3% → 67.4%) is real and should be treated as the honest
|
||||
expected performance, not the training-set number.
|
||||
|
||||
@@ -1,11 +1,16 @@
|
||||
# Rep4 model bake-off + threshold re-tune — experiment log (2026-07-18/19)
|
||||
# Model bake-off + threshold re-tune — experiment log (2026-07-18/19)
|
||||
|
||||
Follow-on to `docs/optimizer-experiments.md`, which used an older, since-superseded
|
||||
scene-union metric. This round uses the **per-second** metric
|
||||
(`scripts/optimizer/second_score.py`) and answers three questions in one 16-run
|
||||
matrix: which embedding model is best, does cast-restriction cut misIDs, and does
|
||||
Follow-on to [the prior optimizer round](optimizer-experiments.md), which used
|
||||
an older, since-superseded scene-union metric. This round uses the **per-second** metric
|
||||
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py))
|
||||
and answers three questions in one 16-run matrix: which embedding model is best, does cast-restriction cut misIDs, and does
|
||||
per-film gallery expansion help.
|
||||
|
||||
(This was the fourth optimizer campaign against the X-Ray corpus, so its on-disk
|
||||
artifacts carry an internal `rep4_` prefix — `experiments/results/rep4_best_*.json`,
|
||||
`experiments/trajectories/rep4_*.jsonl`, and the manifests referenced below. The
|
||||
earlier campaigns used the superseded scene-union metric and were discarded.)
|
||||
|
||||
## Why this experiment, and what it actually delivered
|
||||
|
||||
Four goals going in, and an honest read on each after held-out validation (see
|
||||
@@ -40,14 +45,15 @@ below):
|
||||
| `anneal_sec` | 10.0 | **35.5** | Same reversal; previously thought insensitive. |
|
||||
| `expand_gallery` | false | **true** | Helped recall on the full (unrestricted) gallery for the winning model — opposite of the earlier assumption. |
|
||||
|
||||
These are the **`LVFace-B_Glint360K_full_exp`** winning values — the best result that
|
||||
These are the **`LVFace-B_Glint360K_full_exp`** winning values, applied to
|
||||
[`src/config.hpp`](https://REPOLINK/src/config.hpp) — the best result that
|
||||
uses only features already live in the running app (full gallery, no cast
|
||||
restriction; see below for why restricted mode isn't applied even though it scored
|
||||
higher).
|
||||
|
||||
## Why re-run at all
|
||||
|
||||
`docs/optimizer-experiments.md`'s scene-union metric hid out-of-cast false positives
|
||||
[The prior round](optimizer-experiments.md)'s scene-union metric hid out-of-cast false positives
|
||||
behind a gallery∩cast recall mask, and the earlier 9-film benchmark was scene-level
|
||||
(union over a whole X-Ray scene), not a fair per-timepoint comparison. SESSION_STATE
|
||||
flagged the old scene-metric bake-off numbers (R50≈LVFace≈MBF ~85%) as superseded.
|
||||
@@ -55,19 +61,22 @@ This round uses `second_score.py`: uniform per-second sampling, GT = X-Ray scene
|
||||
cast at time *t*, pred = actors whose presence window covers *t*, FPI weighted 10×
|
||||
when the named actor isn't in the film's cast at all (true misID) vs. an in-cast
|
||||
timing slip. FN only counts gallery-known cast (fair recall — 67% of X-Ray cast have
|
||||
no reference embedding, see `gallery-coverage-gap` memory).
|
||||
no reference embedding; see
|
||||
[the prior round's gallery-coverage-gap analysis](optimizer-experiments.md#the-gallery-coverage-gap)).
|
||||
|
||||
## The deadlock that was blocking all of this
|
||||
|
||||
Every replay in this line of work goes through `scripts/optimizer/replay.py`, which
|
||||
runs the real C++ tracker/matcher/scene_tracker nodes inside a Python-assembled KPN
|
||||
network. Before this session, every subprocess replay **timed out at 45s, 100% of
|
||||
Every replay in this line of work goes through
|
||||
[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py),
|
||||
which runs the real C++ tracker/matcher/scene_tracker nodes inside a
|
||||
Python-assembled KPN network. Before this session, every subprocess replay **timed out at 45s, 100% of
|
||||
the time** — not the documented ~20-30% ROCm rocBLAS-GEMM driver flake, but a plain
|
||||
logic bug: `replay.py`'s CLI called `replay(net, ..., stop=False)` to *skip*
|
||||
`net.stop()` (trying to dodge the GEMM deadlock), planning to `os._exit(0)`
|
||||
immediately after. But:
|
||||
|
||||
- `PyNode::stop()` (`external/KPN/include/kpn/python/bindings.hpp`) is the *only*
|
||||
- `PyNode::stop()` ([`include/kpn/python/bindings.hpp`](https://KPNLINK/include/kpn/python/bindings.hpp)
|
||||
in the KPN++ submodule) is the *only*
|
||||
code that sets `stop_flag_ = true` before joining the node's worker thread.
|
||||
- The source node's `run_loop()` has `while (!stop_flag_)` as its only exit
|
||||
condition (it has no input channels, so it never sees a channel-closed signal
|
||||
@@ -90,7 +99,9 @@ F1/precision/recall instead of flat 0.0%.
|
||||
|
||||
## Concurrency tuning
|
||||
|
||||
With the deadlock fixed, `optimize.py` was extended with DE-level parallelism —
|
||||
With the deadlock fixed,
|
||||
[`scripts/optimizer/optimize.py`](https://REPOLINK/scripts/optimizer/optimize.py)
|
||||
was extended with DE-level parallelism —
|
||||
`differential_evolution(..., workers=ThreadPoolExecutor.map)` — so multiple
|
||||
population candidates evaluate concurrently, each spawning its own per-film replay
|
||||
subprocesses (`REPLAY_WORKERS`). Total concurrent GPU replay processes ≈
|
||||
@@ -108,10 +119,10 @@ actually being garbage — a dangerous failure mode, not a crash. **8 concurrent
|
||||
practical ceiling** on this GPU (gfx1100) for this workload. The matrix ran at
|
||||
`REPLAY_WORKERS=4 DE_WORKERS=2`.
|
||||
|
||||
## Training films (rep4) and validation set
|
||||
## Training films and validation set
|
||||
|
||||
9 films total have dumped embeddings across all 4 models. 4 were used for
|
||||
optimization (rep4), leaving 5 held out for validation:
|
||||
optimization, leaving 5 held out for validation:
|
||||
|
||||
- **Lord of War** (64-cast, "clean")
|
||||
- **Scarface** (67-cast, "ensemble/lookalike")
|
||||
@@ -121,7 +132,7 @@ optimization (rep4), leaving 5 held out for validation:
|
||||
Downton Abbey or The Many Saints of Newark)
|
||||
|
||||
Held out: Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of
|
||||
Newark, Valerian and the City of a Thousand Planets. The rep4 numbers below are
|
||||
Newark, Valerian and the City of a Thousand Planets. The training-set numbers below are
|
||||
training-set fit — see "Held-out validation" further down for the real
|
||||
generalization test.
|
||||
|
||||
@@ -162,10 +173,17 @@ includes in-cast timing slips.
|
||||
† old, narrower anneal/extinction bounds (see above) — not directly comparable to
|
||||
the other 12 on those two params.
|
||||
|
||||
The same 16 results as a picture — the two headline effects are visible without
|
||||
reading a single row: filled (restricted) dots stack the top of the ranking for
|
||||
every model color, and yellow (LVFace) leads within both scopes:
|
||||
|
||||
|
||||
|
||||
## Calibration curves — discriminative power, independent of the threshold
|
||||
|
||||
Each model's gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`
|
||||
(embedded directly in the gallery HDF5, see `src/gallery/gallery_calibration.hpp`).
|
||||
(embedded directly in the gallery HDF5, see
|
||||
[`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
|
||||
Plotting all four side by side shows discriminative power directly, independent of
|
||||
whatever `prob_threshold` a particular run happened to use:
|
||||
|
||||
@@ -175,8 +193,10 @@ LVFace-B has both the steepest curve (`a=17.7`, vs. 15.3–16.2 for the ArcFace
|
||||
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs. 0.27–0.31) —
|
||||
it separates same-actor from different-actor pairs more confidently at a lower
|
||||
similarity, consistent with it winning the full-gallery F1 comparison below.
|
||||
Generated by `scripts/docs/calibration_chart.py` (requires each gallery to have
|
||||
been calibrated at least once — run any replay against it first).
|
||||
Generated by
|
||||
[`scripts/docs/calibration_chart.py`](https://REPOLINK/scripts/docs/calibration_chart.py)
|
||||
(requires each gallery to have been calibrated at least once — run any replay
|
||||
against it first).
|
||||
|
||||
## Two effects in isolation: gallery scope, and pose expansion
|
||||
|
||||
@@ -200,7 +220,8 @@ choice itself — which is exactly why cast-restriction becoming a real runtime
|
||||
feature (not just an optimizer trick) is the top item in Caveats below.
|
||||
|
||||
**Pose expansion (promoting a confidently-identified track's novel-pose views into
|
||||
a per-film gallery annex — `track_gallery.hpp`)** is smaller and interacts with
|
||||
a per-film gallery annex — [`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp))**
|
||||
is smaller and interacts with
|
||||
scope rather than acting independently:
|
||||
|
||||
| scope | expansion | F1 | R | misID |
|
||||
@@ -259,6 +280,22 @@ support in the app yet (see Caveats).
|
||||
Decided not to chase further this round (diminishing-returns judgment call) — flag
|
||||
for a future sweep if it matters.
|
||||
|
||||
The ceiling-pinning is visible in the raw search itself. Every one of the 512
|
||||
DE evaluations for the winning combo, plotted over the
|
||||
`prob_threshold` × `extinction_sec` plane:
|
||||
|
||||
|
||||
|
||||
The dark band hugging the top edge *is* the finding: nearly everything scoring
|
||||
well sits at `extinction_sec` ≥ 50, across a wide range of thresholds, and the
|
||||
population converged into a dense cloud around the optimum (threshold ~0.70–0.80,
|
||||
extinction pinned at the 60s bound). Short extinction windows (bottom half) are
|
||||
uniformly pale — under a strict threshold there is simply no good configuration
|
||||
down there. Generated by
|
||||
[`scripts/docs/experiment_charts.py`](https://REPOLINK/scripts/docs/experiment_charts.py)
|
||||
from the DE trajectories (`experiments/trajectories/*.jsonl`, part of the
|
||||
`experiment-data` artifact package).
|
||||
|
||||
## Held-out validation — the number that actually matters
|
||||
|
||||
The 16-combo matrix above is training-set fit. This is the real test: the shipped
|
||||
@@ -276,12 +313,15 @@ Saints of Newark, Valerian and the City of a Thousand Planets), scored the same
|
||||
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 37.0% | 15922 | 4394 | **974** | 23791 |
|
||||
| **macro average (5 films)** | **67.4%** | 85.8% | 57.0% | 56.2% | 116727 | 9046 | 1032 | 127764 |
|
||||
|
||||
|
||||
|
||||
**67.4% held out vs. 75.3% on training** — an ~8pp drop, and a much more informative
|
||||
number than the training-set F1 alone: a **37pp spread between best and worst film**
|
||||
(83.0% vs 46.3%). The config does not generalize uniformly.
|
||||
|
||||
Two films are outright failure cases, and rendering bounding boxes + names on the
|
||||
extracted frames (`replay.py --raw-out` + `dump_error_frames.py --raw`, see
|
||||
extracted frames (`replay.py --raw-out` +
|
||||
[`dump_error_frames.py`](https://REPOLINK/scripts/optimizer/dump_error_frames.py)` --raw`, see
|
||||
Reproduce) turned what looked like a same-scene misidentification into something
|
||||
more precise and more damning:
|
||||
|
||||
@@ -296,29 +336,36 @@ more precise and more damning:
|
||||
happen to overlap.
|
||||
|
||||
|
||||
*`experiments/results/holdout/frames/many_saints/fpi/fpi_t03543.jpg`*
|
||||
*Frame `many_saints/fpi/fpi_t03543.jpg` from the `montage-frames` artifact
|
||||
package (`scripts/artifacts/pull_artifacts.sh montage-frames
|
||||
Many_Saints_of_Newark`).*
|
||||
|
||||
- **Downton Abbey: A New Era** (large ensemble, 36-cast) has high precision (97.8%)
|
||||
but recall collapses to 39.4% (FN=80084, by far the largest of the 5). The frame
|
||||
below is the starkest evidence in this whole experiment: **the matcher named 15
|
||||
actors — all of them wrong — over a completely blank closing title card with no
|
||||
faces on screen at all.**
|
||||
but recall collapses to 39.4% (FN=80084, by far the largest of the 5). Its
|
||||
starkest failure happens where there is nothing to see at all: the film's hard
|
||||
cut into its closing credits, where **the matcher kept reporting 15 actors —
|
||||
all wrong — for nearly a minute of faceless screen.**
|
||||
|
||||
|
||||
*`experiments/results/holdout/frames/downton_abbey/fpi/fpi_t07242.jpg`*
|
||||
Both are the same mechanism, and it can be *measured*, not just screenshotted.
|
||||
Plotting the dump's own per-second `face_count` (detector output, independent
|
||||
of the tracker) against the number of actors the tracker reports, through
|
||||
Downton Abbey's cut to credits:
|
||||
|
||||
Both are the same mechanism, verified directly against the HDF5 dump and the raw
|
||||
per-frame stream (not just inferred from the screenshot): at the Downton Abbey
|
||||
title card (t=7242), the dump's own `face_count` is **0 from t≈7240 onward** — no
|
||||
detector output at all, confirmed independently of the pipeline. Yet all 15 "wrong"
|
||||
actors are still marked visible, each with the *exact same bbox, unchanged to the
|
||||
pixel*, repeated every single frame back to t=7222 (verified for Hugh Bonneville:
|
||||
`(1743.2, 0.0, 171.3, 317.8)` at every sampled second from 7222 through 7279+). That
|
||||
is `SceneTrackerFunc`'s `active_[actor_idx].last_bbox` (`scene_tracker_node.hpp`)
|
||||
being re-emitted unchanged — **this is the extinction state machine working exactly
|
||||
as coded**, not a bug in the logic. The film cuts from a packed group shot straight
|
||||
into ~40+ seconds of blank titles/credits with zero faces, and `extinction_sec=57.4`
|
||||
is comfortably long enough to bridge that entire gap without expiring, so the
|
||||
|
||||
|
||||
From the cut onward the detector sees **zero faces** — yet the tracker holds a
|
||||
perfectly flat plateau of 15 reported identities for 56 seconds, each with the
|
||||
*exact same bbox, unchanged to the pixel* (verified for Hugh Bonneville:
|
||||
`(1743.2, 0.0, 171.3, 317.8)` at every sampled second from 7222 through 7279+).
|
||||
The staircase on the right edge is the extinction window finally expiring,
|
||||
actor by actor. That plateau is `SceneTrackerFunc`'s
|
||||
`active_[actor_idx].last_bbox`
|
||||
([`src/nodes/scene_tracker_node.hpp`](https://REPOLINK/src/nodes/scene_tracker_node.hpp))
|
||||
being re-emitted
|
||||
unchanged — **the extinction state machine working exactly as coded**, not a
|
||||
bug in the logic. The film cuts from a packed group shot straight into ~40+
|
||||
seconds of blank titles/credits with zero faces, and `extinction_sec=57.4` is
|
||||
comfortably long enough to bridge that entire gap without expiring, so the
|
||||
tracker faithfully keeps reporting "last known position" for a cast that is no
|
||||
longer on screen at all.
|
||||
|
||||
@@ -331,11 +378,14 @@ was chosen to avoid. The training-set films apparently didn't have a long enough
|
||||
faceless stretch after a confirmed identity to expose this; the held-out set did.
|
||||
|
||||
Frames for all three films (`benny_joon`, `many_saints`, `downton_abbey` — one strong
|
||||
performer, two failure cases) are under `experiments/results/holdout/frames/`, each
|
||||
performer, two failure cases) are under `experiments/results/holdout/frames/`
|
||||
(not committed — pull per film with `scripts/artifacts/pull_artifacts.sh
|
||||
montage-frames <film-slug>`), each
|
||||
with a `manifest.json` listing the bucket (`best`/`fpi`/`fn`), timestamp, and
|
||||
predicted vs. ground-truth actors for every dumped frame. Frames are annotated with
|
||||
bounding boxes + name/confidence (green = identified, orange = unknown), matching
|
||||
`debug_renderer_node.hpp`'s colour convention. Generated by
|
||||
[`src/nodes/debug_renderer_node.hpp`](https://REPOLINK/src/nodes/debug_renderer_node.hpp)'s
|
||||
colour convention. Generated by
|
||||
`scripts/optimizer/dump_error_frames.py --raw <replay.py --raw-out output>` (see
|
||||
Reproduce).
|
||||
|
||||
@@ -387,6 +437,10 @@ python3 scripts/optimizer/replay.py \
|
||||
--out pred.json --raw-out raw.jsonl --prob-threshold 0.754 --anneal-sec 35.54 \
|
||||
--extinction-sec 57.43 --expand-gallery
|
||||
|
||||
# regenerate the report's charts (16-combo ranking, DE landscape, held-out
|
||||
# per-film F1, Downton ghost timeline) from the artifacts under experiments/
|
||||
python3 scripts/docs/experiment_charts.py --out-dir docs/assets/images
|
||||
|
||||
# dump example frames (best-agreement / FPI / FN) for visual inspection, annotated
|
||||
# with bounding boxes + names (--raw is optional; omit for unannotated frames)
|
||||
python3 scripts/optimizer/dump_error_frames.py \
|
||||
@@ -403,6 +457,6 @@ python3 scripts/optimizer/dump_error_frames.py \
|
||||
--out-dir experiments/results/holdout/frames/<name>_intervals --interval-sec 600
|
||||
```
|
||||
|
||||
See also: `docs/optimizer-experiments.md` (prior round, superseded metric),
|
||||
`experiments/SESSION_STATE.md`, and memory: kpn-python-replay-optimizer,
|
||||
gallery-coverage-gap, xray-validation-results, per-scene-presence-eval-design.
|
||||
See also: [the prior optimizer round](optimizer-experiments.md) (superseded
|
||||
metric) and the session log
|
||||
[`experiments/SESSION_STATE.md`](https://REPOLINK/experiments/SESSION_STATE.md).
|
||||
@@ -45,13 +45,17 @@ GT set = actors X-Ray lists for that scene. Per scene TP/FP/FN, then:
|
||||
scenes count more) → **equal-weight mean across movies** (macro; each film counts
|
||||
the same regardless of length). This is the DE objective.
|
||||
|
||||
Implemented in `scripts/optimizer/scene_score.py`.
|
||||
Implemented in `scripts/optimizer/scene_score.py` — since **removed** along
|
||||
with this metric; its per-second successor is
|
||||
[`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)
|
||||
(see the [bake-off round](model-bakeoff.md)).
|
||||
|
||||
## The gallery coverage gap
|
||||
|
||||
Diagnosing low recall: only **131 of 392** X-Ray cast were in the gallery (33%). Every
|
||||
in-gallery actor HAD embeddings (gallery well-formed) — the gap was pure coverage.
|
||||
`scripts/optimizer/fetch_missing_actors.py` recovers missing actors:
|
||||
[`scripts/optimizer/fetch_missing_actors.py`](https://REPOLINK/scripts/optimizer/fetch_missing_actors.py)
|
||||
recovers missing actors:
|
||||
`nm-id → TMDB /find external_ids → /person/{id}/images → download → embed (sae_embed)`,
|
||||
with a `--wikidata` fallback (P345→P18 Commons photo).
|
||||
|
||||
@@ -71,7 +75,8 @@ face-recognition pipeline vs X-Ray's presence semantics, not a fixable gap.
|
||||
|
||||
## Optimizer
|
||||
|
||||
`scripts/optimizer/optimize.py` — scipy `differential_evolution` over the knob space,
|
||||
[`scripts/optimizer/optimize.py`](https://REPOLINK/scripts/optimizer/optimize.py)
|
||||
— scipy `differential_evolution` over the knob space,
|
||||
each candidate = full replay of all films through the **real** C++ nodes (see the
|
||||
KPN replay architecture below) scored by the metric above. Global objective (one
|
||||
config for all films, not per-film).
|
||||
@@ -91,7 +96,8 @@ the tightly-converged knobs were adopted as defaults.
|
||||
|
||||
The optimizer never re-decodes video. `scene_analyze --dump-embeddings out.h5` runs the
|
||||
expensive half once (decode→detect→align→embed) and dumps per-frame face embeddings
|
||||
+ metadata to HDF5 (`scripts/optimizer/SCHEMA.md`). `scripts/optimizer/replay.py` then
|
||||
+ metadata to HDF5 ([`scripts/optimizer/SCHEMA.md`](https://REPOLINK/scripts/optimizer/SCHEMA.md)).
|
||||
[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py) then
|
||||
replays that dump through the **real** C++ `face_tracker → identity_matcher →
|
||||
scene_tracker` assembled in a Python KPN network (`sae_kpn` nanobind module), varying
|
||||
Config knobs freely — no GPU embedding, no decode. Verified BYTE-EXACT against
|
||||
@@ -110,8 +116,10 @@ python scripts/optimizer/optimize.py --manifest films.json --gallery gallery.jso
|
||||
--params prob_threshold:0.5:0.999 anneal_sec:1:30 extinction_sec:1:15 \
|
||||
--popsize 8 --maxiter 20 --trajectory traj.jsonl --out opt.json
|
||||
# 4. score a fixed config / validate on a held-out set
|
||||
python scripts/optimizer/score_config.py --manifest heldout.json --gallery gallery.json \
|
||||
--config '{"prob_threshold":0.76,"extinction_sec":1.5,"anneal_sec":10}'
|
||||
# (historical: score_config.py and scene_score.py were removed with the
|
||||
# scene-union metric — use scripts/optimizer/second_score.py, per-second)
|
||||
python scripts/optimizer/second_score.py --help
|
||||
```
|
||||
|
||||
See also memory: kpn-python-replay-optimizer, gallery-coverage-gap, xray-validation-*.
|
||||
Superseded by the [model bake-off + re-tune](model-bakeoff.md), which
|
||||
replaced this round's scene-union metric with per-second scoring.
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Pose expansion: does "learning" new poses mid-film help?
|
||||
|
||||
`expand_gallery` (`src/gallery/track_gallery.hpp`) promotes a confidently-identified
|
||||
`expand_gallery` ([`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp))
|
||||
promotes a confidently-identified
|
||||
track's novel-pose reference views into a per-film, in-memory gallery annex — the
|
||||
idea being that once the pipeline is sure who someone is, a pose it hasn't seen
|
||||
before (turned head, different lighting) becomes a free extra reference for
|
||||
@@ -21,8 +22,8 @@ Averaged across all 4 models, on the 4 films used for optimization:
|
||||
In `restricted` mode (matcher's candidate set capped to the film's own credited
|
||||
cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually
|
||||
lower. In `full` mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but
|
||||
misID roughly quadrupled (209 → 864) — see `rep4-optimizer-results.md` for the
|
||||
per-model breakdown. That's the number that motivated this page: **does turning
|
||||
misID roughly quadrupled (209 → 864) — see the
|
||||
[bake-off experiment log](model-bakeoff.md) for the per-model breakdown. That's the number that motivated this page: **does turning
|
||||
expansion on actually change what gets recognised, frame by frame, or is the
|
||||
aggregate F1 shift something else?**
|
||||
|
||||
@@ -83,7 +84,8 @@ mode) doesn't reproduce on held-out data — at minimum it's far smaller than th
|
||||
training-set numbers suggested, and plausibly it's sampling variation from only
|
||||
4 training films rather than a real, generalizable mechanism. This doesn't mean
|
||||
`expand_gallery` never does anything (the mechanism is real — see
|
||||
`track_gallery.hpp`'s promotion logging: tracks *do* get confirmed and views *do*
|
||||
[`track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp)'s
|
||||
promotion logging: tracks *do* get confirmed and views *do*
|
||||
get promoted into the annex on every film tested), only that **whatever effect
|
||||
it has on final per-second identification was too small to detect against 5
|
||||
held-out films** with this scoring method. A cleaner test would need either many
|
||||
@@ -92,6 +94,7 @@ more held-out films or a metric that can see the annex's direct contribution
|
||||
pass had budget for.
|
||||
|
||||
**Practical takeaway**: don't treat the training-set `exp` vs `noexp` numbers in
|
||||
`rep4-optimizer-results.md` as proof that expansion changes real-world behavior
|
||||
the [bake-off experiment log](model-bakeoff.md) as proof that expansion
|
||||
changes real-world behavior
|
||||
in either direction — on the evidence gathered so far, it doesn't move the
|
||||
needle enough to see.
|
||||
|
||||
@@ -34,12 +34,12 @@ lock-gating, not new pipeline logic.
|
||||
| Piece | Where | What it does |
|
||||
|---|---|---|
|
||||
| Analysis engine | `build/scene_analyze` | Video → face detect/align/embed → gallery match → result JSON |
|
||||
| Backend selection | `CMakeLists.txt` (`SAE_INFERENCE_BACKEND`, `SAE_GEMM_BACKEND`) | ORT/TRT + ROCm/CUDA, chosen **at build time** |
|
||||
| Backend selection | [`CMakeLists.txt`](https://REPOLINK/CMakeLists.txt) (`SAE_INFERENCE_BACKEND`, `SAE_GEMM_BACKEND`) | ORT/TRT + ROCm/CUDA, chosen **at build time** |
|
||||
| New-media queue | JRay plugin → `GET /Plugins/JRay/Tasks/Pending` | Backlog of items with no results yet |
|
||||
| Worker loop | `run_from_jellyfin.py --worker` | Poll Pending → run `scene_analyze` → push results |
|
||||
| Worker loop | [`scripts/run_from_jellyfin.py`](https://REPOLINK/scripts/run_from_jellyfin.py)` --worker` | Poll Pending → run `scene_analyze` → push results |
|
||||
| Result push | `PUT /Plugins/JRay/Items/{id}/Truth` | Stores per-actor scene windows back in Jellyfin |
|
||||
| Incremental gallery | `make_jellyfin_gallery.py --merge` | Embeds only cast not already in the gallery |
|
||||
| Secrets loader | `.env` via `sae_env.py` | `JELLYFIN_URL`, `JELLYFIN_API_KEY`, `TMDB_API_KEY` |
|
||||
| Incremental gallery | [`scripts/make_jellyfin_gallery.py`](https://REPOLINK/scripts/make_jellyfin_gallery.py)` --merge` | Embeds only cast not already in the gallery |
|
||||
| Secrets loader | `.env` via [`scripts/sae_env.py`](https://REPOLINK/scripts/sae_env.py) | `JELLYFIN_URL`, `JELLYFIN_API_KEY`, `TMDB_API_KEY` |
|
||||
|
||||
## Installer config
|
||||
|
||||
|
||||
Reference in New Issue
Block a user