From 9c029d43cabe4e938895aeae68f184c0b0367af6 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Tue, 21 Jul 2026 09:12:46 +0200 Subject: [PATCH] docs: deploy from e588597 --- best-model/index.html | 4 ++-- gallery-scope/index.html | 6 +++--- index.html | 4 ++-- lvface-deep-dive/index.html | 20 +++++++++++++------- methodology/index.html | 2 +- model-bakeoff/index.html | 14 +++++++------- pose-expansion/index.html | 4 ++-- search/search_index.json | 2 +- service-conversion/index.html | 8 ++++---- 9 files changed, 35 insertions(+), 29 deletions(-) diff --git a/best-model/index.html b/best-model/index.html index f4b9003..159f333 100644 --- a/best-model/index.html +++ b/best-model/index.html @@ -835,7 +835,7 @@ image count.

First signal: calibration curves

Each gallery carries a fitted Platt sigmoid P(match | cosine similarity) = σ(a·sim + b), stored directly in the gallery HDF5 -(src/gallery/gallery_calibration.hpp). +(src/gallery/gallery_calibration.hpp). This is a property of the embedding space alone, computed from intra- and inter-actor reference-image pairs with no tracking or scene logic involved, so it is a clean first read on discriminative power before running a @@ -1032,7 +1032,7 @@ lead here is a precision-and-recall lead, not a misID one.

is model-specific, since embeddings from different models are not comparable. Any existing gallery built against a different model must be rebuilt from source images before the new default takes effect. -scripts/optimizer/reembed_gallery.py +scripts/optimizer/reembed_gallery.py does this from a reference gallery's cached source images without re-downloading anything.

diff --git a/gallery-scope/index.html b/gallery-scope/index.html index d225888..1a74de3 100644 --- a/gallery-scope/index.html +++ b/gallery-scope/index.html @@ -835,7 +835,7 @@ model-choice effect: LVFace beats r18 by 6.2pp in full mode but beats mbf by

Why this is not the shipped default

Cast restriction is implemented today only as an offline optimizer technique -(scripts/optimizer/cast_restrict.py): +(scripts/optimizer/cast_restrict.py): it pre-builds a filtered gallery file per film using Jellyfin's cast list before the benchmark calls the matcher. There is no runtime "restrict to this title's credited cast" switch in the shipped application; @@ -844,7 +844,7 @@ given.

Building this as a real feature requires:

-

The shipped src/config.hpp defaults use +

The shipped src/config.hpp defaults use the full-mode winner (LVFace-B_Glint360K_full_exp, F1 75.3% training, 67.4% held-out macro) rather than the higher-scoring restricted_exp (78.3%), because 78.3% describes a capability the application does not diff --git a/index.html b/index.html index e0db9a9..e748c41 100644 --- a/index.html +++ b/index.html @@ -855,7 +855,7 @@ credited cast.

four pages above, including how replaying against cached embeddings inside the same KPN network makes a full model and configuration comparison practical, the full results table, and every caveat. This is - where the shipped src/config.hpp + where the shipped src/config.hpp defaults come from.
  • Service conversion (proposal): design sketch for a native idle-GPU worker gated on screen lock, not yet built.
    • @@ -868,7 +868,7 @@ pushed to the Gitea package registry and pulled on demand:

    scripts/artifacts/pull_artifacts.sh experiment-data scripts/artifacts/pull_artifacts.sh montage-frames <film-slug> -

    See scripts/artifacts/push_artifacts.sh +

    See scripts/artifacts/push_artifacts.sh for the upload side, which requires a GITEA_TOKEN with package write scope.

    diff --git a/lvface-deep-dive/index.html b/lvface-deep-dive/index.html index f07ae2d..e3c9307 100644 --- a/lvface-deep-dive/index.html +++ b/lvface-deep-dive/index.html @@ -1165,7 +1165,7 @@ same span (verified for Hugh Bonneville: bbox (1743.2, 0.0, 171.3, 317.8)< unchanged to the pixel, at every sampled second for 57 seconds). The staircase at the right edge is the extinction window expiring, actor by actor. This is SceneTrackerFunc::active_[actor_idx].last_bbox -(src/nodes/scene_tracker_node.hpp) +(src/nodes/scene_tracker_node.hpp) re-emitted as designed. extinction_sec=57.4 was tuned long because bridging is correct on most footage, as in the polygraph scene above. The training films did not contain a faceless stretch long enough to expose the @@ -1245,17 +1245,23 @@ pipeline has no way to draw that distinction from a face crop alone.

    the edge of frame. A marginal, low-confidence lookalike confusion.

    Lovelace: 1 name

    Chloë Sevigny, out-of-cast name in Lovelace

    -

    Chloë Sevigny, t=2451s, 100% confidence. Two boxes are drawn on the same -face: one correctly labeled Amanda Seyfried, one incorrectly labeled Chloë -Sevigny, both at 100%. A single detection producing two competing high- -confidence identities on the same crop.

    +

    Chloë Sevigny, t=2451s, 100% confidence. Amanda Seyfried's track is real and +well-tracked through most of this shot, but her bbox is frozen at the exact +same coordinates for t=2450 and t=2451, one second where her box stopped +updating from a fresh detection. Only one real face is detected at t=2451 +(confirmed against the dump's own per-frame detections), and it is a tight +IoU-1.0 fit under the Chloë Sevigny box, not the Seyfried one. So the green +Seyfried box in this frame is a ghost, re-emitting her last known position +for that one second, and the fresh, wrong detection is Sevigny, landing on +top of it. Not two competing fresh identities on one crop: one ghost and +one fresh misidentification happening to overlap.

    Scarface: 1 name

    Kirstie Alley, out-of-cast name in Scarface

    Kirstie Alley, t=2451s, 89% confidence. Al Pacino is correctly identified in the foreground at 100%; a background face in the same shot is wrongly labeled Kirstie Alley. (The t=2451s here and the Lovelace Chloë Sevigny case above landing on the identical second is a genuine coincidence, verified from -each film's raw stream by first_fpi_frames.py, +each film's raw stream by first_fpi_frames.py, not a transcription slip, two unrelated films whose first out-of-cast name happens to fall at the same timestamp.)

    Summary of the nine

    @@ -1342,7 +1348,7 @@ bridging: every one of these nine is a fresh detection on a real face crop at the second it first appears.

    Where LVFace beat X-Ray

    Not every name marked wrong is actually wrong. -scripts/optimizer/second_score.py +scripts/optimizer/second_score.py scores strictly against X-Ray, and X-Ray has gaps of its own.

    LVFace correctly identifies Germar Terrell Gardner, uncredited by X-Ray

    Germar Terrell Gardner, the same name from the table above, does not appear diff --git a/methodology/index.html b/methodology/index.html index 59f274b..42adfc2 100644 --- a/methodology/index.html +++ b/methodology/index.html @@ -917,7 +917,7 @@ not in X-Ray's cast for this second. Split into two categories:

    The precision reported throughout this report, and therefore the F1 derived from it, puts each FPI_misid into the denominator 10 times (precision = TPI / (TPI + FPI_incast + 10·FPI_misid), -second_score.py). +second_score.py). This is deliberate: the whole point is to punish naming an out-of-film actor far harder than a scene-boundary slip. But it means the P column is not raw precision, and a misID-heavy film's P is depressed diff --git a/model-bakeoff/index.html b/model-bakeoff/index.html index c086089..4a7873d 100644 --- a/model-bakeoff/index.html +++ b/model-bakeoff/index.html @@ -1039,7 +1039,7 @@ caveat needed to trust the numbers.

    Read How we score against X-Ray first for what F1, precision, recall, and misID mean in this report. All numbers below use the per-second metric -(scripts/optimizer/second_score.py).

    +(scripts/optimizer/second_score.py).

    r50 (ArcFace w600k-R50) is excluded from the detailed comparison below. Its gallery was built with roughly 30% fewer reference images per actor than the other three models on the identical source photos (10808 vs 15055 total @@ -1057,8 +1057,8 @@ output, and the cheap stage can be re-run against that cache as many times as needed with different Config values.

    scene_analyze --dump-embeddings out.h5 runs the expensive half once per film and writes per-frame face detections and embeddings to HDF5 -(scripts/optimizer/SCHEMA.md). -scripts/optimizer/replay.py +(scripts/optimizer/SCHEMA.md). +scripts/optimizer/replay.py then re-assembles the real C++ face_tracker, identity_matcher, and scene_tracker nodes into a Python-driven KPN network and replays a film's cached embeddings through them, varying prob_threshold, @@ -1507,7 +1507,7 @@ runtime feature of the application; see into the application would require.

    Pose expansion (promoting a confidently identified track's novel-pose views into a per-film gallery annex, -src/gallery/track_gallery.hpp):

    +src/gallery/track_gallery.hpp):

    @@ -1563,7 +1563,7 @@ and the two methodology bugs caught while checking it.

    Calibration curves

    Each gallery carries a fitted Platt sigmoid P(match | sim) = σ(a·sim + b), stored directly in the gallery HDF5 -(src/gallery/gallery_calibration.hpp). +(src/gallery/gallery_calibration.hpp). This measures discriminative power independent of whatever prob_threshold a given run used:

    Calibrated P(match|similarity) for all four models

    @@ -1572,7 +1572,7 @@ variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs 0.27-0.31), separating same-actor from different-actor pairs more confidently at a lower similarity than any ArcFace variant tested, including r50. Generated by -scripts/docs/calibration_chart.py.

    +scripts/docs/calibration_chart.py.

    Every one of the 512 DE evaluations for the winning LVFace full_exp combo, plotted over the prob_threshold × extinction_sec plane:

    @@ -1632,7 +1632,7 @@ past 60s, or does it plateau there. Not chased further this pass.

    python3 scripts/docs/first_fpi_frames.py

    See also the session log -experiments/SESSION_STATE.md.

    +experiments/SESSION_STATE.md.

    diff --git a/pose-expansion/index.html b/pose-expansion/index.html index 47769c5..202dde2 100644 --- a/pose-expansion/index.html +++ b/pose-expansion/index.html @@ -826,7 +826,7 @@

    Pose expansion: does promoting new poses mid-film help?

    expand_gallery -(src/gallery/track_gallery.hpp) +(src/gallery/track_gallery.hpp) promotes a confidently identified track's novel-pose reference views into a per-film, in-memory gallery annex. The idea: once the pipeline is confident about an identity, a pose it has not seen before (turned head, different @@ -982,7 +982,7 @@ inert config look consequential; both are reasons to distrust a dramatic training-set delta until it survives on held-out films, which this one did not. This does not mean expand_gallery never does anything: the mechanism is real, and -track_gallery.hpp's +track_gallery.hpp's promotion logging confirms tracks get confirmed and views get promoted into the annex on every film tested. It means whatever effect expansion has on final per-second identification was too small to detect against 5 diff --git a/search/search_index.json b/search/search_index.json index 8b514bb..0eb28fb 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"","title":"scene-actor-extraction","text":"

    A face-recognition pipeline that finds when each actor appears on screen in a film or TV episode, built on KPN++ (a C++20 Kahn Process Network library) for the detect, track, match, and scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.

    This is a correctly scored second from a held-out film, one the optimizer never saw during tuning:

    Every visible face is named at 100% confidence (Chris Noth, Hank Azaria, Bobby Cannavale), the background extra is correctly left unnamed, and the two credited cast members without a visible face are correctly reported present but not visible. This matches Amazon X-Ray's own record for this second exactly.

    Results are not uniform across films. The hardest held-out film scores 46% F1. This report documents why: one tunable trade (extinction bridging at hard cuts), one structural limit (X-Ray credits people whose faces never appear on screen), and a small number of cases where the pipeline is correct and X-Ray's ground truth is not. Read how we score against X-Ray first. X-Ray's ground truth is scene-level; the pipeline's output is per-second. That difference shapes every finding below.

    "},{"location":"#findings","title":"Findings","text":""},{"location":"#full-experiment-log","title":"Full experiment log","text":""},{"location":"#reproducing-the-benchmarks","title":"Reproducing the benchmarks","text":"

    Gallery .h5 files, embedding dumps, the X-Ray corpus, montage frame images, and DE trajectories are not committed to this repository. They are pushed to the Gitea package registry and pulled on demand:

    scripts/artifacts/pull_artifacts.sh galleries\nscripts/artifacts/pull_artifacts.sh experiment-data\nscripts/artifacts/pull_artifacts.sh montage-frames <film-slug>\n

    See scripts/artifacts/push_artifacts.sh for the upload side, which requires a GITEA_TOKEN with package write scope.

    "},{"location":"best-model/","title":"Which embedding model is best?","text":"

    Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K, 455MB) were compared. r50 is excluded from the training/held-out comparison below; its gallery has roughly 30% fewer reference images per actor than the other three on the identical source photos, which confounds a direct score comparison (see the full experiment log for detail). It remains in the calibration comparison, which does not depend on the gallery image count.

    "},{"location":"best-model/#first-signal-calibration-curves","title":"First signal: calibration curves","text":"

    Each gallery carries a fitted Platt sigmoid P(match | cosine similarity) = \u03c3(a\u00b7sim + b), stored directly in the gallery HDF5 (src/gallery/gallery_calibration.hpp). This is a property of the embedding space alone, computed from intra- and inter-actor reference-image pairs with no tracking or scene logic involved, so it is a clean first read on discriminative power before running a benchmark.

    model a (steepness) boundary at P=0.5 LVFace-B Glint360K 17.7 sim 0.228 ArcFace w600k-MBF 16.2 sim 0.267 ArcFace w600k-R50 15.4 sim 0.301 ArcFace R18 15.3 sim 0.309

    LVFace has both the steepest transition and the lowest decision boundary, separating same-actor from different-actor reference pairs more confidently at a lower similarity than any ArcFace variant.

    "},{"location":"best-model/#second-signal-held-out-f1","title":"Second signal: held-out F1","text":"

    Each model's own tuned full_exp config, replayed against the 5 films the optimizer never saw and scored the same way:

    film LVFace F1 mbf F1 r18 F1 Benny & Joon 83.0% 78.5% 77.1% Lovelace 77.5% 73.7% 72.2% Valerian and the City of a Thousand Planets 74.1% 70.2% 71.0% Downton Abbey: A New Era 56.2% 55.0% 53.0% The Many Saints of Newark 46.3% 44.5% 42.1% macro average 67.4% 64.4% 63.1%

    LVFace scores highest on all 5 held-out films; the ranking never flips between models. Total misID count across the 5 films: LVFace 1032, mbf 2197, r18 1224. LVFace has less than half mbf's misID total and still scores higher on every film.

    Held-out results are stronger evidence than training results, because training numbers can reflect what the optimizer was tuned to fit rather than general performance. On training data, the ordering is not as clean:

    film LVFace F1 mbf F1 r18 F1 best Caf\u00e9 Society 68.1% 62.2% 60.1% LVFace Lord of War 75.6% 77.2% 75.6% mbf Scarface 71.5% 68.6% 64.1% LVFace Sound of Metal 78.8% 76.5% 71.6% LVFace

    mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either table where LVFace does not score highest. LVFace's training-set macro average (75.3%, see the full experiment log) is not a uniform win across every film it contributes to; the held-out result, where LVFace wins all 5 films outright, is the stronger claim.

    This reverses an earlier, superseded benchmarking pass that used a scene-union metric and found the three models statistically indistinguishable (around 85% each), concluding LVFace was not worth its size. That metric masked out-of-cast false positives behind a gallery-intersect-cast recall filter; the per-second metric used here does not.

    "},{"location":"best-model/#full-training-matrix-picture","title":"Full training-matrix picture","text":"

    Best full-gallery combo per model (all three are full_exp), from the training matrix in the full experiment log:

    model F1 P R misID LVFace-B Glint360K 75.3% 89.7% 65.4% 232 ArcFace w600k-MBF 72.0% 87.7% 61.4% 240 ArcFace R18 69.1% 87.6% 57.7% 242

    LVFace leads within both the restricted and full gallery modes, visible directly in the chart above without reading the table. The three models' misID counts on the full gallery are nearly identical (232/240/242); LVFace's lead here is a precision-and-recall lead, not a misID one.

    "},{"location":"best-model/#operational-note","title":"Operational note","text":"

    Switching the default embedder is not a config change alone; the gallery is model-specific, since embeddings from different models are not comparable. Any existing gallery built against a different model must be rebuilt from source images before the new default takes effect. scripts/optimizer/reembed_gallery.py does this from a reference gallery's cached source images without re-downloading anything.

    "},{"location":"gallery-scope/","title":"Whole gallery vs. cast-restricted gallery","text":"

    Two ways to run the matcher. Full mode scores every detected face against the entire 2418-actor gallery. Restricted mode pre-filters each film's gallery down to just its Jellyfin-credited cast (typically around 15 top-billed actors) before the matcher runs.

    "},{"location":"gallery-scope/#result","title":"Result","text":"

    Averaged across the 3 compared models (r50 excluded, see the full experiment log) and both expansion settings, on the 4 training films:

    scope F1 P R total misID full 71.1% 89.6% 59.6% 1121 restricted 75.9% 90.4% 65.6% 299

    Restriction improves every metric at once, not a precision/recall trade: +4.8pp F1, +6.0pp recall, roughly a quarter the total misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a lookalike false match, and the recall gain shows this does not cost real detections.

    Every model's best-scoring combo in the training matrix uses the restricted gallery:

    See the full experiment log for the complete table. One combo reaches zero true out-of-cast misidentifications, arcface_w600k_mbf_restricted_exp (F1 76.2%), and it is a restricted one, consistent with restriction, not expansion, being what suppresses cross-film confusions.

    The restriction effect (+4.8pp averaged across models) is larger than the model-choice effect: LVFace beats r18 by 6.2pp in full mode but beats mbf by 3.3pp. Restriction is the single strongest lever in the matrix.

    "},{"location":"gallery-scope/#why-this-is-not-the-shipped-default","title":"Why this is not the shipped default","text":"

    Cast restriction is implemented today only as an offline optimizer technique (scripts/optimizer/cast_restrict.py): it pre-builds a filtered gallery file per film using Jellyfin's cast list before the benchmark calls the matcher. There is no runtime \"restrict to this title's credited cast\" switch in the shipped application; scene_analyze always matches against whatever single gallery file it is given.

    Building this as a real feature requires:

    The shipped src/config.hpp defaults use the full-mode winner (LVFace-B_Glint360K_full_exp, F1 75.3% training, 67.4% held-out macro) rather than the higher-scoring restricted_exp (78.3%), because 78.3% describes a capability the application does not have yet.

    "},{"location":"lvface-deep-dive/","title":"Deep dive: LVFace-B Glint360K","text":"

    LVFace won the model comparison (see Which model is best?) and is the shipped default embedder. This page reports how it performs in detail: a baseline of correct output, the two mechanisms behind its errors, and every distinct case where it names someone who is not in the film's credited cast.

    Read How we score against X-Ray first. X-Ray's ground truth is scene-level, not per-frame. A name marked correct in the Offscreen column below is the pipeline correctly reporting scene membership, not a workaround.

    How to read the frames on this page

    The top of each image is the film frame, with a box and name on every face the pipeline matched to a real detection. The panels below are the per-second result against X-Ray. Onscreen lists names attached to a visible face this second. Offscreen lists names the pipeline reports present without a currently visible face. Colors mark the verdict: green correct (TPI), orange wrong (FPI), blue missed (FN).

    "},{"location":"lvface-deep-dive/#baseline-correctly-scored-seconds","title":"Baseline: correctly scored seconds","text":"

    Six faces on screen, all six named correctly, including Penelope Wilton at the edge of the pews and a partly occluded Michelle Dockery. Thirteen more cast members X-Ray lists as present in the scene are correctly reported Offscreen. One miss: Maggie Smith (blue). Score for this second: 0.86.

    The same film's funeral scene: dark clothing, hats, half the faces turned away. Nineteen of the twenty cast members X-Ray lists for this scene score correct: seven named on screen at up to 100% confidence, twelve more reported correctly as present but not visible.

    The pipeline does not require a live face. This is Herbie Hancock at 98% confidence, identified from a face displayed on a screen inside the film, on a video call under a science-fiction HUD overlay.

    "},{"location":"lvface-deep-dive/#training-vs-held-out-the-generalization-gap","title":"Training vs. held-out: the generalization gap","text":"

    The shipped config (prob_threshold=0.754, anneal_sec=35.54, extinction_sec=57.43, expand_gallery=true) was tuned on 4 films. Scored on the 5 films the optimizer never saw:

    film F1 P R TPI FPI misid FN Benny & Joon 83.0% 89.1% 77.7% 15125 1846 0 4337 Lovelace 77.5% 90.3% 67.9% 14990 1085 58 7085 Valerian and the City of a Thousand Planets 74.1% 97.1% 60.0% 18663 548 0 12467 Downton Abbey: A New Era 56.2% 97.8% 39.4% 52027 1173 0 80084 The Many Saints of Newark 46.3% 54.7% 40.1% 15922 4394 974 23791 macro average 67.4% 85.8% 57.0%

    The P column is misID-weighted (each out-of-film name counts 10x in the denominator; see methodology). That weighting is why Many Saints reads 54.7% here despite naming mostly real, present faces: its raw (unweighted) precision is 78.4%, and the gap is entirely its 974 misIDs paying the 10x penalty. The three zero-misID films (Benny & Joon, Downton, Valerian) have identical weighted and raw precision; Lovelace, with 58 misIDs, sits 3pp below its raw 93.3%.

    Held-out F1 is 67.4%, against 75.3% on training, an 8pp drop. The spread between the best and worst held-out film is 37pp. This is not unique to LVFace: the full experiment log shows mbf and r18 with the same shape of spread on the same films, at a uniformly lower level. Two mechanisms explain the spread. Both are shown below with frame-level evidence.

    "},{"location":"lvface-deep-dive/#mechanism-1-extinction-bridging","title":"Mechanism 1: extinction bridging","text":"

    The extinction window keeps a name reported as present for up to extinction_sec after its last real detection. This is deliberate: most gaps in face visibility are short (a turned head, an occlusion, a cut to a reaction shot), and the window bridges them.

    Lovelace's polygraph scene: only Eric Roberts and Amanda Seyfried have visible faces. X-Ray lists eight cast members present. All eight score correct; the other six are reported Offscreen through a stretch where the camera never shows them. The extinction window is why.

    The same mechanism fails at a hard cut into a long stretch with no faces at all. Downton Abbey's recall (39.4%, the worst of the five held-out films) is dominated by this failure. It is verified directly against the raw per-frame stream and the dump's own detection counts, not inferred from the score. Plotting the dump's per-second face_count (detector output, independent of the tracker) against what the tracker reports, through Downton Abbey's hard cut into its closing credits:

    From the cut onward the detector reports zero faces for close to a minute. The tracker continues reporting the previous shot's 15 identities for the same span (verified for Hugh Bonneville: bbox (1743.2, 0.0, 171.3, 317.8), unchanged to the pixel, at every sampled second for 57 seconds). The staircase at the right edge is the extinction window expiring, actor by actor. This is SceneTrackerFunc::active_[actor_idx].last_bbox (src/nodes/scene_tracker_node.hpp) re-emitted as designed. extinction_sec=57.4 was tuned long because bridging is correct on most footage, as in the polygraph scene above. The training films did not contain a faceless stretch long enough to expose the cost side; the held-out set did.

    The extinction window is a scoring concept, not something drawn on screen. The shipped output is presence windows with no bounding boxes. Even the debug overlay used for this report never draws a box for a bridged name: a name inside its extinction window with no current detection appears only as a name in the Offscreen column, the same as every correctly bridged name above.

    A related, smaller effect shows up at rapid cuts:

    Caf\u00e9 Society (a training film), a shot/reverse-shot dialog. The box on Steve Carell's face carries two labels: his own, and Jesse Eisenberg's, left over from the counter-shot a moment earlier. Both names score correct, because both actors are present in this scene per X-Ray. The box position is briefly wrong; the presence claim, which is what the pipeline ships, is right.

    "},{"location":"lvface-deep-dive/#mechanism-2-the-face-vs-presence-ceiling","title":"Mechanism 2: the face-vs-presence ceiling","text":"

    Downton Abbey's recall did not collapse because faces were misread. It collapsed because for most of its 80084 false-negative seconds there was no face to read.

    A newsreel crew moves equipment through the hall. X-Ray credits 22 cast members as present in this scene. None face the camera. Eight still score correct, carried by presence windows from adjacent shots. The other fourteen are missed, and no face-recognition system can recover them, because there is no face in the frame. X-Ray records scene membership; the pipeline measures visible faces. In ensemble scenes these two quantities diverge, and that gap accounts for most of the false-negative count.

    "},{"location":"lvface-deep-dive/#every-distinct-out-of-cast-name","title":"Every distinct out-of-cast name","text":"

    Many Saints of Newark has the largest misID count of any held-out film: 974 seconds, weighted. Rather than characterize this from a single frame, the raw replay stream was searched directly for every name the pipeline reports that is not in the film's credited cast. The same search was run on all 9 films in the benchmark, one rule applied uniformly: find the first second each distinct out-of-cast name appears, and render that exact second.

    Five films produce no such name anywhere in their runtime: Benny & Joon, Caf\u00e9 Society, Downton Abbey, Sound of Metal, Valerian. Zero out-of-cast names across their entire length. Four films produce nine distinct names between them, shown below in full, not a sample.

    "},{"location":"lvface-deep-dive/#the-many-saints-of-newark-4-names","title":"The Many Saints of Newark: 4 names","text":"

    Germar Terrell Gardner, t=848s, 78% confidence. A real, clearly visible background actor. He is not in X-Ray's cast list for this film, but he is credited in Jellyfin's independent cast metadata (see Where LVFace beat X-Ray below). This is a ground-truth gap, not a model error.

    Archie Yates, t=2521s, 78% confidence. A real detected face, a genuine lookalike confusion.

    Zooey Deschanel, t=2819s, 99% confidence. A real detected face at a dinner table, high-confidence lookalike confusion.

    Talia Balsam, t=4551s, 93% confidence. A real detected face. Talia Balsam plays Mrs. Jarecki, a guidance counselor, in this film; she is confirmed on screen by direct inspection of the frame. She does not appear in X-Ray's people.csv for this title. This is a second ground-truth gap in the same film, not a model error.

    Two of these four names are ground-truth gaps (Gardner, Balsam), not misidentifications. The other two (Yates, Deschanel) are genuine embedding errors on real faces.

    "},{"location":"lvface-deep-dive/#lord-of-war-3-names","title":"Lord of War: 3 names","text":"

    David Shumbris, t=418s, 81% confidence. A real face in a dim, low-detail shot under a train track. A genuine lookalike confusion in poor lighting.

    Ronald Reagan, t=1003s, 100% confidence. This is not a lookalike confusion. The detected face is a photograph of Reagan appearing within the shot, not a living actor. The detector and matcher both did their job correctly on the image content in front of them; the error is that a photograph inside the scene is not the same thing as an actor present in the scene, and the pipeline has no way to draw that distinction from a face crop alone.

    Lance Reddick, t=6424s, 78% confidence. A small, distant, low-detail face at the edge of frame. A marginal, low-confidence lookalike confusion.

    "},{"location":"lvface-deep-dive/#lovelace-1-name","title":"Lovelace: 1 name","text":"

    Chlo\u00eb Sevigny, t=2451s, 100% confidence. Two boxes are drawn on the same face: one correctly labeled Amanda Seyfried, one incorrectly labeled Chlo\u00eb Sevigny, both at 100%. A single detection producing two competing high- confidence identities on the same crop.

    "},{"location":"lvface-deep-dive/#scarface-1-name","title":"Scarface: 1 name","text":"

    Kirstie Alley, t=2451s, 89% confidence. Al Pacino is correctly identified in the foreground at 100%; a background face in the same shot is wrongly labeled Kirstie Alley. (The t=2451s here and the Lovelace Chlo\u00eb Sevigny case above landing on the identical second is a genuine coincidence, verified from each film's raw stream by first_fpi_frames.py, not a transcription slip, two unrelated films whose first out-of-cast name happens to fall at the same timestamp.)

    "},{"location":"lvface-deep-dive/#summary-of-the-nine","title":"Summary of the nine","text":"film name t (s) confidence classification Many Saints of Newark Germar Terrell Gardner 848 78% ground-truth gap Many Saints of Newark Archie Yates 2521 78% lookalike confusion Many Saints of Newark Zooey Deschanel 2819 99% lookalike confusion Many Saints of Newark Talia Balsam 4551 93% ground-truth gap Lord of War David Shumbris 418 81% lookalike confusion Lord of War Ronald Reagan 1003 100% photo-in-frame Lord of War Lance Reddick 6424 78% lookalike confusion, marginal Lovelace Chlo\u00eb Sevigny 2451 100% lookalike confusion Scarface Kirstie Alley 2451 89% lookalike confusion

    Of nine distinct out-of-cast names across four films, two are ground-truth gaps, one is a photograph misread as a person, and six are genuine embedding-space confusions on real detected faces. None trace to extinction bridging: every one of these nine is a fresh detection on a real face crop at the second it first appears.

    "},{"location":"lvface-deep-dive/#where-lvface-beat-x-ray","title":"Where LVFace beat X-Ray","text":"

    Not every name marked wrong is actually wrong. scripts/optimizer/second_score.py scores strictly against X-Ray, and X-Ray has gaps of its own.

    Germar Terrell Gardner, the same name from the table above, does not appear in X-Ray's people.csv for The Many Saints of Newark. Jellyfin's independent cast metadata does credit him for this film (cross-checked against experiments/manifests/jellyfin_casts.json from the experiment-data artifact package, a data source entirely separate from X-Ray). Talia Balsam is the same case: confirmed on screen, absent from X-Ray's cast list for this title.

    This extends past uncredited background actors. This is Robert Patrick, top-billed in Lovelace, clearly on screen reading a newspaper, identified at 100%. The frame is scored wrong because X-Ray's people-in-scene list for this specific scene omits him, despite crediting him elsewhere in the film. The identification is correct; the ground truth is missing an entry.

    X-Ray is a large, convenient ground truth. It is not a complete one. The misID and FPI counts reported throughout this document include some fixed amount of noise from gaps in X-Ray itself, in both directions.

    "},{"location":"lvface-deep-dive/#summary","title":"Summary","text":"

    LVFace wins the model comparison on every held-out film. It correctly names 19 of 20 people in a crowded funeral scene and correctly identifies a face displayed on a screen inside the film. Its errors resolve into two mechanisms: extinction bridging, which is correct on most footage and fails specifically at hard cuts into long faceless stretches, and the face-versus-presence ceiling, where X-Ray credits scene membership for people whose faces never appear on screen. Of the nine distinct out-of-cast identifications found across the benchmark, two trace to gaps in X-Ray's own cast data, one is a photograph misread as a person, and six are genuine lookalike confusions on real faces. The held-out generalization gap, 75.3% training to 67.4% held-out, is real and should be treated as the expected operating point, not the training-set figure.

    "},{"location":"methodology/","title":"How we score against X-Ray","text":"

    Every number in this report, every F1 and misID count, comes from one comparison. The comparison has a mismatch at its core that shapes nearly every finding in this report: the ground truth is scene-level, the pipeline's output is per-second, and the two do not mean the same thing. This page documents that comparison once, so the findings pages can rely on it without re-explaining it.

    "},{"location":"methodology/#what-amazon-x-ray-records","title":"What Amazon X-Ray records","text":"

    X-Ray ships three tables per film: scenes.csv (a list of [start, end] timespans), people_in_scenes.csv (which actors are credited in each scene), and people.csv (actor identities). There is no per-frame or per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and X-Ray records one cast list for the entire span, not \"on screen from second 12 to second 30.\"

    To compare this against per-second predictions, second_score.py expands every scene into per-second ground truth by copying the whole scene's cast list onto every second inside it:

    for sn, (t0, t1) in spans.items():\n    cast = scene_cast.get(sn, [])\n    for t in range(int(t0), int(t1)):\n        timeline[t] = cast\n

    That is the entire mechanism. If X-Ray credits five actors to a 30-second scene, all five count as ground truth present for all 30 seconds, including seconds where only one of them is on screen. This is not a simplification introduced by the pipeline; it is the only reading of X-Ray's data that is possible, because X-Ray itself does not record anything finer-grained.

    "},{"location":"methodology/#why-an-offscreen-name-can-be-scored-correct","title":"Why an offscreen name can be scored correct","text":"

    A name listed under Offscreen with a correct (green) label is not the pipeline guessing or padding its score. It is the pipeline correctly answering the question X-Ray actually asks: is this actor part of this scene. It answers that question using a presence window ([start, end], held open across cuts by anneal_sec and extinction_sec), which matches X-Ray's scene-level semantics more closely than a raw per-frame detection would.

    A system that only reported \"this actor is visible in this exact frame\" would score worse against X-Ray's scene-level ground truth, producing a false negative every time the camera cuts away from a character who is still present in the scene. Not because it is wrong about the world, but because it would be answering a stricter, different question than the one X-Ray's data supports. The presence-window design exists specifically to answer X-Ray's actual question.

    "},{"location":"methodology/#what-this-resolves-and-what-it-does-not","title":"What this resolves and what it does not","text":"

    This resolves the semantic mismatch between a scene and an instant. It does not resolve two other limitations, both discussed in the LVFace deep dive.

    The face-vs-presence ceiling. X-Ray credits scene membership regardless of whether a face is ever visible: background crew, characters shot from behind, voice-only presence. No amount of bridging recovers a face that never appears on screen. This is a hard ceiling on recall, not a defect.

    Extinction bridging can overshoot. The same presence-window mechanism that correctly answers \"still in this scene\" during a normal cut can also bridge across a scene boundary it has no way to detect. A hard cut into a different scene with no faces, such as closing credits, carries the previous scene's identities forward until the window expires. This is the mechanism behind Downton Abbey's recall collapse, documented in the deep dive.

    "},{"location":"methodology/#precision-recall-and-the-misid-weighting","title":"Precision, recall, and the misID weighting","text":"

    Per sampled second t:

    TPI (true positive instances): actors both X-Ray and the pipeline agree are present.

    FPI (false positive instances): actors the pipeline reports that are not in X-Ray's cast for this second. Split into two categories:

    Every headline P and F1 is misID-weighted

    The precision reported throughout this report, and therefore the F1 derived from it, puts each FPI_misid into the denominator 10 times (precision = TPI / (TPI + FPI_incast + 10\u00b7FPI_misid), second_score.py). This is deliberate: the whole point is to punish naming an out-of-film actor far harder than a scene-boundary slip. But it means the P column is not raw precision, and a misID-heavy film's P is depressed super-linearly. second_score.py also emits an unweighted precision_raw (always \u2265 the weighted P); where the gap matters, The Many Saints of Newark, weighted P 54.7% vs. raw 78.4%, the LVFace deep dive reports both. When comparing P across films, remember you are comparing a quantity that penalizes misIDs, not just a hit rate.

    FN (false negatives): actors X-Ray lists that the pipeline never reports, counted only for actors who have a gallery reference embedding. Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from 20% to 79% by film (see the full experiment log); an actor with no reference photo can never be recognized regardless of model quality, and counting them as a miss would penalize gallery coverage, not recognition accuracy.

    Two further numbers are reported alongside F1:

    agreement_rate: mean per-second Jaccard overlap (|Pred \u2229 GT| / |Pred \u222a GT|), partial credit. Naming 2 of 3 present actors scores 2/3, not 0.

    exact_match_rate: the fraction of sampled seconds where the pipeline's named set exactly equals X-Ray's, no partial credit. Far harsher, and dominated by recall, since any single missed actor zeroes that second.

    "},{"location":"methodology/#reproduce","title":"Reproduce","text":"
    python3 scripts/optimizer/second_score.py \\\n    --pred pred.json --xray experiments/xray/.../<xray_dir> \\\n    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5\n

    See also the full experiment log for how pred.json is produced, and the LVFace deep dive for what these mechanisms look like frame by frame.

    "},{"location":"model-bakeoff/","title":"Full experiment log","text":"

    This page reports how the pipeline performs across three questions: which embedding model is best, whether restricting the gallery to a film's credited cast helps, and whether promoting confidently identified poses into a per-film gallery annex helps. It also documents the replay architecture that made testing all three questions in one pass practical, and every caveat needed to trust the numbers.

    Read How we score against X-Ray first for what F1, precision, recall, and misID mean in this report. All numbers below use the per-second metric (scripts/optimizer/second_score.py).

    r50 (ArcFace w600k-R50) is excluded from the detailed comparison below. Its gallery was built with roughly 30% fewer reference images per actor than the other three models on the identical source photos (10808 vs 15055 total embeddings across the same 2418 actors), which confounds any direct comparison of its scores against the others. It remains in the calibration curve comparison, which does not depend on the training benchmark.

    "},{"location":"model-bakeoff/#why-replay-makes-this-affordable","title":"Why replay makes this affordable","text":"

    Decoding video and running face detection, alignment, and embedding is the expensive part of this pipeline. Everything downstream of that (tracking, identity matching, scene aggregation) is cheap. KPN++'s node/network structure means those two stages are separate components connected by typed channels, so the expensive stage can run once per film, cache its output, and the cheap stage can be re-run against that cache as many times as needed with different Config values.

    scene_analyze --dump-embeddings out.h5 runs the expensive half once per film and writes per-frame face detections and embeddings to HDF5 (scripts/optimizer/SCHEMA.md). scripts/optimizer/replay.py then re-assembles the real C++ face_tracker, identity_matcher, and scene_tracker nodes into a Python-driven KPN network and replays a film's cached embeddings through them, varying prob_threshold, anneal_sec, extinction_sec, and expand_gallery freely. No GPU inference and no video decode happen during a replay; each one completes in seconds. This is what makes a 512-evaluation differential-evolution search per model, per gallery mode, per expansion setting, tractable, and what made the full held-out validation across three models in this report possible in one session rather than requiring three full re-encodes of the benchmark set.

    optimize.py runs differential_evolution over this replay function as its objective, with DE-level parallelism (multiple candidate configs evaluated concurrently, each spawning its own replay subprocesses) on top of it. The practical ceiling on this machine's GPU was 8 concurrent replay processes; 9 silently degraded every score to 0.0% (well-formed output, wrong numbers, not a crash), so optimize.py was run at REPLAY_WORKERS=4 DE_WORKERS=2.

    "},{"location":"model-bakeoff/#search-space","title":"Search space","text":"

    popsize=10, maxiter=15 per combo (3 parameters, up to 512 evaluations, usually stopping earlier on DE's convergence tolerance). anneal_sec/extinction_sec bounds were widened from 1-30/1-15 to 1-60/1-60 partway through the sweep. r50's 4 combos finished before the widening and used the old, narrower bounds; this is one more reason r50 is excluded from direct comparison here.

    "},{"location":"model-bakeoff/#training-films-and-held-out-films","title":"Training films and held-out films","text":"

    9 films have dumped embeddings across all 4 models. 4 were used for optimization:

    5 were held out, never seen by any optimizer run:

    "},{"location":"model-bakeoff/#gallery-coverage-per-film","title":"Gallery coverage per film","text":"

    The gallery has reference embeddings for 2418 actors, but coverage of any given film's credited cast varies widely. This was previously reported as one flat number (67% of X-Ray cast lacking a reference embedding, averaged across the whole benchmark); the per-film breakdown is:

    film cast credited in gallery coverage Lord of War 64 13 20.3% Scarface 67 15 22.4% The Many Saints of Newark 48 13 27.1% Caf\u00e9 Society 62 17 27.4% Lovelace 42 15 35.7% Valerian and the City of a Thousand Planets 36 13 36.1% Benny & Joon 23 12 52.2% Downton Abbey: A New Era 36 22 61.1% Sound of Metal 14 11 78.6%

    Two training films (Lord of War, Scarface) have the worst coverage in the set, 20-22%. Their training-set F1 numbers below are partly capped by missing references, not purely by model quality. Downton Abbey has 61% coverage, the second-best in the benchmark, yet the worst held-out recall of any film (39.4%, LVFace). Its recall problem is not primarily a coverage problem; it is the extinction-bridging failure documented in the LVFace deep dive. Reproduce with scripts/docs/gallery_coverage_per_film.py.

    "},{"location":"model-bakeoff/#training-results-3-models-2-gallery-modes-2-expansion-settings","title":"Training results, 3 models \u00d7 2 gallery modes \u00d7 2 expansion settings","text":"

    Ranked by F1. misid = FPI_misid, the count of true wrong-actor identifications (naming someone not in the film's cast at all), distinct from FPI, which also includes in-cast timing slips.

    Each combo's row is its best full-coverage evaluation: the highest-F1 DE evaluation in which all 4 training films replayed without a timeout (see Dropped-film scoring below for why this qualifier is load-bearing and not the same as argmax F1 over the raw sweep).

    combo F1 P R TPI FPI misid FN LVFace-B_Glint360K_restricted_exp 78.3% 91.0% 68.9% 42830 3782 60 19492 LVFace-B_Glint360K_restricted_noexp 76.7% 91.5% 66.2% 41149 3400 59 21173 arcface_w600k_mbf_restricted_exp 76.2% 90.0% 66.2% 64328 7480 0 33234 arcface_r18_restricted_exp 75.5% 87.6% 66.5% 41399 5666 60 20923 LVFace-B_Glint360K_full_exp 75.3% 89.7% 65.4% 47757 3407 232 26966 arcface_w600k_mbf_restricted_noexp 75.0% 91.1% 63.9% 39752 3465 60 22570 arcface_r18_restricted_noexp 73.5% 91.3% 61.7% 38299 3220 60 24023 LVFace-B_Glint360K_full_noexp 72.3% 88.3% 61.8% 40363 3503 244 25850 arcface_w600k_mbf_full_exp 72.0% 87.7% 61.4% 39875 3729 240 26338 arcface_w600k_mbf_full_noexp 71.0% 93.2% 57.9% 41699 2472 56 33024 arcface_r18_full_exp 69.1% 87.6% 57.7% 37342 3119 242 28871 arcface_r18_full_noexp 66.6% 91.3% 53.1% 34314 2362 107 31899

    The two clearest patterns: every model's best-scoring combo uses the restricted gallery, and LVFace leads within both gallery modes. full_exp (the shipped combination) is the best-scoring option that uses only features the running application currently supports; restriction is not wired into the application yet (see Whole vs. cast-restricted gallery).

    "},{"location":"model-bakeoff/#a-scoring-bug-worth-recording-dropped-film-evaluations","title":"A scoring bug worth recording: dropped-film evaluations","text":"

    The numbers above are corrected ones. The raw rep4_best_*.json files, and an earlier version of this table, reported a different arcface_w600k_mbf_full_noexp row: 74.2% F1 at TPI 12645, a third the TPI of every sibling combo. That was not a better config; it was an artifact of how the optimizer aggregates.

    optimize.py builds each candidate's score from only the films whose replay subprocess returned (per_film = [m for m in ex.map(_one, films) if m is not None]), then averages F1/precision/recall and sums TPI/FPI/misID over just those survivors. When a film's replay times out (the sweep ran near the 8-process concurrency ceiling, so this happened intermittently), that film silently drops from both. A candidate whose hardest film timed out is therefore scored on an easier subset, and differential evolution, maximizing that score, will happily converge onto exactly such a candidate. For mbf_full_noexp the reported winner was one of 7 evaluations (out of 512) whose TPI had collapsed to a partial-film subset; its median-coverage evaluations sit around 51686 TPI.

    The fix here was to re-derive each combo's best row from its DE trajectory (experiments/trajectories/rep4_*.jsonl), keeping only evaluations within 30% of that combo's median TPI (full 4-film coverage) before taking the best F1. This needs no re-running, the honest best configuration was already in the sweep, just not the one argmax F1 selected. Three combos moved: mbf_full_noexp 74.2% \u2192 71.0%, LVFace_full_noexp 72.4% \u2192 72.3% (and its misID, 0 \u2192 244, was itself a dropped-film artifact), mbf_restricted_exp 76.5% \u2192 76.2%. The shipped LVFace full_exp winner was unaffected, its reported evaluation already had full coverage (TPI 47757 \u2248 median). experiment_charts.py applies the same clean_best filter, so every figure on this page matches the corrected table. The underlying optimize.py aggregation is also being fixed so a dropped-film evaluation can never be selected as a winner again.

    "},{"location":"model-bakeoff/#per-film-training-breakdown","title":"Per-film training breakdown","text":"

    The 75.3% LVFace training figure is a macro average across 4 films, not a uniform result:

    film LVFace F1 mbf F1 r18 F1 best model Caf\u00e9 Society 68.1% 62.2% 60.1% LVFace Lord of War 75.6% 77.2% 75.6% mbf Scarface 71.5% 68.6% 64.1% LVFace Sound of Metal 78.8% 76.5% 71.6% LVFace

    LVFace does not win every training film. mbf scores higher on Lord of War (77.2% vs 75.6%). LVFace's own training-film range is 68.1% to 78.8%, a 10.7pp spread, smaller than the 37pp spread seen on held-out films but real. Reproduce with scripts/docs/run_holdout_all_models.py --films training.

    "},{"location":"model-bakeoff/#held-out-validation-all-3-models","title":"Held-out validation, all 3 models","text":"

    The training matrix above is training-set fit. Each model's own tuned full_exp config was replayed against the 5 held-out films, scored the same way:

    film LVFace F1 mbf F1 r18 F1 Benny & Joon 83.0% 78.5% 77.1% Lovelace 77.5% 73.7% 72.2% Valerian and the City of a Thousand Planets 74.1% 70.2% 71.0% Downton Abbey: A New Era 56.2% 55.0% 53.0% The Many Saints of Newark 46.3% 44.5% 42.1% macro average 67.4% 64.4% 63.1%

    LVFace scores highest on every one of the 5 held-out films; the ranking never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18 1224. LVFace has less than half mbf's misID count while also scoring higher on every film. This directly confirms the model choice out of sample; it is not inferred from the training numbers alone. See the LVFace deep dive for frame-level detail on where and why LVFace still fails on the two worst films. Reproduce with scripts/docs/run_holdout_all_models.py.

    "},{"location":"model-bakeoff/#two-effects-in-isolation-gallery-scope-and-pose-expansion","title":"Two effects in isolation: gallery scope and pose expansion","text":"

    Averaging across the 3 compared models (r50 excluded) isolates each variable from model choice.

    Gallery scope, averaged over both expansion settings and all 3 models (6 evaluations per row):

    scope F1 P R total misID full 71.1% 89.6% 59.6% 1121 restricted 75.9% 90.4% 65.6% 299

    Restriction improves every metric at once. This is not a precision/recall trade: +4.8pp F1, +6.0pp recall, and roughly a quarter the misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a lookalike false match, and the recall gain shows this does not cost real detections. Restriction is currently an offline optimizer technique, not a runtime feature of the application; see Whole vs. cast-restricted gallery for what building it into the application would require.

    Pose expansion (promoting a confidently identified track's novel-pose views into a per-film gallery annex, src/gallery/track_gallery.hpp):

    scope expansion F1 R misID full off 70.0% 57.6% 407 full on 72.1% 61.5% 714 restricted off 75.1% 63.9% 179 restricted on 76.7% 67.2% 120

    In restricted mode, expansion is a clean win: +1.6pp F1, +3.3pp recall, misID drops. The annex only competes against the film's own roughly 15-actor cast, so a new pose of a known actor is unlikely to be confused with someone else. In full mode, expansion buys +2.1pp F1 and +3.9pp recall but at a real cost: misID rises from 407 to 714 as the same new-pose view now competes against the full 2418-actor gallery, where a confidently learned pose is more likely to match the wrong person. On the full gallery it is a recall-vs-misID trade, not a free gain. This training-set effect did not reproduce on held-out data; see Does pose expansion help? for the full held-out test and the two methodology bugs caught while checking it.

    "},{"location":"model-bakeoff/#calibration-curves","title":"Calibration curves","text":"

    Each gallery carries a fitted Platt sigmoid P(match | sim) = \u03c3(a\u00b7sim + b), stored directly in the gallery HDF5 (src/gallery/gallery_calibration.hpp). This measures discriminative power independent of whatever prob_threshold a given run used:

    LVFace has the steepest curve (a=17.7 vs 15.3-16.2 for the ArcFace variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs 0.27-0.31), separating same-actor from different-actor pairs more confidently at a lower similarity than any ArcFace variant tested, including r50. Generated by scripts/docs/calibration_chart.py.

    "},{"location":"model-bakeoff/#extinction-and-anneal-window-search","title":"Extinction and anneal window search","text":"

    Every one of the 512 DE evaluations for the winning LVFace full_exp combo, plotted over the prob_threshold \u00d7 extinction_sec plane:

    Nearly everything scoring well sits at extinction_sec above 50, across a wide range of thresholds. Short extinction windows are uniformly weaker: under a strict threshold, there is no good configuration in that region of the search space. The optimizer converged with anneal_sec=59.2, extinction_sec=59.2, about 99% of the widened 60s bound, which raises an open question not resolved in this round: does performance keep improving past 60s, or does it plateau there. Not chased further this pass.

    "},{"location":"model-bakeoff/#caveats","title":"Caveats","text":""},{"location":"model-bakeoff/#reproduce","title":"Reproduce","text":"
    # 4-film training matrix, all 4 models \u00d7 2 gallery modes \u00d7 2 expansion settings\nbash experiments/run_rep4_subprocess.sh\n\n# single combo\nSAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \\\n    --manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \\\n    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \\\n    --params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \\\n    --popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json\n\n# held-out validation, all 3 models, 5 films\npython3 scripts/docs/run_holdout_all_models.py --out docs_data/holdout_all_models.json\n\n# per-film training breakdown, all 3 models, 4 films\npython3 scripts/docs/run_holdout_all_models.py --films training --out docs_data/training_per_film.json\n\n# gallery coverage per film\npython3 scripts/docs/gallery_coverage_per_film.py --out docs_data/gallery_coverage_per_film.json\n\n# regenerate this page's charts from experiments/ artifacts\npython3 scripts/docs/experiment_charts.py --out-dir docs/assets/images\n\n# one frame per distinct out-of-cast name across all 9 films (used in the deep dive)\npython3 scripts/docs/first_fpi_frames.py\n

    See also the session log experiments/SESSION_STATE.md.

    "},{"location":"pose-expansion/","title":"Pose expansion: does promoting new poses mid-film help?","text":"

    expand_gallery (src/gallery/track_gallery.hpp) promotes a confidently identified track's novel-pose reference views into a per-film, in-memory gallery annex. The idea: once the pipeline is confident about an identity, a pose it has not seen before (turned head, different lighting) becomes an extra reference for recognizing that actor again later in the same film, without touching the baked gallery.

    "},{"location":"pose-expansion/#training-set-signal","title":"Training-set signal","text":"

    Averaged across the 3 compared models (r50 excluded), on the 4 films used for optimization. These are the corrected, full-coverage figures, see the dropped-film note in the experiment log for why an earlier version of this table overstated the full-mode misID jump (209 \u2192 864) that was itself partly a truncation artifact:

    scope expansion F1 R misID full off 70.0% 57.6% 407 full on 72.1% 61.5% 714 restricted off 75.1% 63.9% 179 restricted on 76.7% 67.2% 120

    In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp recall, lower misID. In full mode it looks like a recall-for-misID trade: +2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See the full experiment log for the per-model breakdown. This asymmetry motivated the question below: does turning expansion on change what gets recognized frame by frame, or is the aggregate F1 shift coming from something else.

    "},{"location":"pose-expansion/#held-out-test","title":"Held-out test","text":"

    Same model, same tuned config, expand_gallery toggled on vs. off, nothing else changed, full gallery mode, per-second scoring against X-Ray. This isolates expansion from every other variable that differs between the training-set rows above.

    LVFace-B Glint360K, all 5 held-out films:

    film F1 (exp) F1 (noexp) TPI delta FN delta Benny & Joon 83.0% 83.0% -2 +2 Downton Abbey: A New Era 56.1% 56.2% -7 +7 Lovelace 77.5% 77.4% +33 -33 The Many Saints of Newark 46.3% 46.3% +2 -2 Valerian and the City of a Thousand Planets 74.1% 74.1% +2 -2

    ArcFace R18, Benny & Joon, r18's own tuned config: F1 77.1% for both, TPI and FN identical, FPI differs by 2.

    Every film, both models tested: F1 differs by 0.1-0.2pp, TPI/FN swings are in the tens out of tens of thousands. This is noise, not a signal. Expansion made no measurable difference to per-second on-screen identification on any held-out film tested.

    "},{"location":"pose-expansion/#two-methodology-bugs-caught-during-this-check","title":"Two methodology bugs caught during this check","text":"

    Getting to the table above required catching two wrong turns, both worth recording because they are exactly the kind of error that produces a false positive \"expansion helped\" finding.

    1. Timeout truncation. The first Downton Abbey exp replay was cut off by a 60-second subprocess timeout at about 76% through the film (5589 of 7368 expected seconds). This silent data loss produced a large, convincing-looking TPI gap (47938 vs 52032) purely because one run was missing a quarter of the film. Caught by comparing n_seconds between runs before trusting any score delta; fixed by re-running with a longer timeout.
    2. Bbox-matching bug. An early per-second raw-annotation diff matched each exp detection to the first noexp detection with IoU above 0.5, not the best-overlapping one. With 3 faces close together in frame, this produced spurious disagreements (for example \"exp says Aidan Quinn, noexp says Johnny Depp\" at the same second) that vanished once the match used the best-IoU candidate instead of the first one. Both configs had actually output the same three names at the same three boxes.

    Both bugs independently pointed toward \"expansion is doing something,\" and both were artifacts of the comparison harness, not the pipeline. Before trusting a dramatic before/after diff, check that both runs cover the same seconds and that entities are matched by best overlap, not first found.

    "},{"location":"pose-expansion/#conclusion","title":"Conclusion","text":"

    The training-set aggregate effect, particularly the full-mode misID increase, does not reproduce on held-out data. At minimum it is far smaller than the training-set numbers suggested; it may be sampling variation from only 4 training films rather than a generalizable mechanism. Note the same class of harness bug appears twice in this investigation, the timeout truncation in bug #1 above, and the dropped-film aggregation that inflated the raw training-set misID figures. Both make an inert config look consequential; both are reasons to distrust a dramatic training-set delta until it survives on held-out films, which this one did not. This does not mean expand_gallery never does anything: the mechanism is real, and track_gallery.hpp's promotion logging confirms tracks get confirmed and views get promoted into the annex on every film tested. It means whatever effect expansion has on final per-second identification was too small to detect against 5 held-out films with this scoring method. A cleaner test would need either more held-out films or a metric that can see the annex's direct contribution, such as tagging which reference embedding won each match; neither was in scope for this pass.

    Do not treat the training-set exp/noexp numbers in the full experiment log as proof that expansion changes real-world behavior in either direction. On the evidence gathered so far, it does not move the needle enough to see.

    "},{"location":"service-conversion/","title":"Conversion to service \u2014 a native idle-GPU worker","text":"

    Status: design / proposal. Nothing here is built yet.

    "},{"location":"service-conversion/#the-idea","title":"The idea","text":"

    Turn the CLI tools into a turnkey batch worker that uses the machine's idle GPU: it analyses newly-added Jellyfin media when you're not using the computer (screen locked), and stops the instant you come back. It's an overnight job on your own Linux box.

    No Docker. This runs on your own machine with your own drivers, so a container buys little and costs a lot: GPU passthrough (nvidia-container-toolkit, or /dev/kfd+/dev/dri+video group for ROCm) is the single most fragile part of a containerised setup, and it exists only because of the container. Natively, the GPU just works with the drivers you already have, and the media paths Jellyfin reports are just real paths \u2014 no re-mounting. So we ship a native installer instead of an image builder.

    Two deliverables:

    1. An installer \u2014 scripts/build_install.py. Detects your distro, ensures the GPU/build dependencies are present (via dnf/pacman), compiles scene_analyze for your GPU, and installs the binary + Python glue + two systemd user units under ~/.local.
    2. A screen-lock gate \u2014 one of those systemd units watches logind lock/unlock and starts/stops the worker. Lock \u2192 analyse. Unlock \u2192 stop.
    "},{"location":"service-conversion/#what-already-exists-reuse-dont-rebuild","title":"What already exists (reuse, don't rebuild)","text":"

    The processing loop is already implemented \u2014 this is packaging, building, and lock-gating, not new pipeline logic.

    Piece Where What it does Analysis engine build/scene_analyze Video \u2192 face detect/align/embed \u2192 gallery match \u2192 result JSON Backend selection CMakeLists.txt (SAE_INFERENCE_BACKEND, SAE_GEMM_BACKEND) ORT/TRT + ROCm/CUDA, chosen at build time New-media queue JRay plugin \u2192 GET /Plugins/JRay/Tasks/Pending Backlog of items with no results yet Worker loop scripts/run_from_jellyfin.py--worker Poll Pending \u2192 run scene_analyze \u2192 push results Result push PUT /Plugins/JRay/Items/{id}/Truth Stores per-actor scene windows back in Jellyfin Incremental gallery scripts/make_jellyfin_gallery.py--merge Embeds only cast not already in the gallery Secrets loader .env via scripts/sae_env.pyJELLYFIN_URL, JELLYFIN_API_KEY, TMDB_API_KEY"},{"location":"service-conversion/#installer-config","title":"Installer config","text":"

    One file. Build-time settings (fixed when we compile) vs. run-time settings (in the worker's .env, editable without recompiling).

    # install.yaml \u2014 consumed by scripts/build_install.py\n\nplatform: nvidia            # nvidia | amd | cpu \u2192 picks the cmake backend\nmodel:\n  arcface: LVFace-B_Glint360K.onnx   # embedder compiled against; gallery MUST match\nschedule:\n  gallery_scan_interval: 24h         # incremental --merge cadence; 0 disables the scanner\nprefix: ~/.local                     # install root (bin, share, systemd user units)\n\n# runtime (written to the worker .env, not compiled in):\nruntime:\n  jellyfin_url: http://localhost:8096\n  # JELLYFIN_API_KEY / TMDB_API_KEY are filled into .env by hand after install\n

    Secrets never go in the repo or a build artifact \u2014 the installer writes a .env under the install prefix with blanks for the keys, and you fill them in once. sae_env.py already loads it.

    Model \u21c4 gallery coupling (guard, don't just document): embeddings from different recognition models aren't interchangeable. We compile against one embedder; the gallery must be built with the same one. Stamp the embedder name into gallery.json, and have the worker refuse to start if the gallery's embedder \u2260 the configured model.arcface, rather than silently mismatching.

    "},{"location":"service-conversion/#dependencies-via-the-system-package-manager","title":"Dependencies via the system package manager","text":"

    The heavy build/runtime deps (OpenCV, ffmpeg, the GPU stack) are best provided by the distro, not vendored. The installer ships a per-distro dependency list and either installs them or prints the exact command. Targets: Fedora (dnf) and Arch (pacman) first.

    Dependency Fedora (dnf) Arch (pacman) OpenCV opencv-developencv ffmpeg ffmpeg-free/ffmpeg (RPM Fusion) ffmpeg CMake / toolchain cmake gcc-c++cmake gcc CUDA + TensorRT (nvidia) NVIDIA CUDA repo + libnvinfer-*cuda, tensorrt ROCm (amd) rocm-hip-sdk / rocblas-develrocm-hip-sdk, rocblas ONNX Runtime not packaged \u2014 installer fetches a pinned release tarball into the prefix AUR onnxruntime (or same pinned-tarball fallback)

    So the flow is: detect distro \u2192 check each package \u2192 install via the native manager (or print sudo dnf install \u2026 / sudo pacman -S \u2026), with ONNX Runtime as the one known gap the installer fills itself (a pinned upstream release extracted under the install prefix, so it doesn't depend on a system package that may not exist). CUDA/ROCm being present is assumed \u2014 you already run a GPU desktop; the installer verifies and points you at the vendor repo if not.

    "},{"location":"service-conversion/#what-build_installpy-does","title":"What build_install.py does","text":"
    build_install.py install.yaml\n  \u2502\n  \u251c\u2500 detect distro (dnf vs pacman) and platform from config\n  \u251c\u2500 ensure deps: install via manager, or print the exact command; fetch ONNX Runtime if needed\n  \u251c\u2500 cmake + build scene_analyze with the platform's backend flags:\n  \u2502     nvidia \u2192 -DSAE_INFERENCE_BACKEND=TRT -DSAE_GEMM_BACKEND=CUDA\n  \u2502     amd    \u2192 -DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=ROCM\n  \u2502     cpu    \u2192 -DSAE_INFERENCE_BACKEND=ORT  (CPU EP; slow, for smoke tests)\n  \u251c\u2500 install into <prefix>:\n  \u2502     bin/sae-scene-analyze           the compiled binary\n  \u2502     share/sae-worker/               Python glue + a venv (requests, etc.), models/\n  \u2502     share/sae-worker/.env           runtime config (keys blank, url from config)\n  \u251c\u2500 install systemd --user units:\n  \u2502     sae-worker.service              runs the worker + gallery-scan supervisor\n  \u2502     sae-lock-gate.service           watches logind lock/unlock, start/stops the worker\n  \u2514\u2500 print next steps (edit .env, `systemctl --user enable --now sae-lock-gate`)\n
    "},{"location":"service-conversion/#the-worker-service-supervisor","title":"The worker service (supervisor)","text":"

    sae-worker.service runs a small Python supervisor as its main process:

    "},{"location":"service-conversion/#the-lock-gate","title":"The lock gate","text":"

    sae-lock-gate.service runs a tiny watcher that subscribes to logind lock/unlock signals and drives the worker service:

    screen locks   \u2192 systemctl --user start sae-worker.service\nscreen unlocks \u2192 systemctl --user stop  sae-worker.service   (SIGTERM)\n

    Screen-lock is the only signal \u2014 deliberately. We don't also gate on GPU/CPU load, because our own worker is the load: a load threshold would form a feedback loop (worker starts \u2192 GPU spikes \u2192 threshold trips \u2192 worker stops \u2192 load drops \u2192 restart \u2192 \u2026). Lock state is external to what the worker does, so it can't oscillate.

    Signal source is desktop-dependent: logind Lock/Unlock (GNOME/KDE via loginctl/D-Bus) covers most setups; a swayidle/xss-lock hook is the fallback for wlroots/X-only compositors. The installer picks based on what's present.

    "},{"location":"service-conversion/#on-resume-hard-stop-re-queue-its-free","title":"On resume: hard stop + re-queue (it's free)","text":"

    Stopping the worker mid-analysis costs nothing to reschedule, because of how the JRay queue works: an item only leaves /Tasks/Pending once its results are pushed (push_truth). A worker stopped mid-scene_analyze simply leaves that item Pending \u2014 next lock picks it up again. No re-queue bookkeeping.

    Two small correctness requirements (the only worker changes needed):

    1. Never push a partial result. Already true \u2014 push_truth runs only after scene_analyze returns; a killed run pushes nothing. \u2713 (keep it that way).
    2. Clean up on signal. process_item writes a temp filtered-gallery file and unlinks it in a finally; a SIGKILL skips finally. Fix: write temps under a dir the worker wipes on start, and/or a SIGTERM handler that unlinks before exit. Minor.

    Accepted trade-off: a partially-analysed title restarts from scratch next lock. Fine for an overnight/idle workload; no mid-video checkpointing.

    "},{"location":"service-conversion/#the-end-to-end-ux","title":"The end-to-end UX","text":"
    # once: build + install for your GPU + model\n./scripts/build_install.py install.yaml\n    # detects Fedora/Arch, ensures deps, compiles, installs units under ~/.local\n\n# once: set your keys, enable the gate\n$EDITOR ~/.local/share/sae-worker/.env     # JELLYFIN_API_KEY, TMDB_API_KEY\nsystemctl --user enable --now sae-lock-gate.service\n\n# from then on: nothing. Lock your screen \u2192 it analyses. Unlock \u2192 it stops.\n

    No Docker, no GPU passthrough config, no media re-mounting \u2014 the worker sees the same filesystem and GPU as everything else on the box.

    "},{"location":"service-conversion/#implementation-plan-follow-up-commits","title":"Implementation plan (follow-up commits)","text":"

    Ordered so each step stands alone:

    1. installer skeleton \u2014 scripts/build_install.py: parse install.yaml, distro detect, dependency check/print (start with cpu platform so it builds without a GPU), cmake+build, copy into prefix.
    2. supervisor + cleanup \u2014 scripts/service.py (worker loop + gallery-scan timer + SIGTERM); temp-file cleanup fix in run_from_jellyfin.py.
    3. systemd units + lock gate \u2014 generate/install sae-worker.service, sae-lock-gate.service, and the logind lock watcher.
    4. gallery/model guard \u2014 stamp embedder into gallery.json; startup mismatch check.
    5. platform + distro matrix \u2014 nvidia/amd backends; dnf/pacman dep lists; ONNX Runtime fetch fallback.
    6. docs \u2014 README \"Run on your idle GPU\" section.
    "},{"location":"service-conversion/#settled-decisions","title":"Settled decisions","text":""},{"location":"service-conversion/#open-questions","title":"Open questions","text":"

    (none blocking \u2014 the spec above is buildable as-is.)

    "}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"","title":"scene-actor-extraction","text":"

    A face-recognition pipeline that finds when each actor appears on screen in a film or TV episode, built on KPN++ (a C++20 Kahn Process Network library) for the detect, track, match, and scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.

    This is a correctly scored second from a held-out film, one the optimizer never saw during tuning:

    Every visible face is named at 100% confidence (Chris Noth, Hank Azaria, Bobby Cannavale), the background extra is correctly left unnamed, and the two credited cast members without a visible face are correctly reported present but not visible. This matches Amazon X-Ray's own record for this second exactly.

    Results are not uniform across films. The hardest held-out film scores 46% F1. This report documents why: one tunable trade (extinction bridging at hard cuts), one structural limit (X-Ray credits people whose faces never appear on screen), and a small number of cases where the pipeline is correct and X-Ray's ground truth is not. Read how we score against X-Ray first. X-Ray's ground truth is scene-level; the pipeline's output is per-second. That difference shapes every finding below.

    "},{"location":"#findings","title":"Findings","text":""},{"location":"#full-experiment-log","title":"Full experiment log","text":""},{"location":"#reproducing-the-benchmarks","title":"Reproducing the benchmarks","text":"

    Gallery .h5 files, embedding dumps, the X-Ray corpus, montage frame images, and DE trajectories are not committed to this repository. They are pushed to the Gitea package registry and pulled on demand:

    scripts/artifacts/pull_artifacts.sh galleries\nscripts/artifacts/pull_artifacts.sh experiment-data\nscripts/artifacts/pull_artifacts.sh montage-frames <film-slug>\n

    See scripts/artifacts/push_artifacts.sh for the upload side, which requires a GITEA_TOKEN with package write scope.

    "},{"location":"best-model/","title":"Which embedding model is best?","text":"

    Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K, 455MB) were compared. r50 is excluded from the training/held-out comparison below; its gallery has roughly 30% fewer reference images per actor than the other three on the identical source photos, which confounds a direct score comparison (see the full experiment log for detail). It remains in the calibration comparison, which does not depend on the gallery image count.

    "},{"location":"best-model/#first-signal-calibration-curves","title":"First signal: calibration curves","text":"

    Each gallery carries a fitted Platt sigmoid P(match | cosine similarity) = \u03c3(a\u00b7sim + b), stored directly in the gallery HDF5 (src/gallery/gallery_calibration.hpp). This is a property of the embedding space alone, computed from intra- and inter-actor reference-image pairs with no tracking or scene logic involved, so it is a clean first read on discriminative power before running a benchmark.

    model a (steepness) boundary at P=0.5 LVFace-B Glint360K 17.7 sim 0.228 ArcFace w600k-MBF 16.2 sim 0.267 ArcFace w600k-R50 15.4 sim 0.301 ArcFace R18 15.3 sim 0.309

    LVFace has both the steepest transition and the lowest decision boundary, separating same-actor from different-actor reference pairs more confidently at a lower similarity than any ArcFace variant.

    "},{"location":"best-model/#second-signal-held-out-f1","title":"Second signal: held-out F1","text":"

    Each model's own tuned full_exp config, replayed against the 5 films the optimizer never saw and scored the same way:

    film LVFace F1 mbf F1 r18 F1 Benny & Joon 83.0% 78.5% 77.1% Lovelace 77.5% 73.7% 72.2% Valerian and the City of a Thousand Planets 74.1% 70.2% 71.0% Downton Abbey: A New Era 56.2% 55.0% 53.0% The Many Saints of Newark 46.3% 44.5% 42.1% macro average 67.4% 64.4% 63.1%

    LVFace scores highest on all 5 held-out films; the ranking never flips between models. Total misID count across the 5 films: LVFace 1032, mbf 2197, r18 1224. LVFace has less than half mbf's misID total and still scores higher on every film.

    Held-out results are stronger evidence than training results, because training numbers can reflect what the optimizer was tuned to fit rather than general performance. On training data, the ordering is not as clean:

    film LVFace F1 mbf F1 r18 F1 best Caf\u00e9 Society 68.1% 62.2% 60.1% LVFace Lord of War 75.6% 77.2% 75.6% mbf Scarface 71.5% 68.6% 64.1% LVFace Sound of Metal 78.8% 76.5% 71.6% LVFace

    mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either table where LVFace does not score highest. LVFace's training-set macro average (75.3%, see the full experiment log) is not a uniform win across every film it contributes to; the held-out result, where LVFace wins all 5 films outright, is the stronger claim.

    This reverses an earlier, superseded benchmarking pass that used a scene-union metric and found the three models statistically indistinguishable (around 85% each), concluding LVFace was not worth its size. That metric masked out-of-cast false positives behind a gallery-intersect-cast recall filter; the per-second metric used here does not.

    "},{"location":"best-model/#full-training-matrix-picture","title":"Full training-matrix picture","text":"

    Best full-gallery combo per model (all three are full_exp), from the training matrix in the full experiment log:

    model F1 P R misID LVFace-B Glint360K 75.3% 89.7% 65.4% 232 ArcFace w600k-MBF 72.0% 87.7% 61.4% 240 ArcFace R18 69.1% 87.6% 57.7% 242

    LVFace leads within both the restricted and full gallery modes, visible directly in the chart above without reading the table. The three models' misID counts on the full gallery are nearly identical (232/240/242); LVFace's lead here is a precision-and-recall lead, not a misID one.

    "},{"location":"best-model/#operational-note","title":"Operational note","text":"

    Switching the default embedder is not a config change alone; the gallery is model-specific, since embeddings from different models are not comparable. Any existing gallery built against a different model must be rebuilt from source images before the new default takes effect. scripts/optimizer/reembed_gallery.py does this from a reference gallery's cached source images without re-downloading anything.

    "},{"location":"gallery-scope/","title":"Whole gallery vs. cast-restricted gallery","text":"

    Two ways to run the matcher. Full mode scores every detected face against the entire 2418-actor gallery. Restricted mode pre-filters each film's gallery down to just its Jellyfin-credited cast (typically around 15 top-billed actors) before the matcher runs.

    "},{"location":"gallery-scope/#result","title":"Result","text":"

    Averaged across the 3 compared models (r50 excluded, see the full experiment log) and both expansion settings, on the 4 training films:

    scope F1 P R total misID full 71.1% 89.6% 59.6% 1121 restricted 75.9% 90.4% 65.6% 299

    Restriction improves every metric at once, not a precision/recall trade: +4.8pp F1, +6.0pp recall, roughly a quarter the total misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a lookalike false match, and the recall gain shows this does not cost real detections.

    Every model's best-scoring combo in the training matrix uses the restricted gallery:

    See the full experiment log for the complete table. One combo reaches zero true out-of-cast misidentifications, arcface_w600k_mbf_restricted_exp (F1 76.2%), and it is a restricted one, consistent with restriction, not expansion, being what suppresses cross-film confusions.

    The restriction effect (+4.8pp averaged across models) is larger than the model-choice effect: LVFace beats r18 by 6.2pp in full mode but beats mbf by 3.3pp. Restriction is the single strongest lever in the matrix.

    "},{"location":"gallery-scope/#why-this-is-not-the-shipped-default","title":"Why this is not the shipped default","text":"

    Cast restriction is implemented today only as an offline optimizer technique (scripts/optimizer/cast_restrict.py): it pre-builds a filtered gallery file per film using Jellyfin's cast list before the benchmark calls the matcher. There is no runtime \"restrict to this title's credited cast\" switch in the shipped application; scene_analyze always matches against whatever single gallery file it is given.

    Building this as a real feature requires:

    The shipped src/config.hpp defaults use the full-mode winner (LVFace-B_Glint360K_full_exp, F1 75.3% training, 67.4% held-out macro) rather than the higher-scoring restricted_exp (78.3%), because 78.3% describes a capability the application does not have yet.

    "},{"location":"lvface-deep-dive/","title":"Deep dive: LVFace-B Glint360K","text":"

    LVFace won the model comparison (see Which model is best?) and is the shipped default embedder. This page reports how it performs in detail: a baseline of correct output, the two mechanisms behind its errors, and every distinct case where it names someone who is not in the film's credited cast.

    Read How we score against X-Ray first. X-Ray's ground truth is scene-level, not per-frame. A name marked correct in the Offscreen column below is the pipeline correctly reporting scene membership, not a workaround.

    How to read the frames on this page

    The top of each image is the film frame, with a box and name on every face the pipeline matched to a real detection. The panels below are the per-second result against X-Ray. Onscreen lists names attached to a visible face this second. Offscreen lists names the pipeline reports present without a currently visible face. Colors mark the verdict: green correct (TPI), orange wrong (FPI), blue missed (FN).

    "},{"location":"lvface-deep-dive/#baseline-correctly-scored-seconds","title":"Baseline: correctly scored seconds","text":"

    Six faces on screen, all six named correctly, including Penelope Wilton at the edge of the pews and a partly occluded Michelle Dockery. Thirteen more cast members X-Ray lists as present in the scene are correctly reported Offscreen. One miss: Maggie Smith (blue). Score for this second: 0.86.

    The same film's funeral scene: dark clothing, hats, half the faces turned away. Nineteen of the twenty cast members X-Ray lists for this scene score correct: seven named on screen at up to 100% confidence, twelve more reported correctly as present but not visible.

    The pipeline does not require a live face. This is Herbie Hancock at 98% confidence, identified from a face displayed on a screen inside the film, on a video call under a science-fiction HUD overlay.

    "},{"location":"lvface-deep-dive/#training-vs-held-out-the-generalization-gap","title":"Training vs. held-out: the generalization gap","text":"

    The shipped config (prob_threshold=0.754, anneal_sec=35.54, extinction_sec=57.43, expand_gallery=true) was tuned on 4 films. Scored on the 5 films the optimizer never saw:

    film F1 P R TPI FPI misid FN Benny & Joon 83.0% 89.1% 77.7% 15125 1846 0 4337 Lovelace 77.5% 90.3% 67.9% 14990 1085 58 7085 Valerian and the City of a Thousand Planets 74.1% 97.1% 60.0% 18663 548 0 12467 Downton Abbey: A New Era 56.2% 97.8% 39.4% 52027 1173 0 80084 The Many Saints of Newark 46.3% 54.7% 40.1% 15922 4394 974 23791 macro average 67.4% 85.8% 57.0%

    The P column is misID-weighted (each out-of-film name counts 10x in the denominator; see methodology). That weighting is why Many Saints reads 54.7% here despite naming mostly real, present faces: its raw (unweighted) precision is 78.4%, and the gap is entirely its 974 misIDs paying the 10x penalty. The three zero-misID films (Benny & Joon, Downton, Valerian) have identical weighted and raw precision; Lovelace, with 58 misIDs, sits 3pp below its raw 93.3%.

    Held-out F1 is 67.4%, against 75.3% on training, an 8pp drop. The spread between the best and worst held-out film is 37pp. This is not unique to LVFace: the full experiment log shows mbf and r18 with the same shape of spread on the same films, at a uniformly lower level. Two mechanisms explain the spread. Both are shown below with frame-level evidence.

    "},{"location":"lvface-deep-dive/#mechanism-1-extinction-bridging","title":"Mechanism 1: extinction bridging","text":"

    The extinction window keeps a name reported as present for up to extinction_sec after its last real detection. This is deliberate: most gaps in face visibility are short (a turned head, an occlusion, a cut to a reaction shot), and the window bridges them.

    Lovelace's polygraph scene: only Eric Roberts and Amanda Seyfried have visible faces. X-Ray lists eight cast members present. All eight score correct; the other six are reported Offscreen through a stretch where the camera never shows them. The extinction window is why.

    The same mechanism fails at a hard cut into a long stretch with no faces at all. Downton Abbey's recall (39.4%, the worst of the five held-out films) is dominated by this failure. It is verified directly against the raw per-frame stream and the dump's own detection counts, not inferred from the score. Plotting the dump's per-second face_count (detector output, independent of the tracker) against what the tracker reports, through Downton Abbey's hard cut into its closing credits:

    From the cut onward the detector reports zero faces for close to a minute. The tracker continues reporting the previous shot's 15 identities for the same span (verified for Hugh Bonneville: bbox (1743.2, 0.0, 171.3, 317.8), unchanged to the pixel, at every sampled second for 57 seconds). The staircase at the right edge is the extinction window expiring, actor by actor. This is SceneTrackerFunc::active_[actor_idx].last_bbox (src/nodes/scene_tracker_node.hpp) re-emitted as designed. extinction_sec=57.4 was tuned long because bridging is correct on most footage, as in the polygraph scene above. The training films did not contain a faceless stretch long enough to expose the cost side; the held-out set did.

    The extinction window is a scoring concept, not something drawn on screen. The shipped output is presence windows with no bounding boxes. Even the debug overlay used for this report never draws a box for a bridged name: a name inside its extinction window with no current detection appears only as a name in the Offscreen column, the same as every correctly bridged name above.

    A related, smaller effect shows up at rapid cuts:

    Caf\u00e9 Society (a training film), a shot/reverse-shot dialog. The box on Steve Carell's face carries two labels: his own, and Jesse Eisenberg's, left over from the counter-shot a moment earlier. Both names score correct, because both actors are present in this scene per X-Ray. The box position is briefly wrong; the presence claim, which is what the pipeline ships, is right.

    "},{"location":"lvface-deep-dive/#mechanism-2-the-face-vs-presence-ceiling","title":"Mechanism 2: the face-vs-presence ceiling","text":"

    Downton Abbey's recall did not collapse because faces were misread. It collapsed because for most of its 80084 false-negative seconds there was no face to read.

    A newsreel crew moves equipment through the hall. X-Ray credits 22 cast members as present in this scene. None face the camera. Eight still score correct, carried by presence windows from adjacent shots. The other fourteen are missed, and no face-recognition system can recover them, because there is no face in the frame. X-Ray records scene membership; the pipeline measures visible faces. In ensemble scenes these two quantities diverge, and that gap accounts for most of the false-negative count.

    "},{"location":"lvface-deep-dive/#every-distinct-out-of-cast-name","title":"Every distinct out-of-cast name","text":"

    Many Saints of Newark has the largest misID count of any held-out film: 974 seconds, weighted. Rather than characterize this from a single frame, the raw replay stream was searched directly for every name the pipeline reports that is not in the film's credited cast. The same search was run on all 9 films in the benchmark, one rule applied uniformly: find the first second each distinct out-of-cast name appears, and render that exact second.

    Five films produce no such name anywhere in their runtime: Benny & Joon, Caf\u00e9 Society, Downton Abbey, Sound of Metal, Valerian. Zero out-of-cast names across their entire length. Four films produce nine distinct names between them, shown below in full, not a sample.

    "},{"location":"lvface-deep-dive/#the-many-saints-of-newark-4-names","title":"The Many Saints of Newark: 4 names","text":"

    Germar Terrell Gardner, t=848s, 78% confidence. A real, clearly visible background actor. He is not in X-Ray's cast list for this film, but he is credited in Jellyfin's independent cast metadata (see Where LVFace beat X-Ray below). This is a ground-truth gap, not a model error.

    Archie Yates, t=2521s, 78% confidence. A real detected face, a genuine lookalike confusion.

    Zooey Deschanel, t=2819s, 99% confidence. A real detected face at a dinner table, high-confidence lookalike confusion.

    Talia Balsam, t=4551s, 93% confidence. A real detected face. Talia Balsam plays Mrs. Jarecki, a guidance counselor, in this film; she is confirmed on screen by direct inspection of the frame. She does not appear in X-Ray's people.csv for this title. This is a second ground-truth gap in the same film, not a model error.

    Two of these four names are ground-truth gaps (Gardner, Balsam), not misidentifications. The other two (Yates, Deschanel) are genuine embedding errors on real faces.

    "},{"location":"lvface-deep-dive/#lord-of-war-3-names","title":"Lord of War: 3 names","text":"

    David Shumbris, t=418s, 81% confidence. A real face in a dim, low-detail shot under a train track. A genuine lookalike confusion in poor lighting.

    Ronald Reagan, t=1003s, 100% confidence. This is not a lookalike confusion. The detected face is a photograph of Reagan appearing within the shot, not a living actor. The detector and matcher both did their job correctly on the image content in front of them; the error is that a photograph inside the scene is not the same thing as an actor present in the scene, and the pipeline has no way to draw that distinction from a face crop alone.

    Lance Reddick, t=6424s, 78% confidence. A small, distant, low-detail face at the edge of frame. A marginal, low-confidence lookalike confusion.

    "},{"location":"lvface-deep-dive/#lovelace-1-name","title":"Lovelace: 1 name","text":"

    Chlo\u00eb Sevigny, t=2451s, 100% confidence. Amanda Seyfried's track is real and well-tracked through most of this shot, but her bbox is frozen at the exact same coordinates for t=2450 and t=2451, one second where her box stopped updating from a fresh detection. Only one real face is detected at t=2451 (confirmed against the dump's own per-frame detections), and it is a tight IoU-1.0 fit under the Chlo\u00eb Sevigny box, not the Seyfried one. So the green Seyfried box in this frame is a ghost, re-emitting her last known position for that one second, and the fresh, wrong detection is Sevigny, landing on top of it. Not two competing fresh identities on one crop: one ghost and one fresh misidentification happening to overlap.

    "},{"location":"lvface-deep-dive/#scarface-1-name","title":"Scarface: 1 name","text":"

    Kirstie Alley, t=2451s, 89% confidence. Al Pacino is correctly identified in the foreground at 100%; a background face in the same shot is wrongly labeled Kirstie Alley. (The t=2451s here and the Lovelace Chlo\u00eb Sevigny case above landing on the identical second is a genuine coincidence, verified from each film's raw stream by first_fpi_frames.py, not a transcription slip, two unrelated films whose first out-of-cast name happens to fall at the same timestamp.)

    "},{"location":"lvface-deep-dive/#summary-of-the-nine","title":"Summary of the nine","text":"film name t (s) confidence classification Many Saints of Newark Germar Terrell Gardner 848 78% ground-truth gap Many Saints of Newark Archie Yates 2521 78% lookalike confusion Many Saints of Newark Zooey Deschanel 2819 99% lookalike confusion Many Saints of Newark Talia Balsam 4551 93% ground-truth gap Lord of War David Shumbris 418 81% lookalike confusion Lord of War Ronald Reagan 1003 100% photo-in-frame Lord of War Lance Reddick 6424 78% lookalike confusion, marginal Lovelace Chlo\u00eb Sevigny 2451 100% lookalike confusion Scarface Kirstie Alley 2451 89% lookalike confusion

    Of nine distinct out-of-cast names across four films, two are ground-truth gaps, one is a photograph misread as a person, and six are genuine embedding-space confusions on real detected faces. None trace to extinction bridging: every one of these nine is a fresh detection on a real face crop at the second it first appears.

    "},{"location":"lvface-deep-dive/#where-lvface-beat-x-ray","title":"Where LVFace beat X-Ray","text":"

    Not every name marked wrong is actually wrong. scripts/optimizer/second_score.py scores strictly against X-Ray, and X-Ray has gaps of its own.

    Germar Terrell Gardner, the same name from the table above, does not appear in X-Ray's people.csv for The Many Saints of Newark. Jellyfin's independent cast metadata does credit him for this film (cross-checked against experiments/manifests/jellyfin_casts.json from the experiment-data artifact package, a data source entirely separate from X-Ray). Talia Balsam is the same case: confirmed on screen, absent from X-Ray's cast list for this title.

    This extends past uncredited background actors. This is Robert Patrick, top-billed in Lovelace, clearly on screen reading a newspaper, identified at 100%. The frame is scored wrong because X-Ray's people-in-scene list for this specific scene omits him, despite crediting him elsewhere in the film. The identification is correct; the ground truth is missing an entry.

    X-Ray is a large, convenient ground truth. It is not a complete one. The misID and FPI counts reported throughout this document include some fixed amount of noise from gaps in X-Ray itself, in both directions.

    "},{"location":"lvface-deep-dive/#summary","title":"Summary","text":"

    LVFace wins the model comparison on every held-out film. It correctly names 19 of 20 people in a crowded funeral scene and correctly identifies a face displayed on a screen inside the film. Its errors resolve into two mechanisms: extinction bridging, which is correct on most footage and fails specifically at hard cuts into long faceless stretches, and the face-versus-presence ceiling, where X-Ray credits scene membership for people whose faces never appear on screen. Of the nine distinct out-of-cast identifications found across the benchmark, two trace to gaps in X-Ray's own cast data, one is a photograph misread as a person, and six are genuine lookalike confusions on real faces. The held-out generalization gap, 75.3% training to 67.4% held-out, is real and should be treated as the expected operating point, not the training-set figure.

    "},{"location":"methodology/","title":"How we score against X-Ray","text":"

    Every number in this report, every F1 and misID count, comes from one comparison. The comparison has a mismatch at its core that shapes nearly every finding in this report: the ground truth is scene-level, the pipeline's output is per-second, and the two do not mean the same thing. This page documents that comparison once, so the findings pages can rely on it without re-explaining it.

    "},{"location":"methodology/#what-amazon-x-ray-records","title":"What Amazon X-Ray records","text":"

    X-Ray ships three tables per film: scenes.csv (a list of [start, end] timespans), people_in_scenes.csv (which actors are credited in each scene), and people.csv (actor identities). There is no per-frame or per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and X-Ray records one cast list for the entire span, not \"on screen from second 12 to second 30.\"

    To compare this against per-second predictions, second_score.py expands every scene into per-second ground truth by copying the whole scene's cast list onto every second inside it:

    for sn, (t0, t1) in spans.items():\n    cast = scene_cast.get(sn, [])\n    for t in range(int(t0), int(t1)):\n        timeline[t] = cast\n

    That is the entire mechanism. If X-Ray credits five actors to a 30-second scene, all five count as ground truth present for all 30 seconds, including seconds where only one of them is on screen. This is not a simplification introduced by the pipeline; it is the only reading of X-Ray's data that is possible, because X-Ray itself does not record anything finer-grained.

    "},{"location":"methodology/#why-an-offscreen-name-can-be-scored-correct","title":"Why an offscreen name can be scored correct","text":"

    A name listed under Offscreen with a correct (green) label is not the pipeline guessing or padding its score. It is the pipeline correctly answering the question X-Ray actually asks: is this actor part of this scene. It answers that question using a presence window ([start, end], held open across cuts by anneal_sec and extinction_sec), which matches X-Ray's scene-level semantics more closely than a raw per-frame detection would.

    A system that only reported \"this actor is visible in this exact frame\" would score worse against X-Ray's scene-level ground truth, producing a false negative every time the camera cuts away from a character who is still present in the scene. Not because it is wrong about the world, but because it would be answering a stricter, different question than the one X-Ray's data supports. The presence-window design exists specifically to answer X-Ray's actual question.

    "},{"location":"methodology/#what-this-resolves-and-what-it-does-not","title":"What this resolves and what it does not","text":"

    This resolves the semantic mismatch between a scene and an instant. It does not resolve two other limitations, both discussed in the LVFace deep dive.

    The face-vs-presence ceiling. X-Ray credits scene membership regardless of whether a face is ever visible: background crew, characters shot from behind, voice-only presence. No amount of bridging recovers a face that never appears on screen. This is a hard ceiling on recall, not a defect.

    Extinction bridging can overshoot. The same presence-window mechanism that correctly answers \"still in this scene\" during a normal cut can also bridge across a scene boundary it has no way to detect. A hard cut into a different scene with no faces, such as closing credits, carries the previous scene's identities forward until the window expires. This is the mechanism behind Downton Abbey's recall collapse, documented in the deep dive.

    "},{"location":"methodology/#precision-recall-and-the-misid-weighting","title":"Precision, recall, and the misID weighting","text":"

    Per sampled second t:

    TPI (true positive instances): actors both X-Ray and the pipeline agree are present.

    FPI (false positive instances): actors the pipeline reports that are not in X-Ray's cast for this second. Split into two categories:

    Every headline P and F1 is misID-weighted

    The precision reported throughout this report, and therefore the F1 derived from it, puts each FPI_misid into the denominator 10 times (precision = TPI / (TPI + FPI_incast + 10\u00b7FPI_misid), second_score.py). This is deliberate: the whole point is to punish naming an out-of-film actor far harder than a scene-boundary slip. But it means the P column is not raw precision, and a misID-heavy film's P is depressed super-linearly. second_score.py also emits an unweighted precision_raw (always \u2265 the weighted P); where the gap matters, The Many Saints of Newark, weighted P 54.7% vs. raw 78.4%, the LVFace deep dive reports both. When comparing P across films, remember you are comparing a quantity that penalizes misIDs, not just a hit rate.

    FN (false negatives): actors X-Ray lists that the pipeline never reports, counted only for actors who have a gallery reference embedding. Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from 20% to 79% by film (see the full experiment log); an actor with no reference photo can never be recognized regardless of model quality, and counting them as a miss would penalize gallery coverage, not recognition accuracy.

    Two further numbers are reported alongside F1:

    agreement_rate: mean per-second Jaccard overlap (|Pred \u2229 GT| / |Pred \u222a GT|), partial credit. Naming 2 of 3 present actors scores 2/3, not 0.

    exact_match_rate: the fraction of sampled seconds where the pipeline's named set exactly equals X-Ray's, no partial credit. Far harsher, and dominated by recall, since any single missed actor zeroes that second.

    "},{"location":"methodology/#reproduce","title":"Reproduce","text":"
    python3 scripts/optimizer/second_score.py \\\n    --pred pred.json --xray experiments/xray/.../<xray_dir> \\\n    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5\n

    See also the full experiment log for how pred.json is produced, and the LVFace deep dive for what these mechanisms look like frame by frame.

    "},{"location":"model-bakeoff/","title":"Full experiment log","text":"

    This page reports how the pipeline performs across three questions: which embedding model is best, whether restricting the gallery to a film's credited cast helps, and whether promoting confidently identified poses into a per-film gallery annex helps. It also documents the replay architecture that made testing all three questions in one pass practical, and every caveat needed to trust the numbers.

    Read How we score against X-Ray first for what F1, precision, recall, and misID mean in this report. All numbers below use the per-second metric (scripts/optimizer/second_score.py).

    r50 (ArcFace w600k-R50) is excluded from the detailed comparison below. Its gallery was built with roughly 30% fewer reference images per actor than the other three models on the identical source photos (10808 vs 15055 total embeddings across the same 2418 actors), which confounds any direct comparison of its scores against the others. It remains in the calibration curve comparison, which does not depend on the training benchmark.

    "},{"location":"model-bakeoff/#why-replay-makes-this-affordable","title":"Why replay makes this affordable","text":"

    Decoding video and running face detection, alignment, and embedding is the expensive part of this pipeline. Everything downstream of that (tracking, identity matching, scene aggregation) is cheap. KPN++'s node/network structure means those two stages are separate components connected by typed channels, so the expensive stage can run once per film, cache its output, and the cheap stage can be re-run against that cache as many times as needed with different Config values.

    scene_analyze --dump-embeddings out.h5 runs the expensive half once per film and writes per-frame face detections and embeddings to HDF5 (scripts/optimizer/SCHEMA.md). scripts/optimizer/replay.py then re-assembles the real C++ face_tracker, identity_matcher, and scene_tracker nodes into a Python-driven KPN network and replays a film's cached embeddings through them, varying prob_threshold, anneal_sec, extinction_sec, and expand_gallery freely. No GPU inference and no video decode happen during a replay; each one completes in seconds. This is what makes a 512-evaluation differential-evolution search per model, per gallery mode, per expansion setting, tractable, and what made the full held-out validation across three models in this report possible in one session rather than requiring three full re-encodes of the benchmark set.

    optimize.py runs differential_evolution over this replay function as its objective, with DE-level parallelism (multiple candidate configs evaluated concurrently, each spawning its own replay subprocesses) on top of it. The practical ceiling on this machine's GPU was 8 concurrent replay processes; 9 silently degraded every score to 0.0% (well-formed output, wrong numbers, not a crash), so optimize.py was run at REPLAY_WORKERS=4 DE_WORKERS=2.

    "},{"location":"model-bakeoff/#search-space","title":"Search space","text":"

    popsize=10, maxiter=15 per combo (3 parameters, up to 512 evaluations, usually stopping earlier on DE's convergence tolerance). anneal_sec/extinction_sec bounds were widened from 1-30/1-15 to 1-60/1-60 partway through the sweep. r50's 4 combos finished before the widening and used the old, narrower bounds; this is one more reason r50 is excluded from direct comparison here.

    "},{"location":"model-bakeoff/#training-films-and-held-out-films","title":"Training films and held-out films","text":"

    9 films have dumped embeddings across all 4 models. 4 were used for optimization:

    5 were held out, never seen by any optimizer run:

    "},{"location":"model-bakeoff/#gallery-coverage-per-film","title":"Gallery coverage per film","text":"

    The gallery has reference embeddings for 2418 actors, but coverage of any given film's credited cast varies widely. This was previously reported as one flat number (67% of X-Ray cast lacking a reference embedding, averaged across the whole benchmark); the per-film breakdown is:

    film cast credited in gallery coverage Lord of War 64 13 20.3% Scarface 67 15 22.4% The Many Saints of Newark 48 13 27.1% Caf\u00e9 Society 62 17 27.4% Lovelace 42 15 35.7% Valerian and the City of a Thousand Planets 36 13 36.1% Benny & Joon 23 12 52.2% Downton Abbey: A New Era 36 22 61.1% Sound of Metal 14 11 78.6%

    Two training films (Lord of War, Scarface) have the worst coverage in the set, 20-22%. Their training-set F1 numbers below are partly capped by missing references, not purely by model quality. Downton Abbey has 61% coverage, the second-best in the benchmark, yet the worst held-out recall of any film (39.4%, LVFace). Its recall problem is not primarily a coverage problem; it is the extinction-bridging failure documented in the LVFace deep dive. Reproduce with scripts/docs/gallery_coverage_per_film.py.

    "},{"location":"model-bakeoff/#training-results-3-models-2-gallery-modes-2-expansion-settings","title":"Training results, 3 models \u00d7 2 gallery modes \u00d7 2 expansion settings","text":"

    Ranked by F1. misid = FPI_misid, the count of true wrong-actor identifications (naming someone not in the film's cast at all), distinct from FPI, which also includes in-cast timing slips.

    Each combo's row is its best full-coverage evaluation: the highest-F1 DE evaluation in which all 4 training films replayed without a timeout (see Dropped-film scoring below for why this qualifier is load-bearing and not the same as argmax F1 over the raw sweep).

    combo F1 P R TPI FPI misid FN LVFace-B_Glint360K_restricted_exp 78.3% 91.0% 68.9% 42830 3782 60 19492 LVFace-B_Glint360K_restricted_noexp 76.7% 91.5% 66.2% 41149 3400 59 21173 arcface_w600k_mbf_restricted_exp 76.2% 90.0% 66.2% 64328 7480 0 33234 arcface_r18_restricted_exp 75.5% 87.6% 66.5% 41399 5666 60 20923 LVFace-B_Glint360K_full_exp 75.3% 89.7% 65.4% 47757 3407 232 26966 arcface_w600k_mbf_restricted_noexp 75.0% 91.1% 63.9% 39752 3465 60 22570 arcface_r18_restricted_noexp 73.5% 91.3% 61.7% 38299 3220 60 24023 LVFace-B_Glint360K_full_noexp 72.3% 88.3% 61.8% 40363 3503 244 25850 arcface_w600k_mbf_full_exp 72.0% 87.7% 61.4% 39875 3729 240 26338 arcface_w600k_mbf_full_noexp 71.0% 93.2% 57.9% 41699 2472 56 33024 arcface_r18_full_exp 69.1% 87.6% 57.7% 37342 3119 242 28871 arcface_r18_full_noexp 66.6% 91.3% 53.1% 34314 2362 107 31899

    The two clearest patterns: every model's best-scoring combo uses the restricted gallery, and LVFace leads within both gallery modes. full_exp (the shipped combination) is the best-scoring option that uses only features the running application currently supports; restriction is not wired into the application yet (see Whole vs. cast-restricted gallery).

    "},{"location":"model-bakeoff/#a-scoring-bug-worth-recording-dropped-film-evaluations","title":"A scoring bug worth recording: dropped-film evaluations","text":"

    The numbers above are corrected ones. The raw rep4_best_*.json files, and an earlier version of this table, reported a different arcface_w600k_mbf_full_noexp row: 74.2% F1 at TPI 12645, a third the TPI of every sibling combo. That was not a better config; it was an artifact of how the optimizer aggregates.

    optimize.py builds each candidate's score from only the films whose replay subprocess returned (per_film = [m for m in ex.map(_one, films) if m is not None]), then averages F1/precision/recall and sums TPI/FPI/misID over just those survivors. When a film's replay times out (the sweep ran near the 8-process concurrency ceiling, so this happened intermittently), that film silently drops from both. A candidate whose hardest film timed out is therefore scored on an easier subset, and differential evolution, maximizing that score, will happily converge onto exactly such a candidate. For mbf_full_noexp the reported winner was one of 7 evaluations (out of 512) whose TPI had collapsed to a partial-film subset; its median-coverage evaluations sit around 51686 TPI.

    The fix here was to re-derive each combo's best row from its DE trajectory (experiments/trajectories/rep4_*.jsonl), keeping only evaluations within 30% of that combo's median TPI (full 4-film coverage) before taking the best F1. This needs no re-running, the honest best configuration was already in the sweep, just not the one argmax F1 selected. Three combos moved: mbf_full_noexp 74.2% \u2192 71.0%, LVFace_full_noexp 72.4% \u2192 72.3% (and its misID, 0 \u2192 244, was itself a dropped-film artifact), mbf_restricted_exp 76.5% \u2192 76.2%. The shipped LVFace full_exp winner was unaffected, its reported evaluation already had full coverage (TPI 47757 \u2248 median). experiment_charts.py applies the same clean_best filter, so every figure on this page matches the corrected table. The underlying optimize.py aggregation is also being fixed so a dropped-film evaluation can never be selected as a winner again.

    "},{"location":"model-bakeoff/#per-film-training-breakdown","title":"Per-film training breakdown","text":"

    The 75.3% LVFace training figure is a macro average across 4 films, not a uniform result:

    film LVFace F1 mbf F1 r18 F1 best model Caf\u00e9 Society 68.1% 62.2% 60.1% LVFace Lord of War 75.6% 77.2% 75.6% mbf Scarface 71.5% 68.6% 64.1% LVFace Sound of Metal 78.8% 76.5% 71.6% LVFace

    LVFace does not win every training film. mbf scores higher on Lord of War (77.2% vs 75.6%). LVFace's own training-film range is 68.1% to 78.8%, a 10.7pp spread, smaller than the 37pp spread seen on held-out films but real. Reproduce with scripts/docs/run_holdout_all_models.py --films training.

    "},{"location":"model-bakeoff/#held-out-validation-all-3-models","title":"Held-out validation, all 3 models","text":"

    The training matrix above is training-set fit. Each model's own tuned full_exp config was replayed against the 5 held-out films, scored the same way:

    film LVFace F1 mbf F1 r18 F1 Benny & Joon 83.0% 78.5% 77.1% Lovelace 77.5% 73.7% 72.2% Valerian and the City of a Thousand Planets 74.1% 70.2% 71.0% Downton Abbey: A New Era 56.2% 55.0% 53.0% The Many Saints of Newark 46.3% 44.5% 42.1% macro average 67.4% 64.4% 63.1%

    LVFace scores highest on every one of the 5 held-out films; the ranking never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18 1224. LVFace has less than half mbf's misID count while also scoring higher on every film. This directly confirms the model choice out of sample; it is not inferred from the training numbers alone. See the LVFace deep dive for frame-level detail on where and why LVFace still fails on the two worst films. Reproduce with scripts/docs/run_holdout_all_models.py.

    "},{"location":"model-bakeoff/#two-effects-in-isolation-gallery-scope-and-pose-expansion","title":"Two effects in isolation: gallery scope and pose expansion","text":"

    Averaging across the 3 compared models (r50 excluded) isolates each variable from model choice.

    Gallery scope, averaged over both expansion settings and all 3 models (6 evaluations per row):

    scope F1 P R total misID full 71.1% 89.6% 59.6% 1121 restricted 75.9% 90.4% 65.6% 299

    Restriction improves every metric at once. This is not a precision/recall trade: +4.8pp F1, +6.0pp recall, and roughly a quarter the misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a lookalike false match, and the recall gain shows this does not cost real detections. Restriction is currently an offline optimizer technique, not a runtime feature of the application; see Whole vs. cast-restricted gallery for what building it into the application would require.

    Pose expansion (promoting a confidently identified track's novel-pose views into a per-film gallery annex, src/gallery/track_gallery.hpp):

    scope expansion F1 R misID full off 70.0% 57.6% 407 full on 72.1% 61.5% 714 restricted off 75.1% 63.9% 179 restricted on 76.7% 67.2% 120

    In restricted mode, expansion is a clean win: +1.6pp F1, +3.3pp recall, misID drops. The annex only competes against the film's own roughly 15-actor cast, so a new pose of a known actor is unlikely to be confused with someone else. In full mode, expansion buys +2.1pp F1 and +3.9pp recall but at a real cost: misID rises from 407 to 714 as the same new-pose view now competes against the full 2418-actor gallery, where a confidently learned pose is more likely to match the wrong person. On the full gallery it is a recall-vs-misID trade, not a free gain. This training-set effect did not reproduce on held-out data; see Does pose expansion help? for the full held-out test and the two methodology bugs caught while checking it.

    "},{"location":"model-bakeoff/#calibration-curves","title":"Calibration curves","text":"

    Each gallery carries a fitted Platt sigmoid P(match | sim) = \u03c3(a\u00b7sim + b), stored directly in the gallery HDF5 (src/gallery/gallery_calibration.hpp). This measures discriminative power independent of whatever prob_threshold a given run used:

    LVFace has the steepest curve (a=17.7 vs 15.3-16.2 for the ArcFace variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs 0.27-0.31), separating same-actor from different-actor pairs more confidently at a lower similarity than any ArcFace variant tested, including r50. Generated by scripts/docs/calibration_chart.py.

    "},{"location":"model-bakeoff/#extinction-and-anneal-window-search","title":"Extinction and anneal window search","text":"

    Every one of the 512 DE evaluations for the winning LVFace full_exp combo, plotted over the prob_threshold \u00d7 extinction_sec plane:

    Nearly everything scoring well sits at extinction_sec above 50, across a wide range of thresholds. Short extinction windows are uniformly weaker: under a strict threshold, there is no good configuration in that region of the search space. The optimizer converged with anneal_sec=59.2, extinction_sec=59.2, about 99% of the widened 60s bound, which raises an open question not resolved in this round: does performance keep improving past 60s, or does it plateau there. Not chased further this pass.

    "},{"location":"model-bakeoff/#caveats","title":"Caveats","text":""},{"location":"model-bakeoff/#reproduce","title":"Reproduce","text":"
    # 4-film training matrix, all 4 models \u00d7 2 gallery modes \u00d7 2 expansion settings\nbash experiments/run_rep4_subprocess.sh\n\n# single combo\nSAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \\\n    --manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \\\n    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \\\n    --params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \\\n    --popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json\n\n# held-out validation, all 3 models, 5 films\npython3 scripts/docs/run_holdout_all_models.py --out docs_data/holdout_all_models.json\n\n# per-film training breakdown, all 3 models, 4 films\npython3 scripts/docs/run_holdout_all_models.py --films training --out docs_data/training_per_film.json\n\n# gallery coverage per film\npython3 scripts/docs/gallery_coverage_per_film.py --out docs_data/gallery_coverage_per_film.json\n\n# regenerate this page's charts from experiments/ artifacts\npython3 scripts/docs/experiment_charts.py --out-dir docs/assets/images\n\n# one frame per distinct out-of-cast name across all 9 films (used in the deep dive)\npython3 scripts/docs/first_fpi_frames.py\n

    See also the session log experiments/SESSION_STATE.md.

    "},{"location":"pose-expansion/","title":"Pose expansion: does promoting new poses mid-film help?","text":"

    expand_gallery (src/gallery/track_gallery.hpp) promotes a confidently identified track's novel-pose reference views into a per-film, in-memory gallery annex. The idea: once the pipeline is confident about an identity, a pose it has not seen before (turned head, different lighting) becomes an extra reference for recognizing that actor again later in the same film, without touching the baked gallery.

    "},{"location":"pose-expansion/#training-set-signal","title":"Training-set signal","text":"

    Averaged across the 3 compared models (r50 excluded), on the 4 films used for optimization. These are the corrected, full-coverage figures, see the dropped-film note in the experiment log for why an earlier version of this table overstated the full-mode misID jump (209 \u2192 864) that was itself partly a truncation artifact:

    scope expansion F1 R misID full off 70.0% 57.6% 407 full on 72.1% 61.5% 714 restricted off 75.1% 63.9% 179 restricted on 76.7% 67.2% 120

    In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp recall, lower misID. In full mode it looks like a recall-for-misID trade: +2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See the full experiment log for the per-model breakdown. This asymmetry motivated the question below: does turning expansion on change what gets recognized frame by frame, or is the aggregate F1 shift coming from something else.

    "},{"location":"pose-expansion/#held-out-test","title":"Held-out test","text":"

    Same model, same tuned config, expand_gallery toggled on vs. off, nothing else changed, full gallery mode, per-second scoring against X-Ray. This isolates expansion from every other variable that differs between the training-set rows above.

    LVFace-B Glint360K, all 5 held-out films:

    film F1 (exp) F1 (noexp) TPI delta FN delta Benny & Joon 83.0% 83.0% -2 +2 Downton Abbey: A New Era 56.1% 56.2% -7 +7 Lovelace 77.5% 77.4% +33 -33 The Many Saints of Newark 46.3% 46.3% +2 -2 Valerian and the City of a Thousand Planets 74.1% 74.1% +2 -2

    ArcFace R18, Benny & Joon, r18's own tuned config: F1 77.1% for both, TPI and FN identical, FPI differs by 2.

    Every film, both models tested: F1 differs by 0.1-0.2pp, TPI/FN swings are in the tens out of tens of thousands. This is noise, not a signal. Expansion made no measurable difference to per-second on-screen identification on any held-out film tested.

    "},{"location":"pose-expansion/#two-methodology-bugs-caught-during-this-check","title":"Two methodology bugs caught during this check","text":"

    Getting to the table above required catching two wrong turns, both worth recording because they are exactly the kind of error that produces a false positive \"expansion helped\" finding.

    1. Timeout truncation. The first Downton Abbey exp replay was cut off by a 60-second subprocess timeout at about 76% through the film (5589 of 7368 expected seconds). This silent data loss produced a large, convincing-looking TPI gap (47938 vs 52032) purely because one run was missing a quarter of the film. Caught by comparing n_seconds between runs before trusting any score delta; fixed by re-running with a longer timeout.
    2. Bbox-matching bug. An early per-second raw-annotation diff matched each exp detection to the first noexp detection with IoU above 0.5, not the best-overlapping one. With 3 faces close together in frame, this produced spurious disagreements (for example \"exp says Aidan Quinn, noexp says Johnny Depp\" at the same second) that vanished once the match used the best-IoU candidate instead of the first one. Both configs had actually output the same three names at the same three boxes.

    Both bugs independently pointed toward \"expansion is doing something,\" and both were artifacts of the comparison harness, not the pipeline. Before trusting a dramatic before/after diff, check that both runs cover the same seconds and that entities are matched by best overlap, not first found.

    "},{"location":"pose-expansion/#conclusion","title":"Conclusion","text":"

    The training-set aggregate effect, particularly the full-mode misID increase, does not reproduce on held-out data. At minimum it is far smaller than the training-set numbers suggested; it may be sampling variation from only 4 training films rather than a generalizable mechanism. Note the same class of harness bug appears twice in this investigation, the timeout truncation in bug #1 above, and the dropped-film aggregation that inflated the raw training-set misID figures. Both make an inert config look consequential; both are reasons to distrust a dramatic training-set delta until it survives on held-out films, which this one did not. This does not mean expand_gallery never does anything: the mechanism is real, and track_gallery.hpp's promotion logging confirms tracks get confirmed and views get promoted into the annex on every film tested. It means whatever effect expansion has on final per-second identification was too small to detect against 5 held-out films with this scoring method. A cleaner test would need either more held-out films or a metric that can see the annex's direct contribution, such as tagging which reference embedding won each match; neither was in scope for this pass.

    Do not treat the training-set exp/noexp numbers in the full experiment log as proof that expansion changes real-world behavior in either direction. On the evidence gathered so far, it does not move the needle enough to see.

    "},{"location":"service-conversion/","title":"Conversion to service \u2014 a native idle-GPU worker","text":"

    Status: design / proposal. Nothing here is built yet.

    "},{"location":"service-conversion/#the-idea","title":"The idea","text":"

    Turn the CLI tools into a turnkey batch worker that uses the machine's idle GPU: it analyses newly-added Jellyfin media when you're not using the computer (screen locked), and stops the instant you come back. It's an overnight job on your own Linux box.

    No Docker. This runs on your own machine with your own drivers, so a container buys little and costs a lot: GPU passthrough (nvidia-container-toolkit, or /dev/kfd+/dev/dri+video group for ROCm) is the single most fragile part of a containerised setup, and it exists only because of the container. Natively, the GPU just works with the drivers you already have, and the media paths Jellyfin reports are just real paths \u2014 no re-mounting. So we ship a native installer instead of an image builder.

    Two deliverables:

    1. An installer \u2014 scripts/build_install.py. Detects your distro, ensures the GPU/build dependencies are present (via dnf/pacman), compiles scene_analyze for your GPU, and installs the binary + Python glue + two systemd user units under ~/.local.
    2. A screen-lock gate \u2014 one of those systemd units watches logind lock/unlock and starts/stops the worker. Lock \u2192 analyse. Unlock \u2192 stop.
    "},{"location":"service-conversion/#what-already-exists-reuse-dont-rebuild","title":"What already exists (reuse, don't rebuild)","text":"

    The processing loop is already implemented \u2014 this is packaging, building, and lock-gating, not new pipeline logic.

    Piece Where What it does Analysis engine build/scene_analyze Video \u2192 face detect/align/embed \u2192 gallery match \u2192 result JSON Backend selection CMakeLists.txt (SAE_INFERENCE_BACKEND, SAE_GEMM_BACKEND) ORT/TRT + ROCm/CUDA, chosen at build time New-media queue JRay plugin \u2192 GET /Plugins/JRay/Tasks/Pending Backlog of items with no results yet Worker loop scripts/run_from_jellyfin.py--worker Poll Pending \u2192 run scene_analyze \u2192 push results Result push PUT /Plugins/JRay/Items/{id}/Truth Stores per-actor scene windows back in Jellyfin Incremental gallery scripts/make_jellyfin_gallery.py--merge Embeds only cast not already in the gallery Secrets loader .env via scripts/sae_env.pyJELLYFIN_URL, JELLYFIN_API_KEY, TMDB_API_KEY"},{"location":"service-conversion/#installer-config","title":"Installer config","text":"

    One file. Build-time settings (fixed when we compile) vs. run-time settings (in the worker's .env, editable without recompiling).

    # install.yaml \u2014 consumed by scripts/build_install.py\n\nplatform: nvidia            # nvidia | amd | cpu \u2192 picks the cmake backend\nmodel:\n  arcface: LVFace-B_Glint360K.onnx   # embedder compiled against; gallery MUST match\nschedule:\n  gallery_scan_interval: 24h         # incremental --merge cadence; 0 disables the scanner\nprefix: ~/.local                     # install root (bin, share, systemd user units)\n\n# runtime (written to the worker .env, not compiled in):\nruntime:\n  jellyfin_url: http://localhost:8096\n  # JELLYFIN_API_KEY / TMDB_API_KEY are filled into .env by hand after install\n

    Secrets never go in the repo or a build artifact \u2014 the installer writes a .env under the install prefix with blanks for the keys, and you fill them in once. sae_env.py already loads it.

    Model \u21c4 gallery coupling (guard, don't just document): embeddings from different recognition models aren't interchangeable. We compile against one embedder; the gallery must be built with the same one. Stamp the embedder name into gallery.json, and have the worker refuse to start if the gallery's embedder \u2260 the configured model.arcface, rather than silently mismatching.

    "},{"location":"service-conversion/#dependencies-via-the-system-package-manager","title":"Dependencies via the system package manager","text":"

    The heavy build/runtime deps (OpenCV, ffmpeg, the GPU stack) are best provided by the distro, not vendored. The installer ships a per-distro dependency list and either installs them or prints the exact command. Targets: Fedora (dnf) and Arch (pacman) first.

    Dependency Fedora (dnf) Arch (pacman) OpenCV opencv-developencv ffmpeg ffmpeg-free/ffmpeg (RPM Fusion) ffmpeg CMake / toolchain cmake gcc-c++cmake gcc CUDA + TensorRT (nvidia) NVIDIA CUDA repo + libnvinfer-*cuda, tensorrt ROCm (amd) rocm-hip-sdk / rocblas-develrocm-hip-sdk, rocblas ONNX Runtime not packaged \u2014 installer fetches a pinned release tarball into the prefix AUR onnxruntime (or same pinned-tarball fallback)

    So the flow is: detect distro \u2192 check each package \u2192 install via the native manager (or print sudo dnf install \u2026 / sudo pacman -S \u2026), with ONNX Runtime as the one known gap the installer fills itself (a pinned upstream release extracted under the install prefix, so it doesn't depend on a system package that may not exist). CUDA/ROCm being present is assumed \u2014 you already run a GPU desktop; the installer verifies and points you at the vendor repo if not.

    "},{"location":"service-conversion/#what-build_installpy-does","title":"What build_install.py does","text":"
    build_install.py install.yaml\n  \u2502\n  \u251c\u2500 detect distro (dnf vs pacman) and platform from config\n  \u251c\u2500 ensure deps: install via manager, or print the exact command; fetch ONNX Runtime if needed\n  \u251c\u2500 cmake + build scene_analyze with the platform's backend flags:\n  \u2502     nvidia \u2192 -DSAE_INFERENCE_BACKEND=TRT -DSAE_GEMM_BACKEND=CUDA\n  \u2502     amd    \u2192 -DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=ROCM\n  \u2502     cpu    \u2192 -DSAE_INFERENCE_BACKEND=ORT  (CPU EP; slow, for smoke tests)\n  \u251c\u2500 install into <prefix>:\n  \u2502     bin/sae-scene-analyze           the compiled binary\n  \u2502     share/sae-worker/               Python glue + a venv (requests, etc.), models/\n  \u2502     share/sae-worker/.env           runtime config (keys blank, url from config)\n  \u251c\u2500 install systemd --user units:\n  \u2502     sae-worker.service              runs the worker + gallery-scan supervisor\n  \u2502     sae-lock-gate.service           watches logind lock/unlock, start/stops the worker\n  \u2514\u2500 print next steps (edit .env, `systemctl --user enable --now sae-lock-gate`)\n
    "},{"location":"service-conversion/#the-worker-service-supervisor","title":"The worker service (supervisor)","text":"

    sae-worker.service runs a small Python supervisor as its main process:

    "},{"location":"service-conversion/#the-lock-gate","title":"The lock gate","text":"

    sae-lock-gate.service runs a tiny watcher that subscribes to logind lock/unlock signals and drives the worker service:

    screen locks   \u2192 systemctl --user start sae-worker.service\nscreen unlocks \u2192 systemctl --user stop  sae-worker.service   (SIGTERM)\n

    Screen-lock is the only signal \u2014 deliberately. We don't also gate on GPU/CPU load, because our own worker is the load: a load threshold would form a feedback loop (worker starts \u2192 GPU spikes \u2192 threshold trips \u2192 worker stops \u2192 load drops \u2192 restart \u2192 \u2026). Lock state is external to what the worker does, so it can't oscillate.

    Signal source is desktop-dependent: logind Lock/Unlock (GNOME/KDE via loginctl/D-Bus) covers most setups; a swayidle/xss-lock hook is the fallback for wlroots/X-only compositors. The installer picks based on what's present.

    "},{"location":"service-conversion/#on-resume-hard-stop-re-queue-its-free","title":"On resume: hard stop + re-queue (it's free)","text":"

    Stopping the worker mid-analysis costs nothing to reschedule, because of how the JRay queue works: an item only leaves /Tasks/Pending once its results are pushed (push_truth). A worker stopped mid-scene_analyze simply leaves that item Pending \u2014 next lock picks it up again. No re-queue bookkeeping.

    Two small correctness requirements (the only worker changes needed):

    1. Never push a partial result. Already true \u2014 push_truth runs only after scene_analyze returns; a killed run pushes nothing. \u2713 (keep it that way).
    2. Clean up on signal. process_item writes a temp filtered-gallery file and unlinks it in a finally; a SIGKILL skips finally. Fix: write temps under a dir the worker wipes on start, and/or a SIGTERM handler that unlinks before exit. Minor.

    Accepted trade-off: a partially-analysed title restarts from scratch next lock. Fine for an overnight/idle workload; no mid-video checkpointing.

    "},{"location":"service-conversion/#the-end-to-end-ux","title":"The end-to-end UX","text":"
    # once: build + install for your GPU + model\n./scripts/build_install.py install.yaml\n    # detects Fedora/Arch, ensures deps, compiles, installs units under ~/.local\n\n# once: set your keys, enable the gate\n$EDITOR ~/.local/share/sae-worker/.env     # JELLYFIN_API_KEY, TMDB_API_KEY\nsystemctl --user enable --now sae-lock-gate.service\n\n# from then on: nothing. Lock your screen \u2192 it analyses. Unlock \u2192 it stops.\n

    No Docker, no GPU passthrough config, no media re-mounting \u2014 the worker sees the same filesystem and GPU as everything else on the box.

    "},{"location":"service-conversion/#implementation-plan-follow-up-commits","title":"Implementation plan (follow-up commits)","text":"

    Ordered so each step stands alone:

    1. installer skeleton \u2014 scripts/build_install.py: parse install.yaml, distro detect, dependency check/print (start with cpu platform so it builds without a GPU), cmake+build, copy into prefix.
    2. supervisor + cleanup \u2014 scripts/service.py (worker loop + gallery-scan timer + SIGTERM); temp-file cleanup fix in run_from_jellyfin.py.
    3. systemd units + lock gate \u2014 generate/install sae-worker.service, sae-lock-gate.service, and the logind lock watcher.
    4. gallery/model guard \u2014 stamp embedder into gallery.json; startup mismatch check.
    5. platform + distro matrix \u2014 nvidia/amd backends; dnf/pacman dep lists; ONNX Runtime fetch fallback.
    6. docs \u2014 README \"Run on your idle GPU\" section.
    "},{"location":"service-conversion/#settled-decisions","title":"Settled decisions","text":""},{"location":"service-conversion/#open-questions","title":"Open questions","text":"

    (none blocking \u2014 the spec above is buildable as-is.)

    "}]} \ No newline at end of file diff --git a/service-conversion/index.html b/service-conversion/index.html index 56a7f8a..7918c2d 100644 --- a/service-conversion/index.html +++ b/service-conversion/index.html @@ -1035,7 +1035,7 @@ lock-gating, not new pipeline logic.

    - + @@ -1045,7 +1045,7 @@ lock-gating, not new pipeline logic.

    - + @@ -1055,12 +1055,12 @@ lock-gating, not new pipeline logic.

    - + - +
    Backend selectionCMakeLists.txt (SAE_INFERENCE_BACKEND, SAE_GEMM_BACKEND)CMakeLists.txt (SAE_INFERENCE_BACKEND, SAE_GEMM_BACKEND) ORT/TRT + ROCm/CUDA, chosen at build time
    Worker loopscripts/run_from_jellyfin.py--workerscripts/run_from_jellyfin.py--worker Poll Pending → run scene_analyze → push results
    Incremental galleryscripts/make_jellyfin_gallery.py--mergescripts/make_jellyfin_gallery.py--merge Embeds only cast not already in the gallery
    Secrets loader.env via scripts/sae_env.py.env via scripts/sae_env.py JELLYFIN_URL, JELLYFIN_API_KEY, TMDB_API_KEY