A face-recognition pipeline that finds when each actor appears on screen in a film or TV episode \u2014 built on KPN++ (a C++20 Kahn Process Network library) for the detect \u2192 track \u2192 match \u2192 scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
This is a perfect X-Ray second, on a film the optimizer never saw:
Every visible face named at 100% \u2014 Chris Noth, Hank Azaria, Bobby Cannavale \u2014 the background extra honestly left unnamed, and the two credited cast without a visible face correctly carried as present off-screen by the tracker's presence windows. That's the pipeline exactly reproducing Amazon X-Ray's record for this second.
It doesn't always go like that: the hardest held-out film scores 46% F1, and the report is honest about why \u2014 one tunable trade (extinction bridging at hard cuts), one structural ceiling (X-Ray credits people whose faces never appear), and a few cases where the pipeline is right and X-Ray is wrong. The evidence for all of it is in the pages below.
"},{"location":"#start-here-four-questions-this-bake-off-answers","title":"Start here \u2014 four questions this bake-off answers","text":"Which model is best?
Calibration curves first (discriminative power, independent of any threshold), then F1 on the actual benchmark. LVFace-B Glint360K wins both.
Whole vs. cast-restricted gallery
Restricting the matcher to a film's credited cast is a clean win on every axis (+3.3pp F1, less than a third the misIDs) \u2014 but isn't a shipped runtime feature yet.
Does pose expansion help?
A convincing training-set effect that didn't reproduce on 5 held-out films once two methodology bugs were caught and fixed. An honest null result, not a forced narrative.
Deep dive: LVFace-B Glint360K
The held-out generalization gap, how the error budget decomposes (extinction bridging at hard cuts, X-Ray's scene-membership vs. on-screen-face ceiling), and the frames where the pipeline is right and the ground truth is wrong.
src/config.hpp defaults come from.Gallery .h5 files, embedding dumps, the X-Ray corpus, montage frame images, and DE trajectories are not committed to this repository \u2014 they're pushed to the Gitea package registry and pulled on demand:
scripts/artifacts/pull_artifacts.sh galleries\nscripts/artifacts/pull_artifacts.sh experiment-data\nscripts/artifacts/pull_artifacts.sh montage-frames <film-slug>\nSee scripts/artifacts/push_artifacts.sh for the upload side (requires a GITEA_TOKEN with package write scope).
Four candidates went into the bake-off: three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K), a Vision-Transformer embedder that's a drop-in replacement for ArcFace's [N,3,112,112] input / 512-d output. The open question: is LVFace (455MB) actually better, or just the biggest?
Each gallery carries a fitted Platt sigmoid P(match | cosine similarity) = \u03c3(a\u00b7sim + b), embedded directly in the gallery's HDF5 file (src/gallery/gallery_calibration.hpp). This is a property of the embedding space alone \u2014 computed from intra/inter-actor reference-image pairs, no tracking or scene logic involved \u2014 so it's a clean first read on discriminative power before running a single benchmark.
a (steepness) boundary at P=0.5 LVFace-B Glint360K 17.7 sim 0.228 ArcFace w600k-MBF 16.2 sim 0.267 ArcFace w600k-R50 15.4 sim 0.301 ArcFace R18 15.3 sim 0.309 LVFace has both the steepest transition and the lowest decision boundary \u2014 it separates same-actor from different-actor reference pairs more confidently, at a lower similarity threshold, than any ArcFace variant. That's a genuine head start before the tracking/scoring pipeline is even involved.
"},{"location":"best-model/#second-signal-f1-on-the-actual-benchmark","title":"Second signal: F1 on the actual benchmark","text":"Best full-gallery (no cast-restriction) result per model, from the 16-combo bake-off matrix (full experiment log):
model F1 P R misID LVFace-B Glint360K 75.3% 89.7% 65.4% 232 ArcFace w600k-MBF 74.2% 87.4% 64.4% 57 ArcFace R18 69.1% 87.6% 57.7% 242 ArcFace w600k-R50 68.5% 94.0% 54.1% 150The full 16-combo picture makes the model ordering visible at a glance \u2014 LVFace (yellow) tops both the restricted and full columns, and R18 (green) props up the bottom of the full-gallery ranking:
LVFace wins outright, with the highest recall of any full-mode combo. This reverses an earlier conclusion from a prior (superseded) benchmarking pass using a scene-union metric, which found the three models statistically indistinguishable (~85% each) and concluded LVFace wasn't worth its size \u2014 that metric hid out-of-cast false positives behind a gallery\u2229cast recall mask (see the prior optimizer round); the per-second metric used here does not.
Held-out validation (5 films never seen by the optimizer) confirms LVFace's lead holds up out of sample \u2014 see the LVFace deep dive for the full breakdown, including where it fails.
"},{"location":"best-model/#caveat-model-choice-is-an-operational-change","title":"Caveat: model choice is an operational change","text":"Switching the default embedder isn't just flipping a config value \u2014 the gallery itself is model-specific (embeddings from different models aren't comparable), so any existing gallery built against ArcFace w600k-R50 needs to be rebuilt from source images against LVFace before the new default takes effect. scripts/optimizer/reembed_gallery.py does this from a reference gallery's cached source images without re-downloading anything.
Two ways to run the matcher: full scores every detected face against the entire library gallery (2418 actors across the 9-film benchmark set); restricted pre-filters each film's gallery down to just its Jellyfin-credited cast (typically ~15 top-billed actors) before the matcher ever runs.
"},{"location":"gallery-scope/#the-result","title":"The result","text":"Averaged across all 4 models and both expansion settings, on the 4 bake-off training films:
scope F1 P R total misID (8 evals) full 71.2% 91.1% 59.0% 1073 restricted 74.5% 92.2% 62.9% 329This is not a precision/recall trade \u2014 restriction wins on every axis at once: +3.3pp F1, +3.9pp recall, and less than a third the total misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a look-alike false match (an actor who happens to share enough facial structure with someone in the film, but isn't actually in it), and the recall gain shows it isn't costing real detections to get there.
Per-model, every single model's best-scoring combo in the full 16-way matrix is a restricted variant \u2014 visible directly in the ranking below (filled dots = restricted, open = full; the filled dots cluster at the top for every color):
See the full table in the bake-off experiment log. Two combos hit zero true out-of-cast misidentifications: arcface_w600k_mbf_restricted_exp (F1 76.5%) and, in full mode, LVFace-B_Glint360K_full_noexp (F1 72.4%) \u2014 restriction isn't the only way to reach misid=0, but it's the more reliable one.
Cast-restriction is implemented today only as an offline optimizer technique (scripts/optimizer/cast_restrict.py): it pre-builds a filtered gallery file per film, using Jellyfin's own cast list, before the benchmark ever calls the matcher. There's no runtime \"restrict matching to this title's credited cast\" switch in the shipped application \u2014 scene_analyze always matches against whatever single gallery file it's given.
Building that as a real feature would need, at minimum:
scripts/run_from_jellyfin.py already does this same lookup for its own filter_gallery-based restriction path, just not wired into scene_analyze itself as a first-class option).This is why the shipped src/config.hpp defaults use the full-mode winner (LVFace-B_Glint360K_full_exp, F1 75.3% training / 67.4% held-out macro) rather than the higher-scoring restricted_exp (78.3%) \u2014 the 78.3% number describes a capability the app doesn't have yet, not what actually ships.
LVFace won the model bake-off (see Which model is best?) and is the shipped default embedder. This page is the honest accounting of how it actually performs \u2014 what a good second looks like, where the errors actually come from, and two cases where the ground truth itself is wrong and LVFace is right.
How to read the frames on this page
The top is the film frame, with a box and name on every face the pipeline identified. The bottom panels are the per-second verdict against X-Ray: Onscreen lists faces named in the frame, Offscreen lists cast X-Ray marks present in the scene without a visible face \u2014 presence carried by the tracker's windows, not by a detection. Colors are the score: green = correct (TPI), orange = wrong (FPI), blue = missed (FN).
"},{"location":"lvface-deep-dive/#what-good-looks-like","title":"What good looks like","text":"Six faces on screen, all six named correctly \u2014 including Penelope Wilton at the edge of the pews and a half-occluded Michelle Dockery \u2014 while thirteen more cast members X-Ray marks present in the scene are correctly carried as \"Offscreen\" by their presence windows. One miss in the whole frame: Maggie Smith (blue). Score for this second: 0.86.
The same film's funeral gathering: mourning dress, hats, half the faces turned. Nineteen of the twenty cast X-Ray lists for this scene are scored correctly \u2014 seven named on screen at up to 100% confidence, twelve more correctly held as present off-screen.
And the pipeline doesn't need the face to be real:
That's Herbie Hancock at 98% \u2014 as a face on a screen inside the movie, over a sci-fi HUD overlay, during a video call in Valerian. A face is a face, whether it's in the room or on the bridge's comms display.
"},{"location":"lvface-deep-dive/#training-vs-held-out-the-generalization-gap","title":"Training vs. held-out: the generalization gap","text":"The shipped config (prob_threshold=0.754, anneal_sec=35.54, extinction_sec=57.43, expand_gallery=true) was tuned against 4 films. Scored against the 5 films the optimizer never saw:
67.4% held-out vs. 75.3% on training \u2014 an ~8pp drop, and a 37pp spread between the best and worst held-out film. The config does not generalize uniformly, and the spread traces to two mechanisms, both visible frame by frame below.
"},{"location":"lvface-deep-dive/#mechanism-1-extinction-bridging-usually-right-wrong-at-hard-cuts","title":"Mechanism 1: extinction bridging \u2014 usually right, wrong at hard cuts","text":"The extinction window keeps an identity alive through seconds where no face is detectable. Most of the time this is exactly what you want, and it's where a lot of the TPI count comes from:
Lovelace's polygraph scene: only Eric Roberts and Amanda Seyfried have visible faces, but X-Ray lists eight cast present \u2014 and all eight score green, the other six correctly carried by presence windows through a scene where the camera never shows them. A perfect second, and the extinction/anneal machinery is why.
The same mechanism has a failure case: a hard cut into long faceless footage. Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst recall of the five) are dominated by it \u2014 verified directly against the raw per-frame stream and the HDF5 dump's own detection counts, not inferred from the score alone. This is not a malfunction: the tracker is doing exactly what its window is for; the footage just stops cooperating. In the debug overlay (which draws a bridged identity's last-known bbox, unlike the shipped output, which emits presence windows and no boxes at all) the bridged state is visible spatially:
Debug-overlay rendering (dump_error_frames.py --raw): \"Jon Bernthal\", \"Joey Diaz\" and \"Billy Magnussen\" are extinction-bridged identities from the previous shot, drawn frozen over the wall and the hanging plates. Frame many_saints/fpi/fpi_t03543.jpg, montage-frames artifact package.
The cost is measurable, not just visible. Downton Abbey's hard cut into its closing credits, plotting the dump's own per-second face_count (detector output, independent of the tracker) against what the tracker reports:
From the cut onward the detector sees zero faces for nearly a minute \u2014 and the tracker keeps reporting the last shot's 15 identities the whole time (verified for Hugh Bonneville: bbox (1743.2, 0.0, 171.3, 317.8), unchanged to the pixel, at every sampled second for 57+ seconds). The staircase at the right edge is the extinction window expiring actor by actor. That plateau is SceneTrackerFunc::active_[actor_idx].last_bbox (src/nodes/scene_tracker_node.hpp) re-emitted as designed: extinction_sec=57.4 was tuned long because bridging wins on most footage (see the polygraph frame above) \u2014 the training films just never contained a faceless stretch long enough to show the cost side, and the held-out set did.
The same track-continuation machinery has one milder spatial artifact, worth knowing when reading these frames:
Caf\u00e9 Society (a training film), a shot/reverse-shot dialog: that is Steve Carell wearing both his own label and Jesse Eisenberg's.
At a rapid cut, the previous shot's track can linger for a beat at nearly the same screen position the new face occupies \u2014 here Jesse Eisenberg's box from the counter-shot lands on Steve Carell. Note what the score panel says, though: both actors are green, because both are present in this dialog scene per X-Ray. The spatial label is briefly wrong; the per-second presence claim \u2014 the thing the pipeline actually ships \u2014 is right. It's the same trade as the extinction window: track continuation smooths over cuts, and 1 fps sampling occasionally catches the seam.
"},{"location":"lvface-deep-dive/#mechanism-2-the-face-vs-presence-ceiling","title":"Mechanism 2: the face-vs-presence ceiling","text":"Downton Abbey's recall didn't collapse because faces were misread \u2014 it collapsed because for most of its 80084 FN-seconds there was no face to read:
A newsreel crew hauls equipment through the hall: X-Ray credits 22 cast as present in this scene; not one face looks at the camera. Eight are still scored green (windows bridging from adjacent shots) \u2014 the other fourteen are blue FNs that no face-recognition pipeline could ever recover. X-Ray encodes scene membership; the pipeline measures on-screen faces. In ensemble films those two definitions diverge massively, and that gap \u2014 not identification error \u2014 is most of what the FN column counts.
Same ceiling from the other side: Michela De Rossi in frame but turned away, five cast correctly bridged as offscreen (green), four blue FNs \u2014 and one orange we'll come back to below.
"},{"location":"lvface-deep-dive/#where-lvface-beat-x-ray","title":"Where LVFace beat X-Ray","text":"Not every orange in these frames is actually wrong. scripts/optimizer/second_score.py scores strictly against X-Ray \u2014 but X-Ray itself has holes, and the pipeline found two kinds.
Germar Terrell Gardner \u2014 a real, clean, high-confidence detection \u2014 is counted as an out-of-cast misID because he doesn't appear in X-Ray's people.csv for The Many Saints of Newark at all. But Jellyfin's independent cast metadata does credit him for this exact film (cross-checked via experiments/manifests/jellyfin_casts.json from the experiment-data artifact package, a completely separate data source from X-Ray). That's also him in orange in the frame above \u2014 every one of those \"errors\" is the pipeline being right about a person X-Ray forgot.
And it isn't only uncredited bit-parts. That is Robert Patrick \u2014 top-billed in Lovelace, unmistakably on screen, reading his newspaper, identified at 100% \u2014 scored orange because X-Ray's people-in-scene list for this scene doesn't include him. The identification is flawless; the ground truth missed an actor sitting in the middle of the frame.
This doesn't mean every flagged misID is secretly correct \u2014 Many Saints' 974-count total is still overwhelmingly extinction bridging at cuts, not uncredited cameos. But the X-Ray corpus is a convenient, large-scale ground truth, not a perfect one, and the misID/FPI numbers in these tables carry an irreducible noise floor from ground-truth gaps in both directions.
"},{"location":"lvface-deep-dive/#summary","title":"Summary","text":"LVFace is the right default: it wins the model comparison outright, it names 19 of 20 correctly across a hat-heavy funeral crowd, and it recognises a face on a screen inside the movie. Its error budget decomposes into two understood mechanisms \u2014 extinction bridging at hard cuts (a tunable trade, not a bug) and the face-vs-presence ceiling baked into X-Ray's semantics \u2014 plus a nonzero slice where the pipeline is right and the ground truth is wrong. The held-out generalization gap (75.3% \u2192 67.4%) is real and should be treated as the honest expected performance, not the training-set number.
"},{"location":"model-bakeoff/","title":"Model bake-off + threshold re-tune \u2014 experiment log (2026-07-18/19)","text":"Follow-on to the prior optimizer round, which used an older, since-superseded scene-union metric. This round uses the per-second metric (scripts/optimizer/second_score.py) and answers three questions in one 16-run matrix: which embedding model is best, does cast-restriction cut misIDs, and does per-film gallery expansion help.
(This was the fourth optimizer campaign against the X-Ray corpus, so its on-disk artifacts carry an internal rep4_ prefix \u2014 experiments/results/rep4_best_*.json, experiments/trajectories/rep4_*.jsonl, and the manifests referenced below. The earlier campaigns used the superseded scene-union metric and were discarded.)
Four goals going in, and an honest read on each after held-out validation (see below):
extinction_sec/threshold interactions and gallery coverage, not the embedder.prob_threshold \"earns\" a longer extinction window before it starts hurting).src/config.hpp","text":"knob old default new default why arcface_model arcface_w600k_r50.onnx LVFace-B_Glint360K.onnx Best F1 in the full-gallery bake-off (75.3% vs r50's 68.5%). LVFace was worth its size. prob_threshold 0.76 0.754 Re-tuned for LVFace + per-second metric. extinction_sec 1.5 57.4 Reverses the earlier \"short is better\" finding \u2014 see below. anneal_sec 10.0 35.5 Same reversal; previously thought insensitive. expand_gallery false true Helped recall on the full (unrestricted) gallery for the winning model \u2014 opposite of the earlier assumption. These are the LVFace-B_Glint360K_full_exp winning values, applied to src/config.hpp \u2014 the best result that uses only features already live in the running app (full gallery, no cast restriction; see below for why restricted mode isn't applied even though it scored higher).
The prior round's scene-union metric hid out-of-cast false positives behind a gallery\u2229cast recall mask, and the earlier 9-film benchmark was scene-level (union over a whole X-Ray scene), not a fair per-timepoint comparison. SESSION_STATE flagged the old scene-metric bake-off numbers (R50\u2248LVFace\u2248MBF ~85%) as superseded. This round uses second_score.py: uniform per-second sampling, GT = X-Ray scene's cast at time t, pred = actors whose presence window covers t, FPI weighted 10\u00d7 when the named actor isn't in the film's cast at all (true misID) vs. an in-cast timing slip. FN only counts gallery-known cast (fair recall \u2014 67% of X-Ray cast have no reference embedding; see the prior round's gallery-coverage-gap analysis).
Every replay in this line of work goes through scripts/optimizer/replay.py, which runs the real C++ tracker/matcher/scene_tracker nodes inside a Python-assembled KPN network. Before this session, every subprocess replay timed out at 45s, 100% of the time \u2014 not the documented ~20-30% ROCm rocBLAS-GEMM driver flake, but a plain logic bug: replay.py's CLI called replay(net, ..., stop=False) to skip net.stop() (trying to dodge the GEMM deadlock), planning to os._exit(0) immediately after. But:
PyNode::stop() (include/kpn/python/bindings.hpp in the KPN++ submodule) is the only code that sets stop_flag_ = true before joining the node's worker thread.run_loop() has while (!stop_flag_) as its only exit condition (it has no input channels, so it never sees a channel-closed signal either).stop() meant stop_flag_ never became true. When replay() returned, its local net went out of scope immediately, running ~PyNetwork \u2192 ~PyNode \u2192 thread_.join() synchronously inside replay()'s own call frame \u2014 before main() ever got control back to run os._exit(0).Root-caused via gdb -p <pid> -batch -ex \"thread apply all bt\" on a hung process: the main thread was stuck in ~PyNode's jthread::join(); the worker thread was in an ordinary time.sleep() inside the Python source callback, waiting for a stop signal that was never sent. The two HSA kfd_wait_on_events threads visible in the same trace are normal ROCm runtime housekeeping, not evidence of a wedged GPU kernel.
Fix: replay.py now calls replay(..., stop=True) (the removed stop=False + os._exit workaround was actively harmful). Verified 3/3 clean runs at ~8s each (down from a guaranteed 45s timeout), and a full DE sweep producing real, sensible F1/precision/recall instead of flat 0.0%.
With the deadlock fixed, scripts/optimizer/optimize.py was extended with DE-level parallelism \u2014 differential_evolution(..., workers=ThreadPoolExecutor.map) \u2014 so multiple population candidates evaluate concurrently, each spawning its own per-film replay subprocesses (REPLAY_WORKERS). Total concurrent GPU replay processes \u2248 DE_WORKERS \u00d7 REPLAY_WORKERS.
REPLAY_WORKERS=3, no DE parallelism) baseline, GPU underutilised 6 (DE_WORKERS=2 \u00d7 REPLAY_WORKERS=3) clean, real scores, ~1 isolated timeout per run 8 (DE_WORKERS=2 \u00d7 REPLAY_WORKERS=4, 4-film manifest) clean, real scores 9 (DE_WORKERS=3 \u00d7 REPLAY_WORKERS=3) broken \u2014 every replay blew past the 45s timeout, all scores silently degraded to 0.0% 9 concurrent replays looks like valid output (well-formed JSON, a real number) while actually being garbage \u2014 a dangerous failure mode, not a crash. 8 concurrent is the practical ceiling on this GPU (gfx1100) for this workload. The matrix ran at REPLAY_WORKERS=4 DE_WORKERS=2.
9 films total have dumped embeddings across all 4 models. 4 were used for optimization, leaving 5 held out for validation:
Held out: Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of Newark, Valerian and the City of a Thousand Planets. The training-set numbers below are training-set fit \u2014 see \"Held-out validation\" further down for the real generalization test.
"},{"location":"model-bakeoff/#search-space-and-de-settings","title":"Search space and DE settings","text":"popsize=10, maxiter=15 (3 params \u2192 \u2264480 evals/combo ceiling; DE's tol convergence usually stops earlier). anneal_sec/extinction_sec bounds were widened from 1\u201330/1\u201315 to 1\u201360/1\u201360 mid-run (see below) \u2014 the 4 arcface_w600k_r50 combos finished before the widening and still use the old, narrower bounds, so they are not directly comparable to the other 12 on those two params. Re-running r50 with the wider bounds was deferred (diminishing-returns judgment call, not yet done).
Ranked by F1. misid = FPI_misid, count of true wrong-actor identifications (an actor named who isn't in the film's cast at all) \u2014 distinct from FPI, which includes in-cast timing slips.
\u2020 old, narrower anneal/extinction bounds (see above) \u2014 not directly comparable to the other 12 on those two params.
The same 16 results as a picture \u2014 the two headline effects are visible without reading a single row: filled (restricted) dots stack the top of the ranking for every model color, and yellow (LVFace) leads within both scopes:
"},{"location":"model-bakeoff/#calibration-curves-discriminative-power-independent-of-the-threshold","title":"Calibration curves \u2014 discriminative power, independent of the threshold","text":"Each model's gallery carries a fitted Platt sigmoid P(match | sim) = \u03c3(a\u00b7sim + b) (embedded directly in the gallery HDF5, see src/gallery/gallery_calibration.hpp). Plotting all four side by side shows discriminative power directly, independent of whatever prob_threshold a particular run happened to use:
LVFace-B has both the steepest curve (a=17.7, vs. 15.3\u201316.2 for the ArcFace variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs. 0.27\u20130.31) \u2014 it separates same-actor from different-actor pairs more confidently at a lower similarity, consistent with it winning the full-gallery F1 comparison below. Generated by scripts/docs/calibration_chart.py (requires each gallery to have been calibrated at least once \u2014 run any replay against it first).
The matrix crosses two independent variables \u2014 averaging across all 4 models isolates each one from model choice:
Gallery scope (whole 2418-actor gallery vs. restricted to the film's credited cast) \u2014 averaged over both expansion settings and all 4 models:
scope F1 P R total misID (16 evals\u21928 each) full 71.2% 91.1% 59.0% 1073 restricted 74.5% 92.2% 62.9% 329Restriction wins outright on every axis \u2014 not a precision/recall trade, a clean win: +3.3pp F1, +3.9pp recall, and less than a third the total misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a look-alike false match, and (per the recall gain) doesn't cost real detections. This is the single cleanest signal in the whole matrix \u2014 stronger than the model choice itself \u2014 which is exactly why cast-restriction becoming a real runtime feature (not just an optimizer trick) is the top item in Caveats below.
Pose expansion (promoting a confidently-identified track's novel-pose views into a per-film gallery annex \u2014 src/gallery/track_gallery.hpp) is smaller and interacts with scope rather than acting independently:
In restricted mode, expansion is a clean win (+1.8pp F1, +3.2pp recall, misID actually drops) \u2014 the annex only ever competes against the film's own ~15-actor cast, so a \"confidently identified, new pose\" view is unlikely to be mistaken for someone else. In full mode, expansion buys essentially nothing on F1 (71.2% \u2192 71.2%, recall +1.4pp) while quadrupling misIDs (209 \u2192 864): a novel-pose view promoted into the annex now competes against the whole 2418-actor gallery, so a \"confident\" identity is confident against the wrong universe of candidates \u2014 the expansion mechanism is \"learning\" a pose correctly, but the enlarged evidence pool makes it easier for that learned pose to look like a plausible match for a different actor. Practical takeaway: gallery expansion should be paired with cast restriction, not used on the full gallery \u2014 the version currently shipped as default (full_exp, see TL;DR) sits in the worse of these four cells for this specific knob, even though it's the best available combo without cast-restriction support in the app yet (see Caveats).
restricted. It isn't just precision-safe: arcface_w600k_mbf_restricted_exp and LVFace-B_Glint360K_full_noexp both hit misid=0 \u2014 zero true wrong-actor identifications. But restriction is an offline optimizer technique, not a live app feature \u2014 it pre-filters each film's gallery to its Jellyfin-credited cast before the matcher ever runs; there's no runtime \"restrict to this film's cast\" switch in the app today. Implementing it for real is future work, tracked separately from this defaults update.expand_gallery) is mode-dependent. It helps on restricted galleries (smaller, so novel-pose promotion adds real signal) and on LVFace's full gallery, but hurts r50 and mbf in full mode (compare arcface_w600k_r50_full_exp 68.5% vs full_noexp 71.6%). Don't assume it's a free win \u2014 model- and mode-dependent.anneal_sec/extinction_sec kept pinning at the search ceiling. With the original 1\u201330/1\u201315 bounds, 3 of 4 r50 combos landed at ~93-98% of the upper bound. Widened to 1\u201360/1\u201360 mid-run (after the r50 combos had already finished) \u2014 every subsequent combo's best config landed at ~90%+ of the new ceiling too (e.g. the LVFace winner: ann=59.2, ext=59.2, both ~99% of 60). The likely mechanism: a strict prob_threshold \"earns\" a long extinction/anneal window \u2014 once false matches are rare, a long window just bridges real presence gaps (occlusion, turned face) instead of smearing false positives into later scenes, which is what made short windows look better under the old, laxer thresholds. Open question, not resolved: does this keep climbing past 60s, or does it actually plateau there? Decided not to chase further this round (diminishing-returns judgment call) \u2014 flag for a future sweep if it matters.The ceiling-pinning is visible in the raw search itself. Every one of the 512 DE evaluations for the winning combo, plotted over the prob_threshold \u00d7 extinction_sec plane:
The dark band hugging the top edge is the finding: nearly everything scoring well sits at extinction_sec \u2265 50, across a wide range of thresholds, and the population converged into a dense cloud around the optimum (threshold ~0.70\u20130.80, extinction pinned at the 60s bound). Short extinction windows (bottom half) are uniformly pale \u2014 under a strict threshold there is simply no good configuration down there. Generated by scripts/docs/experiment_charts.py from the DE trajectories (experiments/trajectories/*.jsonl, part of the experiment-data artifact package).
The 16-combo matrix above is training-set fit. This is the real test: the shipped config (LVFace-B_Glint360K_full_exp \u2014 prob_threshold=0.754, anneal_sec=35.5, extinction_sec=57.4, expand_gallery=true) replayed against the 5 films never seen by the optimizer (Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of Newark, Valerian and the City of a Thousand Planets), scored the same way.
67.4% held out vs. 75.3% on training \u2014 an ~8pp drop, and a much more informative number than the training-set F1 alone: a 37pp spread between best and worst film (83.0% vs 46.3%). The config does not generalize uniformly.
Two films are outright failure cases, and rendering bounding boxes + names on the extracted frames (replay.py --raw-out + dump_error_frames.py--raw, see Reproduce) turned what looked like a same-scene misidentification into something more precise and more damning:
Frame many_saints/fpi/fpi_t03543.jpg from the montage-frames artifact package (scripts/artifacts/pull_artifacts.sh montage-frames Many_Saints_of_Newark).
Both are the same mechanism, and it can be measured, not just screenshotted. Plotting the dump's own per-second face_count (detector output, independent of the tracker) against the number of actors the tracker reports, through Downton Abbey's cut to credits:
From the cut onward the detector sees zero faces \u2014 yet the tracker holds a perfectly flat plateau of 15 reported identities for 56 seconds, each with the exact same bbox, unchanged to the pixel (verified for Hugh Bonneville: (1743.2, 0.0, 171.3, 317.8) at every sampled second from 7222 through 7279+). The staircase on the right edge is the extinction window finally expiring, actor by actor. That plateau is SceneTrackerFunc's active_[actor_idx].last_bbox (src/nodes/scene_tracker_node.hpp) being re-emitted unchanged \u2014 the extinction state machine working exactly as coded, not a bug in the logic. The film cuts from a packed group shot straight into ~40+ seconds of blank titles/credits with zero faces, and extinction_sec=57.4 is comfortably long enough to bridge that entire gap without expiring, so the tracker faithfully keeps reporting \"last known position\" for a cast that is no longer on screen at all.
This reframes the \"long extinction window wins\" DE-search pattern (see above): it isn't unambiguously good. It buys recall by bridging real gaps (occlusion, turned face) in some films, but on others \u2014 specifically, hard cuts into long faceless footage \u2014 it manufactures a frozen-bbox ghost the tracker has no way to verify, precisely the failure mode the original short-extinction-window default (1.5s) was chosen to avoid. The training-set films apparently didn't have a long enough faceless stretch after a confirmed identity to expose this; the held-out set did.
Frames for all three films (benny_joon, many_saints, downton_abbey \u2014 one strong performer, two failure cases) are under experiments/results/holdout/frames/ (not committed \u2014 pull per film with scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>), each with a manifest.json listing the bucket (best/fpi/fn), timestamp, and predicted vs. ground-truth actors for every dumped frame. Frames are annotated with bounding boxes + name/confidence (green = identified, orange = unknown), matching src/nodes/debug_renderer_node.hpp's colour convention. Generated by scripts/optimizer/dump_error_frames.py --raw <replay.py --raw-out output> (see Reproduce).
dump_error_frames.py --interval-sec 600 also supports a per-N-second sweep instead of the fixed best/fpi/fn buckets: one best (highest Jaccard) and one worst (lowest Jaccard) frame per 10-minute window across the whole film, e.g. experiments/results/holdout/frames/many_saints_intervals/ (13 windows \u00d7 2 = 26 frames for the ~2h Many Saints runtime) \u2014 a way to sample \"how are we doing\" evenly across a film's runtime rather than only at its most extreme seconds.
anneal_sec/extinction_sec.full_exp, not the higher-scoring restricted_exp, because cast-restriction isn't a real runtime feature yet (see above). The 78.3% F1 number is not what the shipped defaults will produce \u2014 75.3% is.full_exp is the best full-gallery combo, but not the safest. Per the isolated-effects analysis above, expand_gallery=true only cleanly pays off when paired with cast-restriction; on the full gallery it's flat on F1 while ~4x-ing misIDs (209\u2192864, averaged across models). full_noexp scores lower (72.4% vs 75.3% for LVFace) but with zero true misIDs and higher precision (94.2% vs 89.7%). Kept full_exp as shipped since it's the highest-F1 option available without cast-restriction, but this is a real F1-vs-safety trade, not a strictly-better choice \u2014 worth revisiting if misID rate matters more than the last few points of F1 for a given deployment.# 4-film matrix, all 4 models \u00d7 2 modes \u00d7 2 expansion settings\nbash experiments/run_rep4_subprocess.sh\n\n# single combo\nSAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \\\n --manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \\\n --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \\\n --params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \\\n --popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json\n\n# replay the shipped config against a held-out film \u2014 --raw-out is needed to draw\n# bboxes later (the merged pred.json has no per-frame bbox, only actor windows)\npython3 scripts/optimizer/replay.py \\\n --dump experiments/dumps/LVFace-B_Glint360K/dump_<slug>.h5 \\\n --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \\\n --out pred.json --raw-out raw.jsonl --prob-threshold 0.754 --anneal-sec 35.54 \\\n --extinction-sec 57.43 --expand-gallery\n\n# regenerate the report's charts (16-combo ranking, DE landscape, held-out\n# per-film F1, Downton ghost timeline) from the artifacts under experiments/\npython3 scripts/docs/experiment_charts.py --out-dir docs/assets/images\n\n# dump example frames (best-agreement / FPI / FN) for visual inspection, annotated\n# with bounding boxes + names (--raw is optional; omit for unannotated frames)\npython3 scripts/optimizer/dump_error_frames.py \\\n --pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \\\n --movie \"<path to source video>\" \\\n --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \\\n --out-dir experiments/results/holdout/frames/<name> --n-per-bucket 4\n\n# or: one best + one worst frame per 10-minute window across the whole film\npython3 scripts/optimizer/dump_error_frames.py \\\n --pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \\\n --movie \"<path to source video>\" \\\n --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \\\n --out-dir experiments/results/holdout/frames/<name>_intervals --interval-sec 600\nSee also: the prior optimizer round (superseded metric) and the session log experiments/SESSION_STATE.md.
Record of the July 2026 work that tuned the pipeline's recognition/tracking defaults against ground-truth per-scene actor presence, and the tooling built to do it.
"},{"location":"optimizer-experiments/#tldr-what-changed","title":"TL;DR \u2014 what changed","text":"knob old default new default whyprob_threshold 0.99 0.76 0.99 was far too strict \u2014 halved recall for a fraction of a precision point. DE optimum, tightly converged. extinction_sec 5.0 1.5 Long extinction smears presence into later scenes \u2192 FPs. DE converged tightly low. anneal_sec 10.0 10.0 (unchanged) DE found it insensitive (F1 flat \u00b10.3pp across 3\u201326s) \u2014 kept the round default. detector_conf 0.5 0.5 (unchanged) Sweep showed raising it only trades recall for precision at a net F1 loss \u2014 near-threshold detections are real faces, not phantoms. Net effect on the 9-film benchmark (strict per-scene, augmented gallery): recall 58% \u2192 ~72%, F1 70% \u2192 ~76%, precision ~85%, at no meaningful precision cost.
"},{"location":"optimizer-experiments/#ground-truth","title":"Ground truth","text":"Public scene-level Amazon X-Ray dataset (Zenodo DOI 10.5281/zenodo.17659734, CC-BY-4.0): per movie, people.csv (name_id/person/character), scenes.csv (scene/start/end ms), people_in_scenes.csv. Films matched to the library by an authoritative Jellyfin ID join (query /Items?IncludeItemTypes=Movie&Fields= ProviderIds,Path, join Imdb/Tmdb against X-Ray metadata) \u2014 NOT fuzzy title matching, which collides badly (TV episodes vs same-named films). 9 genuine films with source video on disk: Benny & Joon, Caf\u00e9 Society, Downton Abbey: A New Era, Lord of War, Lovelace, The Many Saints of Newark, Scarface, Sound of Metal, Valerian.
Comparison unit is the X-Ray scene, not sampled timepoints. For each scene [start,end]: predicted set = union of actors detected anywhere in the span; GT set = actors X-Ray lists for that scene. Per scene TP/FP/FN, then:
prob_threshold to the 0.50 floor \u2014 a metric artifact. Counting them is essential.recall (fair) and recall_strict (all) are reported.Implemented in scripts/optimizer/scene_score.py \u2014 since removed along with this metric; its per-second successor is scripts/optimizer/second_score.py (see the bake-off round).
Diagnosing low recall: only 131 of 392 X-Ray cast were in the gallery (33%). Every in-gallery actor HAD embeddings (gallery well-formed) \u2014 the gap was pure coverage. scripts/optimizer/fetch_missing_actors.py recovers missing actors: nm-id \u2192 TMDB /find external_ids \u2192 /person/{id}/images \u2192 download \u2192 embed (sae_embed), with a --wikidata fallback (P345\u2192P18 Commons photo).
Coverage vs detectability. Adding references lifted recall (58\u219268% at fixed config) but modestly. Per-film drill-down (Lord of War: 12 actors recovered, only 1 had a detectable on-camera face) showed most missing cast are a detectability gap \u2014 X-Ray credits them as cast-in-scene (incl. off-camera/background), but their face never appears clearly for the pipeline to detect. This is a fundamental ceiling of a face-recognition pipeline vs X-Ray's presence semantics, not a fixable gap.
"},{"location":"optimizer-experiments/#optimizer","title":"Optimizer","text":"scripts/optimizer/optimize.py \u2014 scipy differential_evolution over the knob space, each candidate = full replay of all films through the real C++ nodes (see the KPN replay architecture below) scored by the metric above. Global objective (one config for all films, not per-film).
Convergence stability (augmented gallery, 233 evals):
knob top-20 range verdictprob_threshold 0.69\u20130.83 (\u03c3 0.05) TIGHT \u2014 trust 0.76 extinction_sec 1.0\u20132.2 (\u03c3 0.33) TIGHT \u2014 trust 1.5 anneal_sec 3.1\u201326.3 (\u03c3 6.4) LOOSE \u2014 insensitive, not hard-coded F1 varied only 0.3pp across the top-20 \u2192 objective is flat near the optimum, so only the tightly-converged knobs were adopted as defaults.
"},{"location":"optimizer-experiments/#replay-architecture-how-the-sweep-is-cheap","title":"Replay architecture (how the sweep is cheap)","text":"The optimizer never re-decodes video. scene_analyze --dump-embeddings out.h5 runs the expensive half once (decode\u2192detect\u2192align\u2192embed) and dumps per-frame face embeddings + metadata to HDF5 (scripts/optimizer/SCHEMA.md). scripts/optimizer/replay.py then replays that dump through the real C++ face_tracker \u2192 identity_matcher \u2192 scene_tracker assembled in a Python KPN network (sae_kpn nanobind module), varying Config knobs freely \u2014 no GPU embedding, no decode. Verified BYTE-EXACT against scene_analyze's own output. The dumps are gallery-independent, so testing the augmented gallery needed no re-dump. detector_conf is replayable UPWARD only (the dump floor is 0.5).
# 1. dump (once per film, needs video)\nscene_analyze --movie <f> --gallery gallery.json --dump-embeddings dump.h5 --fps 1\n# 2. build films manifest by Jellyfin ID join (see scripts/optimizer notes)\n# 3. optimize\npython scripts/optimizer/optimize.py --manifest films.json --gallery gallery.json \\\n --params prob_threshold:0.5:0.999 anneal_sec:1:30 extinction_sec:1:15 \\\n --popsize 8 --maxiter 20 --trajectory traj.jsonl --out opt.json\n# 4. score a fixed config / validate on a held-out set\n# (historical: score_config.py and scene_score.py were removed with the\n# scene-union metric \u2014 use scripts/optimizer/second_score.py, per-second)\npython scripts/optimizer/second_score.py --help\nSuperseded by the model bake-off + re-tune, which replaced this round's scene-union metric with per-second scoring.
"},{"location":"pose-expansion/","title":"Pose expansion: does \"learning\" new poses mid-film help?","text":"expand_gallery (src/gallery/track_gallery.hpp) promotes a confidently-identified track's novel-pose reference views into a per-film, in-memory gallery annex \u2014 the idea being that once the pipeline is sure who someone is, a pose it hasn't seen before (turned head, different lighting) becomes a free extra reference for recognising that actor again later in the same film, without touching the baked gallery.
Averaged across all 4 models, on the 4 films used for optimization:
scope expansion F1 R misID full off 71.2% 58.3% 209 full on 71.2% 59.7% 864 restricted off 73.6% 61.3% 194 restricted on 75.4% 64.5% 135In restricted mode (matcher's candidate set capped to the film's own credited cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually lower. In full mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but misID roughly quadrupled (209 \u2192 864) \u2014 see the bake-off experiment log for the per-model breakdown. That's the number that motivated this page: does turning expansion on actually change what gets recognised, frame by frame, or is the aggregate F1 shift something else?
Same model + same tuned config, expand_gallery toggled on vs. off, nothing else changed \u2014 full gallery mode, per-second scoring against X-Ray. This isolates expansion from every other variable (config, model, threshold) that differs between the training-set exp/noexp rows above.
LVFace-B Glint360K, all 5 held-out films (films never seen by the optimizer):
film F1 (exp) F1 (noexp) TPI \u0394 FN \u0394 Benny & Joon 83.0% 83.0% -2 +2 Downton Abbey: A New Era 56.1% 56.2% -7 +7 Lovelace 77.5% 77.4% +33 -33 The Many Saints of Newark 46.3% 46.3% +2 -2 Valerian and the City of a Thousand Planets 74.1% 74.1% +2 -2ArcFace R18 (Benny & Joon, r18's own tuned config): F1 77.1% for both, TPI/FN identical, FPI differs by 2 (noise).
Every film, both models tested: F1 within 0.1\u20130.2pp, TPI/FN swings in the tens out of tens of thousands. That's noise, not a signal \u2014 expansion made no measurable difference to per-second onscreen identification anywhere it was tested on unseen data.
"},{"location":"pose-expansion/#two-bugs-this-required-catching-this-sections-own-methodology","title":"Two bugs this required catching (this section's own methodology)","text":"Getting to the clean table above took two wrong turns, both worth recording since they're exactly the kind of error that produces a false positive \"look, expansion helped!\" finding:
exp replay was cut off by a 60s subprocess timeout at ~76% through the film (5589 of 7368 expected seconds) \u2014 a genuinely large, silent data loss that showed up as a large, convincing-looking TPI gap (47938 vs 52032) purely because one run had a quarter of the film missing. Caught by comparing n_seconds between runs before trusting any score delta; fixed by re-running with a longer timeout.exp detection to the first noexp detection with IoU > 0.5, not the best-overlapping one. With 3 faces close together in frame, this produced spurious \"disagreements\" (e.g. \"exp says Aidan Quinn, noexp says Johnny Depp\" at the same seconds) that vanished entirely once the match picked the true best-IoU candidate \u2014 both configs had actually output the exact same three names at the exact same three boxes.Both bugs independently pointed toward \"expansion is doing something,\" and both were artifacts of the comparison harness, not the pipeline. Worth remembering when a before/after diff looks dramatic: check that the two runs actually cover the same seconds, and match entities by best overlap, not first-found.
"},{"location":"pose-expansion/#what-this-means","title":"What this means","text":"The training-set aggregate effect (particularly the ~4x misID increase in full mode) doesn't reproduce on held-out data \u2014 at minimum it's far smaller than the training-set numbers suggested, and plausibly it's sampling variation from only 4 training films rather than a real, generalizable mechanism. This doesn't mean expand_gallery never does anything (the mechanism is real \u2014 see track_gallery.hpp's promotion logging: tracks do get confirmed and views do get promoted into the annex on every film tested), only that whatever effect it has on final per-second identification was too small to detect against 5 held-out films with this scoring method. A cleaner test would need either many more held-out films or a metric that can see the annex's direct contribution (e.g. tagging which reference embedding won each match), neither of which this pass had budget for.
Practical takeaway: don't treat the training-set exp vs noexp numbers in the bake-off experiment log as proof that expansion changes real-world behavior in either direction \u2014 on the evidence gathered so far, it doesn't move the needle enough to see.
Status: design / proposal. Nothing here is built yet.
"},{"location":"service-conversion/#the-idea","title":"The idea","text":"Turn the CLI tools into a turnkey batch worker that uses the machine's idle GPU: it analyses newly-added Jellyfin media when you're not using the computer (screen locked), and stops the instant you come back. It's an overnight job on your own Linux box.
No Docker. This runs on your own machine with your own drivers, so a container buys little and costs a lot: GPU passthrough (nvidia-container-toolkit, or /dev/kfd+/dev/dri+video group for ROCm) is the single most fragile part of a containerised setup, and it exists only because of the container. Natively, the GPU just works with the drivers you already have, and the media paths Jellyfin reports are just real paths \u2014 no re-mounting. So we ship a native installer instead of an image builder.
Two deliverables:
scripts/build_install.py. Detects your distro, ensures the GPU/build dependencies are present (via dnf/pacman), compiles scene_analyze for your GPU, and installs the binary + Python glue + two systemd user units under ~/.local.The processing loop is already implemented \u2014 this is packaging, building, and lock-gating, not new pipeline logic.
Piece Where What it does Analysis enginebuild/scene_analyze Video \u2192 face detect/align/embed \u2192 gallery match \u2192 result JSON Backend selection CMakeLists.txt (SAE_INFERENCE_BACKEND, SAE_GEMM_BACKEND) ORT/TRT + ROCm/CUDA, chosen at build time New-media queue JRay plugin \u2192 GET /Plugins/JRay/Tasks/Pending Backlog of items with no results yet Worker loop scripts/run_from_jellyfin.py--worker Poll Pending \u2192 run scene_analyze \u2192 push results Result push PUT /Plugins/JRay/Items/{id}/Truth Stores per-actor scene windows back in Jellyfin Incremental gallery scripts/make_jellyfin_gallery.py--merge Embeds only cast not already in the gallery Secrets loader .env via scripts/sae_env.py JELLYFIN_URL, JELLYFIN_API_KEY, TMDB_API_KEY"},{"location":"service-conversion/#installer-config","title":"Installer config","text":"One file. Build-time settings (fixed when we compile) vs. run-time settings (in the worker's .env, editable without recompiling).
# install.yaml \u2014 consumed by scripts/build_install.py\n\nplatform: nvidia # nvidia | amd | cpu \u2192 picks the cmake backend\nmodel:\n arcface: LVFace-B_Glint360K.onnx # embedder compiled against; gallery MUST match\nschedule:\n gallery_scan_interval: 24h # incremental --merge cadence; 0 disables the scanner\nprefix: ~/.local # install root (bin, share, systemd user units)\n\n# runtime (written to the worker .env, not compiled in):\nruntime:\n jellyfin_url: http://localhost:8096\n # JELLYFIN_API_KEY / TMDB_API_KEY are filled into .env by hand after install\nSecrets never go in the repo or a build artifact \u2014 the installer writes a .env under the install prefix with blanks for the keys, and you fill them in once. sae_env.py already loads it.
Model \u21c4 gallery coupling (guard, don't just document): embeddings from different recognition models aren't interchangeable. We compile against one embedder; the gallery must be built with the same one. Stamp the embedder name into gallery.json, and have the worker refuse to start if the gallery's embedder \u2260 the configured model.arcface, rather than silently mismatching.
The heavy build/runtime deps (OpenCV, ffmpeg, the GPU stack) are best provided by the distro, not vendored. The installer ships a per-distro dependency list and either installs them or prints the exact command. Targets: Fedora (dnf) and Arch (pacman) first.
Dependency Fedora (dnf) Arch (pacman) OpenCVopencv-devel opencv ffmpeg ffmpeg-free/ffmpeg (RPM Fusion) ffmpeg CMake / toolchain cmake gcc-c++ cmake gcc CUDA + TensorRT (nvidia) NVIDIA CUDA repo + libnvinfer-* cuda, tensorrt ROCm (amd) rocm-hip-sdk / rocblas-devel rocm-hip-sdk, rocblas ONNX Runtime not packaged \u2014 installer fetches a pinned release tarball into the prefix AUR onnxruntime (or same pinned-tarball fallback) So the flow is: detect distro \u2192 check each package \u2192 install via the native manager (or print sudo dnf install \u2026 / sudo pacman -S \u2026), with ONNX Runtime as the one known gap the installer fills itself (a pinned upstream release extracted under the install prefix, so it doesn't depend on a system package that may not exist). CUDA/ROCm being present is assumed \u2014 you already run a GPU desktop; the installer verifies and points you at the vendor repo if not.
build_install.py does","text":"build_install.py install.yaml\n \u2502\n \u251c\u2500 detect distro (dnf vs pacman) and platform from config\n \u251c\u2500 ensure deps: install via manager, or print the exact command; fetch ONNX Runtime if needed\n \u251c\u2500 cmake + build scene_analyze with the platform's backend flags:\n \u2502 nvidia \u2192 -DSAE_INFERENCE_BACKEND=TRT -DSAE_GEMM_BACKEND=CUDA\n \u2502 amd \u2192 -DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=ROCM\n \u2502 cpu \u2192 -DSAE_INFERENCE_BACKEND=ORT (CPU EP; slow, for smoke tests)\n \u251c\u2500 install into <prefix>:\n \u2502 bin/sae-scene-analyze the compiled binary\n \u2502 share/sae-worker/ Python glue + a venv (requests, etc.), models/\n \u2502 share/sae-worker/.env runtime config (keys blank, url from config)\n \u251c\u2500 install systemd --user units:\n \u2502 sae-worker.service runs the worker + gallery-scan supervisor\n \u2502 sae-lock-gate.service watches logind lock/unlock, start/stops the worker\n \u2514\u2500 print next steps (edit .env, `systemctl --user enable --now sae-lock-gate`)\nsae-worker.service runs a small Python supervisor as its main process:
run_from_jellyfin.py --worker) \u2014 the hot path,gallery_scan_interval, runs make_jellyfin_gallery.py --merge, repeats,sae-lock-gate.service runs a tiny watcher that subscribes to logind lock/unlock signals and drives the worker service:
screen locks \u2192 systemctl --user start sae-worker.service\nscreen unlocks \u2192 systemctl --user stop sae-worker.service (SIGTERM)\nScreen-lock is the only signal \u2014 deliberately. We don't also gate on GPU/CPU load, because our own worker is the load: a load threshold would form a feedback loop (worker starts \u2192 GPU spikes \u2192 threshold trips \u2192 worker stops \u2192 load drops \u2192 restart \u2192 \u2026). Lock state is external to what the worker does, so it can't oscillate.
Signal source is desktop-dependent: logind Lock/Unlock (GNOME/KDE via loginctl/D-Bus) covers most setups; a swayidle/xss-lock hook is the fallback for wlroots/X-only compositors. The installer picks based on what's present.
Stopping the worker mid-analysis costs nothing to reschedule, because of how the JRay queue works: an item only leaves /Tasks/Pending once its results are pushed (push_truth). A worker stopped mid-scene_analyze simply leaves that item Pending \u2014 next lock picks it up again. No re-queue bookkeeping.
Two small correctness requirements (the only worker changes needed):
push_truth runs only after scene_analyze returns; a killed run pushes nothing. \u2713 (keep it that way).process_item writes a temp filtered-gallery file and unlinks it in a finally; a SIGKILL skips finally. Fix: write temps under a dir the worker wipes on start, and/or a SIGTERM handler that unlinks before exit. Minor.Accepted trade-off: a partially-analysed title restarts from scratch next lock. Fine for an overnight/idle workload; no mid-video checkpointing.
"},{"location":"service-conversion/#the-end-to-end-ux","title":"The end-to-end UX","text":"# once: build + install for your GPU + model\n./scripts/build_install.py install.yaml\n # detects Fedora/Arch, ensures deps, compiles, installs units under ~/.local\n\n# once: set your keys, enable the gate\n$EDITOR ~/.local/share/sae-worker/.env # JELLYFIN_API_KEY, TMDB_API_KEY\nsystemctl --user enable --now sae-lock-gate.service\n\n# from then on: nothing. Lock your screen \u2192 it analyses. Unlock \u2192 it stops.\nNo Docker, no GPU passthrough config, no media re-mounting \u2014 the worker sees the same filesystem and GPU as everything else on the box.
"},{"location":"service-conversion/#implementation-plan-follow-up-commits","title":"Implementation plan (follow-up commits)","text":"Ordered so each step stands alone:
scripts/build_install.py: parse install.yaml, distro detect, dependency check/print (start with cpu platform so it builds without a GPU), cmake+build, copy into prefix.scripts/service.py (worker loop + gallery-scan timer + SIGTERM); temp-file cleanup fix in run_from_jellyfin.py.sae-worker.service, sae-lock-gate.service, and the logind lock watcher.gallery.json; startup mismatch check.amd platform, and its CPU execution provider covers the cpu smoke-test fallback too, so one download handles both. (nvidia uses raw TRT and doesn't need ORT.)dnf/pacman invocation \u2014 auto-install. The installer runs sudo dnf install \u2026 / sudo pacman -S \u2026 itself (prompting for sudo), rather than only printing the command. It still prints what it's about to install first.apt) is out of scope.(none blocking \u2014 the spec above is buildable as-is.)
"}]}