dtourolle
1141172b04
Twenty commits, of which the one that matters for this repo is the filter and router losslessness. FilterNode and RouterNode were the last nodes on a data path still using the throwing push() and swallowing the exception, so `decimate` — which passes EOF by predicate — discarded that token whenever its output was full. Which it reliably is: the embedder is the slowest node in the chain. Nothing downstream ever received EOF, `done` was never set, and the run had to be killed. That is the wedge. The rest, in rough order of how much they affect a run here: - A sentinel could be delivered ahead of a value still queued behind it, losing that value to any consumer treating EOF as a hard stop. - Two fire_once invocations for one node could overlap, which breaks the one-slot park: a parked value can be overwritten with no drop recorded. - A node's push/space callbacks were written while a running neighbour read them — ten ThreadSanitizer races, and the root cause of the startup lost wake that AR-004 records as closed by a level-triggered re-check. - shutdown() polled every channel in the graph with no deadline, and could fail to terminate outright on an index underflow in the fill calculation. - Submitting to a stopped pool indexed a cleared vector: a segfault, which reproduced 12 runs in 20 once sources stopped before their consumers. - stop() returned while a firing was still touching the node's members. - Idle pool workers burned ~6.5 cores while one task ran (1991 ms of CPU against 0.4 ms). Latent here, since every node owns a private one-thread pool, but not for anything using a shared one. Two consequences worth holding onto. `decimate` is now a backpressure point rather than a relief valve, so the source throttles to the face branch instead of quietly thinning it — the intended behaviour, but it changes the shape of a loaded run and has not been benchmarked. And a channel now carries at most one undelivered sentinel; a second is refused and reported rather than silently overwriting the first. Verified: KPN 148/148 with ThreadSanitizer clean across six runs, this repo 138/138 with every target building warning-free against the new headers. TRACES: AR-004 | SR-002
Scene Actor Extraction
Identifies actors in movie files and produces X-ray-style scene annotations compatible with Jellyfin. Built on a KPN++ pipeline with ArcFace/LVFace embeddings and a tracked-identity matcher.
67.4% macro-F1 against Amazon X-Ray ground truth, on 5 films never seen by the optimizer (89.7% P / 65.4% R training-set; see the generalization-gap discussion in the deep dive). Full benchmark write-up, model comparison, and failure-mode analysis: https://pages.tourolle.paris/dtourolle/scene-actor-extraction/
A perfect X-Ray second on a held-out film (never used for threshold tuning):
every visible face named at 100%, the background extra honestly left unnamed,
and the two credited cast without a visible face correctly carried as present
off-screen. Bottom panels show the per-second verdict against Amazon X-Ray
(green = correct, orange = wrong, blue = missed).
It also doesn't care whether the face is in the room:
Herbie Hancock at 98% — as a face on a screen inside the movie, under a
sci-fi HUD overlay.
How it works
- Build a gallery — download actor headshots from TMDB/IMDB, embed them with ArcFace or LVFace (
build_gallery/scripts/make_gallery.py). - Analyze a movie —
scene_analyzedecodes frames at configurable FPS, detects faces (SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON. - Output — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.
Dependencies
| Dependency | Role |
|---|---|
| KPN++ | Pipeline backbone (nodes, networks) |
| OpenCV 4 | Video decode, image ops, DNN inference |
| ONNX Runtime | SCRFD face detector (dynamic shape nodes unsupported by cv::dnn) |
| TensorRT + CUDA runtime + cuBLAS | Optional TRT engines for SCRFD/ArcFace (--detector-engine/--arcface-engine); identity_matcher's GPU gallery scan |
| FFmpeg (libav*) | NVDEC hardware video decode + colour conversion |
| nlohmann/json | JSON I/O |
| nanobind | Python bindings for sae_embed |
Build
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
This also builds sae_embed, a Python module (via nanobind) that loads the
SCRFD detector and ArcFace embedder once and exposes a reusable embed()
method. The gallery-builder scripts (make_gallery.py,
make_jellyfin_gallery.py, movienet_eval.py) import it directly — there is
no subprocess fallback, so if it's missing they exit with a build instruction:
cmake --build build --target sae_embed
Optional flags:
| Flag | Default | Effect |
|---|---|---|
-DSAE_WEB_DEBUG=ON |
OFF | Enables KPN web debug UI at localhost:9090 |
Models
The ONNX model weights live in models/ (tracked via Git LFS):
LVFace-B_Glint360K.onnx— LVFace embedder (ViT backbone, ICCV 2025), the default (best F1 in the rep4 model bake-off, seedocs/rep4-optimizer-results.md)arcface_w600k_r50.onnx— ArcFace embedder, previous defaultarcface_w600k_mbf.onnx,arcface_r18.onnx— lighter ArcFace alternativesface_detection_yunet_2023mar.onnx— YuNet face detectorscrfd_500m_bnkps.onnx— SCRFD face detector
LVFace
LVFace is a Vision-Transformer face
recognition model. The LVFace-B_Glint360K.onnx export shares ArcFace's I/O
contract (112×112 aligned BGR crop → L2-normalised 512-d embedding) and its
(x − 127.5)/128 input scaling, so it slots straight into the existing embedder
— just point --arcface-model at it:
./build/scene_analyze --arcface-model models/LVFace-B_Glint360K.onnx \
--gallery gallery.h5 --movie movie.mp4
Important: embeddings from different recognition models are not interchangeable. A gallery (and its calibration cache) must be built with the same embedder used for analysis — rebuild the gallery with
--arcface models/LVFace-B_Glint360K.onnxbefore analysing with LVFace.
If they are missing (e.g. LFS not fetched), re-download them with:
bash scripts/download_models.sh
Model licensing: the model weights carry their own licenses, separate from this project's MIT license, and are redistributed here under those upstream terms. Several — notably the InsightFace "buffalo" models (ArcFace / SCRFD) — are licensed for non-commercial research use only. Review and comply with each model's license before use.
Binaries
| Binary | Description |
|---|---|
scene_analyze |
Main analysis pipeline, writes JSON output |
scene_analyze_debug |
Same as above + per-frame annotated JPEGs (SAE_DEBUG=1) |
scene_preview |
Live OpenCV display window while analysing |
build_gallery |
Offline gallery builder from a directory of images |
embed_faces |
CLI: image(s) → embedding JSON, used by gallery-builder scripts |
sae_embed |
Python module (nanobind) used by gallery-builder scripts — loads SCRFD+ArcFace once |
scene_analyze
./build/scene_analyze --gallery gallery.h5 --movie movie.mp4 [options]
Key options:
| Flag | Default | Description |
|---|---|---|
--fps |
1 | Frames per second to sample (5–10 recommended for tracking) |
--prob-threshold |
0.5 | Minimum calibrated match probability |
--match-threshold |
— | Raw cosine similarity threshold (fallback) |
--extinction |
5s | How long a track persists after last detection |
--track-alpha |
— | IoU vs. embedding weight in Hungarian assignment |
--track-min-iou |
— | Minimum IoU gate for spatial assignment |
--track-max-embed |
— | Maximum embedding distance gate |
--track-max-missing |
— | Frames a track survives without a detection |
Gallery builders
Per-movie (TMDB):
python3 scripts/make_gallery.py --tmdb-key <TMDB_KEY> --movie-id <TMDB_ID> --output gallery.h5
Fetches cast images from TMDB and embeds them via sae_embed.
Whole-library (Jellyfin):
python3 scripts/make_jellyfin_gallery.py \
--jellyfin-url http://jellyfin.local:8096 \
--api-key <API_KEY> \
--output gallery.h5
Scans every Movie/Series in Jellyfin, collects the unique cast across the
whole library, downloads each actor's headshot directly from Jellyfin (no
TMDB key needed), and embeds them via sae_embed into one global
gallery.h5. Since identity_matcher scores faces against the entire
gallery, scene_analyze can then recognise any actor from your library in
any film — not just the cast listed for that one title. Pass --merge on
later runs to only embed actors newly added to the library. Pass
--tmdb-key to fall back to TMDB profile images for actors with no usable
image cached in Jellyfin.
Jellyfin/TMDB lookups and image downloads for different actors run
concurrently (--workers, default 8). Embedding is GPU-bound, so it's
gated separately via --embed-concurrency (default 1) — only that many
embed calls run at once while other actors' downloads continue in the
background.
To restrict a single-title run to that title's credited cast (faster, fewer look-alike mismatches), filter the global gallery first:
python3 scripts/filter_gallery.py \
--gallery gallery.h5 \
--jellyfin-url http://jellyfin.local:8096 \
--api-key <API_KEY> \
--title "The Matrix" \
--output gallery_matrix.h5
Running directly from Jellyfin
scripts/run_from_jellyfin.py resolves a title to its media file via the
Jellyfin API, filters the gallery to that title's cast, and runs
scene_analyze in one step. Requires this tool to run on a host that shares
Jellyfin's media mount (it uses the item's on-disk Path, not a stream URL):
python3 scripts/run_from_jellyfin.py \
--jellyfin-url http://jellyfin.local:8096 \
--api-key <API_KEY> \
--title "The Matrix" \
--gallery gallery.h5 \
-- --fps 5 --verbosity 2
Anything after -- is passed through to scene_analyze unchanged. Pass
--no-filter to use the gallery as-is (skip per-title cast filtering), or
--item-id instead of --title to skip the search.
After a successful run, the output JSON is pushed to the JRay Jellyfin
plugin's Truth endpoint
(PUT /Plugins/JRay/Items/{itemId}/Truth) so
Jellyfin picks it up immediately, using --api-key (must be an
Administrator key for the push to succeed). Pass --no-push to skip
this and only write --output locally (e.g. for local debugging).
Worker mode
Pass --worker instead of --item-id/--title to run this as an extraction
worker: it polls the JRay plugin's GET /Plugins/JRay/Tasks/Pending endpoint
for a random batch of items with no truth data yet, processes each one, and
pushes the result back. The endpoint's sampling spreads work across the
backlog without any server-side task tracking, so any number of workers can
poll the same library concurrently.
python3 scripts/run_from_jellyfin.py \
--jellyfin-url http://jellyfin.local:8096 \
--api-key <ADMIN_API_KEY> \
--gallery whole_gallery.h5 \
--worker \
-- --fps 5
--poll-limit— batch size requested fromTasks/Pending(default 10, max 100)--poll-interval— seconds to sleep between polls when the backlog is empty (default 60)--once— process a single batch and exit instead of looping forever
A failure on one item (bad path, push rejected, etc.) is logged and the worker moves on to the next item rather than exiting.
Output format
Minimal (default) — Jellyfin-ready:
[
{ "actor": "Name", "start": 12.0, "end": 45.5 }
]
Standard — per-frame detail with bounding boxes, similarity scores, and track IDs.
Evaluation
Scripts in eval/ and scripts/movienet_*.py support benchmarking against the MovieNet dataset.
License
The source code in this repository is licensed under the MIT License.
The MIT license covers only the code. The model weights in models/ (see
Models) are redistributed under their own licenses — several for
non-commercial research use only. Third-party libraries this software links
against (OpenCV, ONNX Runtime, FFmpeg, TensorRT/CUDA, nlohmann/json, nanobind,
and others) likewise carry their own licenses.