Identifies actors in movie files and produces X-ray-style scene annotations compatible with Jellyfin. Built on a KPN++ pipeline with ArcFace embeddings and a tracked-identity matcher.

How it works

Build a gallery — download actor headshots from TMDB/IMDB, embed them with ArcFace (build_gallery / scripts/make_gallery.py).
Analyze a movie — scene_analyze decodes frames at configurable FPS, detects faces (SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON.
Output — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.

Dependencies

Dependency	Role
KPN++	Pipeline backbone (nodes, networks)
OpenCV 4	Video decode, image ops, DNN inference
ONNX Runtime	SCRFD face detector (dynamic shape nodes unsupported by cv::dnn)
TensorRT + CUDA runtime + cuBLAS	Optional TRT engines for SCRFD/ArcFace (`--detector-engine`/`--arcface-engine`); identity_matcher's GPU gallery scan
FFmpeg (libav*)	NVDEC hardware video decode + colour conversion
nlohmann/json	JSON I/O
nanobind	Python bindings for `sae_embed`

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

This also builds sae_embed, a Python module (via nanobind) that loads the SCRFD detector and ArcFace embedder once and exposes a reusable embed() method. The gallery-builder scripts (make_gallery.py, make_jellyfin_gallery.py, movienet_eval.py) import it directly — there is no subprocess fallback, so if it's missing they exit with a build instruction:

cmake --build build --target sae_embed

Optional flags:

Flag	Default	Effect
`-DSAE_WEB_DEBUG=ON`	OFF	Enables KPN web debug UI at `localhost:9090`

Models

Download the required ONNX models:

bash scripts/download_models.sh

Models are placed in external/:

arcface_w600k_r50.onnx — primary ArcFace embedder
arcface_w600k_mbf.onnx, arcface_r18.onnx — lighter alternatives
face_detection_yunet_2023mar.onnx — YuNet face detector
scrfd_500m_bnkps.onnx — SCRFD face detector

Binaries

Binary	Description
`scene_analyze`	Main analysis pipeline, writes JSON output
`scene_analyze_debug`	Same as above + per-frame annotated JPEGs (`SAE_DEBUG=1`)
`scene_preview`	Live OpenCV display window while analysing
`build_gallery`	Offline gallery builder from a directory of images
`embed_faces`	CLI: image(s) → embedding JSON, used by gallery-builder scripts
`sae_embed`	Python module (nanobind) used by gallery-builder scripts — loads SCRFD+ArcFace once

`scene_analyze`

./build/scene_analyze --gallery gallery.json --input movie.mp4 [options]

Key options:

Flag	Default	Description
`--fps`	1	Frames per second to sample (5–10 recommended for tracking)
`--prob-threshold`	0.5	Minimum calibrated match probability
`--match-threshold`	—	Raw cosine similarity threshold (fallback)
`--extinction`	5s	How long a track persists after last detection
`--track-alpha`	—	IoU vs. embedding weight in Hungarian assignment
`--track-min-iou`	—	Minimum IoU gate for spatial assignment
`--track-max-embed`	—	Maximum embedding distance gate
`--track-max-missing`	—	Frames a track survives without a detection

Gallery builders

Per-movie (TMDB):

python3 scripts/make_gallery.py --tmdb-bearer <JWT> --movie-id <TMDB_ID> --output gallery.json

Fetches cast images from TMDB and embeds them via sae_embed.

Whole-library (Jellyfin):

python3 scripts/make_jellyfin_gallery.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --output gallery.json

Scans every Movie/Series in Jellyfin, collects the unique cast across the whole library, downloads each actor's headshot directly from Jellyfin (no TMDB key needed), and embeds them via sae_embed into one global gallery.json. Since identity_matcher scores faces against the entire gallery, scene_analyze can then recognise any actor from your library in any film — not just the cast listed for that one title. Pass --merge on later runs to only embed actors newly added to the library. Pass --tmdb-key to fall back to TMDB profile images for actors with no usable image cached in Jellyfin.

Jellyfin/TMDB lookups and image downloads for different actors run concurrently (--workers, default 8). Embedding is GPU-bound, so it's gated separately via --embed-concurrency (default 1) — only that many embed calls run at once while other actors' downloads continue in the background.

To restrict a single-title run to that title's credited cast (faster, fewer look-alike mismatches), filter the global gallery first:

python3 scripts/filter_gallery.py \
    --gallery gallery.json \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --title "The Matrix" \
    --output gallery_matrix.json

Running directly from Jellyfin

scripts/run_from_jellyfin.py resolves a title to its media file via the Jellyfin API, filters the gallery to that title's cast, and runs scene_analyze in one step. Requires this tool to run on a host that shares Jellyfin's media mount (it uses the item's on-disk Path, not a stream URL):

python3 scripts/run_from_jellyfin.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --title "The Matrix" \
    --gallery gallery.json \
    -- --fps 5 --verbosity 2

Anything after -- is passed through to scene_analyze unchanged. Pass --no-filter to use the gallery as-is (skip per-title cast filtering), or --item-id instead of --title to skip the search.

After a successful run, the output JSON is pushed to the JRay Jellyfin plugin's Truth endpoint (PUT /Plugins/JRay/Items/{itemId}/Truth) so Jellyfin picks it up immediately, using --api-key (must be an Administrator key for the push to succeed). Pass --no-push to skip this and only write --output locally (e.g. for local debugging).

Worker mode

Pass --worker instead of --item-id/--title to run this as an extraction worker: it polls the JRay plugin's GET /Plugins/JRay/Tasks/Pending endpoint for a random batch of items with no truth data yet, processes each one, and pushes the result back. The endpoint's sampling spreads work across the backlog without any server-side task tracking, so any number of workers can poll the same library concurrently.

python3 scripts/run_from_jellyfin.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <ADMIN_API_KEY> \
    --gallery whole_gallery.json \
    --worker \
    -- --fps 5

--poll-limit — batch size requested from Tasks/Pending (default 10, max 100)
--poll-interval — seconds to sleep between polls when the backlog is empty (default 60)
--once — process a single batch and exit instead of looping forever

A failure on one item (bad path, push rejected, etc.) is logged and the worker moves on to the next item rather than exiting.

Output format

Minimal (default) — Jellyfin-ready:

[
  { "actor": "Name", "start": 12.0, "end": 45.5 }
]

Standard — per-frame detail with bounding boxes, similarity scores, and track IDs.

Pipeline topology

frame_source → face_detector → face_aligner → embedder
    → face_tracker → identity_matcher → scene_tracker → result_sink

Debug/preview branches fan out automatically from identity_matcher.

Evaluation

Scripts in eval/ and scripts/movienet_*.py support benchmarking against the MovieNet dataset.

README.md Unescape Escape

Scene Actor Extraction