Scene Actor Extraction

Identifies actors in movie files and produces X-ray-style scene annotations compatible with Jellyfin. Built on a KPN++ pipeline with ArcFace embeddings and a tracked-identity matcher.

How it works

1. Build a gallery — download actor headshots from TMDB/IMDB, embed them with ArcFace (build_gallery / scripts/make_gallery.py).
2. Analyze a movie — scene_analyze decodes frames at configurable FPS, detects faces (SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON.
3. Output — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.

Dependencies

Dependency	Role
KPN++	Pipeline backbone (nodes, networks)
OpenCV 4	Video decode, image ops, DNN inference
ONNX Runtime	SCRFD face detector (dynamic shape nodes unsupported by cv::dnn)
TensorRT + CUDA runtime + cuBLAS	Optional TRT engines for SCRFD/ArcFace (--detector-engine/--arcface-engine); identity_matcher's GPU gallery scan
FFmpeg (libav*)	NVDEC hardware video decode + colour conversion
nlohmann/json	JSON I/O
nanobind	Python bindings for sae_embed

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

This also builds sae_embed, a Python module (via nanobind) that loads the SCRFD detector and ArcFace embedder once and exposes a reusable embed() method. The gallery-builder scripts (make_gallery.py, make_jellyfin_gallery.py, movienet_eval.py) import it directly — there is no subprocess fallback, so if it's missing they exit with a build instruction:

cmake --build build --target sae_embed

Optional flags:

Flag	Default	Effect
-DSAE_WEB_DEBUG=ON	OFF	Enables KPN web debug UI at localhost:9090

Models

The ONNX model weights live in models/ (tracked via Git LFS):

• arcface_w600k_r50.onnx — primary ArcFace embedder
• arcface_w600k_mbf.onnx, arcface_r18.onnx — lighter alternatives
• face_detection_yunet_2023mar.onnx — YuNet face detector
• scrfd_500m_bnkps.onnx — SCRFD face detector

If they are missing (e.g. LFS not fetched), re-download them with:

bash scripts/download_models.sh

Model licensing: the model weights carry their own licenses, separate from this project's MIT license, and are redistributed here under those upstream terms. Several — notably the InsightFace "buffalo" models (ArcFace / SCRFD) — are licensed for non-commercial research use only. Review and comply with each model's license before use.

Binaries

Binary	Description
scene_analyze	Main analysis pipeline, writes JSON output
scene_analyze_debug	Same as above + per-frame annotated JPEGs (SAE_DEBUG=1)
scene_preview	Live OpenCV display window while analysing
build_gallery	Offline gallery builder from a directory of images
embed_faces	CLI: image(s) → embedding JSON, used by gallery-builder scripts
sae_embed	Python module (nanobind) used by gallery-builder scripts — loads SCRFD+ArcFace once

scene_analyze

./build/scene_analyze --gallery gallery.json --input movie.mp4 [options]

Key options:

Flag	Default	Description
--fps	1	Frames per second to sample (5–10 recommended for tracking)
--prob-threshold	0.5	Minimum calibrated match probability
--match-threshold	—	Raw cosine similarity threshold (fallback)
--extinction	5s	How long a track persists after last detection
--track-alpha	—	IoU vs. embedding weight in Hungarian assignment
--track-min-iou	—	Minimum IoU gate for spatial assignment
--track-max-embed	—	Maximum embedding distance gate
--track-max-missing	—	Frames a track survives without a detection

Gallery builders

Per-movie (TMDB):

python3 scripts/make_gallery.py --tmdb-bearer <JWT> --movie-id <TMDB_ID> --output gallery.json

Fetches cast images from TMDB and embeds them via sae_embed.

Whole-library (Jellyfin):

python3 scripts/make_jellyfin_gallery.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --output gallery.json

Scans every Movie/Series in Jellyfin, collects the unique cast across the whole library, downloads each actor's headshot directly from Jellyfin (no TMDB key needed), and embeds them via sae_embed into one global gallery.json. Since identity_matcher scores faces against the entire gallery, scene_analyze can then recognise any actor from your library in any film — not just the cast listed for that one title. Pass --merge on later runs to only embed actors newly added to the library. Pass --tmdb-key to fall back to TMDB profile images for actors with no usable image cached in Jellyfin.

Jellyfin/TMDB lookups and image downloads for different actors run concurrently (--workers, default 8). Embedding is GPU-bound, so it's gated separately via --embed-concurrency (default 1) — only that many embed calls run at once while other actors' downloads continue in the background.

To restrict a single-title run to that title's credited cast (faster, fewer look-alike mismatches), filter the global gallery first:

python3 scripts/filter_gallery.py \
    --gallery gallery.json \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --title "The Matrix" \
    --output gallery_matrix.json

Running directly from Jellyfin

scripts/run_from_jellyfin.py resolves a title to its media file via the Jellyfin API, filters the gallery to that title's cast, and runs scene_analyze in one step. Requires this tool to run on a host that shares Jellyfin's media mount (it uses the item's on-disk Path, not a stream URL):

python3 scripts/run_from_jellyfin.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --title "The Matrix" \
    --gallery gallery.json \
    -- --fps 5 --verbosity 2

Anything after -- is passed through to scene_analyze unchanged. Pass --no-filter to use the gallery as-is (skip per-title cast filtering), or --item-id instead of --title to skip the search.

After a successful run, the output JSON is pushed to the JRay Jellyfin plugin's Truth endpoint (PUT /Plugins/JRay/Items/{itemId}/Truth) so Jellyfin picks it up immediately, using --api-key (must be an Administrator key for the push to succeed). Pass --no-push to skip this and only write --output locally (e.g. for local debugging).

Worker mode

Pass --worker instead of --item-id/--title to run this as an extraction worker: it polls the JRay plugin's GET /Plugins/JRay/Tasks/Pending endpoint for a random batch of items with no truth data yet, processes each one, and pushes the result back. The endpoint's sampling spreads work across the backlog without any server-side task tracking, so any number of workers can poll the same library concurrently.

python3 scripts/run_from_jellyfin.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <ADMIN_API_KEY> \
    --gallery whole_gallery.json \
    --worker \
    -- --fps 5

• --poll-limit — batch size requested from Tasks/Pending (default 10, max 100)
• --poll-interval — seconds to sleep between polls when the backlog is empty (default 60)
• --once — process a single batch and exit instead of looping forever

A failure on one item (bad path, push rejected, etc.) is logged and the worker moves on to the next item rather than exiting.

Output format

Minimal (default) — Jellyfin-ready:

[
  { "actor": "Name", "start": 12.0, "end": 45.5 }
]

Standard — per-frame detail with bounding boxes, similarity scores, and track IDs.

Pipeline topology

frame_source → face_detector → face_aligner → embedder
    → face_tracker → identity_matcher → scene_tracker → result_sink

Debug/preview branches fan out automatically from identity_matcher.

Evaluation

Scripts in eval/ and scripts/movienet_*.py support benchmarking against the MovieNet dataset.

License

The source code in this repository is licensed under the MIT License.

The MIT license covers only the code. The model weights in models/ (see Models) are redistributed under their own licenses — several for non-commercial research use only. Third-party libraries this software links against (OpenCV, ONNX Runtime, FFmpeg, TensorRT/CUDA, nlohmann/json, nanobind, and others) likewise carry their own licenses.