Scene Actor Extraction

Identifies actors in movie files and produces X-ray-style scene annotations compatible with Jellyfin. Built on a KPN++ pipeline with ArcFace embeddings and a tracked-identity matcher.

How it works

1. Build a gallery — download actor headshots from TMDB/IMDB, embed them with ArcFace (build_gallery / scripts/make_gallery.py).
2. Analyze a movie — scene_analyze decodes frames at configurable FPS, detects faces (YuNet/SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON.
3. Output — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.

Dependencies

Dependency	Role
KPN++	Pipeline backbone (nodes, networks)
OpenCV 4	Video decode, image ops, DNN inference, YuNet face detection
ONNX Runtime	SCRFD face detector (dynamic shape nodes unsupported by cv::dnn)
nlohmann/json	JSON I/O

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

This also builds sae_embed, a Python module (via nanobind) that loads the SCRFD detector and ArcFace embedder once and exposes a reusable embed() method. The gallery-builder scripts (make_gallery.py, make_jellyfin_gallery.py, movienet_eval.py) import it directly — there is no subprocess fallback, so if it's missing they exit with a build instruction:

cmake --build build --target sae_embed

Optional flags:

Flag	Default	Effect
-DSAE_WEB_DEBUG=ON	OFF	Enables KPN web debug UI at localhost:9090

Models

Download the required ONNX models:

bash scripts/download_models.sh

Models are placed in external/:

• arcface_w600k_r50.onnx — primary ArcFace embedder
• arcface_w600k_mbf.onnx, arcface_r18.onnx — lighter alternatives
• face_detection_yunet_2023mar.onnx — YuNet face detector
• scrfd_500m_bnkps.onnx — SCRFD face detector

Binaries

Binary	Description
scene_analyze	Main analysis pipeline, writes JSON output
scene_analyze_debug	Same as above + per-frame annotated JPEGs (SAE_DEBUG=1)
scene_preview	Live OpenCV display window while analysing
build_gallery	Offline gallery builder from a directory of images
sae_embed	Python module (nanobind) used by gallery-builder scripts — loads SCRFD+ArcFace once

scene_analyze

./build/scene_analyze --gallery gallery.json --input movie.mp4 [options]

Key options:

Flag	Default	Description
--fps	1	Frames per second to sample (5–10 recommended for tracking)
--prob-threshold	0.5	Minimum calibrated match probability
--match-threshold	—	Raw cosine similarity threshold (fallback)
--extinction	5s	How long a track persists after last detection
--track-alpha	—	IoU vs. embedding weight in Hungarian assignment
--track-min-iou	—	Minimum IoU gate for spatial assignment
--track-max-embed	—	Maximum embedding distance gate
--track-max-missing	—	Frames a track survives without a detection
--track-min-frames	3	Observations before a track's mean embedding is used for matching

Gallery builders

Per-movie (TMDB):

python3 scripts/make_gallery.py --tmdb-bearer <JWT> --movie-id <TMDB_ID> --output gallery.json

Fetches cast images from TMDB and embeds them via sae_embed.

Whole-library (Jellyfin):

python3 scripts/make_jellyfin_gallery.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --output gallery.json

Scans every Movie/Series in Jellyfin, collects the unique cast across the whole library, downloads each actor's headshot directly from Jellyfin (no TMDB key needed), and embeds them via sae_embed into one global gallery.json. Since identity_matcher scores faces against the entire gallery, scene_analyze can then recognise any actor from your library in any film — not just the cast listed for that one title. Pass --merge on later runs to only embed actors newly added to the library. Pass --tmdb-key to fall back to TMDB profile images for actors with no usable image cached in Jellyfin.

Jellyfin/TMDB lookups and image downloads for different actors run concurrently (--workers, default 8). Embedding is GPU-bound, so it's gated separately via --embed-concurrency (default 1) — only that many embed calls run at once while other actors' downloads continue in the background.

To restrict a single-title run to that title's credited cast (faster, fewer look-alike mismatches), filter the global gallery first:

python3 scripts/filter_gallery.py \
    --gallery gallery.json \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --title "The Matrix" \
    --output gallery_matrix.json

Running directly from Jellyfin

scripts/run_from_jellyfin.py resolves a title to its media file via the Jellyfin API, filters the gallery to that title's cast, and runs scene_analyze in one step. Requires this tool to run on a host that shares Jellyfin's media mount (it uses the item's on-disk Path, not a stream URL):

python3 scripts/run_from_jellyfin.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --title "The Matrix" \
    --gallery gallery.json \
    -- --fps 5 --verbosity 2

Anything after -- is passed through to scene_analyze unchanged. Pass --no-filter to use the gallery as-is (skip per-title cast filtering), or --item-id instead of --title to skip the search.

Output format

Minimal (default) — Jellyfin-ready:

[
  { "actor": "Name", "start": 12.0, "end": 45.5 }
]

Standard — per-frame detail with bounding boxes, similarity scores, and track IDs.

Pipeline topology

frame_source → face_detector → face_aligner → embedder
    → face_tracker → identity_matcher → scene_tracker → result_sink

Debug/preview branches fan out automatically from identity_matcher.

Evaluation

Scripts in eval/ and scripts/movienet_*.py support benchmarking against the MovieNet dataset.