scene-actor-extraction/README.md

Scene Actor Extraction

Identifies actors in movie files and produces X-ray-style scene annotations compatible with Jellyfin. Built on a KPN++ pipeline with ArcFace embeddings and a tracked-identity matcher.

How it works

1. Build a gallery — download actor headshots from TMDB/IMDB, embed them with ArcFace (build_gallery / scripts/make_gallery.py).
2. Analyze a movie — scene_analyze decodes frames at configurable FPS, detects faces (SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON.
3. Output — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.

Dependencies

Dependency	Role
KPN++	Pipeline backbone (nodes, networks)
OpenCV 4	Video decode, image ops, DNN inference
ONNX Runtime	SCRFD face detector (dynamic shape nodes unsupported by cv::dnn)
TensorRT + CUDA runtime + cuBLAS	Optional TRT engines for SCRFD/ArcFace (--detector-engine/--arcface-engine); identity_matcher's GPU gallery scan
FFmpeg (libav*)	NVDEC hardware video decode + colour conversion
nlohmann/json	JSON I/O
nanobind	Python bindings for sae_embed

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

This also builds sae_embed, a Python module (via nanobind) that loads the SCRFD detector and ArcFace embedder once and exposes a reusable embed() method. The gallery-builder scripts (make_gallery.py, make_jellyfin_gallery.py, movienet_eval.py) import it directly — there is no subprocess fallback, so if it's missing they exit with a build instruction:

cmake --build build --target sae_embed

Optional flags:

Flag	Default	Effect
-DSAE_WEB_DEBUG=ON	OFF	Enables KPN web debug UI at localhost:9090

Models

The ONNX model weights live in models/ (tracked via Git LFS):

• LVFace-B_Glint360K.onnx — LVFace embedder (ViT backbone, ICCV 2025), the default (best F1 in the rep4 model bake-off, see docs/rep4-optimizer-results.md)
• arcface_w600k_r50.onnx — ArcFace embedder, previous default
• arcface_w600k_mbf.onnx, arcface_r18.onnx — lighter ArcFace alternatives
• face_detection_yunet_2023mar.onnx — YuNet face detector
• scrfd_500m_bnkps.onnx — SCRFD face detector

LVFace

LVFace is a Vision-Transformer face recognition model. The LVFace-B_Glint360K.onnx export shares ArcFace's I/O contract (112×112 aligned BGR crop → L2-normalised 512-d embedding) and its (x − 127.5)/128 input scaling, so it slots straight into the existing embedder — just point --arcface-model at it:

./build/scene_analyze --arcface-model models/LVFace-B_Glint360K.onnx \
    --gallery gallery.json --input movie.mp4

Important: embeddings from different recognition models are not interchangeable. A gallery (and its calibration cache) must be built with the same embedder used for analysis — rebuild the gallery with --arcface models/LVFace-B_Glint360K.onnx before analysing with LVFace.

If they are missing (e.g. LFS not fetched), re-download them with:

bash scripts/download_models.sh

Model licensing: the model weights carry their own licenses, separate from this project's MIT license, and are redistributed here under those upstream terms. Several — notably the InsightFace "buffalo" models (ArcFace / SCRFD) — are licensed for non-commercial research use only. Review and comply with each model's license before use.

Binaries

Binary	Description
scene_analyze	Main analysis pipeline, writes JSON output
scene_analyze_debug	Same as above + per-frame annotated JPEGs (SAE_DEBUG=1)
scene_preview	Live OpenCV display window while analysing
build_gallery	Offline gallery builder from a directory of images
embed_faces	CLI: image(s) → embedding JSON, used by gallery-builder scripts
sae_embed	Python module (nanobind) used by gallery-builder scripts — loads SCRFD+ArcFace once

scene_analyze

./build/scene_analyze --gallery gallery.json --input movie.mp4 [options]

Key options:

Flag	Default	Description
--fps	1	Frames per second to sample (5–10 recommended for tracking)
--prob-threshold	0.5	Minimum calibrated match probability
--match-threshold	—	Raw cosine similarity threshold (fallback)
--extinction	5s	How long a track persists after last detection
--track-alpha	—	IoU vs. embedding weight in Hungarian assignment
--track-min-iou	—	Minimum IoU gate for spatial assignment
--track-max-embed	—	Maximum embedding distance gate
--track-max-missing	—	Frames a track survives without a detection

Gallery builders

Per-movie (TMDB):

python3 scripts/make_gallery.py --tmdb-bearer <JWT> --movie-id <TMDB_ID> --output gallery.json

Fetches cast images from TMDB and embeds them via sae_embed.

Whole-library (Jellyfin):

python3 scripts/make_jellyfin_gallery.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --output gallery.json

Scans every Movie/Series in Jellyfin, collects the unique cast across the whole library, downloads each actor's headshot directly from Jellyfin (no TMDB key needed), and embeds them via sae_embed into one global gallery.json. Since identity_matcher scores faces against the entire gallery, scene_analyze can then recognise any actor from your library in any film — not just the cast listed for that one title. Pass --merge on later runs to only embed actors newly added to the library. Pass --tmdb-key to fall back to TMDB profile images for actors with no usable image cached in Jellyfin.

Jellyfin/TMDB lookups and image downloads for different actors run concurrently (--workers, default 8). Embedding is GPU-bound, so it's gated separately via --embed-concurrency (default 1) — only that many embed calls run at once while other actors' downloads continue in the background.

To restrict a single-title run to that title's credited cast (faster, fewer look-alike mismatches), filter the global gallery first:

python3 scripts/filter_gallery.py \
    --gallery gallery.json \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --title "The Matrix" \
    --output gallery_matrix.json

Running directly from Jellyfin

scripts/run_from_jellyfin.py resolves a title to its media file via the Jellyfin API, filters the gallery to that title's cast, and runs scene_analyze in one step. Requires this tool to run on a host that shares Jellyfin's media mount (it uses the item's on-disk Path, not a stream URL):

python3 scripts/run_from_jellyfin.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <API_KEY> \
    --title "The Matrix" \
    --gallery gallery.json \
    -- --fps 5 --verbosity 2

Anything after -- is passed through to scene_analyze unchanged. Pass --no-filter to use the gallery as-is (skip per-title cast filtering), or --item-id instead of --title to skip the search.

After a successful run, the output JSON is pushed to the JRay Jellyfin plugin's Truth endpoint (PUT /Plugins/JRay/Items/{itemId}/Truth) so Jellyfin picks it up immediately, using --api-key (must be an Administrator key for the push to succeed). Pass --no-push to skip this and only write --output locally (e.g. for local debugging).

Worker mode

Pass --worker instead of --item-id/--title to run this as an extraction worker: it polls the JRay plugin's GET /Plugins/JRay/Tasks/Pending endpoint for a random batch of items with no truth data yet, processes each one, and pushes the result back. The endpoint's sampling spreads work across the backlog without any server-side task tracking, so any number of workers can poll the same library concurrently.

python3 scripts/run_from_jellyfin.py \
    --jellyfin-url http://jellyfin.local:8096 \
    --api-key <ADMIN_API_KEY> \
    --gallery whole_gallery.json \
    --worker \
    -- --fps 5

• --poll-limit — batch size requested from Tasks/Pending (default 10, max 100)
• --poll-interval — seconds to sleep between polls when the backlog is empty (default 60)
• --once — process a single batch and exit instead of looping forever

A failure on one item (bad path, push rejected, etc.) is logged and the worker moves on to the next item rather than exiting.

Output format

Minimal (default) — Jellyfin-ready:

[
  { "actor": "Name", "start": 12.0, "end": 45.5 }
]

Standard — per-frame detail with bounding boxes, similarity scores, and track IDs.

Pipeline topology

frame_source → face_detector → face_aligner → embedder
    → face_tracker → identity_matcher → scene_tracker → result_sink

Debug/preview branches fan out automatically from identity_matcher.

Evaluation

Scripts in eval/ and scripts/movienet_*.py support benchmarking against the MovieNet dataset.

License

The source code in this repository is licensed under the MIT License.

The MIT license covers only the code. The model weights in models/ (see Models) are redistributed under their own licenses — several for non-commercial research use only. Third-party libraries this software links against (OpenCV, ONNX Runtime, FFmpeg, TensorRT/CUDA, nlohmann/json, nanobind, and others) likewise carry their own licenses.