177 lines
6.2 KiB
Markdown
177 lines
6.2 KiB
Markdown
# Scene Actor Extraction
|
||
|
||
Identifies actors in movie files and produces X-ray-style scene annotations compatible with [Jellyfin](https://jellyfin.org/). Built on a KPN++ pipeline with ArcFace embeddings and a tracked-identity matcher.
|
||
|
||
## How it works
|
||
|
||
1. **Build a gallery** — download actor headshots from TMDB/IMDB, embed them with ArcFace (`build_gallery` / `scripts/make_gallery.py`).
|
||
2. **Analyze a movie** — `scene_analyze` decodes frames at configurable FPS, detects faces (YuNet/SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON.
|
||
3. **Output** — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.
|
||
|
||
## Dependencies
|
||
|
||
| Dependency | Role |
|
||
|---|---|
|
||
| KPN++ | Pipeline backbone (nodes, networks) |
|
||
| OpenCV 4 | Video decode, image ops, DNN inference, YuNet face detection |
|
||
| ONNX Runtime | SCRFD face detector (dynamic shape nodes unsupported by cv::dnn) |
|
||
| nlohmann/json | JSON I/O |
|
||
|
||
## Build
|
||
|
||
```bash
|
||
cmake -B build -DCMAKE_BUILD_TYPE=Release
|
||
cmake --build build -j$(nproc)
|
||
```
|
||
|
||
This also builds `sae_embed`, a Python module (via nanobind) that loads the
|
||
SCRFD detector and ArcFace embedder once and exposes a reusable `embed()`
|
||
method. The gallery-builder scripts (`make_gallery.py`,
|
||
`make_jellyfin_gallery.py`, `movienet_eval.py`) import it directly — there is
|
||
no subprocess fallback, so if it's missing they exit with a build instruction:
|
||
|
||
```bash
|
||
cmake --build build --target sae_embed
|
||
```
|
||
|
||
Optional flags:
|
||
|
||
| Flag | Default | Effect |
|
||
|---|---|---|
|
||
| `-DSAE_WEB_DEBUG=ON` | OFF | Enables KPN web debug UI at `localhost:9090` |
|
||
|
||
## Models
|
||
|
||
Download the required ONNX models:
|
||
|
||
```bash
|
||
bash scripts/download_models.sh
|
||
```
|
||
|
||
Models are placed in `external/`:
|
||
- `arcface_w600k_r50.onnx` — primary ArcFace embedder
|
||
- `arcface_w600k_mbf.onnx`, `arcface_r18.onnx` — lighter alternatives
|
||
- `face_detection_yunet_2023mar.onnx` — YuNet face detector
|
||
- `scrfd_500m_bnkps.onnx` — SCRFD face detector
|
||
|
||
## Binaries
|
||
|
||
| Binary | Description |
|
||
|---|---|
|
||
| `scene_analyze` | Main analysis pipeline, writes JSON output |
|
||
| `scene_analyze_debug` | Same as above + per-frame annotated JPEGs (`SAE_DEBUG=1`) |
|
||
| `scene_preview` | Live OpenCV display window while analysing |
|
||
| `build_gallery` | Offline gallery builder from a directory of images |
|
||
| `sae_embed` | Python module (nanobind) used by gallery-builder scripts — loads SCRFD+ArcFace once |
|
||
|
||
### `scene_analyze`
|
||
|
||
```bash
|
||
./build/scene_analyze --gallery gallery.json --input movie.mp4 [options]
|
||
```
|
||
|
||
Key options:
|
||
|
||
| Flag | Default | Description |
|
||
|---|---|---|
|
||
| `--fps` | 1 | Frames per second to sample (5–10 recommended for tracking) |
|
||
| `--prob-threshold` | 0.5 | Minimum calibrated match probability |
|
||
| `--match-threshold` | — | Raw cosine similarity threshold (fallback) |
|
||
| `--extinction` | 5s | How long a track persists after last detection |
|
||
| `--track-alpha` | — | IoU vs. embedding weight in Hungarian assignment |
|
||
| `--track-min-iou` | — | Minimum IoU gate for spatial assignment |
|
||
| `--track-max-embed` | — | Maximum embedding distance gate |
|
||
| `--track-max-missing` | — | Frames a track survives without a detection |
|
||
| `--track-min-frames` | 3 | Observations before a track's mean embedding is used for matching |
|
||
|
||
### Gallery builders
|
||
|
||
**Per-movie (TMDB):**
|
||
|
||
```bash
|
||
python3 scripts/make_gallery.py --tmdb-bearer <JWT> --movie-id <TMDB_ID> --output gallery.json
|
||
```
|
||
|
||
Fetches cast images from TMDB and embeds them via `sae_embed`.
|
||
|
||
**Whole-library (Jellyfin):**
|
||
|
||
```bash
|
||
python3 scripts/make_jellyfin_gallery.py \
|
||
--jellyfin-url http://jellyfin.local:8096 \
|
||
--api-key <API_KEY> \
|
||
--output gallery.json
|
||
```
|
||
|
||
Scans every Movie/Series in Jellyfin, collects the unique cast across the
|
||
whole library, downloads each actor's headshot directly from Jellyfin (no
|
||
TMDB key needed), and embeds them via `sae_embed` into one global
|
||
gallery.json. Since `identity_matcher` scores faces against the entire
|
||
gallery, `scene_analyze` can then recognise any actor from your library in
|
||
any film — not just the cast listed for that one title. Pass `--merge` on
|
||
later runs to only embed actors newly added to the library. Pass
|
||
`--tmdb-key` to fall back to TMDB profile images for actors with no usable
|
||
image cached in Jellyfin.
|
||
|
||
Jellyfin/TMDB lookups and image downloads for different actors run
|
||
concurrently (`--workers`, default 8). Embedding is GPU-bound, so it's
|
||
gated separately via `--embed-concurrency` (default 1) — only that many
|
||
embed calls run at once while other actors' downloads continue in the
|
||
background.
|
||
|
||
To restrict a single-title run to that title's credited cast (faster, fewer
|
||
look-alike mismatches), filter the global gallery first:
|
||
|
||
```bash
|
||
python3 scripts/filter_gallery.py \
|
||
--gallery gallery.json \
|
||
--jellyfin-url http://jellyfin.local:8096 \
|
||
--api-key <API_KEY> \
|
||
--title "The Matrix" \
|
||
--output gallery_matrix.json
|
||
```
|
||
|
||
## Running directly from Jellyfin
|
||
|
||
`scripts/run_from_jellyfin.py` resolves a title to its media file via the
|
||
Jellyfin API, filters the gallery to that title's cast, and runs
|
||
`scene_analyze` in one step. Requires this tool to run on a host that shares
|
||
Jellyfin's media mount (it uses the item's on-disk `Path`, not a stream URL):
|
||
|
||
```bash
|
||
python3 scripts/run_from_jellyfin.py \
|
||
--jellyfin-url http://jellyfin.local:8096 \
|
||
--api-key <API_KEY> \
|
||
--title "The Matrix" \
|
||
--gallery gallery.json \
|
||
-- --fps 5 --verbosity 2
|
||
```
|
||
|
||
Anything after `--` is passed through to `scene_analyze` unchanged. Pass
|
||
`--no-filter` to use the gallery as-is (skip per-title cast filtering), or
|
||
`--item-id` instead of `--title` to skip the search.
|
||
|
||
## Output format
|
||
|
||
**Minimal** (default) — Jellyfin-ready:
|
||
```json
|
||
[
|
||
{ "actor": "Name", "start": 12.0, "end": 45.5 }
|
||
]
|
||
```
|
||
|
||
**Standard** — per-frame detail with bounding boxes, similarity scores, and track IDs.
|
||
|
||
## Pipeline topology
|
||
|
||
```
|
||
frame_source → face_detector → face_aligner → embedder
|
||
→ face_tracker → identity_matcher → scene_tracker → result_sink
|
||
```
|
||
|
||
Debug/preview branches fan out automatically from `identity_matcher`.
|
||
|
||
## Evaluation
|
||
|
||
Scripts in `eval/` and `scripts/movienet_*.py` support benchmarking against the MovieNet dataset.
|