dtourolle
d753062c6c
Source (KPN++ pipeline nodes, ArcFace embedders, SCRFD/YuNet detectors, gallery builder), build scripts, and eval artifacts. - external/KPN as a git submodule (gitea.tourolle.paris/dtourolle/KPN) - ONNX models tracked via Git LFS (models/*.onnx) - generated outputs, TensorRT engines, reference repos, and media ignored
108 lines
3.6 KiB
Markdown
108 lines
3.6 KiB
Markdown
# Scene Actor Extraction
|
||
|
||
Identifies actors in movie files and produces X-ray-style scene annotations compatible with [Jellyfin](https://jellyfin.org/). Built on a KPN++ pipeline with ArcFace embeddings and a tracked-identity matcher.
|
||
|
||
## How it works
|
||
|
||
1. **Build a gallery** — download actor headshots from TMDB/IMDB, embed them with ArcFace (`build_gallery` / `scripts/make_gallery.py`).
|
||
2. **Analyze a movie** — `scene_analyze` decodes frames at configurable FPS, detects faces (YuNet/SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON.
|
||
3. **Output** — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.
|
||
|
||
## Dependencies
|
||
|
||
| Dependency | Role |
|
||
|---|---|
|
||
| KPN++ | Pipeline backbone (nodes, networks) |
|
||
| OpenCV 4 | Video decode, image ops, DNN inference, YuNet face detection |
|
||
| ONNX Runtime | SCRFD face detector (dynamic shape nodes unsupported by cv::dnn) |
|
||
| nlohmann/json | JSON I/O |
|
||
|
||
## Build
|
||
|
||
```bash
|
||
cmake -B build -DCMAKE_BUILD_TYPE=Release
|
||
cmake --build build -j$(nproc)
|
||
```
|
||
|
||
Optional flags:
|
||
|
||
| Flag | Default | Effect |
|
||
|---|---|---|
|
||
| `-DSAE_WEB_DEBUG=ON` | OFF | Enables KPN web debug UI at `localhost:9090` |
|
||
|
||
## Models
|
||
|
||
Download the required ONNX models:
|
||
|
||
```bash
|
||
bash scripts/download_models.sh
|
||
```
|
||
|
||
Models are placed in `external/`:
|
||
- `arcface_w600k_r50.onnx` — primary ArcFace embedder
|
||
- `arcface_w600k_mbf.onnx`, `arcface_r18.onnx` — lighter alternatives
|
||
- `face_detection_yunet_2023mar.onnx` — YuNet face detector
|
||
- `scrfd_500m_bnkps.onnx` — SCRFD face detector
|
||
|
||
## Binaries
|
||
|
||
| Binary | Description |
|
||
|---|---|
|
||
| `scene_analyze` | Main analysis pipeline, writes JSON output |
|
||
| `scene_analyze_debug` | Same as above + per-frame annotated JPEGs (`SAE_DEBUG=1`) |
|
||
| `scene_preview` | Live OpenCV display window while analysing |
|
||
| `build_gallery` | Offline gallery builder from a directory of images |
|
||
| `embed_faces` | Standalone embedder used by gallery scripts |
|
||
|
||
### `scene_analyze`
|
||
|
||
```bash
|
||
./build/scene_analyze --gallery gallery.json --input movie.mp4 [options]
|
||
```
|
||
|
||
Key options:
|
||
|
||
| Flag | Default | Description |
|
||
|---|---|---|
|
||
| `--fps` | 1 | Frames per second to sample (5–10 recommended for tracking) |
|
||
| `--prob-threshold` | 0.5 | Minimum calibrated match probability |
|
||
| `--match-threshold` | — | Raw cosine similarity threshold (fallback) |
|
||
| `--extinction` | 5s | How long a track persists after last detection |
|
||
| `--track-alpha` | — | IoU vs. embedding weight in Hungarian assignment |
|
||
| `--track-min-iou` | — | Minimum IoU gate for spatial assignment |
|
||
| `--track-max-embed` | — | Maximum embedding distance gate |
|
||
| `--track-max-missing` | — | Frames a track survives without a detection |
|
||
| `--track-min-frames` | 3 | Observations before a track's mean embedding is used for matching |
|
||
|
||
### Gallery builder
|
||
|
||
```bash
|
||
python3 scripts/make_gallery.py --tmdb-bearer <JWT> --movie-id <TMDB_ID> --output gallery.json
|
||
```
|
||
|
||
Fetches cast images from TMDB and embeds them via `embed_faces`.
|
||
|
||
## Output format
|
||
|
||
**Minimal** (default) — Jellyfin-ready:
|
||
```json
|
||
[
|
||
{ "actor": "Name", "start": 12.0, "end": 45.5 }
|
||
]
|
||
```
|
||
|
||
**Standard** — per-frame detail with bounding boxes, similarity scores, and track IDs.
|
||
|
||
## Pipeline topology
|
||
|
||
```
|
||
frame_source → face_detector → face_aligner → embedder
|
||
→ face_tracker → identity_matcher → scene_tracker → result_sink
|
||
```
|
||
|
||
Debug/preview branches fan out automatically from `identity_matcher`.
|
||
|
||
## Evaluation
|
||
|
||
Scripts in `eval/` and `scripts/movienet_*.py` support benchmarking against the MovieNet dataset.
|