Initial commit: scene-actor-extraction pipeline

Source (KPN++ pipeline nodes, ArcFace embedders, SCRFD/YuNet detectors, gallery builder), build scripts, and eval artifacts. - external/KPN as a git submodule (gitea.tourolle.paris/dtourolle/KPN) - ONNX models tracked via Git LFS (models/*.onnx) - generated outputs, TensorRT engines, reference repos, and media ignored
2026-06-12 15:29:01 +02:00
commit d753062c6c
50 changed files with 10100 additions and 0 deletions
@@ -0,0 +1,107 @@
+# Scene Actor Extraction
+
+Identifies actors in movie files and produces X-ray-style scene annotations compatible with [Jellyfin](https://jellyfin.org/). Built on a KPN++ pipeline with ArcFace embeddings and a tracked-identity matcher.
+
+## How it works
+
+1. **Build a gallery** — download actor headshots from TMDB/IMDB, embed them with ArcFace (`build_gallery` / `scripts/make_gallery.py`).
+2. **Analyze a movie** — `scene_analyze` decodes frames at configurable FPS, detects faces (YuNet/SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON.
+3. **Output** — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.
+
+## Dependencies
+
+| Dependency | Role |
+|---|---|
+| KPN++ | Pipeline backbone (nodes, networks) |
+| OpenCV 4 | Video decode, image ops, DNN inference, YuNet face detection |
+| ONNX Runtime | SCRFD face detector (dynamic shape nodes unsupported by cv::dnn) |
+| nlohmann/json | JSON I/O |
+
+## Build
+
+```bash
+cmake -B build -DCMAKE_BUILD_TYPE=Release
+cmake --build build -j$(nproc)
+```
+
+Optional flags:
+
+| Flag | Default | Effect |
+|---|---|---|
+| `-DSAE_WEB_DEBUG=ON` | OFF | Enables KPN web debug UI at `localhost:9090` |
+
+## Models
+
+Download the required ONNX models:
+
+```bash
+bash scripts/download_models.sh
+```
+
+Models are placed in `external/`:
+- `arcface_w600k_r50.onnx` — primary ArcFace embedder
+- `arcface_w600k_mbf.onnx`, `arcface_r18.onnx` — lighter alternatives
+- `face_detection_yunet_2023mar.onnx` — YuNet face detector
+- `scrfd_500m_bnkps.onnx` — SCRFD face detector
+
+## Binaries
+
+| Binary | Description |
+|---|---|
+| `scene_analyze` | Main analysis pipeline, writes JSON output |
+| `scene_analyze_debug` | Same as above + per-frame annotated JPEGs (`SAE_DEBUG=1`) |
+| `scene_preview` | Live OpenCV display window while analysing |
+| `build_gallery` | Offline gallery builder from a directory of images |
+| `embed_faces` | Standalone embedder used by gallery scripts |
+
+### `scene_analyze`
+
+```bash
+./build/scene_analyze --gallery gallery.json --input movie.mp4 [options]
+```
+
+Key options:
+
+| Flag | Default | Description |
+|---|---|---|
+| `--fps` | 1 | Frames per second to sample (5–10 recommended for tracking) |
+| `--prob-threshold` | 0.5 | Minimum calibrated match probability |
+| `--match-threshold` | — | Raw cosine similarity threshold (fallback) |
+| `--extinction` | 5s | How long a track persists after last detection |
+| `--track-alpha` | — | IoU vs. embedding weight in Hungarian assignment |
+| `--track-min-iou` | — | Minimum IoU gate for spatial assignment |
+| `--track-max-embed` | — | Maximum embedding distance gate |
+| `--track-max-missing` | — | Frames a track survives without a detection |
+| `--track-min-frames` | 3 | Observations before a track's mean embedding is used for matching |
+
+### Gallery builder
+
+```bash
+python3 scripts/make_gallery.py --tmdb-bearer <JWT> --movie-id <TMDB_ID> --output gallery.json
+```
+
+Fetches cast images from TMDB and embeds them via `embed_faces`.
+
+## Output format
+
+**Minimal** (default) — Jellyfin-ready:
+```json
+[
+  { "actor": "Name", "start": 12.0, "end": 45.5 }
+]
+```
+
+**Standard** — per-frame detail with bounding boxes, similarity scores, and track IDs.
+
+## Pipeline topology
+
+```
+frame_source → face_detector → face_aligner → embedder
+    → face_tracker → identity_matcher → scene_tracker → result_sink
+```
+
+Debug/preview branches fan out automatically from `identity_matcher`.
+
+## Evaluation
+
+Scripts in `eval/` and `scripts/movienet_*.py` support benchmarking against the MovieNet dataset.