docs: richer report — data figures, success/failure frames, commit-pinned repo links
- experiment_charts.py generates 4 figures from experiments/ artifacts: held-out per-film F1, 16-combo ranking, DE search landscape, and the Downton detector-vs-tracker ghost timeline (replaces the blank title-card screenshot) - new frames: 19-correct wedding shot (success case), Many Saints ghost-vs-unknown frame (three error classes in one image) - rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only as the on-disk artifact prefix, explained once - repo file references are now links via https://REPOLINK/<path> placeholders; build_site.sh pins them to the HEAD commit's raw URLs and fails the build if a linked path doesn't exist at HEAD - drop references to removed scripts (scene_score.py, score_config.py) and to session-memory names; mark artifact-registry paths with their pull commands - commit readme_example.jpg + pipeline_topology.svg so README renders on the plain Gitea repo view - deploy_pages.sh: push built site/ to the gitea-pages branch
This commit is contained in:
@@ -1,13 +1,25 @@
|
||||
# Scene Actor Extraction
|
||||
|
||||
Identifies actors in movie files and produces X-ray-style scene annotations compatible with [Jellyfin](https://jellyfin.org/). Built on a KPN++ pipeline with ArcFace embeddings and a tracked-identity matcher.
|
||||
Identifies actors in movie files and produces X-ray-style scene annotations compatible with [Jellyfin](https://jellyfin.org/). Built on a KPN++ pipeline with ArcFace/LVFace embeddings and a tracked-identity matcher.
|
||||
|
||||
**67.4% macro-F1 against Amazon X-Ray ground truth**, on 5 films never seen by
|
||||
the optimizer (89.7% P / 65.4% R training-set; see the generalization-gap
|
||||
discussion in the [deep dive](https://pages.tourolle.paris/dtourolle/scene-actor-extraction/lvface-deep-dive/)).
|
||||
Full benchmark write-up, model comparison, and failure-mode analysis:
|
||||
**https://pages.tourolle.paris/dtourolle/scene-actor-extraction/**
|
||||
|
||||

|
||||
*A held-out film (never used for threshold tuning) — three actors correctly
|
||||
identified with calibrated confidence scores.*
|
||||
|
||||
## How it works
|
||||
|
||||
1. **Build a gallery** — download actor headshots from TMDB/IMDB, embed them with ArcFace (`build_gallery` / `scripts/make_gallery.py`).
|
||||
1. **Build a gallery** — download actor headshots from TMDB/IMDB, embed them with ArcFace or LVFace (`build_gallery` / `scripts/make_gallery.py`).
|
||||
2. **Analyze a movie** — `scene_analyze` decodes frames at configurable FPS, detects faces (SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON.
|
||||
3. **Output** — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.
|
||||
|
||||

|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Role |
|
||||
@@ -63,7 +75,7 @@ contract (112×112 aligned BGR crop → L2-normalised 512-d embedding) and its
|
||||
|
||||
```bash
|
||||
./build/scene_analyze --arcface-model models/LVFace-B_Glint360K.onnx \
|
||||
--gallery gallery.json --input movie.mp4
|
||||
--gallery gallery.h5 --movie movie.mp4
|
||||
```
|
||||
|
||||
> **Important:** embeddings from different recognition models are not
|
||||
@@ -97,7 +109,7 @@ bash scripts/download_models.sh
|
||||
### `scene_analyze`
|
||||
|
||||
```bash
|
||||
./build/scene_analyze --gallery gallery.json --input movie.mp4 [options]
|
||||
./build/scene_analyze --gallery gallery.h5 --movie movie.mp4 [options]
|
||||
```
|
||||
|
||||
Key options:
|
||||
@@ -118,7 +130,7 @@ Key options:
|
||||
**Per-movie (TMDB):**
|
||||
|
||||
```bash
|
||||
python3 scripts/make_gallery.py --tmdb-bearer <JWT> --movie-id <TMDB_ID> --output gallery.json
|
||||
python3 scripts/make_gallery.py --tmdb-key <TMDB_KEY> --movie-id <TMDB_ID> --output gallery.h5
|
||||
```
|
||||
|
||||
Fetches cast images from TMDB and embeds them via `sae_embed`.
|
||||
@@ -129,13 +141,13 @@ Fetches cast images from TMDB and embeds them via `sae_embed`.
|
||||
python3 scripts/make_jellyfin_gallery.py \
|
||||
--jellyfin-url http://jellyfin.local:8096 \
|
||||
--api-key <API_KEY> \
|
||||
--output gallery.json
|
||||
--output gallery.h5
|
||||
```
|
||||
|
||||
Scans every Movie/Series in Jellyfin, collects the unique cast across the
|
||||
whole library, downloads each actor's headshot directly from Jellyfin (no
|
||||
TMDB key needed), and embeds them via `sae_embed` into one global
|
||||
gallery.json. Since `identity_matcher` scores faces against the entire
|
||||
gallery.h5. Since `identity_matcher` scores faces against the entire
|
||||
gallery, `scene_analyze` can then recognise any actor from your library in
|
||||
any film — not just the cast listed for that one title. Pass `--merge` on
|
||||
later runs to only embed actors newly added to the library. Pass
|
||||
@@ -153,11 +165,11 @@ look-alike mismatches), filter the global gallery first:
|
||||
|
||||
```bash
|
||||
python3 scripts/filter_gallery.py \
|
||||
--gallery gallery.json \
|
||||
--gallery gallery.h5 \
|
||||
--jellyfin-url http://jellyfin.local:8096 \
|
||||
--api-key <API_KEY> \
|
||||
--title "The Matrix" \
|
||||
--output gallery_matrix.json
|
||||
--output gallery_matrix.h5
|
||||
```
|
||||
|
||||
## Running directly from Jellyfin
|
||||
@@ -172,7 +184,7 @@ python3 scripts/run_from_jellyfin.py \
|
||||
--jellyfin-url http://jellyfin.local:8096 \
|
||||
--api-key <API_KEY> \
|
||||
--title "The Matrix" \
|
||||
--gallery gallery.json \
|
||||
--gallery gallery.h5 \
|
||||
-- --fps 5 --verbosity 2
|
||||
```
|
||||
|
||||
@@ -200,7 +212,7 @@ poll the same library concurrently.
|
||||
python3 scripts/run_from_jellyfin.py \
|
||||
--jellyfin-url http://jellyfin.local:8096 \
|
||||
--api-key <ADMIN_API_KEY> \
|
||||
--gallery whole_gallery.json \
|
||||
--gallery whole_gallery.h5 \
|
||||
--worker \
|
||||
-- --fps 5
|
||||
```
|
||||
@@ -223,15 +235,6 @@ worker moves on to the next item rather than exiting.
|
||||
|
||||
**Standard** — per-frame detail with bounding boxes, similarity scores, and track IDs.
|
||||
|
||||
## Pipeline topology
|
||||
|
||||
```
|
||||
frame_source → face_detector → face_aligner → embedder
|
||||
→ face_tracker → identity_matcher → scene_tracker → result_sink
|
||||
```
|
||||
|
||||
Debug/preview branches fan out automatically from `identity_matcher`.
|
||||
|
||||
## Evaluation
|
||||
|
||||
Scripts in `eval/` and `scripts/movienet_*.py` support benchmarking against the MovieNet dataset.
|
||||
|
||||
Reference in New Issue
Block a user