docs: richer report — data figures, success/failure frames, commit-pinned repo links

- experiment_charts.py generates 4 figures from experiments/ artifacts:
  held-out per-film F1, 16-combo ranking, DE search landscape, and the
  Downton detector-vs-tracker ghost timeline (replaces the blank
  title-card screenshot)
- new frames: 19-correct wedding shot (success case), Many Saints
  ghost-vs-unknown frame (three error classes in one image)
- rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only
  as the on-disk artifact prefix, explained once
- repo file references are now links via https://REPOLINK/<path>
  placeholders; build_site.sh pins them to the HEAD commit's raw URLs
  and fails the build if a linked path doesn't exist at HEAD
- drop references to removed scripts (scene_score.py, score_config.py)
  and to session-memory names; mark artifact-registry paths with their
  pull commands
- commit readme_example.jpg + pipeline_topology.svg so README renders
  on the plain Gitea repo view
- deploy_pages.sh: push built site/ to the gitea-pages branch
This commit is contained in:
2026-07-19 22:06:56 +02:00
parent 4925443e56
commit b1efefac6f
17 changed files with 702 additions and 146 deletions
+23 -20
View File
@@ -1,13 +1,25 @@
# Scene Actor Extraction
Identifies actors in movie files and produces X-ray-style scene annotations compatible with [Jellyfin](https://jellyfin.org/). Built on a KPN++ pipeline with ArcFace embeddings and a tracked-identity matcher.
Identifies actors in movie files and produces X-ray-style scene annotations compatible with [Jellyfin](https://jellyfin.org/). Built on a KPN++ pipeline with ArcFace/LVFace embeddings and a tracked-identity matcher.
**67.4% macro-F1 against Amazon X-Ray ground truth**, on 5 films never seen by
the optimizer (89.7% P / 65.4% R training-set; see the generalization-gap
discussion in the [deep dive](https://pages.tourolle.paris/dtourolle/scene-actor-extraction/lvface-deep-dive/)).
Full benchmark write-up, model comparison, and failure-mode analysis:
**https://pages.tourolle.paris/dtourolle/scene-actor-extraction/**
![Example: correctly identified actors in a held-out film](docs/assets/images/readme_example.jpg)
*A held-out film (never used for threshold tuning) — three actors correctly
identified with calibrated confidence scores.*
## How it works
1. **Build a gallery** — download actor headshots from TMDB/IMDB, embed them with ArcFace (`build_gallery` / `scripts/make_gallery.py`).
1. **Build a gallery** — download actor headshots from TMDB/IMDB, embed them with ArcFace or LVFace (`build_gallery` / `scripts/make_gallery.py`).
2. **Analyze a movie**`scene_analyze` decodes frames at configurable FPS, detects faces (SCRFD), tracks them across cuts, matches identities against the gallery using calibrated similarity, and writes time-window JSON.
3. **Output** — minimal mode produces Jellyfin-ready actor name + time-window JSON; standard mode adds per-frame bbox, similarity, and track data.
![Pipeline topology](docs/assets/images/pipeline_topology.svg)
## Dependencies
| Dependency | Role |
@@ -63,7 +75,7 @@ contract (112×112 aligned BGR crop → L2-normalised 512-d embedding) and its
```bash
./build/scene_analyze --arcface-model models/LVFace-B_Glint360K.onnx \
--gallery gallery.json --input movie.mp4
--gallery gallery.h5 --movie movie.mp4
```
> **Important:** embeddings from different recognition models are not
@@ -97,7 +109,7 @@ bash scripts/download_models.sh
### `scene_analyze`
```bash
./build/scene_analyze --gallery gallery.json --input movie.mp4 [options]
./build/scene_analyze --gallery gallery.h5 --movie movie.mp4 [options]
```
Key options:
@@ -118,7 +130,7 @@ Key options:
**Per-movie (TMDB):**
```bash
python3 scripts/make_gallery.py --tmdb-bearer <JWT> --movie-id <TMDB_ID> --output gallery.json
python3 scripts/make_gallery.py --tmdb-key <TMDB_KEY> --movie-id <TMDB_ID> --output gallery.h5
```
Fetches cast images from TMDB and embeds them via `sae_embed`.
@@ -129,13 +141,13 @@ Fetches cast images from TMDB and embeds them via `sae_embed`.
python3 scripts/make_jellyfin_gallery.py \
--jellyfin-url http://jellyfin.local:8096 \
--api-key <API_KEY> \
--output gallery.json
--output gallery.h5
```
Scans every Movie/Series in Jellyfin, collects the unique cast across the
whole library, downloads each actor's headshot directly from Jellyfin (no
TMDB key needed), and embeds them via `sae_embed` into one global
gallery.json. Since `identity_matcher` scores faces against the entire
gallery.h5. Since `identity_matcher` scores faces against the entire
gallery, `scene_analyze` can then recognise any actor from your library in
any film — not just the cast listed for that one title. Pass `--merge` on
later runs to only embed actors newly added to the library. Pass
@@ -153,11 +165,11 @@ look-alike mismatches), filter the global gallery first:
```bash
python3 scripts/filter_gallery.py \
--gallery gallery.json \
--gallery gallery.h5 \
--jellyfin-url http://jellyfin.local:8096 \
--api-key <API_KEY> \
--title "The Matrix" \
--output gallery_matrix.json
--output gallery_matrix.h5
```
## Running directly from Jellyfin
@@ -172,7 +184,7 @@ python3 scripts/run_from_jellyfin.py \
--jellyfin-url http://jellyfin.local:8096 \
--api-key <API_KEY> \
--title "The Matrix" \
--gallery gallery.json \
--gallery gallery.h5 \
-- --fps 5 --verbosity 2
```
@@ -200,7 +212,7 @@ poll the same library concurrently.
python3 scripts/run_from_jellyfin.py \
--jellyfin-url http://jellyfin.local:8096 \
--api-key <ADMIN_API_KEY> \
--gallery whole_gallery.json \
--gallery whole_gallery.h5 \
--worker \
-- --fps 5
```
@@ -223,15 +235,6 @@ worker moves on to the next item rather than exiting.
**Standard** — per-frame detail with bounding boxes, similarity scores, and track IDs.
## Pipeline topology
```
frame_source → face_detector → face_aligner → embedder
→ face_tracker → identity_matcher → scene_tracker → result_sink
```
Debug/preview branches fan out automatically from `identity_matcher`.
## Evaluation
Scripts in `eval/` and `scripts/movienet_*.py` support benchmarking against the MovieNet dataset.