362 lines
16 KiB
Markdown
362 lines
16 KiB
Markdown
# JRay
|
|
|
|
A Jellyfin plugin that brings an actor-overlay (think Amazon "X-Ray") feature to your media: pause a movie and JRay shows you which actors are on screen at that exact moment.
|
|
|
|
JRay reads "truth" files produced offline by the
|
|
[scene-actor-extraction](https://github.com/dtourolle/scene-actor-extraction)
|
|
pipeline (face detection + recognition) and exposes an API to query which actors
|
|
are visible at a given timestamp. A small overlay, injected into the Jellyfin web
|
|
client, displays the result when you pause playback.
|
|
|
|
## Status
|
|
|
|
**Alpha** — JRay works end to end (sidecar truth files, remote truth push, and the
|
|
pause overlay) but is early software and the truth-file schema may still change.
|
|
The overlay relies on patching the web client's `index.html`, which is inherently
|
|
a little fragile across Jellyfin versions (see [Important Notes](#important-notes)).
|
|
|
|
## Quick Install
|
|
|
|
Add this repository URL in Jellyfin (Dashboard → Plugins → Repositories):
|
|
|
|
```
|
|
https://gitea.tourolle.paris/dtourolle/jRay/raw/branch/master/manifest.json
|
|
```
|
|
|
|
Then install "JRay" from the plugin catalog and restart Jellyfin.
|
|
|
|
## Features
|
|
|
|
- **Pause overlay** — pause a movie or episode in the web client and see the
|
|
actors currently on screen, without leaving the player.
|
|
- **Sidecar truth files** — drop a `Movie.jray.json` next to `Movie.mkv` and JRay
|
|
picks it up automatically (suffix configurable).
|
|
- **Remote truth push** — for servers that can't run the extraction pipeline
|
|
locally, a remote worker can `PUT` truth data over HTTP. Managed (pushed) data
|
|
takes precedence over sidecar files.
|
|
- **Work discovery API** — a remote worker can poll for a random batch of library
|
|
items that still need processing, so the backlog spreads naturally across
|
|
workers without server-side task tracking.
|
|
- **Extensible "context at time t" envelope** — the per-timestamp response is
|
|
designed to grow (locations, trivia, …) without breaking existing clients.
|
|
- **In-memory caching** — loaded truth files are cached with a configurable TTL.
|
|
- **Toggleable overlay** — disabling the overlay also cleanly removes the injected
|
|
script from the web client.
|
|
|
|
## How It Works
|
|
|
|
```
|
|
scene-actor-extraction Jellyfin server web client
|
|
(offline pipeline) (browser)
|
|
┌────────────────────┐ ┌──────────────────┐ ┌────────────┐
|
|
│ face detection + │ truth │ JRay plugin │ jray?t= │ pause │
|
|
│ recognition │ ───────► │ - sidecar reader │ ◄─────── │ overlay │
|
|
│ result_sink_node │ file │ - managed store │ actors │ script │
|
|
└────────────────────┘ │ - REST API │ ───────► └────────────┘
|
|
│ PUT /Truth (remote) │ - web patcher │
|
|
└────────────────────────►└──────────────────┘
|
|
```
|
|
|
|
1. The extraction pipeline analyses a film offline and emits a **truth file**
|
|
listing each detected actor and the time windows they're on screen.
|
|
2. JRay loads that truth file either from a **sidecar** next to the media
|
|
(`Movie.jray.json`) or from a **managed store** populated via the push API.
|
|
3. On startup JRay injects a small `<script>` into the web client's `index.html`.
|
|
When you pause, the script calls JRay for the current item and timestamp and
|
|
renders the on-screen actors as an overlay.
|
|
|
|
## Truth File Format
|
|
|
|
For a media file `Movie.mkv`, JRay looks for a sibling `Movie.jray.json` (suffix
|
|
configurable). Schema (`schema_version: 1`, minimal verbosity):
|
|
|
|
```json
|
|
{
|
|
"schema_version": 1,
|
|
"movie": "/path/to/Movie.mkv",
|
|
"sample_fps": 1,
|
|
"anneal_sec": 2,
|
|
"actors": [
|
|
{
|
|
"name": "Tom Hanks",
|
|
"imdb_id": "nm0000158",
|
|
"tmdb_id": "31",
|
|
"jellyfin_id": "abc123-guid",
|
|
"scenes": [[12.0, 45.0], [102.5, 150.0]]
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
An actor is considered visible at timestamp `t` (seconds) if any of their
|
|
`scenes` windows satisfies `start <= t <= end`. JRay prefers `jellyfin_id` (a
|
|
Jellyfin Person GUID) when present, otherwise resolves `imdb_id`/`tmdb_id`
|
|
against the item's People `ProviderIds`.
|
|
|
|
See [SPEC.md](SPEC.md) for the full schema and field-by-field reference.
|
|
|
|
## API
|
|
|
|
All routes are served under `/Plugins/JRay`. Authentication uses Jellyfin's
|
|
standard scheme — pass a token (a user access token or an API key) as either
|
|
the `X-Emby-Token: <token>` header or `Authorization: MediaBrowser Token="<token>"`.
|
|
|
|
| Method & Route | Auth | Description |
|
|
| --- | --- | --- |
|
|
| `GET /Items/{itemId}/jray?t={seconds}` | user | "Context at time t" envelope (on-screen actors), or `404`. |
|
|
| `GET /Items/{itemId}/Timeline` | user | Full truth file for an item, or `404` if none. |
|
|
| `PUT /Items/{itemId}/Truth` | admin | Push managed truth data (schema v1). `204` on success, `400` on bad schema. |
|
|
| `DELETE /Items/{itemId}/Truth` | admin | Remove managed truth data (idempotent, `204`). Falls back to sidecar. |
|
|
| `GET /Tasks/Pending?limit=10` | admin | Random sample of items still needing truth data (default 10, max 100). |
|
|
| `GET /ClientScript` | anon | The pause-overlay script injected into the web client. |
|
|
|
|
- **user** — any authenticated Jellyfin user token.
|
|
- **admin** — a token belonging to a user with the **Administrator** role
|
|
(create an API key under Dashboard → API Keys).
|
|
- **anon** — no authentication required.
|
|
|
|
## Client-Side Integration
|
|
|
|
JRay is designed so that *any* Jellyfin client (not just the bundled web overlay)
|
|
can build an actor-overlay feature. The integration is two calls: figure out
|
|
**what is playing and where**, then ask JRay **who is on screen**.
|
|
|
|
### 1. Query on-screen actors: `GET /Items/{itemId}/jray?t={seconds}`
|
|
|
|
Given a Jellyfin item id and a playback position in **seconds**, returns the
|
|
actors visible at that timestamp. This is the only call most clients need.
|
|
|
|
**Request**
|
|
|
|
```
|
|
GET /Plugins/JRay/Items/abc123-guid/jray?t=87.5
|
|
X-Emby-Token: <user-or-api-token>
|
|
```
|
|
|
|
**Response — `200 OK`**
|
|
|
|
```json
|
|
{
|
|
"actors": [
|
|
{
|
|
"name": "Tom Hanks",
|
|
"imdb_id": "nm0000158",
|
|
"tmdb_id": "31",
|
|
"jellyfin_id": "abc123-guid"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
- `actors` may be an **empty array** when no one is on screen at `t` — that's a
|
|
`200`, not a `404`.
|
|
- `404 Not Found` means the item has **no truth data at all** (no managed upload
|
|
and no sidecar file). Treat this as "JRay isn't available for this item" and
|
|
silently skip — don't surface an error to the viewer.
|
|
- The id fields are `""` when unresolved. Prefer `jellyfin_id` (a Jellyfin Person
|
|
GUID) to deep-link into the library or fetch a headshot; fall back to
|
|
`imdb_id` / `tmdb_id` for external links.
|
|
- The top-level object is an **extensible envelope**: future releases may add
|
|
sibling keys (e.g. `locations`, `trivia`) alongside `actors`. **Ignore unknown
|
|
keys** so your client keeps working across versions.
|
|
|
|
### 2. (Optional) Pre-fetch the whole timeline: `GET /Items/{itemId}/Timeline`
|
|
|
|
Returns the complete truth file (the [schema above](#truth-file-format)) — every
|
|
actor with all their scene windows. Use this if you'd rather fetch once and
|
|
compute "who's on screen" client-side (e.g. to drive a scrubber-bar heatmap)
|
|
instead of polling `jray?t=` on each pause. `404` if no truth data exists.
|
|
|
|
### Reference implementation (web client)
|
|
|
|
The bundled overlay ([`Web/jray-overlay.js`](Jellyfin.Plugin.JRay/Web/jray-overlay.js))
|
|
shows the full pattern using Jellyfin's `ApiClient`, and is a good template for a
|
|
custom client:
|
|
|
|
```js
|
|
// 1. Find what's playing and the current position (in seconds).
|
|
var sessions = await ApiClient.ajax({
|
|
url: ApiClient.getUrl('Sessions', { DeviceId: ApiClient.deviceId() }),
|
|
type: 'GET', dataType: 'json'
|
|
});
|
|
var s = sessions[0];
|
|
var itemId = s.NowPlayingItem.Id;
|
|
var t = (s.PlayState.PositionTicks || 0) / 10000000; // ticks → seconds
|
|
|
|
// 2. Ask JRay who is on screen. ApiClient adds the auth token for you.
|
|
var ctx = await ApiClient.ajax({
|
|
url: ApiClient.getUrl('Plugins/JRay/Items/' + itemId + '/jray', { t: t }),
|
|
type: 'GET', dataType: 'json'
|
|
});
|
|
|
|
// 3. Render ctx.actors. A 404 (no truth data) rejects the promise — swallow it.
|
|
```
|
|
|
|
Notes for client authors, learned from the reference overlay:
|
|
|
|
- **Position is in seconds.** Jellyfin reports `PositionTicks` (100 ns units);
|
|
divide by `10_000_000` before passing as `t`.
|
|
- **Fail silently.** A `404` or any network error must never interrupt playback —
|
|
just render nothing.
|
|
- **Enrich via the core API.** JRay returns ids, not images/bios. Use
|
|
`jellyfin_id` with the standard Jellyfin item/image endpoints (e.g.
|
|
`ApiClient.getItem(...)` / `getImageUrl(...)`) to show headshots and overviews,
|
|
and deep-link to `#/details?id=<jellyfin_id>`.
|
|
- **Refresh on player events.** The overlay recomputes on `pause` and clears on
|
|
`play` / `playing` / `seeking`. Poll `jray?t=` again after a seek rather than
|
|
reusing a stale result.
|
|
|
|
### Pushing truth data (remote extraction workers)
|
|
|
|
For the full remote-worker workflow — authenticate, resolve the item id by
|
|
`Path`, and `PUT` the truth file — see
|
|
[SPEC.md](SPEC.md#client-pushing-results-from-a-remote-extraction-worker).
|
|
These endpoints require an **Administrator** token.
|
|
|
|
## Configuration
|
|
|
|
Configure JRay in Jellyfin Dashboard → Plugins → JRay:
|
|
|
|
- **Truth File Suffix** — filename suffix used to locate sidecar truth files next
|
|
to media (default `.jray.json`, e.g. `Movie.mkv` → `Movie.jray.json`).
|
|
- **Cache Duration (minutes)** — how long a loaded truth file is cached in memory
|
|
before being re-read from disk (default `60`).
|
|
- **Enable pause overlay** — whether JRay injects its overlay script into the web
|
|
client's `index.html`. Disabling it removes any previously injected script
|
|
(default `on`).
|
|
|
|
## Building the Plugin
|
|
|
|
### Prerequisites
|
|
|
|
- [.NET SDK 9.0](https://dotnet.microsoft.com/en-us/download/dotnet)
|
|
- Jellyfin 10.9.0 or later (built against `Jellyfin.Controller` 10.11.5)
|
|
|
|
### Build Steps
|
|
|
|
```bash
|
|
dotnet publish Jellyfin.Plugin.JRay/Jellyfin.Plugin.JRay.csproj -c Release
|
|
```
|
|
|
|
The compiled `Jellyfin.Plugin.JRay.dll` will be under
|
|
`Jellyfin.Plugin.JRay/bin/Release/net9.0/publish/`.
|
|
|
|
A reproducible build via the bundled Docker image is also available:
|
|
|
|
```bash
|
|
docker build -f Dockerfile.builder -t jray-builder .
|
|
```
|
|
|
|
## Manual Installation
|
|
|
|
1. Build the plugin (see above).
|
|
2. Copy the published output into a `JRay` subfolder of your Jellyfin plugins
|
|
directory (e.g. `~/.local/share/jellyfin/plugins/JRay/` on Linux, or
|
|
`%LOCALAPPDATA%\jellyfin\plugins\JRay\` on Windows).
|
|
3. Restart Jellyfin.
|
|
4. Configure JRay in Dashboard → Plugins → JRay.
|
|
|
|
## Technical Architecture
|
|
|
|
### Directory Structure
|
|
|
|
```
|
|
Jellyfin.Plugin.JRay/
|
|
├── Configuration/
|
|
│ ├── PluginConfiguration.cs # Suffix, cache TTL, overlay toggle
|
|
│ └── configPage.html # Dashboard config page (embedded resource)
|
|
├── Controllers/
|
|
│ ├── ActorsController.cs # /Timeline and /jray?t= read endpoints
|
|
│ ├── TruthController.cs # PUT/DELETE managed truth data
|
|
│ ├── TasksController.cs # /Tasks/Pending work discovery
|
|
│ └── WebController.cs # /ClientScript overlay script
|
|
├── Models/
|
|
│ ├── TruthFile.cs # Root truth-file schema (schema_version 1)
|
|
│ ├── TruthActor.cs # Per-actor entry with scene windows
|
|
│ ├── ActorAtTime.cs # Actor entry in the "context at t" envelope
|
|
│ ├── JRayContext.cs # Extensible "context at time t" envelope
|
|
│ └── PendingExtractionItem.cs # Item descriptor for the work-discovery API
|
|
├── Services/
|
|
│ ├── Interfaces/
|
|
│ │ ├── ITruthDataService.cs
|
|
│ │ └── IManagedTruthStore.cs
|
|
│ ├── TruthDataService.cs # Resolves + caches truth (managed > sidecar)
|
|
│ ├── ManagedTruthStore.cs # Storage for pushed/managed truth data
|
|
│ └── WebClientPatchService.cs # Injects/removes overlay script in index.html
|
|
├── Web/
|
|
│ └── jray-overlay.js # Pause-overlay client script (embedded resource)
|
|
├── ServiceRegistrator.cs # DI registration
|
|
└── Plugin.cs # Plugin entry point; applies web patch on load
|
|
```
|
|
|
|
### Key Components
|
|
|
|
1. **Truth Data Service** (`TruthDataService`): resolves truth data for an item —
|
|
managed (pushed) data takes precedence over a sidecar file — and caches the
|
|
result in memory with the configured TTL.
|
|
2. **Managed Truth Store** (`ManagedTruthStore`): stores truth data pushed via the
|
|
API, independently of the media library filesystem.
|
|
3. **Controllers**: REST endpoints for reading (`ActorsController`), pushing
|
|
(`TruthController`), work discovery (`TasksController`), and serving the
|
|
overlay script (`WebController`).
|
|
4. **Web Client Patch Service** (`WebClientPatchService`): injects (or removes) the
|
|
`<script>` tag in the web client's `index.html`, marked with `<!-- jray-overlay -->`
|
|
so it's idempotent. Re-applied whenever configuration changes.
|
|
5. **Overlay Script** (`jray-overlay.js`): listens for the player's pause event,
|
|
calls `jray?t=`, and renders the on-screen actors.
|
|
|
|
## Important Notes
|
|
|
|
- **Web client overlay is a patch, not a hook.** Jellyfin has no plugin hook for
|
|
player UI, so JRay edits the web client's `index.html` directly. This generally
|
|
survives across restarts but may need re-applying after a Jellyfin web update;
|
|
toggling the overlay setting off and on re-runs the patch.
|
|
- **Truth data is produced offline.** JRay does not run face detection itself — it
|
|
consumes truth files from the
|
|
[scene-actor-extraction](https://github.com/dtourolle/scene-actor-extraction)
|
|
pipeline. No extraction = no overlay.
|
|
- **Schema version 1 only.** JRay accepts `schema_version: 1`. The schema may
|
|
change in future releases; bump-aware clients should send the version they
|
|
produced.
|
|
- **Remote path mapping.** Workers pushing truth match items by `Path`, so they
|
|
must see media at the same path Jellyfin does (translate paths first if mounts
|
|
differ).
|
|
|
|
## Contributing
|
|
|
|
Contributions welcome! This project is hosted on a self-hosted
|
|
[Gitea](https://gitea.tourolle.paris/dtourolle/jRay) instance.
|
|
|
|
**You don't need a separate account** — you can sign in with your existing GitHub
|
|
account. On the [sign-in page](https://gitea.tourolle.paris/user/login), choose
|
|
**"Sign in with GitHub"** to register and log in via GitHub OAuth. Once signed in,
|
|
you can:
|
|
|
|
- **Raise issues** — report bugs or request features on the
|
|
[issue tracker](https://gitea.tourolle.paris/dtourolle/jRay/issues).
|
|
- **Contribute code** — fork the repository, push a branch, and open a pull request.
|
|
|
|
## License
|
|
|
|
JRay is licensed under the **GNU General Public License v3.0**. See the
|
|
[LICENSE](LICENSE) file for details.
|
|
|
|
Because Jellyfin plugins link against the GPLv3-licensed Jellyfin NuGet packages,
|
|
the compiled plugin is necessarily GPLv3 as well.
|
|
|
|
## Acknowledgments
|
|
|
|
This plugin was developed partly using
|
|
[Claude Code](https://docs.anthropic.com/en/docs/claude-code) by Anthropic.
|
|
|
|
Built on the [Jellyfin plugin template](https://github.com/jellyfin/jellyfin-plugin-template)
|
|
and powered by the
|
|
[scene-actor-extraction](https://github.com/dtourolle/scene-actor-extraction)
|
|
pipeline.
|
|
|
|
## References
|
|
|
|
- [Jellyfin Plugin Documentation](https://jellyfin.org/docs/general/server/plugins/)
|
|
- [scene-actor-extraction pipeline](https://github.com/dtourolle/scene-actor-extraction)
|
|
- [JRay truth file specification](SPEC.md)
|