Duncan Tourolle f11e5508ad update readme and spec for current release

2026-06-28 12:16:07 +02:00

16 KiB

Raw Blame History

JRay

A Jellyfin plugin that brings an actor-overlay (think Amazon "X-Ray") feature to your media: pause a movie and JRay shows you which actors are on screen at that exact moment.

JRay reads "truth" files produced offline by the scene-actor-extraction pipeline (face detection + recognition) and exposes an API to query which actors are visible at a given timestamp. A small overlay, injected into the Jellyfin web client, displays the result when you pause playback.

Status

Alpha — JRay works end to end (sidecar truth files, remote truth push, and the pause overlay) but is early software and the truth-file schema may still change. The overlay relies on patching the web client's index.html, which is inherently a little fragile across Jellyfin versions (see Important Notes).

Quick Install

Add this repository URL in Jellyfin (Dashboard → Plugins → Repositories):

https://gitea.tourolle.paris/dtourolle/jRay/raw/branch/master/manifest.json

Then install "JRay" from the plugin catalog and restart Jellyfin.

Features

Pause overlay — pause a movie or episode in the web client and see the actors currently on screen, without leaving the player.
Sidecar truth files — drop a Movie.jray.json next to Movie.mkv and JRay picks it up automatically (suffix configurable).
Remote truth push — for servers that can't run the extraction pipeline locally, a remote worker can PUT truth data over HTTP. Managed (pushed) data takes precedence over sidecar files.
Work discovery API — a remote worker can poll for a random batch of library items that still need processing, so the backlog spreads naturally across workers without server-side task tracking.
Extensible "context at time t" envelope — the per-timestamp response is designed to grow (locations, trivia, …) without breaking existing clients.
In-memory caching — loaded truth files are cached with a configurable TTL.
Toggleable overlay — disabling the overlay also cleanly removes the injected script from the web client.

How It Works

  scene-actor-extraction            Jellyfin server                 web client
  (offline pipeline)                                                (browser)
 ┌────────────────────┐           ┌──────────────────┐           ┌────────────┐
 │ face detection +   │  truth    │   JRay plugin    │  jray?t=  │  pause      │
 │ recognition        │ ───────►  │ - sidecar reader │ ◄───────  │  overlay    │
 │ result_sink_node   │  file     │ - managed store  │  actors   │  script     │
 └────────────────────┘           │ - REST API       │ ───────►  └────────────┘
        │ PUT /Truth (remote)     │ - web patcher    │
        └────────────────────────►└──────────────────┘

The extraction pipeline analyses a film offline and emits a truth file listing each detected actor and the time windows they're on screen.
JRay loads that truth file either from a sidecar next to the media (Movie.jray.json) or from a managed store populated via the push API.
On startup JRay injects a small <script> into the web client's index.html. When you pause, the script calls JRay for the current item and timestamp and renders the on-screen actors as an overlay.

Truth File Format

For a media file Movie.mkv, JRay looks for a sibling Movie.jray.json (suffix configurable). Schema (schema_version: 1, minimal verbosity):

{
  "schema_version": 1,
  "movie": "/path/to/Movie.mkv",
  "sample_fps": 1,
  "anneal_sec": 2,
  "actors": [
    {
      "name": "Tom Hanks",
      "imdb_id": "nm0000158",
      "tmdb_id": "31",
      "jellyfin_id": "abc123-guid",
      "scenes": [[12.0, 45.0], [102.5, 150.0]]
    }
  ]
}

An actor is considered visible at timestamp t (seconds) if any of their scenes windows satisfies start <= t <= end. JRay prefers jellyfin_id (a Jellyfin Person GUID) when present, otherwise resolves imdb_id/tmdb_id against the item's People ProviderIds.

See SPEC.md for the full schema and field-by-field reference.

API

All routes are served under /Plugins/JRay. Authentication uses Jellyfin's standard scheme — pass a token (a user access token or an API key) as either the X-Emby-Token: <token> header or Authorization: MediaBrowser Token="<token>".

Method & Route	Auth	Description
`GET /Items/{itemId}/jray?t={seconds}`	user	"Context at time t" envelope (on-screen actors), or `404`.
`GET /Items/{itemId}/Timeline`	user	Full truth file for an item, or `404` if none.
`PUT /Items/{itemId}/Truth`	admin	Push managed truth data (schema v1). `204` on success, `400` on bad schema.
`DELETE /Items/{itemId}/Truth`	admin	Remove managed truth data (idempotent, `204`). Falls back to sidecar.
`GET /Tasks/Pending?limit=10`	admin	Random sample of items still needing truth data (default 10, max 100).
`GET /ClientScript`	anon	The pause-overlay script injected into the web client.

user — any authenticated Jellyfin user token.
admin — a token belonging to a user with the Administrator role (create an API key under Dashboard → API Keys).
anon — no authentication required.

Client-Side Integration

JRay is designed so that any Jellyfin client (not just the bundled web overlay) can build an actor-overlay feature. The integration is two calls: figure out what is playing and where, then ask JRay who is on screen.

1. Query on-screen actors: `GET /Items/{itemId}/jray?t={seconds}`

Given a Jellyfin item id and a playback position in seconds, returns the actors visible at that timestamp. This is the only call most clients need.

Request

GET /Plugins/JRay/Items/abc123-guid/jray?t=87.5
X-Emby-Token: <user-or-api-token>

Response — 200 OK

{
  "actors": [
    {
      "name": "Tom Hanks",
      "imdb_id": "nm0000158",
      "tmdb_id": "31",
      "jellyfin_id": "abc123-guid"
    }
  ]
}

actors may be an empty array when no one is on screen at t — that's a 200, not a 404.
404 Not Found means the item has no truth data at all (no managed upload and no sidecar file). Treat this as "JRay isn't available for this item" and silently skip — don't surface an error to the viewer.
The id fields are "" when unresolved. Prefer jellyfin_id (a Jellyfin Person GUID) to deep-link into the library or fetch a headshot; fall back to imdb_id / tmdb_id for external links.
The top-level object is an extensible envelope: future releases may add sibling keys (e.g. locations, trivia) alongside actors. Ignore unknown keys so your client keeps working across versions.

2. (Optional) Pre-fetch the whole timeline: `GET /Items/{itemId}/Timeline`

Returns the complete truth file (the schema above) — every actor with all their scene windows. Use this if you'd rather fetch once and compute "who's on screen" client-side (e.g. to drive a scrubber-bar heatmap) instead of polling jray?t= on each pause. 404 if no truth data exists.

Reference implementation (web client)

The bundled overlay (Web/jray-overlay.js) shows the full pattern using Jellyfin's ApiClient, and is a good template for a custom client:

// 1. Find what's playing and the current position (in seconds).
var sessions = await ApiClient.ajax({
    url: ApiClient.getUrl('Sessions', { DeviceId: ApiClient.deviceId() }),
    type: 'GET', dataType: 'json'
});
var s = sessions[0];
var itemId = s.NowPlayingItem.Id;
var t = (s.PlayState.PositionTicks || 0) / 10000000; // ticks → seconds

// 2. Ask JRay who is on screen. ApiClient adds the auth token for you.
var ctx = await ApiClient.ajax({
    url: ApiClient.getUrl('Plugins/JRay/Items/' + itemId + '/jray', { t: t }),
    type: 'GET', dataType: 'json'
});

// 3. Render ctx.actors. A 404 (no truth data) rejects the promise — swallow it.

Notes for client authors, learned from the reference overlay:

Position is in seconds. Jellyfin reports PositionTicks (100 ns units); divide by 10_000_000 before passing as t.
Fail silently. A 404 or any network error must never interrupt playback — just render nothing.
Enrich via the core API. JRay returns ids, not images/bios. Use jellyfin_id with the standard Jellyfin item/image endpoints (e.g. ApiClient.getItem(...) / getImageUrl(...)) to show headshots and overviews, and deep-link to #/details?id=<jellyfin_id>.
Refresh on player events. The overlay recomputes on pause and clears on play / playing / seeking. Poll jray?t= again after a seek rather than reusing a stale result.

Pushing truth data (remote extraction workers)

For the full remote-worker workflow — authenticate, resolve the item id by Path, and PUT the truth file — see SPEC.md. These endpoints require an Administrator token.

Configuration

Configure JRay in Jellyfin Dashboard → Plugins → JRay:

Truth File Suffix — filename suffix used to locate sidecar truth files next to media (default .jray.json, e.g. Movie.mkv → Movie.jray.json).
Cache Duration (minutes) — how long a loaded truth file is cached in memory before being re-read from disk (default 60).
Enable pause overlay — whether JRay injects its overlay script into the web client's index.html. Disabling it removes any previously injected script (default on).

Building the Plugin

Prerequisites

.NET SDK 9.0
Jellyfin 10.9.0 or later (built against Jellyfin.Controller 10.11.5)

Build Steps

dotnet publish Jellyfin.Plugin.JRay/Jellyfin.Plugin.JRay.csproj -c Release

The compiled Jellyfin.Plugin.JRay.dll will be under Jellyfin.Plugin.JRay/bin/Release/net9.0/publish/.

A reproducible build via the bundled Docker image is also available:

docker build -f Dockerfile.builder -t jray-builder .

Manual Installation

Build the plugin (see above).
Copy the published output into a JRay subfolder of your Jellyfin plugins directory (e.g. ~/.local/share/jellyfin/plugins/JRay/ on Linux, or %LOCALAPPDATA%\jellyfin\plugins\JRay\ on Windows).
Restart Jellyfin.
Configure JRay in Dashboard → Plugins → JRay.

Technical Architecture

Directory Structure

Jellyfin.Plugin.JRay/
├── Configuration/
│   ├── PluginConfiguration.cs     # Suffix, cache TTL, overlay toggle
│   └── configPage.html            # Dashboard config page (embedded resource)
├── Controllers/
│   ├── ActorsController.cs        # /Timeline and /jray?t= read endpoints
│   ├── TruthController.cs         # PUT/DELETE managed truth data
│   ├── TasksController.cs         # /Tasks/Pending work discovery
│   └── WebController.cs           # /ClientScript overlay script
├── Models/
│   ├── TruthFile.cs               # Root truth-file schema (schema_version 1)
│   ├── TruthActor.cs              # Per-actor entry with scene windows
│   ├── ActorAtTime.cs            # Actor entry in the "context at t" envelope
│   ├── JRayContext.cs             # Extensible "context at time t" envelope
│   └── PendingExtractionItem.cs   # Item descriptor for the work-discovery API
├── Services/
│   ├── Interfaces/
│   │   ├── ITruthDataService.cs
│   │   └── IManagedTruthStore.cs
│   ├── TruthDataService.cs        # Resolves + caches truth (managed > sidecar)
│   ├── ManagedTruthStore.cs       # Storage for pushed/managed truth data
│   └── WebClientPatchService.cs   # Injects/removes overlay script in index.html
├── Web/
│   └── jray-overlay.js            # Pause-overlay client script (embedded resource)
├── ServiceRegistrator.cs          # DI registration
└── Plugin.cs                      # Plugin entry point; applies web patch on load

Key Components

Truth Data Service (TruthDataService): resolves truth data for an item — managed (pushed) data takes precedence over a sidecar file — and caches the result in memory with the configured TTL.
Managed Truth Store (ManagedTruthStore): stores truth data pushed via the API, independently of the media library filesystem.
Controllers: REST endpoints for reading (ActorsController), pushing (TruthController), work discovery (TasksController), and serving the overlay script (WebController).
Web Client Patch Service (WebClientPatchService): injects (or removes) the <script> tag in the web client's index.html, marked with  so it's idempotent. Re-applied whenever configuration changes.
Overlay Script (jray-overlay.js): listens for the player's pause event, calls jray?t=, and renders the on-screen actors.

Important Notes

Web client overlay is a patch, not a hook. Jellyfin has no plugin hook for player UI, so JRay edits the web client's index.html directly. This generally survives across restarts but may need re-applying after a Jellyfin web update; toggling the overlay setting off and on re-runs the patch.
Truth data is produced offline. JRay does not run face detection itself — it consumes truth files from the scene-actor-extraction pipeline. No extraction = no overlay.
Schema version 1 only. JRay accepts schema_version: 1. The schema may change in future releases; bump-aware clients should send the version they produced.
Remote path mapping. Workers pushing truth match items by Path, so they must see media at the same path Jellyfin does (translate paths first if mounts differ).

Contributing

Contributions welcome! This project is hosted on a self-hosted Gitea instance.

You don't need a separate account — you can sign in with your existing GitHub account. On the sign-in page, choose "Sign in with GitHub" to register and log in via GitHub OAuth. Once signed in, you can:

Raise issues — report bugs or request features on the issue tracker.
Contribute code — fork the repository, push a branch, and open a pull request.

License

JRay is licensed under the GNU General Public License v3.0. See the LICENSE file for details.

Because Jellyfin plugins link against the GPLv3-licensed Jellyfin NuGet packages, the compiled plugin is necessarily GPLv3 as well.

Acknowledgments

This plugin was developed partly using Claude Code by Anthropic.

Built on the Jellyfin plugin template and powered by the scene-actor-extraction pipeline.

16 KiB Raw Blame History