jRay/SPEC.md

JRay truth file format

JRay reads "truth" files produced offline by the scene-actor-extraction pipeline (result_sink_node, Verbosity::minimal, schema_version: 1).

File location

For a media file Movie.mkv, the pipeline writes a sibling file Movie.jray.json (suffix configurable in the plugin settings, default .jray.json). The plugin resolves this path from the Jellyfin item's media source path by stripping the extension and appending the suffix.

JSON schema (schema_version 1, minimal verbosity)

{
  "schema_version": 1,
  "movie": "/path/to/Movie.mkv",
  "sample_fps": 1,
  "anneal_sec": 2,
  "actors": [
    {
      "name": "Tom Hanks",
      "imdb_id": "nm0000158",
      "tmdb_id": "31",
      "jellyfin_id": "abc123-guid",
      "scenes": [[12.0, 45.0], [102.5, 150.0]]
    }
  ]
}

• schema_version: integer, bump on breaking changes. JRay should refuse (or warn) on a version it doesn't understand.
• movie: absolute path to the source media file at extraction time (informational only).
• sample_fps: frames-per-second the pipeline sampled at.
• anneal_sec: gap (in seconds) below which consecutive detections of the same actor were merged into a single scene window.
• actors[]: one entry per actor detected anywhere in the film.
  • name: display name from the gallery.
  • imdb_id / tmdb_id / jellyfin_id: identity keys, each "" if not resolved. JRay should prefer jellyfin_id (a Jellyfin Person item GUID) when non-empty, and otherwise resolve imdb_id/tmdb_id against the item's People ProviderIds.
  • scenes: list of [start_sec, end_sec] windows (inclusive) during which the actor is on screen.

Querying "who's on screen at time t"

For a given timestamp t (seconds), an actor is visible if any of their scenes windows satisfies start <= t <= end.

API

All read endpoints below require an authenticated Jellyfin user token (passed as X-Emby-Token or Authorization: MediaBrowser Token="..."); the admin endpoints additionally require the Administrator role. Only GET /Plugins/JRay/ClientScript is anonymous.

GET /Plugins/JRay/Items/{itemId}/Timeline

Returns the full truth file (schema above) for an item, or 404 if no truth data exists (neither a managed upload nor a sidecar file). Requires an authenticated user token.

GET /Plugins/JRay/Items/{itemId}/jray?t={seconds}

Returns an extensible "context at time t" envelope, or 404 if no truth data exists for the item. Requires an authenticated user token:

{
  "actors": [
    { "name": "Tom Hanks", "imdb_id": "nm0000158", "tmdb_id": "31", "jellyfin_id": "abc123-guid" }
  ]
}

Future fields (e.g. locations, trivia) will be added to this object without changing the route, so clients should ignore unknown keys.

PUT /Plugins/JRay/Items/{itemId}/Truth

For servers that cannot run the extraction pipeline locally, a remote worker may push truth data directly. Requires an administrator API key. Body is a truth file (schema above). Returns 204 on success, or 400 if schema_version is not 1.

This "managed" truth data takes precedence over any sidecar Movie.jray.json file for the same item, and is stored independently of the media library filesystem.

DELETE /Plugins/JRay/Items/{itemId}/Truth

Removes managed truth data for an item (idempotent, always returns 204). The item falls back to its sidecar truth file, if any, on subsequent reads. Requires an administrator API key.

GET /Plugins/JRay/ClientScript

Serves the pause-overlay script that JRay injects into the web client's index.html (see below). Anonymous access.

GET /Plugins/JRay/Tasks/Pending?limit=10

Lets a remote extraction worker discover what to work on next. Returns a random sample (default 10, max 100) of movies/episodes in the library that have no truth data yet (neither a managed upload nor a sidecar file):

[
  { "item_id": "abc123-guid", "path": "/data/movies/Movie.mkv", "name": "Movie" }
]

Requires an administrator API key. The sample is random and unordered, so repeated polling naturally spreads work across the backlog without needing server-side task tracking; an empty array means there's nothing left to do (or every remaining item is a virtual/missing-path item that JRay can't process).

Client: pushing results from a remote extraction worker

A worker that runs the extraction pipeline on a different machine than Jellyfin (i.e. it cannot write a Movie.jray.json sidecar next to the media file) can push results directly over HTTP.

To find work, poll GET /Plugins/JRay/Tasks/Pending?limit=10 (see above) for a random batch of items that still need processing, instead of walking the whole library and checking each item's Timeline/sidecar yourself.

1. Authenticate

Create an Administrator API key in Jellyfin (Dashboard → API Keys), and send it on every request as either:

X-Emby-Token: <api-key>

or:

Authorization: MediaBrowser Token="<api-key>"

2. Resolve the Jellyfin item id

The push endpoint is keyed by the Jellyfin item GUID, not by file path. To find it for Movie.mkv:

GET /Items?Recursive=true&Fields=Path&IncludeItemTypes=Movie,Episode

(use &ParentId=<library-id> to narrow the search if the library is large). Each returned item DTO has Id (the GUID) and Path. Match Path against the absolute path of the file you just processed — note this requires the worker to see the file at the same path Jellyfin does (same mount/share); translate paths first if the worker mounts the library elsewhere.

This mapping is stable until the file is moved/re-scanned, so the worker should cache path -> itemId and only re-resolve on a cache miss.

3. Push the truth file

PUT /Plugins/JRay/Items/{itemId}/Truth
Content-Type: application/json

<truth file JSON, schema_version 1, as produced by result_sink_node>

• 204 No Content — stored. Takes effect immediately (any cached read for this item is invalidated server-side).
• 400 Bad Request — schema_version is not 1.
• 401/403 — API key missing or not an administrator.

The PUT is idempotent (replaces any existing managed truth for the item), so the worker can safely retry on network errors.

4. (Optional) Remove pushed data

DELETE /Plugins/JRay/Items/{itemId}/Truth

Always returns 204. The item falls back to a sidecar Movie.jray.json (if any) on the next read.

Web client pause overlay

Since Jellyfin has no plugin hook for player UI, JRay injects <script defer src="/Plugins/JRay/ClientScript"></script> into the web client's index.html on startup (idempotent, marked with <!-- jray-overlay -->). The injected script listens for the video player's pause event, calls jray?t= for the current item and timestamp, and renders a small overlay listing on-screen actors. This can be disabled via the plugin's "Enable pause overlay" setting, which also removes the injected script.