jRay/SPEC.md

JRay truth file format

JRay reads "truth" files produced offline by the scene-actor-extraction pipeline (result_sink_node, Verbosity::minimal, schema_version: 1).

File location

For a media file Movie.mkv, the pipeline writes a sibling file Movie.jray.json (suffix configurable in the plugin settings, default .jray.json). The plugin resolves this path from the Jellyfin item's media source path by stripping the extension and appending the suffix.

JSON schema (schema_version 1, minimal verbosity)

{
  "schema_version": 1,
  "movie": "/path/to/Movie.mkv",
  "sample_fps": 1,
  "anneal_sec": 2,
  "actors": [
    {
      "name": "Tom Hanks",
      "imdb_id": "nm0000158",
      "tmdb_id": "31",
      "jellyfin_id": "abc123-guid",
      "scenes": [[12.0, 45.0], [102.5, 150.0]]
    }
  ]
}

• schema_version: integer, bump on breaking changes. JRay should refuse (or warn) on a version it doesn't understand.
• movie: absolute path to the source media file at extraction time (informational only).
• sample_fps: frames-per-second the pipeline sampled at.
• anneal_sec: gap (in seconds) below which consecutive detections of the same actor were merged into a single scene window.
• actors[]: one entry per actor detected anywhere in the film.
  • name: display name from the gallery.
  • imdb_id / tmdb_id / jellyfin_id: identity keys, each "" if not resolved. JRay should prefer jellyfin_id (a Jellyfin Person item GUID) when non-empty, and otherwise resolve imdb_id/tmdb_id against the item's People ProviderIds.
  • scenes: list of [start_sec, end_sec] windows (inclusive) during which the actor is on screen.

Querying "who's on screen at time t"

For a given timestamp t (seconds), an actor is visible if any of their scenes windows satisfies start <= t <= end.

API

All read endpoints below require an authenticated Jellyfin user token (passed as X-Emby-Token or Authorization: MediaBrowser Token="..."); the admin endpoints additionally require the Administrator role. Only GET /Plugins/JRay/ClientScript is anonymous.

GET /Plugins/JRay/Items/{itemId}/Timeline

Returns the full truth file (schema above) for an item, or 404 if no truth data exists (neither a managed upload nor a sidecar file). Requires an authenticated user token.

GET /Plugins/JRay/Items/{itemId}/jray?t={seconds}

Returns an extensible "context at time t" envelope, or 404 if no truth data exists for the item. Requires an authenticated user token:

{
  "actors": [
    { "name": "Tom Hanks", "imdb_id": "nm0000158", "tmdb_id": "31", "jellyfin_id": "abc123-guid" }
  ]
}

Future fields (e.g. locations, trivia) will be added to this object without changing the route, so clients should ignore unknown keys.

PUT /Plugins/JRay/Items/{itemId}/Truth

For servers that cannot run the extraction pipeline locally, a remote worker may push truth data directly. Requires an administrator API key. Body is a truth file (schema above). Returns 204 on success, or 400 if schema_version is not 1.

This "managed" truth data takes precedence over any sidecar Movie.jray.json file for the same item, and is stored independently of the media library filesystem.

DELETE /Plugins/JRay/Items/{itemId}/Truth

Removes managed truth data for an item (idempotent, always returns 204). The item falls back to its sidecar truth file, if any, on subsequent reads. Requires an administrator API key.

GET /Plugins/JRay/ClientScript

Serves the pause-overlay script that JRay injects into the web client's index.html (see below). Anonymous access.

GET /Plugins/JRay/Tasks/Pending?limit=10

Lets a remote extraction worker discover what to work on next. Returns a random sample (default 10, max 100) of movies/episodes in the library that have no truth data yet (neither a managed upload nor a sidecar file):

[
  { "item_id": "abc123-guid", "path": "/data/movies/Movie.mkv", "name": "Movie" }
]

Requires an administrator API key. The sample is random, so repeated polling naturally spreads work across the backlog without needing server-side task tracking; an empty array means there's nothing left to do (or every remaining item is a virtual/missing-path item that JRay can't process).

Prioritise/ignore rules apply here. Items covered by an ignore rule are never returned. Items covered by a prioritise rule are returned ahead of un-prioritised items (still randomised within each tier). See Prioritise / ignore rules below. Rules only affect this work-discovery endpoint — they never change the overlay or the read endpoints, so an item you ignore for extraction still shows its overlay if truth data happens to exist for it.

Prioritise / ignore rules

Admins can steer the work-discovery queue with a small set of rules. Each rule targets a genre, a series, or a single item, and either prioritises (moves matching items to the front of Tasks/Pending) or ignores them (hides them from Tasks/Pending entirely). This is how you say "never extract anime", "process this series first", or "skip this one movie".

Rule resolution for an item picks the most specific matching scope: Item overrides Series, which overrides Genre. A rule is uniquely keyed by its scope + value, and setting a rule for an existing scope+value replaces it — so a single target can never be both prioritised and ignored. (An item can still be pulled in two directions across scopes, e.g. a prioritised series in an ignored genre; specificity resolves that — the series rule wins.)

Rules are persisted to policy.json under the plugin's configuration directory. A rule object:

{ "scope": "Genre", "value": "Anime", "action": "Ignore", "label": "Anime" }

• scope: "Genre", "Series", or "Item".
• value: a genre name (for Genre), a series id GUID (for Series), or an item id GUID (for Item). Genre matching is case-insensitive.
• action: "Prioritise" or "Ignore".
• label: optional human-readable label shown in the config UI (informational).

All endpoints below require an Administrator API key.

GET /Plugins/JRay/Policy/Rules

Returns all configured rules as a JSON array of rule objects.

PUT /Plugins/JRay/Policy/Rules

Adds or replaces a rule (body is a single rule object). Replaces any existing rule with the same scope + value. Returns 204, or 400 if value is empty.

DELETE /Plugins/JRay/Policy/Rules?scope={scope}&value={value}

Removes the rule matching scope + value. Idempotent, always returns 204.

Coverage overview

GET /Plugins/JRay/Coverage

Returns how much of the library has truth data, overall and broken down by media type (Film vs TV) and by genre. Requires an Administrator API key.

{
  "total":  { "total": 1200, "covered": 300, "pending": 850, "prioritised": 40, "ignored": 50 },
  "by_media_type": [
    { "label": "Film", "counts": { "total": 400, "covered": 200, "pending": 190, "prioritised": 10, "ignored": 10 } },
    { "label": "TV",   "counts": { "total": 800, "covered": 100, "pending": 660, "prioritised": 30, "ignored": 40 } }
  ],
  "by_genre": [
    { "label": "Anime", "counts": { "total": 120, "covered": 0, "pending": 0, "prioritised": 0, "ignored": 120 } }
  ]
}

Each counts object buckets items as: covered (has truth data), pending (needs processing and not ignored; prioritised is the subset of pending under a prioritise rule), and ignored (excluded by an ignore rule). total is the sum. A useful "percent done" is covered / (total - ignored), so ignoring a genre or series does not drag the percentage down — ignored items are treated as intentionally out of scope.

An item counts toward every genre it carries, so genre rows can overlap and their totals need not sum to the library total.

Pickers for the config UI

Three helper endpoints populate the rule editor's dropdowns (all require an Administrator API key, all return [{ "value": ..., "label": ... }]):

• GET /Plugins/JRay/Coverage/Genres — distinct genres present on movies/episodes.
• GET /Plugins/JRay/Coverage/Series — series in the library (value is the series id).
• GET /Plugins/JRay/Coverage/Items?search={term}&limit={n} — movies/episodes whose name matches term (value is the item id; limit default 25, max 100). An empty/absent search returns [].

Client: pushing results from a remote extraction worker

A worker that runs the extraction pipeline on a different machine than Jellyfin (i.e. it cannot write a Movie.jray.json sidecar next to the media file) can push results directly over HTTP.

To find work, poll GET /Plugins/JRay/Tasks/Pending?limit=10 (see above) for a random batch of items that still need processing, instead of walking the whole library and checking each item's Timeline/sidecar yourself.

1. Authenticate

Create an Administrator API key in Jellyfin (Dashboard → API Keys), and send it on every request as either:

X-Emby-Token: <api-key>

or:

Authorization: MediaBrowser Token="<api-key>"

2. Resolve the Jellyfin item id

The push endpoint is keyed by the Jellyfin item GUID, not by file path. To find it for Movie.mkv:

GET /Items?Recursive=true&Fields=Path&IncludeItemTypes=Movie,Episode

(use &ParentId=<library-id> to narrow the search if the library is large). Each returned item DTO has Id (the GUID) and Path. Match Path against the absolute path of the file you just processed — note this requires the worker to see the file at the same path Jellyfin does (same mount/share); translate paths first if the worker mounts the library elsewhere.

This mapping is stable until the file is moved/re-scanned, so the worker should cache path -> itemId and only re-resolve on a cache miss.

3. Push the truth file

PUT /Plugins/JRay/Items/{itemId}/Truth
Content-Type: application/json

<truth file JSON, schema_version 1, as produced by result_sink_node>

• 204 No Content — stored. Takes effect immediately (any cached read for this item is invalidated server-side).
• 400 Bad Request — schema_version is not 1.
• 401/403 — API key missing or not an administrator.

The PUT is idempotent (replaces any existing managed truth for the item), so the worker can safely retry on network errors.

4. (Optional) Remove pushed data

DELETE /Plugins/JRay/Items/{itemId}/Truth

Always returns 204. The item falls back to a sidecar Movie.jray.json (if any) on the next read.

Web client pause overlay

Since Jellyfin has no plugin hook for player UI, JRay injects <script defer src="/Plugins/JRay/ClientScript"></script> into the web client's index.html on startup (idempotent, marked with <!-- jray-overlay -->). The injected script listens for the video player's pause event, calls jray?t= for the current item and timestamp, and renders a small overlay listing on-screen actors. This can be disabled via the plugin's "Enable pause overlay" setting, which also removes the injected script.