jRay/SPEC.md

# JRay truth file format

JRay reads "truth" files produced offline by the
[scene-actor-extraction](https://github.com/dtourolle/scene-actor-extraction)
pipeline (`result_sink_node`, `Verbosity::minimal`, `schema_version: 1`).

## File location

For a media file `Movie.mkv`, the pipeline writes a sibling file
`Movie.jray.json` (suffix configurable in the plugin settings, default
`.jray.json`). The plugin resolves this path from the Jellyfin item's media
source path by stripping the extension and appending the suffix.

## JSON schema (schema_version 1, minimal verbosity)

```json
{
  "schema_version": 1,
  "movie": "/path/to/Movie.mkv",
  "sample_fps": 1,
  "anneal_sec": 2,
  "actors": [
    {
      "name": "Tom Hanks",
      "imdb_id": "nm0000158",
      "tmdb_id": "31",
      "jellyfin_id": "abc123-guid",
      "scenes": [[12.0, 45.0], [102.5, 150.0]]
    }
  ]
}
```

- `schema_version`: integer, bump on breaking changes. JRay should refuse (or
  warn) on a version it doesn't understand.
- `movie`: absolute path to the source media file at extraction time (informational only).
- `sample_fps`: frames-per-second the pipeline sampled at.
- `anneal_sec`: gap (in seconds) below which consecutive detections of the
  same actor were merged into a single scene window.
- `actors[]`: one entry per actor detected anywhere in the film.
  - `name`: display name from the gallery.
  - `imdb_id` / `tmdb_id` / `jellyfin_id`: identity keys, each `""` if not
    resolved. JRay should prefer `jellyfin_id` (a Jellyfin Person item GUID)
    when non-empty, and otherwise resolve `imdb_id`/`tmdb_id` against the
    item's People `ProviderIds`.
  - `scenes`: list of `[start_sec, end_sec]` windows (inclusive) during which
    the actor is on screen.

## Querying "who's on screen at time t"

For a given timestamp `t` (seconds), an actor is visible if any of their
`scenes` windows satisfies `start <= t <= end`.

## API

All read endpoints below require an authenticated Jellyfin user token (passed as
`X-Emby-Token` or `Authorization: MediaBrowser Token="..."`); the admin endpoints
additionally require the **Administrator** role. Only `GET /Plugins/JRay/ClientScript`
is anonymous.

### `GET /Plugins/JRay/Items/{itemId}/Timeline`

Returns the full truth file (schema above) for an item, or `404` if no truth
data exists (neither a managed upload nor a sidecar file). Requires an
authenticated user token.

### `GET /Plugins/JRay/Items/{itemId}/jray?t={seconds}`

Returns an extensible "context at time t" envelope, or `404` if no truth
data exists for the item. Requires an authenticated user token:

```json
{
  "actors": [
    { "name": "Tom Hanks", "imdb_id": "nm0000158", "tmdb_id": "31", "jellyfin_id": "abc123-guid" }
  ]
}
```

Future fields (e.g. `locations`, `trivia`) will be added to this object
without changing the route, so clients should ignore unknown keys.

### `PUT /Plugins/JRay/Items/{itemId}/Truth`

For servers that cannot run the extraction pipeline locally, a remote worker
may push truth data directly. Requires an administrator API key. Body is a
truth file (schema above). Returns `204` on success, or `400` if
`schema_version` is not `1`.

This "managed" truth data takes precedence over any sidecar
`Movie.jray.json` file for the same item, and is stored independently of the
media library filesystem.

### `DELETE /Plugins/JRay/Items/{itemId}/Truth`

Removes managed truth data for an item (idempotent, always returns `204`).
The item falls back to its sidecar truth file, if any, on subsequent reads.
Requires an administrator API key.

### `GET /Plugins/JRay/ClientScript`

Serves the pause-overlay script that JRay injects into the web client's
`index.html` (see below). Anonymous access.

### `GET /Plugins/JRay/Tasks/Pending?limit=10`

Lets a remote extraction worker discover what to work on next. Returns a
random sample (default 10, max 100) of movies/episodes in the library that
have no truth data yet (neither a managed upload nor a sidecar file):

```json
[
  { "item_id": "abc123-guid", "path": "/data/movies/Movie.mkv", "name": "Movie" }
]
```

Requires an administrator API key. The sample is random and unordered, so
repeated polling naturally spreads work across the backlog without needing
server-side task tracking; an empty array means there's nothing left to do
(or every remaining item is a virtual/missing-path item that JRay can't
process).

## Client: pushing results from a remote extraction worker

A worker that runs the extraction pipeline on a different machine than
Jellyfin (i.e. it cannot write a `Movie.jray.json` sidecar next to the media
file) can push results directly over HTTP.

To find work, poll `GET /Plugins/JRay/Tasks/Pending?limit=10` (see above) for
a random batch of items that still need processing, instead of walking the
whole library and checking each item's `Timeline`/sidecar yourself.

### 1. Authenticate

Create an **Administrator** API key in Jellyfin (Dashboard → API Keys), and
send it on every request as either:

```
X-Emby-Token: <api-key>
```

or:

```
Authorization: MediaBrowser Token="<api-key>"
```

### 2. Resolve the Jellyfin item id

The push endpoint is keyed by the Jellyfin item GUID, not by file path. To
find it for `Movie.mkv`:

```
GET /Items?Recursive=true&Fields=Path&IncludeItemTypes=Movie,Episode
```

(use `&ParentId=<library-id>` to narrow the search if the library is large).
Each returned item DTO has `Id` (the GUID) and `Path`. Match `Path` against
the absolute path of the file you just processed — note this requires the
worker to see the file at the *same path* Jellyfin does (same mount/share);
translate paths first if the worker mounts the library elsewhere.

This mapping is stable until the file is moved/re-scanned, so the worker
should cache `path -> itemId` and only re-resolve on a cache miss.

### 3. Push the truth file

```
PUT /Plugins/JRay/Items/{itemId}/Truth
Content-Type: application/json

<truth file JSON, schema_version 1, as produced by result_sink_node>
```

- `204 No Content` — stored. Takes effect immediately (any cached read for
  this item is invalidated server-side).
- `400 Bad Request` — `schema_version` is not `1`.
- `401`/`403` — API key missing or not an administrator.

The `PUT` is idempotent (replaces any existing managed truth for the item),
so the worker can safely retry on network errors.

### 4. (Optional) Remove pushed data

```
DELETE /Plugins/JRay/Items/{itemId}/Truth
```

Always returns `204`. The item falls back to a sidecar `Movie.jray.json` (if
any) on the next read.

## Web client pause overlay

Since Jellyfin has no plugin hook for player UI, JRay injects
`<script defer src="/Plugins/JRay/ClientScript"></script>` into the web
client's `index.html` on startup (idempotent, marked with
`<!-- jray-overlay -->`). The injected script listens for the video player's
pause event, calls `jray?t=` for the current item and timestamp, and renders
a small overlay listing on-screen actors. This can be disabled via the
plugin's "Enable pause overlay" setting, which also removes the injected
script.