jRay/README.md

# JRay

A Jellyfin plugin that brings an actor-overlay (think Amazon "X-Ray") feature to your media: pause a movie and JRay shows you which actors are on screen at that exact moment.

JRay reads "truth" files produced offline by the
[scene-actor-extraction](https://github.com/dtourolle/scene-actor-extraction)
pipeline (face detection + recognition) and exposes an API to query which actors
are visible at a given timestamp. A small overlay, injected into the Jellyfin web
client, displays the result when you pause playback.

## Status

**Alpha** — JRay works end to end (sidecar truth files, remote truth push, and the
pause overlay) but is early software and the truth-file schema may still change.
The overlay relies on patching the web client's `index.html`, which is inherently
a little fragile across Jellyfin versions (see [Important Notes](#important-notes)).

## Quick Install

Add this repository URL in Jellyfin (Dashboard → Plugins → Repositories):

```
https://gitea.tourolle.paris/dtourolle/jRay/raw/branch/master/manifest.json
```

Then install "JRay" from the plugin catalog and restart Jellyfin.

### Required for the overlay: File Transformation

JRay's pause overlay needs a script tag in the web client's `index.html`.
Install [File Transformation](https://github.com/IAmParadox27/jellyfin-plugin-file-transformation)
(repository `https://www.iamparadox.dev/jellyfin/plugins/manifest.json`) **before**
installing JRay. It rewrites the page as it is served, so the file on disk is
never touched — that survives server upgrades and coexists with other plugins
patching the same file.

**There is no fallback.** JRay never edits `index.html` on disk: a patch there
outlives an uninstall, is silently discarded by a web-client upgrade, and races
any other plugin touching the file. Without File Transformation the pause
overlay is simply disabled — every other JRay feature works normally, and the
plugin's configuration page tells you what is missing and how to install it.

Upgrading from an older JRay that did patch `index.html`? It removes its own
patch on startup, so there is nothing to clean up by hand. See
[SPEC.md](SPEC.md) JR-021/JR-022.

## Features

- **Pause overlay** — pause a movie or episode in the web client and see the
  actors currently on screen, without leaving the player.
- **Sidecar truth files** — drop a `Movie.jray.json` next to `Movie.mkv` and JRay
  picks it up automatically (suffix configurable).
- **Remote truth push** — for servers that can't run the extraction pipeline
  locally, a remote worker can `PUT` truth data over HTTP. Managed (pushed) data
  takes precedence over sidecar files.
- **Work discovery API** — a remote worker can poll for a random batch of library
  items that still need processing, so the backlog spreads naturally across
  workers without server-side task tracking.
- **Prioritise / ignore rules** — steer that backlog by genre, series, or
  individual item: ignore what you never want extracted (e.g. anime), or push a
  series to the front of the queue.
- **Coverage overview** — see at a glance how much of your library has actor
  data, broken down by genre and by media type (film vs TV).
- **Extensible "context at time t" envelope** — the per-timestamp response is
  designed to grow (locations, trivia, …) without breaking existing clients.
- **In-memory caching** — loaded truth files are cached with a configurable TTL.
- **Toggleable overlay** — disabling the overlay also cleanly removes the injected
  script from the web client.

## How It Works

```
  scene-actor-extraction            Jellyfin server                 web client
  (offline pipeline)                                                (browser)
 ┌────────────────────┐           ┌──────────────────┐           ┌────────────┐
 │ face detection +   │  truth    │   JRay plugin    │  jray?t=  │  pause      │
 │ recognition        │ ───────►  │ - sidecar reader │ ◄───────  │  overlay    │
 │ result_sink_node   │  file     │ - managed store  │  actors   │  script     │
 └────────────────────┘           │ - REST API       │ ───────►  └────────────┘
        │ PUT /Truth (remote)     │ - web patcher    │
        └────────────────────────►└──────────────────┘
```

1. The extraction pipeline analyses a film offline and emits a **truth file**
   listing each detected actor and the time windows they're on screen.
2. JRay loads that truth file either from a **sidecar** next to the media
   (`Movie.jray.json`) or from a **managed store** populated via the push API.
3. On startup JRay injects a small `<script>` into the web client's `index.html`.
   When you pause, the script calls JRay for the current item and timestamp and
   renders the on-screen actors as an overlay.

## Truth File Format

For a media file `Movie.mkv`, JRay looks for a sibling `Movie.jray.json` (suffix
configurable). Schema (`schema_version: 1`, minimal verbosity):

```json
{
  "schema_version": 1,
  "movie": "/path/to/Movie.mkv",
  "sample_fps": 1,
  "anneal_sec": 2,
  "actors": [
    {
      "name": "Tom Hanks",
      "imdb_id": "nm0000158",
      "tmdb_id": "31",
      "jellyfin_id": "abc123-guid",
      "scenes": [[12.0, 45.0], [102.5, 150.0]]
    }
  ]
}
```

An actor is considered visible at timestamp `t` (seconds) if any of their
`scenes` windows satisfies `start <= t <= end`. JRay prefers `jellyfin_id` (a
Jellyfin Person GUID) when present, otherwise resolves `imdb_id`/`tmdb_id`
against the item's People `ProviderIds`.

See [SPEC.md](SPEC.md) for the full schema and field-by-field reference.

## API

All routes are served under `/Plugins/JRay`. Authentication uses Jellyfin's
standard scheme — pass a token (a user access token or an API key) as either
the `X-Emby-Token: <token>` header or `Authorization: MediaBrowser Token="<token>"`.

| Method & Route | Auth | Description |
| --- | --- | --- |
| `GET /Items/{itemId}/jray?t={seconds}` | user | "Context at time t" envelope (on-screen actors), or `404`. |
| `GET /Items/{itemId}/Timeline` | user | Full truth file for an item, or `404` if none. |
| `PUT /Items/{itemId}/Truth` | admin | Push managed truth data (schema v1). `204` on success, `400` on bad schema. |
| `DELETE /Items/{itemId}/Truth` | admin | Remove managed truth data (idempotent, `204`). Falls back to sidecar. |
| `GET /Tasks/Pending?limit=10` | admin | Random sample of items still needing truth data (default 10, max 100); honours prioritise/ignore rules. |
| `GET /Policy/Rules` | admin | List prioritise/ignore rules. |
| `PUT /Policy/Rules` | admin | Add or replace a rule. `204`, or `400` if `value` is empty. |
| `DELETE /Policy/Rules?scope=&value=` | admin | Remove a rule (idempotent, `204`). |
| `GET /Coverage` | admin | Library coverage totals, plus breakdowns by media type and genre. |
| `GET /Coverage/Genres` \| `/Series` \| `/Items?search=` | admin | Option lists for the config page's rule editor. |
| `GET /ClientScript` | anon | The pause-overlay script injected into the web client. |

- **user** — any authenticated Jellyfin user token.
- **admin** — a token belonging to a user with the **Administrator** role
  (create an API key under Dashboard → API Keys).
- **anon** — no authentication required.

## Client-Side Integration

JRay is designed so that *any* Jellyfin client (not just the bundled web overlay)
can build an actor-overlay feature. The integration is two calls: figure out
**what is playing and where**, then ask JRay **who is on screen**.

### 1. Query on-screen actors: `GET /Items/{itemId}/jray?t={seconds}`

Given a Jellyfin item id and a playback position in **seconds**, returns the
actors visible at that timestamp. This is the only call most clients need.

**Request**

```
GET /Plugins/JRay/Items/abc123-guid/jray?t=87.5
X-Emby-Token: <user-or-api-token>
```

**Response — `200 OK`**

```json
{
  "actors": [
    {
      "name": "Tom Hanks",
      "imdb_id": "nm0000158",
      "tmdb_id": "31",
      "jellyfin_id": "abc123-guid"
    }
  ]
}
```

- `actors` may be an **empty array** when no one is on screen at `t` — that's a
  `200`, not a `404`.
- `404 Not Found` means the item has **no truth data at all** (no managed upload
  and no sidecar file). Treat this as "JRay isn't available for this item" and
  silently skip — don't surface an error to the viewer.
- The id fields are `""` when unresolved. Prefer `jellyfin_id` (a Jellyfin Person
  GUID) to deep-link into the library or fetch a headshot; fall back to
  `imdb_id` / `tmdb_id` for external links.
- The top-level object is an **extensible envelope**: future releases may add
  sibling keys (e.g. `locations`, `trivia`) alongside `actors`. **Ignore unknown
  keys** so your client keeps working across versions.

### 2. (Optional) Pre-fetch the whole timeline: `GET /Items/{itemId}/Timeline`

Returns the complete truth file (the [schema above](#truth-file-format)) — every
actor with all their scene windows. Use this if you'd rather fetch once and
compute "who's on screen" client-side (e.g. to drive a scrubber-bar heatmap)
instead of polling `jray?t=` on each pause. `404` if no truth data exists.

### Reference implementation (web client)

The bundled overlay ([`Web/jray-overlay.js`](Jellyfin.Plugin.JRay/Web/jray-overlay.js))
shows the full pattern using Jellyfin's `ApiClient`, and is a good template for a
custom client:

```js
// 1. Find what's playing and the current position (in seconds).
var sessions = await ApiClient.ajax({
    url: ApiClient.getUrl('Sessions', { DeviceId: ApiClient.deviceId() }),
    type: 'GET', dataType: 'json'
});
var s = sessions[0];
var itemId = s.NowPlayingItem.Id;
var t = (s.PlayState.PositionTicks || 0) / 10000000; // ticks → seconds

// 2. Ask JRay who is on screen. ApiClient adds the auth token for you.
var ctx = await ApiClient.ajax({
    url: ApiClient.getUrl('Plugins/JRay/Items/' + itemId + '/jray', { t: t }),
    type: 'GET', dataType: 'json'
});

// 3. Render ctx.actors. A 404 (no truth data) rejects the promise — swallow it.
```

Notes for client authors, learned from the reference overlay:

- **Position is in seconds.** Jellyfin reports `PositionTicks` (100 ns units);
  divide by `10_000_000` before passing as `t`.
- **Fail silently.** A `404` or any network error must never interrupt playback —
  just render nothing.
- **Enrich via the core API.** JRay returns ids, not images/bios. Use
  `jellyfin_id` with the standard Jellyfin item/image endpoints (e.g.
  `ApiClient.getItem(...)` / `getImageUrl(...)`) to show headshots and overviews,
  and deep-link to `#/details?id=<jellyfin_id>`.
- **Refresh on player events.** The overlay recomputes on `pause` and clears on
  `play` / `playing` / `seeking`. Poll `jray?t=` again after a seek rather than
  reusing a stale result.

### Pushing truth data (remote extraction workers)

For the full remote-worker workflow — authenticate, resolve the item id by
`Path`, and `PUT` the truth file — see
[SPEC.md](SPEC.md#client-pushing-results-from-a-remote-extraction-worker).
These endpoints require an **Administrator** token.

## Configuration

Configure JRay in Jellyfin Dashboard → Plugins → JRay:

- **Truth File Suffix** — filename suffix used to locate sidecar truth files next
  to media (default `.jray.json`, e.g. `Movie.mkv` → `Movie.jray.json`).
- **Cache Duration (minutes)** — how long a loaded truth file is cached in memory
  before being re-read from disk (default `60`).
- **Enable pause overlay** — whether JRay injects its overlay script into the web
  client's `index.html`. Disabling it removes any previously injected script
  (default `on`).

The config page also shows a **coverage overview** (how much of your library has
actor data, by genre and media type) and a **prioritise / ignore rules** editor.
Rules steer the work-discovery API (`/Tasks/Pending`) only — they don't affect
the overlay or read endpoints:

- **Ignore** — matching items are never offered to extraction workers. Use it to
  skip content you don't want processed (e.g. a whole genre like anime, a
  specific series, or one movie). Ignored items are excluded from the coverage
  "percent done" so they don't count against you.
- **Prioritise** — matching items jump to the front of the work queue.

A rule targets a **genre**, a **series**, or a single **item**; the most specific
matching rule wins (item > series > genre). Setting a rule for a target that
already has one replaces it, so nothing can be both prioritised and ignored.

## Building the Plugin

### Prerequisites

- [.NET SDK 9.0](https://dotnet.microsoft.com/en-us/download/dotnet)
- Jellyfin 10.9.0 or later (built against `Jellyfin.Controller` 10.11.5)

### Build Steps

```bash
dotnet publish Jellyfin.Plugin.JRay/Jellyfin.Plugin.JRay.csproj -c Release
```

The compiled `Jellyfin.Plugin.JRay.dll` will be under
`Jellyfin.Plugin.JRay/bin/Release/net9.0/publish/`.

A reproducible build via the bundled Docker image is also available:

```bash
docker build -f Dockerfile.builder -t jray-builder .
```

## Manual Installation

1. Build the plugin (see above).
2. Copy the published output into a `JRay` subfolder of your Jellyfin plugins
   directory (e.g. `~/.local/share/jellyfin/plugins/JRay/` on Linux, or
   `%LOCALAPPDATA%\jellyfin\plugins\JRay\` on Windows).
3. Restart Jellyfin.
4. Configure JRay in Dashboard → Plugins → JRay.

## Technical Architecture

### Directory Structure

```
Jellyfin.Plugin.JRay/
├── Configuration/
│   ├── PluginConfiguration.cs     # Suffix, cache TTL, overlay toggle
│   └── configPage.html            # Dashboard config page (embedded resource)
├── Controllers/
│   ├── ActorsController.cs        # /Timeline and /jray?t= read endpoints
│   ├── TruthController.cs         # PUT/DELETE managed truth data
│   ├── TasksController.cs         # /Tasks/Pending work discovery (policy-aware)
│   ├── PolicyController.cs        # /Policy/Rules prioritise/ignore CRUD
│   ├── CoverageController.cs      # /Coverage overview + genre/series/item pickers
│   └── WebController.cs           # /ClientScript overlay script
├── Models/
│   ├── TruthFile.cs               # Root truth-file schema (schema_version 1)
│   ├── TruthActor.cs              # Per-actor entry with scene windows
│   ├── ActorAtTime.cs            # Actor entry in the "context at t" envelope
│   ├── JRayContext.cs             # Extensible "context at time t" envelope
│   ├── PendingExtractionItem.cs   # Item descriptor for the work-discovery API
│   ├── MediaPolicyRule.cs         # A prioritise/ignore rule (+ PolicyScope/PolicyAction)
│   ├── CoverageReport.cs          # Coverage overview (+ CoverageCounts / breakdown rows)
│   └── PickerOption.cs            # value/label option for the config-page pickers
├── Services/
│   ├── Interfaces/
│   │   ├── ITruthDataService.cs
│   │   ├── IManagedTruthStore.cs
│   │   └── IMediaPolicyStore.cs
│   ├── TruthDataService.cs        # Resolves + caches truth (managed > sidecar)
│   ├── ManagedTruthStore.cs       # Storage for pushed/managed truth data
│   ├── MediaPolicyStore.cs        # Persists prioritise/ignore rules (policy.json)
│   ├── PolicyResolver.cs          # Resolves an item's effective rule (item>series>genre)
│   └── WebClientPatchService.cs   # Injects/removes overlay script in index.html
├── Web/
│   └── jray-overlay.js            # Pause-overlay client script (embedded resource)
├── ServiceRegistrator.cs          # DI registration
└── Plugin.cs                      # Plugin entry point; applies web patch on load
```

### Key Components

1. **Truth Data Service** (`TruthDataService`): resolves truth data for an item —
   managed (pushed) data takes precedence over a sidecar file — and caches the
   result in memory with the configured TTL.
2. **Managed Truth Store** (`ManagedTruthStore`): stores truth data pushed via the
   API, independently of the media library filesystem.
3. **Controllers**: REST endpoints for reading (`ActorsController`), pushing
   (`TruthController`), work discovery (`TasksController`), and serving the
   overlay script (`WebController`).
4. **Web Client Patch Service** (`WebClientPatchService`): injects (or removes) the
   `<script>` tag in the web client's `index.html`, marked with `<!-- jray-overlay -->`
   so it's idempotent. Re-applied whenever configuration changes.
5. **Overlay Script** (`jray-overlay.js`): listens for the player's pause event,
   calls `jray?t=`, and renders the on-screen actors.

## Important Notes

- **Web client overlay is a patch, not a hook.** Jellyfin has no plugin hook for
  player UI, so JRay edits the web client's `index.html` directly. This generally
  survives across restarts but may need re-applying after a Jellyfin web update;
  toggling the overlay setting off and on re-runs the patch.
- **Truth data is produced offline.** JRay does not run face detection itself — it
  consumes truth files from the
  [scene-actor-extraction](https://github.com/dtourolle/scene-actor-extraction)
  pipeline. No extraction = no overlay.
- **Schema version 1 only.** JRay accepts `schema_version: 1`. The schema may
  change in future releases; bump-aware clients should send the version they
  produced.
- **Remote path mapping.** Workers pushing truth match items by `Path`, so they
  must see media at the same path Jellyfin does (translate paths first if mounts
  differ).

## Contributing

Contributions welcome! This project is hosted on a self-hosted
[Gitea](https://gitea.tourolle.paris/dtourolle/jRay) instance.

**You don't need a separate account** — you can sign in with your existing GitHub
account. On the [sign-in page](https://gitea.tourolle.paris/user/login), choose
**"Sign in with GitHub"** to register and log in via GitHub OAuth. Once signed in,
you can:

- **Raise issues** — report bugs or request features on the
  [issue tracker](https://gitea.tourolle.paris/dtourolle/jRay/issues).
- **Contribute code** — fork the repository, push a branch, and open a pull request.

## License

JRay is licensed under the **GNU General Public License v3.0**. See the
[LICENSE](LICENSE) file for details.

Because Jellyfin plugins link against the GPLv3-licensed Jellyfin NuGet packages,
the compiled plugin is necessarily GPLv3 as well.

## Acknowledgments

This plugin was developed partly using
[Claude Code](https://docs.anthropic.com/en/docs/claude-code) by Anthropic.

Built on the [Jellyfin plugin template](https://github.com/jellyfin/jellyfin-plugin-template)
and powered by the
[scene-actor-extraction](https://github.com/dtourolle/scene-actor-extraction)
pipeline.

## References

- [Jellyfin Plugin Documentation](https://jellyfin.org/docs/general/server/plugins/)
- [scene-actor-extraction pipeline](https://github.com/dtourolle/scene-actor-extraction)
- [JRay truth file specification](SPEC.md)