dtourolle/jRay
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 6 Wiki Activity

JR-004, JR-005, JR-006: scene-scoped read path

Browse Source 
Presence was decided by a LINQ predicate inline in the controller, so the
semantics SR-002 sets were nowhere stated in code -- the read path complied by
accident rather than by requirement. PresenceLookup is now the unit that
decides, tagged, with the reasoning next to it.

JR-004: windows are served exactly as given. UT-021 pins that [0,10] and
[10,20] are not merged despite looking mergeable -- two windows mean a genuine
departure and return, and collapsing them answers a different question from the
one the truth file asked. UT-022 pins a byte-identical round trip.

JR-005: bounds inclusive at both ends, zero-length windows are real sightings
rather than degenerate ones to discard, overlaps resolve.

The wording was the larger half of JR-005. The overlay rendered a bare list: it
asserted nothing, but told the viewer nothing either, and the default reading of
a paused frame is "these people are on screen" -- exactly what SR-002 forbids.
It now carries an "In this scene" heading. ActorAtTime became ActorInScene, and
README no longer contains "on screen" anywhere; it stated the forbidden reading
outright in seven places, including the opening sentence.

JR-006: measured rather than assumed. UT-023 builds 50 actors x 1000 windows and
asserts the response is bounded by actor count, never window count. The lookup
is a full scan on purpose -- an early exit on `start > t` would exploit the
sortedness the format requires, but would silently under-report the moment one
producer emitted windows out of order. UT-020 pins that unsorted input still
resolves; WindowsAreSorted is a diagnostic, not a correctness dependency.

Third mutation check: making the end bound exclusive fails UT-016 and UT-018 and
nothing else. One character turns an inclusive window into a half-open one,
dropping an actor at exactly the moment a scene ends.

TRACES: UT-016, UT-017, UT-018, UT-019, UT-020, UT-021, UT-022, UT-023
TRACES: JR-004, JR-005, JR-006 | SR-002

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dtourolle
2026-07-31 11:29:46 +02:00
co-authored by Claude Opus 5
parent 305b898b15
commit c04d5a3dcc
11 changed files with 378 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+18 -15
View File
@@ -1,11 +1,11 @@
# JRay
A Jellyfin plugin that brings an actor-overlay (think Amazon "X-Ray") feature to your media: pause a movie and JRay shows you which actors are on screen at that exact moment.
A Jellyfin plugin that brings an actor-overlay (think Amazon "X-Ray") feature to your media: pause a movie and JRay shows you which actors are in the scene you paused in.
JRay reads "truth" files produced offline by the
[scene-actor-extraction](https://github.com/dtourolle/scene-actor-extraction)
pipeline (face detection + recognition) and exposes an API to query which actors
are visible at a given timestamp. A small overlay, injected into the Jellyfin web
are present in the scene at a given timestamp. A small overlay, injected into the Jellyfin web
client, displays the result when you pause playback.
## Status
@@ -47,7 +47,7 @@ patch on startup, so there is nothing to clean up by hand. See
## Features
- **Pause overlay** — pause a movie or episode in the web client and see the
actors currently on screen, without leaving the player.
actors in the current scene, without leaving the player.
- **Sidecar truth files** — drop a `Movie.jray.json` next to `Movie.mkv` and JRay
picks it up automatically (suffix configurable).
- **Remote truth push** — for servers that can't run the extraction pipeline
@@ -82,12 +82,12 @@ patch on startup, so there is nothing to clean up by hand. See
```
1. The extraction pipeline analyses a film offline and emits a **truth file**
listing each detected actor and the time windows they're on screen.
listing each actor and the time windows they are present in the film.
2. JRay loads that truth file either from a **sidecar** next to the media
(`Movie.jray.json`) or from a **managed store** populated via the push API.
3. On startup JRay injects a small `<script>` into the web client's `index.html`.
When you pause, the script calls JRay for the current item and timestamp and
renders the on-screen actors as an overlay.
renders the scene's cast as an overlay.
## Truth File Format
@@ -112,8 +112,11 @@ configurable). Schema (`schema_version: 1`, minimal verbosity):
}
```
An actor is considered visible at timestamp `t` (seconds) if any of their
`scenes` windows satisfies `start <= t <= end`. JRay prefers `jellyfin_id` (a
An actor is present at timestamp `t` (seconds) if any of their `scenes` windows
satisfies `start <= t <= end`. **A window is a claim about scene membership, not
a recognition event** — an actor who has turned away or is off-camera during a
reverse shot is still present, and two windows mean a genuine departure and
return rather than a break in detection. JRay prefers `jellyfin_id` (a
Jellyfin Person GUID) when present, otherwise resolves `imdb_id`/`tmdb_id`
against the item's People `ProviderIds`.
@@ -127,7 +130,7 @@ the `X-Emby-Token: <token>` header or `Authorization: MediaBrowser Token="<token
| Method & Route | Auth | Description |
| --- | --- | --- |
| `GET /Items/{itemId}/jray?t={seconds}` | user | "Context at time t" envelope (on-screen actors), or `404`. |
| `GET /Items/{itemId}/jray?t={seconds}` | user | "Context at time t" envelope (the scene's cast), or `404`. |
| `GET /Items/{itemId}/Timeline` | user | Full truth file for an item, or `404` if none. |
| `PUT /Items/{itemId}/Truth` | admin | Push managed truth data (schema v1). `204` on success, `400` on bad schema. |
| `DELETE /Items/{itemId}/Truth` | admin | Remove managed truth data (idempotent, `204`). Falls back to sidecar. |
@@ -148,12 +151,12 @@ the `X-Emby-Token: <token>` header or `Authorization: MediaBrowser Token="<token
JRay is designed so that *any* Jellyfin client (not just the bundled web overlay)
can build an actor-overlay feature. The integration is two calls: figure out
**what is playing and where**, then ask JRay **who is on screen**.
**what is playing and where**, then ask JRay **who is in the scene**.
### 1. Query on-screen actors: `GET /Items/{itemId}/jray?t={seconds}`
### 1. Query the scene's cast: `GET /Items/{itemId}/jray?t={seconds}`
Given a Jellyfin item id and a playback position in **seconds**, returns the
actors visible at that timestamp. This is the only call most clients need.
the actors in that scene. This is the only call most clients need.
**Request**
@@ -177,7 +180,7 @@ X-Emby-Token: <user-or-api-token>
}
```
- `actors` may be an **empty array** when no one is on screen at `t` — that's a
- `actors` may be an **empty array** when no one is in the scene at `t` — that's a
`200`, not a `404`.
- `404 Not Found` means the item has **no truth data at all** (no managed upload
and no sidecar file). Treat this as "JRay isn't available for this item" and
@@ -193,7 +196,7 @@ X-Emby-Token: <user-or-api-token>
Returns the complete truth file (the [schema above](#truth-file-format)) — every
actor with all their scene windows. Use this if you'd rather fetch once and
compute "who's on screen" client-side (e.g. to drive a scrubber-bar heatmap)
compute "who is in the scene" client-side (e.g. to drive a scrubber-bar heatmap)
instead of polling `jray?t=` on each pause. `404` if no truth data exists.
### Reference implementation (web client)
@@ -212,7 +215,7 @@ var s = sessions[0];
var itemId = s.NowPlayingItem.Id;
var t = (s.PlayState.PositionTicks || 0) / 10000000; // ticks → seconds
// 2. Ask JRay who is on screen. ApiClient adds the auth token for you.
// 2. Ask JRay who is in the scene. ApiClient adds the auth token for you.
var ctx = await ApiClient.ajax({
url: ApiClient.getUrl('Plugins/JRay/Items/' + itemId + '/jray', { t: t }),
type: 'GET', dataType: 'json'
@@ -355,7 +358,7 @@ Jellyfin.Plugin.JRay/
`<script>` tag in the web client's `index.html`, marked with `<!-- jray-overlay -->`
so it's idempotent. Re-applied whenever configuration changes.
5. **Overlay Script** (`jray-overlay.js`): listens for the player's pause event,
calls `jray?t=`, and renders the on-screen actors.
calls `jray?t=`, and renders the scene's cast.
## Important Notes