jRay/SPEC.md

# jRay — software specification

Status: **alpha.** Core read path ships; the schema bump, the exchange client and
the audio signature do not.

This is a *software* spec: its job is to implement the
[system spec](scripts/vendor/jray-project/SPEC.md), which owns everything spanning more than one repo.
Requirements here trace up to an `SR-nnn` or a `PR-nnn`; the prose below is the
detail. The authoritative ID list with status lives in
[`docs/requirements.md`](docs/requirements.md).

jRay is the Jellyfin plugin: it **consumes** presence data, **displays** it in
the player, and **owns the truth-file format** that the other two components
produce and exchange.

---

## 0. Requirements

IDs are `JR-nnn`, zero-padded and **permanent** — a withdrawn requirement keeps
its number, because renumbering is what produces orphan TRACES tags
([system spec](scripts/vendor/jray-project/SPEC.md) §6).

| Group | IDs | Where addressed |
|---|---|---|
| Truth-file format | JR-001 … JR-007 | §1 |
| Sources and precedence | JR-008 … JR-011 | §2 |
| Read API | JR-012 … JR-014 | §3 |
| Work discovery, policy, coverage | JR-015 … JR-019 | §4 |
| Player overlay | JR-020 … JR-024 | §5 |
| Manifest exchange client | JR-025 … JR-037 | §6 |
| Egress and privacy | JR-038 … JR-041 | §7 |
| Audio signature | JR-042 … JR-045 | §8 |
| Human-in-the-loop association | JR-046 | §9 |

**JR-038 … JR-041 exist because `PR-005` had no software row anywhere.** The
system spec notes that "leak nothing about what the user owns" is preserved
structurally — by SR-004 and GR-005 both being prohibitions — and that a goal
held only by prohibitions needs watching. jRay is the component that actually
opens a socket, so it is the right place for that goal to become checkable.

---

## 1. Truth-file format — JR-001 … JR-007

### JR-001 — This document is normative for the format

The truth file is produced by `scene-actor-extraction`, read by this plugin, and
transformed into a Jmanifest by the exchange client. Three repos touch it, so
exactly one must define it, and the [system spec](scripts/vendor/jray-project/SPEC.md) §1 assigns that to
jRay. Other repos reference this section rather than restating the schema.

The truth file is **not** the Jmanifest. It carries installation-local fields
(`movie`, `jellyfin_id`) that the exchange strips, and lacks the portable
identity block the exchange adds. See §6.

**Current:** [`Models/TruthFile.cs`](Jellyfin.Plugin.JRay/Models/TruthFile.cs),
`schema_version: 2` — the schema below is implemented, with
[`Models/TruthScene.cs`](Jellyfin.Plugin.JRay/Models/TruthScene.cs),
[`Models/TruthExtraction.cs`](Jellyfin.Plugin.JRay/Models/TruthExtraction.cs) and
[`Models/TruthCut.cs`](Jellyfin.Plugin.JRay/Models/TruthCut.cs).

**Gap:** the ownership claim is still not *enforced* — nothing checks that the
other two repos reference this section rather than restating the schema. JR-001's
verification tier is `static` for that reason, and no such check exists yet.

### JR-002 — `schema_version: 2`

```json
{
  "schema_version": 2,
  "movie": "/data/movies/Movie.mkv",
  "extraction": {
    "sample_fps": 5,
    "extinction_sec": 12,
    "gallery_size": 1820,
    "gallery_scope": "global",
    "pipeline_version": "scene-actor-extraction 0.4.1"
  },
  "cut": {
    "runtime_sec": 6420.5,
    "audio_signature": "v1:v7fA3k…"
  },
  "actors": [
    {
      "name": "Steve Buscemi",
      "imdb_id": "nm0000114",
      "tmdb_id": "884",
      "jellyfin_id": "abc123-guid",
      "scenes": [
        { "start": 191.6, "end": 209.2, "belief": 0.98, "route": "live" },
        { "start": 438.2, "end": 465.6, "belief": 0.81, "route": "deferred" }
      ]
    }
  ]
}
```

Field notes:

- `schema_version` — **system-level** (SR-003), incremented once per breaking
  change and referenced by the same number in all three repos.
- `movie` — absolute path at extraction time, informational only. Stripped on
  contribution (JR-034).
- `extraction.*` — provenance. `sample_fps`, `gallery_size` and
  `pipeline_version` move here from the top level so the truth file and the
  Jmanifest's `extraction` block have the same shape, rather than differing for
  no reason.
- `extraction.extinction_sec` — **replaces `anneal_sec`**, which is deleted, not
  retained as a vestigial `0`. It is the re-acquisition timeout that shapes
  window extent, so it is what a consumer needs in order to interpret a window.
- `extraction.gallery_scope` — `"global"` or `"limited"`. The strongest single
  quality signal when two manifests compete for one cut.
- `cut.runtime_sec` — the decoded duration the timings came from. Required for
  contribution; the primary alignment guard.
- `cut.audio_signature` — optional, `v1:`-prefixed. See §8.
- `actors[].scenes[]` — objects, not float pairs. `start`/`end` in seconds,
  inclusive, sorted. `belief` is the accumulated posterior that justified the
  claim; `route` is `"live"`, `"deferred"` or `"pooled"` (extraction AR-017).

**Belief is an attribute, not part of identity.** Two servers that validated the
same upload must agree on its `content_id`, and belief is a producer-side
estimate that may legitimately differ between pipeline versions for identical
timings. It replicates the way `audio_signature` does — see the server spec §9a.

**Current:** implemented. `scenes` are `TruthScene` objects carrying `belief` and
`route`; `extraction.*` and `cut.*` are their own types; `anneal_sec` and the
top-level `sample_fps` are **deleted, not zeroed**. `ManifestConverter` carries
belief, route and both blocks through from a fetched manifest rather than
flattening them. **Gap:** none — this shipped with extraction `IR-002` and the
server's `jmanifest_version: 2` as the one coordinated SR-003 bump.

### JR-003 — Unknown `schema_version` is refused, never guessed

**Decision: flag day.** The plugin accepts `schema_version: 2` and rejects
everything else, on every path — sidecar read, managed `PUT`, and fetched
manifest. There is no transitional dual-accept.

All three components are pre-release and move together, and the alternative
carries a cost that outlasts the transition: a v1 read path is the one nobody
exercises, so it is the one that rots, and it would have to be carried through
every subsequent change to the reader.

**The consequence must be stated plainly rather than discovered:** existing v1
sidecar files on disk **stop being read** at the bump, and stay dark until the
library is re-extracted. The plugin logs this per item, naming the file and the
version found, rather than silently reporting no coverage — an item that looks
un-extracted when it was merely stale is the failure mode that wastes a user's
compute.

**Current:** implemented.
[`Services/TruthSchema.cs`](Jellyfin.Plugin.JRay/Services/TruthSchema.cs) holds
the single `SupportedVersion = 2` and is consulted on **all four** paths — sidecar
read, managed store load, managed `PUT`, and the converter that writes a fetched
manifest. Rejections name the file and the version found, per the paragraph above.

Managed truth is checked on load as well as on `PUT`, because data written by an
earlier plugin build is already on disk and never passed today's `PUT`.

One consequence is worth stating: a v1 file usually fails to *deserialise* before
its version can be read, since `scenes` was a float pair and is now an object. So
both the parse-failure path and the version-gate path must name the file — a
rejection the user cannot attribute to a stale sidecar is the failure mode JR-003
exists to prevent. **Gap:** none.

### JR-004 — A window is a scene-membership claim

**This is SR-002, and it binds this plugin harder than it binds anything else,**
because jRay is where the claim reaches a human.

An actor who turns away, is occluded, or is off-camera while the shot cuts to
whoever they are speaking to **is still present**. Two windows mean a genuine
departure and return, not a break in detection. Gaps shorter than
`extinction_sec` were absorbed upstream and are claimed as presence.

The plugin therefore **never reinterprets, merges, splits, or trims windows.**
It stores and serves what it was given. The one permitted transformation is the
timebase offset of JR-030, which shifts every window uniformly and so preserves
the claim.

**Current:** satisfied.
[`PresenceLookup`](Jellyfin.Plugin.JRay/Services/PresenceLookup.cs) is now the
unit that decides presence, so the semantics live in one tagged place instead of
being implied by a LINQ predicate in the controller. UT-021 pins that adjacent
windows such as `[0,10]` and `[10,20]` are *not* merged, and UT-022 that a truth
file round-trips byte-identical. **Gap:** none.

### JR-005 — Query semantics, and how presence is presented

An actor is present at `t` if any window satisfies `start <= t <= end`.

**The presentation must not assert instantaneous visibility.** SR-002 is explicit
that a consumer must never interpret window boundaries as "the face was detected
here", and the overlay is the exact place that misreading would be made
user-visible. "On screen now" is a claim the data does not support; "in this
scene" is the claim it does.

This is a wording requirement, not a hedge — it is the difference between the
product being right and being a worse version of a frame-by-frame detector.

**Current:** satisfied, in logic and in wording. Bounds are inclusive at both
ends (UT-016/017), a zero-length window is a real sighting rather than a
degenerate one to discard (UT-018), and overlapping windows resolve (UT-019).

The wording was the larger half. The overlay now carries an **"In this scene"**
heading — previously it rendered a bare list, which asserted nothing but also
told the viewer nothing, and a viewer's default reading of a paused frame is
"these people are on screen". The model type `ActorAtTime` became `ActorInScene`,
and [`README.md`](README.md) no longer contains the word "on screen" anywhere;
it stated the forbidden reading outright in seven places, including the opening
sentence. **Gap:** none.

### JR-006 — Numerous windows

SR-002 warns that windows may be numerous and consumers must not assume a handful
of long ones. Track-extent presence with a short `extinction_sec` produces many
short windows per actor, and the previous design's few long ones were an artefact
of the over-claiming that was removed.

The read path must therefore treat per-actor windows as a sorted sequence to be
searched, not a short list to be scanned, and the `jray?t=` response must stay
small regardless of how many windows an actor has.

**Current:** satisfied, and now measured rather than assumed. UT-023 builds 50
actors × 1000 windows and asserts the `jray?t=` result is bounded by **actor**
count, never window count — which is what keeps the response small however finely
presence is sliced.

**The lookup is a full scan, deliberately.** An early exit on `start > t` would
exploit the sortedness the format requires, but it would silently under-report
the moment one producer emitted windows out of order — a correctness risk traded
for a saving that does not register at this scale. UT-020 pins that unsorted
input still resolves. `WindowsAreSorted` exists as a diagnostic for surfacing
such a producer bug, not as something correctness depends on.

**Gap:** none for lookup. The whole truth file is still held in memory per cached
item, which is a memory question rather than a query-cost one and is untouched
here.

### JR-007 — Identity is public identifiers

Each actor carries `imdb_id`, `tmdb_id` and `jellyfin_id`, any of which may be
`""`. Resolution prefers `jellyfin_id` (a Jellyfin Person GUID) when non-empty,
and otherwise matches `imdb_id`/`tmdb_id` against the item's People
`ProviderIds`. Never a name alone — names are ambiguous and unstable (SR-001).

`jellyfin_id` is the exception that proves the rule: it is meaningful only on the
instance that produced it, which is exactly why the exchange strips it (JR-034).

**Current:** implemented. **Gap:** none.

---

## 2. Truth-data sources and precedence — JR-008 … JR-011

### JR-008 — Sidecar discovery

For `Movie.mkv`, the plugin looks for `Movie.jray.json` beside it — suffix
configurable, default `.jray.json`, resolved from the item's media source path.

**Current:** implemented. **Gap:** none.

### JR-009 — Managed truth push

`PUT`/`DELETE .../Truth` let a worker that cannot write beside the media file
deliver results over HTTP. Stored under the plugin's configuration directory,
keyed by item id, independent of the library filesystem.

**Current:** implemented. **Gap:** none.

### JR-010 — Precedence and provenance

There are now **three** sources: a sidecar file, a push from a local worker, and
a manifest fetched from a server. Managed truth — pushed *or* fetched — takes
precedence over a sidecar.

**Fetched manifests are stored through the managed store**, so precedence stays a
two-way rule rather than a three-way one, and the read path does not learn about
the exchange at all.

But the three are no longer interchangeable, so **provenance is recorded with the
stored truth**: which source it came from, and for a fetched one, which server
and at what match tier. A `loose`-tier fetch from a third-party server and a
locally-computed sidecar are not the same claim, and JR-036 requires the
difference be surfaceable.

**Current:** satisfied. Two-way precedence in
[`TruthDataService`](Jellyfin.Plugin.JRay/Services/TruthDataService.cs), and
[`TruthProvenance`](Jellyfin.Plugin.JRay/Models/TruthProvenance.cs) records
source, server, tier, applied offset and caveat. `GET .../Provenance` serves it.

Two decisions worth keeping. **Provenance is stored beside the truth file, never
inside it** — injecting fields would mean the bytes served back are not the bytes
the producer wrote, which is the property JR-004 turns on (UT-026 pins this).
And **the applied offset is recorded** because it is otherwise unrecoverable:
once JR-030 shifts the windows they look native, and nothing would say they had
been shifted.

A sidecar's provenance is derived rather than stored — it is local, and its
timestamp is the file's own. **Gap:** none.

### JR-011 — Caching

Loaded truth is cached in memory for a configurable duration. Any write —
managed `PUT`, `DELETE`, or a stored fetch — invalidates that item's entry
immediately, so a push takes effect without waiting for expiry.

**Current:** implemented. **Gap:** none.

---

## 3. Read API — JR-012 … JR-014

### JR-012 — `GET /Plugins/JRay/Items/{itemId}/Timeline`

Returns the full truth file (§1), or `404` if no truth data exists from any
source.

### JR-013 — `GET /Plugins/JRay/Items/{itemId}/jray?t={seconds}`

Returns an extensible "context at time `t`" envelope, or `404`:

```json
{
  "actors": [
    { "name": "Steve Buscemi", "imdb_id": "nm0000114", "tmdb_id": "884", "jellyfin_id": "abc123-guid" }
  ]
}
```

Future fields (`locations`, `trivia`, and per JR-005 a presence caveat) are added
to this object without changing the route, so **clients must ignore unknown
keys**.

### JR-014 — Authorisation

| Route | Requires |
|---|---|
| `Timeline`, `jray?t=` | Authenticated Jellyfin user token |
| `Truth`, `Tasks/*`, `Policy/*`, `Coverage/*`, and §6's fetch routes | **Administrator** role |
| `ClientScript` | Anonymous — it is injected into a page served before login |

**Current:** all three implemented as stated. **Gap:** none.

---

## 4. Work discovery, policy and coverage — JR-015 … JR-019

### JR-015 — `GET /Plugins/JRay/Tasks/Pending?limit=10`

A **random** sample (default 10, max 100) of movies and episodes with no truth
data:

```json
[ { "item_id": "abc123-guid", "path": "/data/movies/Movie.mkv", "name": "Movie" } ]
```

Randomness is the design: repeated polling spreads work across the backlog
without the server tracking who is working on what, and two workers polling
concurrently mostly do not collide. An empty array means nothing is left, or that
everything remaining is a virtual/missing-path item.

### JR-016, JR-017 — Prioritise / ignore rules

Rules steer the queue: each targets a **genre**, a **series**, or an **item**,
and either prioritises (front of the queue) or ignores (hidden entirely). This is
how an admin says "never extract anime", "this series first", or "skip this one".

Resolution picks the **most specific** match: `Item` > `Series` > `Genre`. A rule
is keyed by scope + value, and setting one replaces any existing rule for the
same key — so a single target can never be simultaneously prioritised and
ignored. Cross-scope conflicts (a prioritised series inside an ignored genre) are
resolved by specificity: the series wins.

```json
{ "scope": "Genre", "value": "Anime", "action": "Ignore", "label": "Anime" }
```

`value` is a genre name, a series id GUID, or an item id GUID; genre matching is
case-insensitive. Persisted to `policy.json` in the plugin's configuration
directory.

**JR-017 is the constraint worth stating separately: rules affect work discovery
only.** They never change the overlay or the read endpoints. An item you ignore
for extraction still shows its overlay if truth data happens to exist — because
the rule expresses "don't spend compute here", not "pretend this doesn't exist".

Endpoints: `GET`/`PUT /Plugins/JRay/Policy/Rules`, and
`DELETE /Plugins/JRay/Policy/Rules?scope=&value=` (idempotent).

### JR-018 — `GET /Plugins/JRay/Coverage`

How much of the library has truth data, overall and by media type and genre:

```json
{
  "total":  { "total": 1200, "covered": 300, "pending": 850, "prioritised": 40, "ignored": 50 },
  "by_media_type": [ { "label": "Film", "counts": { "…": 0 } } ],
  "by_genre":      [ { "label": "Anime", "counts": { "…": 0 } } ]
}
```

`prioritised` is a subset of `pending`. Percent done is
`covered / (total - ignored)` — **ignoring a genre does not drag the percentage
down**, because ignored items are intentionally out of scope, not outstanding
work. An item counts toward every genre it carries, so genre rows overlap and
need not sum to the library total.

### JR-019 — Pickers

`Coverage/Genres`, `Coverage/Series`, and `Coverage/Items?search=&limit=`
populate the rule editor's dropdowns, each returning
`[{ "value": …, "label": … }]`. An absent `search` returns `[]`.

**Current (JR-015 … JR-019):** all implemented. **Gap:** none functionally —
these traced to no requirement until this register existed, which by the gate's
own definition read as scope creep. They serve PR-003 (fully automatic, no
per-title manual work): steering a queue is how automation is directed without
becoming per-title labour.

---

## 5. Player overlay — JR-020 … JR-024

### JR-020 — The overlay

Jellyfin has no plugin hook for player UI, so jRay adds
`<script defer src="/Plugins/JRay/ClientScript"></script>` to the web client's
`index.html`. The script listens for the player's pause event, calls `jray?t=`
for the current item and timestamp, and renders the scene's cast.

Per JR-005 it presents scene membership, not instantaneous visibility.

### JR-021 — jRay never injects into `index.html` on disk

**File Transformation is a hard requirement, not a preference. There is no
on-disk patching fallback.**

The prohibition is on **injection**, not on writing: JR-022's migration must
write to the file in order to remove a legacy patch. Stating it as "never
writes" would put the two requirements in contradiction, and the static check
would have to be disabled to let the migration through — so the check is that no
code path *adds* the script tag.

At startup jRay looks for the
[File Transformation](https://github.com/IAmParadox27/jellyfin-plugin-file-transformation)
assembly via `AssemblyLoadContext` and, if present, calls
`Jellyfin.Plugin.FileTransformation.PluginInterface.RegisterTransformation` by
reflection — no compile-time dependency, so jRay loads normally when it is
absent. The payload:

```json
{
  "id": "2c9b5a41-6ad0-4c1e-9f7d-1d1e6b0d5a90",
  "fileNamePattern": "index.html",
  "callbackAssembly": "<jRay assembly full name>",
  "callbackClass": "Jellyfin.Plugin.JRay.Services.FileTransformationRegistration",
  "callbackMethod": "TransformIndexHtml"
}
```

File Transformation matches `callbackAssembly` against `Assembly.FullName`
exactly, so the full display name is sent. The callback is a public static method
taking a payload with a `contents` string and returning the transformed string;
the payload binds with Newtonsoft, which matches property names
case-insensitively. Registration is unconditional at startup — the callback
itself checks the "enable overlay" setting per request, so toggling takes effect
without re-registering.

Writing to `index.html` is rejected because it is destructive in ways a plugin
cannot clean up after:

- **It outlives the plugin.** Uninstalling jRay leaves the patch in a file jRay
  no longer owns.
- **It breaks on upgrade.** A web-client update replaces the file, discarding the
  patch — or preserves one pointing at an endpoint that has since changed.
- **It collides.** Another plugin patching the same file races with jRay, and the
  loser's edit is lost with no diagnostic.
- **It is a second code path**, and it is the one nobody runs, so it is the one
  that rots.

**Current:** satisfied.
[`WebClientPatchService`](Jellyfin.Plugin.JRay/Services/WebClientPatchService.cs)
is removal-only — the injection capability is *deleted*, not switched off, since
dead code with a live signature is what a later refactor re-enables by accident.
Enforced by
[`scripts/checks/no-index-injection.sh`](scripts/checks/no-index-injection.sh),
which was verified to fail on a reintroduced injection rather than merely to pass
today. [`README.md`](README.md) no longer advertises a fallback. **Gap:** none.

### JR-022 — Migrate away from earlier on-disk patches

Users upgrading from a version that patched the file must not be left with a
stale injection. On startup jRay removes any on-disk patch bearing its own
`<!-- jray-overlay -->` marker — unambiguous, and touching nothing another plugin
added.

**Current:** implemented — `WebClientPatchService.RemoveLegacyPatch` runs at
every startup and is a no-op once the marker is gone. The strip itself is
factored out as `RemoveInjection` so it is unit-testable without a filesystem.
**Gap:** no test executes it yet, so this stays `In Progress` rather than
`Done` — there is no test project in this repo.

### JR-023 — A hard dependency Jellyfin cannot resolve

**Jellyfin has no plugin dependency mechanism.** A manifest cannot declare that
another plugin is required, and nothing will install one. File Transformation
documents only an end-user repository URL and a reflection integration for plugin
authors; there is no NuGet-style dependency to take.

**jRay must not bundle the assembly.** A bundled copy would sit in a different
`AssemblyLoadContext` from the real one — precisely the failure the reflection
integration exists to avoid — on top of licensing and version skew. The
dependency is satisfied by the user installing the real plugin.

So:

1. **Detect at startup and say so** — log a warning naming the plugin and its
   install URL, and disable only the overlay. Every other feature works.
2. **Surface it where it can be acted on** — the configuration page shows
   dependency status: satisfied, or missing with the manifest URL
   `https://www.iamparadox.dev/jellyfin/plugins/manifest.json` and a one-line
   instruction. A warning only in the server log is one nobody reads.
3. **State it as a prerequisite in install docs**, before the jRay install step.

Optionally, publish jRay through a repository manifest that also lists File
Transformation, so one repository URL surfaces both. This is not a dependency
mechanism; it removes a step and the chance of installing the wrong thing.

**Current:** all three implemented, and the detection half is tested
(UT-012…015). Startup detection and registration, a warning naming the plugin
and its install URL, and a status banner on the configuration page fed by
`GET /Plugins/JRay/Status/Dependencies`. The README states the dependency before
the install step rather than after it.

Both failure-path log messages previously said the overlay was "falling back to
patching index.html on disk" — a claim JR-021 made false, and the worst place to
leave one: an admin reading it while debugging a missing overlay would go hunting
for a patch that no longer exists. UT-013 asserts the message names the install
URL and does *not* claim a fallback.

**Gap:** the config-page banner is T4, verifiable only against a live server.

### JR-024 — Names render as text, never markup

Every string that reaches the overlay — actor names above all — is rendered as
text. With §6 the source of those strings may be a third-party server, and the
server spec §5a names this the single most important client-side control,
because it holds even when every other check is bypassed.

It is stated here as a plugin requirement because the server cannot enforce it
and the DOM is jRay's.

**Current:** satisfied. [`Web/jray-overlay.js`](Jellyfin.Plugin.JRay/Web/jray-overlay.js)
uses `textContent` throughout — no `innerHTML`, no `insertAdjacentHTML`. **Gap:**
nothing behavioural. It holds today by construction rather than by rule, which
is what the static check exists to keep true once §6 makes remote strings
reachable.

---

## 6. Manifest exchange client — JR-025 … JR-037

The wire format, tiers and server behaviour are specified in
[`../JRay-public-server/SPEC.md`](../JRay-public-server/SPEC.md). **This section
owns the client half**, which previously lived in that document's §9 — an
inversion, since those are obligations on this repo.

A Jmanifest is this truth file plus a portable identity block and a cut
fingerprint, minus the installation-local fields.

### JR-025, JR-026, JR-037 — Server list and resolution

The plugin queries a configured **ordered list**, not a single URL. Per server:
`Url`, `Name`, `Token` (contribution only), `Enabled`, `AllowContribute`,
`TrustLevel` (`Full` / `FetchOnly`). A community entry ships pre-configured but
**disabled**.

**First acceptable wins** — servers are tried in order, and the first result
clearing the configured tier is taken. Order *is* the user's trust ranking, made
explicit. Best-match-across-all would multiply egress and leak the library to
more parties for a gain the ordering already expresses.

**For a series, first-match applies per episode** (JR-026): fetch the bundle from
server 1, then query server 2 only for what is still missing. Series are commonly
split across sources, and this is where multiple servers earn their keep.

**Failure isolation** (JR-037): an unreachable or failing server is skipped after
a short timeout (5 s connect, 30 s read) and marked failed with exponential
backoff. One dead server must never stall a library sweep; failures surface
per-server in the config page.

**Current:**
[`ManifestServer`](Jellyfin.Plugin.JRay/Configuration/ManifestServer.cs) and
[`PluginConfiguration`](Jellyfin.Plugin.JRay/Configuration/PluginConfiguration.cs)
model all of this. **Gap:** nothing consumes them — there is no HTTP client.

### JR-027 … JR-029 — Every server is untrusted

Everything in the server spec §5a is a property of a *correctly operated* server.
Pointing the plugin at an arbitrary URL inherits none of it. jRay therefore
re-applies client-side what a server applies on upload, **including for the
default server**:

- **JR-027 — validate on receipt.** Downloaded manifests go through the same
  strict schema as uploads: unknown fields rejected, sizes capped, and windows
  bounds-checked against the item's real runtime. A manifest is never trusted
  because a server served it.
- **JR-028 — size caps enforced while streaming**, so an unbounded body is
  aborted rather than buffered. 2 MiB single manifest, 25 MiB bundle.
- **JR-029 — HTTPS required** for non-loopback servers, with certificate
  validation never disabled. A plaintext server would let any intermediary
  rewrite actor overlays.
- **`TrustLevel: FetchOnly`** — the default for user-added servers — accepts
  manifests but never contributes and never sends inventory beyond the single
  item queried.

The honest framing for the config page: *adding a third-party server means
trusting its operator not to serve you deliberately wrong actor data.* The
controls above bound the damage to bad overlay content; they cannot make wrong
data right.

### JR-030 — Offsets are applied before storage

When a match carries a non-zero `offset` (the `audio` tier — §8), the plugin
**must** add it to every scene window before storing.

**The stored truth file is always in the local file's own timebase.** This is
what keeps the offset out of the read path entirely: `Timeline`, `jray?t=` and
the overlay never learn that an offset existed. An offset applied at read time
would have to be applied identically in three places and would be wrong in the
fourth.

### JR-031, JR-032 — Endpoints

Mirroring the existing Truth and Tasks controllers:

| Route | Purpose |
|---|---|
| `POST /Plugins/JRay/Items/{itemId}/Fetch` | Resolve across servers in order; on a match at or above the configured tier, apply JR-030 and store via the managed store |
| `POST /Plugins/JRay/Series/{seriesId}/Fetch` | Bundle fetch with per-episode gap-filling |
| `GET /Plugins/JRay/Servers/Status` | Per-server reachability and last error, for the config page |
| `POST /Plugins/JRay/Items/{itemId}/Identify` | Compute the audio signature and search by content, for items of unknown providence |

**JR-032: `Identify` never stores automatically.** It returns candidate titles
with scores and offsets; storing one is a separate confirmation step. Content
identification is a guess about what a file *is*, and a wrong guess silently
attaches another film's cast to it.

### JR-033 — Scheduled sweep

A scheduled task walks items with no truth data and attempts a fetch, reusing the
`Tasks/Pending` backlog logic — including its policy rules — and the **batch**
`exists` endpoint, so a sweep is a handful of requests per server rather than one
per item.

### JR-034, JR-035 — Contribution

On a `PUT .../Truth` from a local worker, if contribution is enabled: strip
`movie` and `jellyfin_id`, attach identity from the item's `ProviderIds` and its
measured runtime, and `POST` to each contribute-enabled server. For a series,
batch into one bundle upload rather than per-episode posts.

**Stripping is a requirement, not hygiene.** `movie` leaks the contributor's
directory layout and `jellyfin_id` is a GUID from their database — meaningless
elsewhere and mildly identifying. The server rejects both, but the plugin must
not send them in the first place.

**Contribution is never fanned out.** A manifest goes only to servers with
`AllowContribute` set, each an explicit choice.

**JR-035:** uploads set `Expect: 100-continue`, so a server rejecting on size or
auth does so before the body is transmitted. This matters most for bundles, where
a rejected upload would otherwise push tens of MiB pointlessly.

### JR-036 — Match tier is the user's dial

The configured minimum tier (`audio` / `runtime` / `loose`) gates what may be
stored. A `loose` match — runtimes within ±30 s — is plausibly a different trim
of the same cut, so it is **surfaced as a caveat in the UI**, not applied
silently. Per JR-010 the tier is recorded with the stored truth, which is what
makes surfacing it possible after the fetch has finished.

**There is no `exact` tier here, and the plugin sends no `video_hash`.** The
server spec §3 defines `exact` as an equal OpenSubtitles file hash, and it is the
strongest *technical* signal available — it identifies a specific file, so it
cannot produce a false positive. That is exactly why it is withdrawn.

A TMDB id discloses "some copy of this film", which is what a library catalogue
discloses. A file hash discloses **this exact release**, which turns a catalogue
lookup into a release-identification service and turns a server's database into a
mapping from file fingerprints to the instances holding them. That is a far more
specific disclosure than PR-005 permits, and a dataset no volunteer operator
should be asked to hold.

The audio signature is the deliberate replacement: derived from *content*, it
identifies the **cut** rather than the copy, so two different encodes of the same
edit agree. It answers the question the exchange needs — "do these timings apply
to this media?" — without answering the one it must not. `audio` is therefore the
top tier.

A server may still hold hashes contributed by other clients; this plugin simply
never participates, and `MatchTier` has no `Exact` member so no code path can
come to depend on one.

**Current:** `MinimumMatchTier` exists in configuration, defaulting to `runtime`,
and `ManifestExchangeClient` rejects a below-tier match. **Gap:** the caveat is
returned by the fetch endpoint but not yet displayed in the overlay.

---

## 7. Egress and privacy — JR-038 … JR-041

Contribution reveals to a server operator that some instance holds a given title.
Fetching reveals the same. That is inherent to the exchange — which is why the
requirements here bound it rather than claim to remove it.

- **JR-038 — opt-in, off by default.** Manifest sharing, contribution and audio
  signatures are three separate switches, all default off, and the pre-configured
  community server ships **disabled**. No traffic leaves an installation until an
  admin acts.
- **JR-039 — no library-wide inventory in one request.** The batch `exists`
  endpoint is capped at 100 items and sweeps are paced. A single request
  enumerating a library is a fingerprint of it, which is the thing PR-005 exists
  to prevent.
- **JR-040 — the config page says plainly that each configured server multiplies
  the exposure.** First-match resolution limits it — later servers are queried
  only for what earlier ones lacked — and that is worth stating too.
- **JR-041 — the plugin never touches gallery data.** No reference faces, no
  embeddings, fetched or stored or transmitted. There is no such code path and
  there must not be one (SR-005). Verified by static check, mirroring the
  server's UR-012.

**Current:** JR-038 holds — every switch defaults off. JR-041 holds vacuously,
there being no gallery code. **Gap:** JR-039 and JR-040 are unimplemented,
alongside the exchange client itself.

---

## 8. Audio signature — JR-042 … JR-045

A content-derived fingerprint from the centre of a media file, used to identify a
file of unknown providence and to recover the time offset between differently
trimmed releases of the same cut. Construction is specified in
[`../JRay-public-server/SPEC.md` §3](../JRay-public-server/SPEC.md) and must be
implemented **exactly**:

1. Decode a 120 s window centred on the midpoint (`runtime/2 ± 60 s`) — avoiding
   logos and cold opens at the head, credits at the tail.
2. Downmix to mono, resample to 11025 Hz.
3. STFT: 4096-sample frame, 1024-sample hop (~93 ms, ~1290 frames), Hann window.
4. Log-magnitude spectrum over 300–3000 Hz.
5. 32 logarithmically spaced bins; record peak-bin index plus a 2-bit energy
   class.
6. One byte per frame → ~1290-byte array, base64-encoded.

**JR-042 — no new dependency.** FFmpeg performs decode, downmix and resample,
using the binary Jellyfin already ships, reached via `IMediaEncoder.EncoderPath`
from `MediaBrowser.Controller.MediaEncoding`. The plugin implements only a small
fixed FFT and bin-peak extraction.

**JR-043 — bit-exactness is verified, not assumed.** The pipeline computes this
signature too (extraction `IR-004`), deliberately: files never processed locally
still get one from the plugin. Two independent implementations of one fingerprint
are only useful if they agree exactly, so a **golden-vector fixture is shared
between the two repos** — a short WAV and its expected signature, committed in
both. It is CPU-only DSP, which is why this cross-repo check can be a binding CI
test rather than an aspiration. Extraction's counterpart is `IR-005`.

**JR-044 — media shorter than 120 s.** The window underflows, so **no signature
is emitted and no sync offset is applied**. Such items fall back to the runtime
tier, which is adequate: a 90-second extra is not content whose cut alignment
matters. Both producers must apply the identical rule, or they diverge
on exactly the short items most likely to be misidentified. Extraction's
counterpart is `IR-007`.

**JR-045 — the signature carries its own `v1:` prefix**, separate from
`schema_version`. Emit and honour it, so a future change to the DSP chain is
*detectable* rather than silently producing non-matching signatures. Extraction's
counterpart is `IR-008`.

Matching — sliding ±600 frames (≈±56 s), scoring the fraction of overlapping
frames whose peak bin matches — is a **consumer** concern and belongs to this
plugin. Offsets are applied client-side per JR-030; manifests are never
rewritten.

**Current:** JR-042 and JR-043 hold. `AudioSignature` implements the construction
above — band table, periodic Hann, radix-2 FFT, band-mean peak, energy class,
packing — and `AudioSignatureService` decodes the centre window by running the
FFmpeg binary `IMediaEncoder.EncoderPath` names, so the plugin gained no
dependency. `fixtures/audio/` holds the extraction repo's three golden-vector
files byte-identically, and the computed signature equals the recorded vector
exactly (UT-038 … UT-044). The binding check regenerates the fixture PCM from
`make_fixture.py`'s arithmetic and verifies it against the recorded decode
checksums, so it runs on a CI host with no codec at all; the two tests that
exercise the real FFmpeg decode self-skip without a binary.

JR-044 and JR-045 hold as well, both halves. `AudioSignatureMatcher` is the
reader they were waiting on: it implements the ±600-frame slide above, scoring
the fraction of overlapping frames whose peak band agrees, and returns the tier
and the offset. `TryParseFrames` refuses any prefix but `v1:`, so a future
producer's `v2:` drops the item to the runtime tier rather than being scored as
if it were understood; and a runtime under 120 s yields no match and therefore no
offset, read off the runtime rather than inferred from a missing string, because
the runtime is what both producers test (UT-045 … UT-052).

The offset has **two terms**, which is easy to miss: the recovered slide, and the
difference between where the two windows are anchored. Both windows are centred
on their own file's midpoint, so unequal runtimes start them at different
absolute times. A release carrying 40 s of extra head material recovers 20 s from
each term.

One parameter is **not** from the specification: an alignment must overlap by at
least 64 frames (~6 s) before its score counts. Without a floor the extreme
offsets compare a handful of frames, where a chance agreement scores 1.0 and
beats the true alignment. It never binds on the real case — two full-length
signatures still overlap by 688 frames at the widest offset.

JR-047 connects it. `ManifestAligner` runs on the fetch path, before anything is
stored: it computes the local file's signature, compares it against the one the
manifest carries, and **applies the local offset in preference to the server's**.
The server has never seen this file — its offset is a runtime-difference
inference at best — and the comparison needs no round trip, so no signature
leaves the instance. `ComputeAudioSignatures` now gates that decode.

Everything that can go wrong degrades to the server's offset rather than
refusing: signatures off, no manifest signature, media under the window, a `v2:`
producer, a missing binary, a decode error. A signature is an enhancement to cut
matching and must never be able to break a fetch. A *genuine* disagreement —
both signatures valid, both items long enough, best alignment still under 0.60 —
is recorded and surfaced as a caveat that outranks the tier's own, since "the
audio does not match" is a stronger statement than "the runtimes differ"; the
manifest is still stored, because the audio may legitimately differ.

"Un-comparable" and "does not match" are kept distinct. A 90-second extra is not
content disagreeing with its manifest, and reporting it as such would be worse
than saying nothing.

The applied offset, the score, the recovered slide, and the local file's own
signature are written beside the truth file (`TruthAlignment`) — the offset is
otherwise unrecoverable once the windows are shifted, and the stored signature
lets a later fetch align without decoding again.

See [`docs/audio-alignment.md`](docs/audio-alignment.md) for the mechanism end to
end.

**Gap:** contribution does not yet attach a signature (JR-034), and the stored
`local_signature` is written but not read back, so today every fetch decodes.
Server-side catalogue matching — `POST /manifests/search` and the `audio` tier —
is the server's UR-009 and is deliberately sequenced after signature coverage
accumulates; nothing here depends on it.

---

## 9. Human-in-the-loop association — JR-046

*Proposed. See [system spec](scripts/vendor/jray-project/SPEC.md) §4, which owns the design.*

The pipeline produces **unidentified tracks** — a face that is genuinely someone,
sustained across many frames, that the gallery cannot name. A user watching the
film usually knows exactly who it is. jRay's contribution is the review UI: show
a cluster's context crops, let the user pick from the title's cast or search
TMDB, and record the association for extraction to ingest into the local gallery.

**The unit of review is a person, not a track.** Unknown tracks are clustered
upstream (`AR-021`), so the question is "who is this person, who appears in these
twelve places?" rather than twelve disconnected questions. One answer resolves
the cluster.

Deliberately left as a single `TBD` row rather than decomposed. It depends on
extraction `AR-021`/`AR-022` landing, and on **system open question 2** — whether
unidentified presence is published in the truth file at all, which determines
whether this UI's work queue arrives with the truth data or needs a separate
channel. Decomposing now would fix an interface against an undecided upstream.

---

## 10. Client: pushing results from a remote worker

Reference material for worker authors; the requirements are JR-009 and JR-015.

**1. Authenticate.** Create an Administrator API key (Dashboard → API Keys) and
send it as `X-Emby-Token: <key>` or
`Authorization: MediaBrowser Token="<key>"`.

**2. Find work.** Poll `GET /Plugins/JRay/Tasks/Pending?limit=10` rather than
walking the library and checking each item.

**3. Resolve the item id.** The push endpoint is keyed by Jellyfin item GUID, not
path:

```
GET /Items?Recursive=true&Fields=Path&IncludeItemTypes=Movie,Episode
```

Match `Path` against the file you processed — which requires the worker to see
the file at the *same path* Jellyfin does; translate first if it mounts the
library elsewhere. The mapping is stable until the file moves, so cache
`path -> itemId` and re-resolve only on a miss.

**4. Push.** `PUT /Plugins/JRay/Items/{itemId}/Truth` with the truth file body.
`204` stored (cache invalidated immediately), `400` unsupported
`schema_version`, `401`/`403` key missing or not an administrator. The `PUT` is
idempotent, so retrying on a network error is safe.

**5. Optionally remove.** `DELETE /Plugins/JRay/Items/{itemId}/Truth` always
returns `204`; the item falls back to its sidecar on the next read.

---

## 11. Open questions

1. **`sample_fps`, `gallery_size` and `pipeline_version` move under
   `extraction.*` in JR-002.** This aligns the truth file with the Jmanifest's
   block of the same name, and the bump is breaking regardless. It is a change
   this spec proposes rather than one inherited from SR-003's list — confirm, or
   keep them top-level.
2. **Does `route` belong in the `jray?t=` envelope?** JR-013 says the response is
   extensible and JR-005 says presentation must not over-claim. Exposing belief
   and route would let the overlay caveat a weak claim, but invites a UI that
   shows a number to a viewer who cannot act on it.
3. **System open question 2 — unidentified presence.** If published, the overlay
   could show "unidentified person" and JR-046 gets its queue from the truth file
   directly. jRay is the consumer that would have to display it, so this repo has
   a position to state.
4. **Test-ID namespacing.** `UT-nnn`/`IT-nnn` are per-component registers, so
   `UT-001` will exist in both this repo and `scene-actor-extraction`. Fine while
   the gate runs per repo; ambiguous the moment a rollup spans them.