dtourolle/JRay-public-server
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(audio): store, serve and match the v1 audio signature

Browse Source 
Completes UR-009. The register recorded the signature as stored; it was
not. `ingest` validated `cut.audio_signature` and wrote NULL, so every
served manifest came back without one — which also meant the plugin's own
alignment (jRay JR-047, Done) had nothing to align against and could never
run. That is the failure mode a status field is least able to catch: every
validation test passed and the feature delivered nothing.

Now stored, coarse-indexed and served back byte-identically, with a held
manifest adopting an incoming signature it lacked (§9a). `audio`-tier
matching runs on every read endpoint via an `audio_signature` parameter,
and `POST /manifests/search` answers the unknown-providence case with a
runtime prefilter, a bounded scan and honest truncation reporting.

Three rules §3 did not previously state, now normative:

- **±1 frame of slack in the score.** scene-actor-extraction VR-014
  measured the exact-frame rule demoting 27 of 40 correctly aligned
  releases to `loose`, because the two windows are cut on their own
  file's frame grid and those grids do not coincide. With ±1 frame all
  40 reach `audio` (worst 0.906) and the strongest false match is
  unmoved at 0.16.
- **The offset has two terms.** Both windows are anchored at their own
  file's runtime/2, so the slide alone is wrong by half the runtime
  difference on every shifted release. A signature without a runtime
  therefore cannot align, and is refused by name rather than answered
  at a lower tier.
- **A signature verdict is final**, including its refusals. Falling back
  to the runtime tier after the audio declined would let a coincidence
  overturn direct evidence, inverting the ordering the tier table exists
  to state.

The slide precomputes each frame's neighbourhood as a 32-bit bin set
rather than re-deriving it across 1201 slides — 3.3 ms to 1.1 ms per
candidate, with a test asserting exact equivalence to the rule written
the obvious way. The 1000-candidate search cap follows from that
measurement as a ~1.1 s ceiling per request, not a round number.

jRay's matcher still implements the pre-slack rule and will label some
alignments `loose` that this server calls `audio`. Nothing misaligns —
JR-047 makes the local answer win — but that register now carries the
follow-up.

TRACES: UR-008, UR-009 | SR-003
This commit is contained in:
dtourolle
2026-07-31 22:43:26 +02:00
parent 7eb5c175af
commit c41253ef5c
24 changed files with 2288 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SPEC.md
+146 -14
View File
@@ -459,9 +459,10 @@ Two signatures are compared by sliding one against the other and taking the
best score:
```
for offset in -600 .. +600 frames: # ±56 s
score(offset) = fraction of overlapping frames whose peak bin matches
best = argmax score
for slide in -600 .. +600 frames: # ±56 s
score(slide) = fraction of overlapping frames whose peak bin matches
the reference within ±1 frame
best = argmax score # smallest |slide| wins a tie
```
| Result | Interpretation |
@@ -483,6 +484,72 @@ Speed-differing releases (PAL 4% speed-up) are **not** handled by a constant
offset and are correctly rejected by the score threshold; a scale-and-offset
search is possible later but is out of scope.
##### Normative v1 matching parameters
As with the construction, prose alone leaves choices that decide whether two
implementations agree. These are pinned for v1:
| Parameter | v1 value |
|---|---|
| Slide range | ±600 frames, inclusive |
| Score tolerance | A frame agrees if the reference carries the same peak bin within **±1 frame** |
| Tie-break | Smallest \|slide\| |
| Minimum overlap | **64 frames** — an alignment thinner than this does not score at all |
| Field compared | Peak bin only; the 2-bit energy class is not scored |
**The ±1 frame tolerance is a correction, not a loosening.** The rule was an
exact frame match until `scene-actor-extraction` VR-014 measured it on real
film audio: over 40 random in-cap offsets every alignment was recovered to the
nearest frame, but **27 of the 40 scored below 0.85 and were demoted to
`loose`** — not because the content disagreed but because the two windows are
cut on their own file's frame grid and those grids do not coincide. The exact
rule therefore scored how nearly two grids happened to line up. With ±1 frame
all 40 reach `audio` (worst 0.906), the strongest false match is unmoved at
0.16, and the offset costs 81 ms of a 500 ms budget. The gap that makes the
thresholds mean anything is untouched; what changed is that a correctly aligned
release now clears them.
**Minimum overlap is not from the construction.** Without a floor the extreme
slides compare a handful of frames, where a chance agreement scores 1.0 and
beats the true alignment. It never binds on the real case — two full-length
signatures still overlap by 688 frames at ±600 — but two implementations that
chose different floors would disagree at the edges, so it is fixed here.
##### The offset has two terms
This is the easiest thing in the feature to get wrong, and the pseudocode above
gives only half of it. Both signatures are cut from **their own file's centre**,
`runtime/2 ± 60 s`, so when the runtimes differ the two windows do not begin at
the same point in the content:
```
offset = (local_runtime - manifest_runtime) / 2 - slide × 1024/11025
└──── window-anchor difference ────┘ └───── recovered ─────┘
```
A release carrying 40 s of extra head material takes 20 s from each term. Using
the slide alone is wrong by half the runtime difference on **every** shifted
release — which is every release this feature exists for.
It follows that **a signature alone cannot produce an alignment**: the anchor
term needs both runtimes. A caller that supplies a signature without a runtime
is refused by name rather than answered at a lower tier, which would look like a
match its own signature had failed to improve.
##### A signature verdict is final
When both sides carry a signature, the audio comparison decides — including
when it declines. There is no fallback to the runtime tiers after a `score <
0.60`, because `audio` outranks `runtime` precisely for being content-derived,
and letting a runtime coincidence overturn direct evidence would invert the
ordering the tier table states. The separation measured in VR-014 — true
matches at 0.906 and above, the strongest false one at 0.16 — is what makes
that safe rather than brave.
A signature on only one side is a different case entirely and falls through to
the runtime tiers, since coverage accumulates gradually and most stored
manifests carry none.
#### Revised tier table
| Tier | Signal | Confidence |
@@ -503,25 +570,57 @@ With no TMDB id at all, a client can search by signature alone:
```
POST /manifests/search
{ "audio_signature": "base64…", "runtime_sec": 6420.5 }
{ "audio_signature": "v1:…", "runtime_sec": 6420.5 }
```
Both fields are required. `runtime_sec` drives the prefilter and supplies the
offset's window-anchor term, so a search without it could rank candidates but
could not align to them.
The server returns candidate matches with scores, offsets and title identity —
letting JRay identify an unidentified file *and* align to it in one step.
letting JRay identify an unidentified file *and* align to it in one step:
```json
{
"results": [
{ "manifest_id": "01HZ...", "match": "audio", "score": 0.97, "offset_sec": -20.0,
"identity": { "type": "movie", "tmdb_id": "504172",
"title": "Road to Bali", "year": 1952 } }
],
"candidates_scored": 412,
"truncated": false
}
```
Results are best-first and capped at 10. Title identity is the half of the
answer `GET /manifests/exists` cannot give, since a caller with no id has
nothing to probe it with — and it stays cut-level like every other tier, so it
says which *work* this is and never which copy.
**This endpoint is a scaling problem, not a correctness one.** A naive
implementation compares against every stored signature. Mitigations:
- Prefilter by runtime (±90 s) before scoring, which eliminates almost
everything.
- Index a **coarse hash** of the signature (e.g. the peak-bin sequence of
every 16th frame) for candidate generation, with full sliding comparison
only on candidates.
everything. Necessarily wider than the ±30 s `loose` tier: here a large
runtime difference is the premise rather than a disqualification.
- Index a **coarse hash** of the signature — the peak-bin sequence of every
16th frame — for candidate generation, with full sliding comparison only on
candidates. Subsampling destroys the alignment for any non-zero slide, so
the coarse key settles the *re-encode* case cheaply and orders the scan; the
runtime prefilter is what carries the shifted releases.
- Bound the scan. A full slide costs ~1.1 ms per candidate, so this server
scores at most 1000 of them — a ~1.1 s ceiling on one request — and reports
`candidates_scored` alongside `truncated`. A truncated scan presented as a
complete one turns "no match" into a claim the server cannot support, and
truncation is the signal that the corpus has outgrown a linear scan.
- Rate-limit hard (§5): this is the most expensive read endpoint and the most
attractive to abuse.
Because it is expensive, `POST /manifests/search` is **optional for a server
to implement**; `GET /federation/capabilities` advertises support.
to implement**; `GET /federation/capabilities` advertises support, alongside
`audio_tier_matching` for the read endpoints, which is not optional — matching
costs one slide against the candidates a title already narrows to, which is
nothing like a corpus-wide search.
#### Validation and abuse
@@ -559,6 +658,12 @@ plugin-side computation first, let signatures accumulate, then enable
`audio`-tier matching and the search endpoint once coverage is useful. Nothing
above needs to land at once.
That sequence is now complete: the plugin computes signatures (`jRay` JR-042),
the server stores and serves them, and both `audio`-tier matching and
`POST /manifests/search` are live. The optionality survives it — a manifest
without a signature is matched by the runtime tiers exactly as before, and
**nothing here may be allowed to break a fetch** (UR-009 is an enhancement).
#### Computing the signature in the JRay plugin
The plugin is the right place for this, and it is the *only* place that covers
@@ -670,7 +775,12 @@ title *and* at what cut-match tier, without transferring the payload.
Query parameters are the same identity + cut parameters as the fetch
endpoints: `tmdb_id` / `imdb_id` (or `series_tmdb_id` + `season` + `episode`),
plus optional `runtime_sec`.
plus optional `runtime_sec` and `audio_signature`.
`audio_signature` is what lets the `audio` tier fire here at all. This endpoint
transfers no payload, so a client cannot align locally from its answer — the
server-side comparison is the only way a sweep can learn that a manifest is not
merely present but *aligned*. It requires `runtime_sec` alongside it (§3).
```json
{ "exists": true, "match": "runtime", "manifest_id": "01HZ...", "actor_count": 34 }
@@ -701,24 +811,46 @@ returning results positionally. This exists specifically so the rate limit in
because the payload does not fit a query string; it is a read and requires no
token.
### `GET /manifests/movie?tmdb_id=&imdb_id=&runtime_sec=`
### `GET /manifests/movie?tmdb_id=&imdb_id=&runtime_sec=&audio_signature=`
Returns the best-matching Jmanifest, or `404` if none clears `loose`.
```json
{ "match": "runtime", "manifest": { "...": "..." } }
{ "match": "runtime", "offset_sec": 0.0, "manifest": { "...": "..." } }
```
`offset_sec` is always present and is non-zero only on the `audio` route — the
runtime tiers know that two cuts are close, never by how much they are
displaced. The served manifest carries `cut.audio_signature` when the server
holds one, which is what a client aligning locally compares against.
`audio_signature` rides in the query string: ~2.3 KB encoded, comfortably inside
any request-line limit. Unlike the withdrawn `video_hash` it is not a file-level
signal, so accepting it on a read does not make the read an oracle for which
*copy* a caller holds (§3, PR-005).
### `GET /manifests/series/{series_tmdb_id}?season=`
Returns a series bundle (§2). `season` optional; omitted means all seasons.
Episode-level cut matching is done client-side against the returned bundle,
since a client pulling a whole series already knows its own runtimes.
### `GET /manifests/episode?series_tmdb_id=&season=&episode=&runtime_sec=`
### `GET /manifests/episode?series_tmdb_id=&season=&episode=&runtime_sec=&audio_signature=`
Single-episode equivalent of the movie endpoint.
### `POST /manifests/search` — UR-009
Identify a file of unknown providence by its audio signature, and align to it.
See §3 "Unknown-providence search" for the body, the response and the cost.
- `200` — results, possibly empty. An empty list is an answer, not an error
- `400` — the signature is malformed, or fails the same structural rules an
upload would (§6)
- `404` — this server does not offer search; `GET /federation/capabilities`
says so up front
- `429` — rate limited (§5)
### `POST /manifests`
Contribute a manifest. Body is a Jmanifest. Requires an API token (§5).