Initial implementation: core vertical slice
Implements the core of SPEC.md — the manifest exchange, less audio-tier matching (§3) and federation (§9a), both of which the spec sequences as later work. - §2 Jmanifest format and series bundles - §3 cut matching: exact / runtime / loose tiers - §4 API, less POST /manifests/search - §5 rate limiting; §5a trust model, anonymous bearer tokens - §6 upload validation, all four stages - §7 relational storage, no JSON blob on the write path - §8 Rust + Axum + SQLite, single serialized writer, in-process job queue - §9a content addressing, computed on upload Reconciled against the system spec: - anneal_sec removed, withdrawn upstream by AR-012/AR-013. Presence follows track extent, so a track survives its own gaps and there is nothing to anneal. Its successor extinction_sec and the new gallery_scope are accepted and stored; scope enters the §7 ranking. A manifest still carrying anneal_sec is a hard 400, not silently ignored — it came from a pipeline whose window semantics differ from what this server assumes. - Audio signature: media under 120 s now emits no signature at all, matching scene-actor-extraction IR-007. The earlier §3 draft allowed a shortened window under 150 s, which was the weaker rule — a caller-varying length is the property SR-004 forbids. - UR IDs regularised to UR-nnn; docs/requirements.md registers 32 requirements, each tracing to an SR-nnn or PR-nnn. 189 tests: unit, end-to-end through the real router, and an injection suite covering SQL, JSON, header and Unicode payloads. Writing that suite found two real gaps, both fixed here: compatibility homoglyphs passed the §5a character class, and a one-frame audio signature was accepted on a feature-length item. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,220 @@
|
||||
# JRay public server
|
||||
|
||||
A community manifest exchange for JRay. Jellyfin servers running the JRay plugin
|
||||
pull actor-timeline manifests ("Jmanifests") for titles they own instead of
|
||||
running the CV pipeline locally, and optionally contribute the manifests they
|
||||
generate back.
|
||||
|
||||
See [SPEC.md](SPEC.md) for the design. Section references throughout the code
|
||||
point at it.
|
||||
|
||||
The community instance is **`https://jray.tourolle.paris`**. The JRay plugin
|
||||
ships with it pre-configured but **disabled** — §9 requires that no traffic leave
|
||||
an installation until an admin opts in, so the default entry exists to save the
|
||||
admin from typing a URL, not to enable sharing on their behalf. Set
|
||||
`JRAY_SERVER_ID=jray.tourolle.paris` when deploying that instance: it becomes the
|
||||
`origin` stamped on manifests it first accepts (§9a) and the salt for report IP
|
||||
hashes.
|
||||
|
||||
## Status
|
||||
|
||||
First implementation pass: the **core vertical slice**, reconciled against the
|
||||
[system spec](../SPEC.md).
|
||||
|
||||
Per-requirement status is in [`docs/requirements.md`](docs/requirements.md) —
|
||||
32 requirements (`UR-001..018`, `DR-001..014`), each tracing up to an `SR-nnn`
|
||||
or `PR-nnn`. `UR-015..018` are the pending SR-003 schema bump and are marked
|
||||
`Planned` rather than omitted.
|
||||
|
||||
Implemented:
|
||||
|
||||
- §2 Jmanifest format and series bundles
|
||||
- §3 cut matching — `exact` / `runtime` / `loose` tiers
|
||||
- §4 the API surface, less `POST /manifests/search`
|
||||
- §5 rate limiting, in-process fixed-window counters
|
||||
- §5a trust model — anonymous bearer tokens, closed-vocabulary storage,
|
||||
automatic revocation
|
||||
- §6 upload validation, all four stages
|
||||
- §7 relational storage, no JSON blobs on the write path
|
||||
- §8 Rust + Axum + SQLite, single serialized writer, in-process job queue
|
||||
- §9a content addressing (`content_id`), computed on upload
|
||||
|
||||
Reconciled with the system spec (see `docs/requirements.md` for the detail):
|
||||
|
||||
- **`anneal_sec` removed.** Withdrawn upstream by AR-012/AR-013 — presence now
|
||||
follows track extent, so a track survives its own gaps and there is nothing to
|
||||
anneal. Its successor `extinction_sec` and the new `gallery_scope` are
|
||||
accepted, stored, and (for scope) used in §7 ranking. A manifest still
|
||||
carrying `anneal_sec` is now a hard `400`, not silently ignored: it was
|
||||
produced by a pipeline whose window semantics differ from what this server
|
||||
assumes.
|
||||
- **Audio signature: the 120 s rule now matches both producers.** An earlier
|
||||
draft of §3 allowed a shortened window for items under 150 s; that conflicted
|
||||
with `scene-actor-extraction` IR-007 and was the weaker rule, since a
|
||||
caller-varying length is the property SR-004 forbids. Items under 120 s now
|
||||
send no signature at all.
|
||||
|
||||
Deferred:
|
||||
|
||||
- §3 audio signatures — the field is **accepted, validated and stored**, and
|
||||
`content_id` already excludes it, but `audio`-tier matching and
|
||||
`POST /manifests/search` are not wired up. This follows §3's own recommended
|
||||
sequencing: ship the plugin-side computation first, let signatures accumulate,
|
||||
then enable matching once coverage is useful.
|
||||
- §9a federation endpoints (`/federation/*`) and the pull worker. The schema
|
||||
columns (`content_id`, `origin`, `ingested_from`, `peers`) are in place, and
|
||||
`ingest::persist` is already the shared path a pull would reuse.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
cargo run
|
||||
```
|
||||
|
||||
Configuration is entirely environment variables:
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `JRAY_BIND` | `127.0.0.1:8080` | Listen address. Terminate TLS at the operator's proxy (§8) |
|
||||
| `JRAY_DB` | `jray.db` | SQLite path. WAL mode, created on first run |
|
||||
| `JRAY_TMDB_API_KEY` | — | **Hard dependency for UR-3.** Without it, uploads stay `pending` and are never listed |
|
||||
| `JRAY_TMDB_BASE_URL` | `https://api.themoviedb.org/3` | Override for testing |
|
||||
| `JRAY_TRUSTED_PROXIES` | — | Comma-separated proxy IPs whose `X-Forwarded-For` is honoured. **Not default-on**: §5 rate limiting and report attribution key on client IP, so a spoofable header defeats both |
|
||||
| `JRAY_SERVER_ID` | `localhost` | This server's identity, used as manifest `origin` and as the report IP-hash salt |
|
||||
| `JRAY_REQUEST_TIMEOUT_SEC` | `30` | Request timeout so a slow bundle query fails fast |
|
||||
| `JRAY_JOB_BATCH` | `8` | Cast-check jobs leased per worker tick |
|
||||
| `JRAY_JOB_POLL_SEC` | `5` | Worker poll interval |
|
||||
| `JRAY_LOG` | `info` | `tracing` filter |
|
||||
|
||||
Contributing requires a token (§5a) — an anonymous bearer capability, not an
|
||||
account. Self-issue one:
|
||||
|
||||
```sh
|
||||
curl -sX POST -H 'content-type: application/json' -d '{}' \
|
||||
http://127.0.0.1:8080/api/v1/tokens
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
§8's deployment notes are requirements, not suggestions:
|
||||
|
||||
- Enforce the body cap at **both** the proxy and the app. `client_max_body_size`
|
||||
(nginx) / `request_body max_size` (Caddy) should match §6 stage 1, so oversized
|
||||
uploads are dropped at the edge and never occupy an application worker. The app
|
||||
must also be safe when run without a proxy, which it is.
|
||||
- Set `JRAY_TRUSTED_PROXIES` to the proxy's address, or `X-Forwarded-For` is
|
||||
ignored and every client behind it shares one rate-limit bucket.
|
||||
- **Back up the SQLite file with `VACUUM INTO` or the backup API** — never a
|
||||
plain file copy of a live WAL database. Manifests represent real CV compute.
|
||||
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
cargo test # 178 tests
|
||||
cargo deny check # advisories, licences, bans, sources
|
||||
```
|
||||
|
||||
Unit tests per module, plus two integration suites:
|
||||
|
||||
- `tests/api.rs` — end-to-end through the real router: status codes, headers, and
|
||||
the properties that only hold if the layers compose correctly (per-route body
|
||||
caps, rate-limit surfaces, the strict schema actually reaching uploads).
|
||||
- `tests/injection.rs` — that hostile input cannot escape its layer: SQL payloads
|
||||
in query parameters, path segments, JSON bodies, bearer tokens and report notes;
|
||||
JSON structure abuse; CRLF header injection; path traversal; and Unicode tricks
|
||||
against the §5a character class.
|
||||
|
||||
Security-relevant properties are asserted rather than assumed —
|
||||
`movie`/`jellyfin_id` rejection, the §5a character class defeating base64/hex
|
||||
smuggling, per-route body caps, a lying `Content-Length` not bypassing the cap,
|
||||
and a forged `X-Forwarded-For` not resetting a rate-limit budget.
|
||||
|
||||
**On injection specifically.** Two independent defences apply, and they fail
|
||||
differently, so both are tested:
|
||||
|
||||
1. **Parameterised queries.** Every value reaches SQLite through `params![]`. The
|
||||
only `format!`-built SQL interpolates two compile-time constants (a column list
|
||||
and a status literal) — no runtime input ever becomes SQL syntax. This is what
|
||||
actually prevents injection.
|
||||
2. **Closed-vocabulary validation.** Identifiers are regex-constrained and free
|
||||
text is limited to a closed character class, so most payloads never reach the
|
||||
query layer at all.
|
||||
|
||||
The injection suite would still pass on defence 1 alone, which is deliberate: if
|
||||
validation were ever loosened, the tests should not silently start depending on it.
|
||||
|
||||
Writing that suite found two real gaps, both since fixed:
|
||||
|
||||
- Compatibility homoglyphs (`𝐒𝐭𝐞𝐯𝐞`, `Actor`) passed the §5a class. They are
|
||||
letters by Unicode category and NFC does not fold them — only NFKC would. Beyond
|
||||
name spoofing, a fullwidth-digit alphabet would have reopened the encoding
|
||||
channel the "no digits" rule exists to close.
|
||||
- A one-frame `audio_signature` was accepted on a feature-length manifest, making
|
||||
the field the variable-length container §3 explicitly forbids. The length floor
|
||||
is now derived from the declared runtime, keeping §3's genuine short-item
|
||||
exception without trusting the client's length.
|
||||
|
||||
Two tests worth knowing about:
|
||||
|
||||
- `content_id::tests::golden_vector_hash_is_stable` locks the §9a canonical form.
|
||||
**The JRay plugin must reproduce it byte-identically**; §8 notes the extraction
|
||||
side is Python, so this can no longer be one shared implementation and must be
|
||||
cross-tested instead. `GOLDEN_VECTORS` is that fixture, and its hash was
|
||||
verified against an independent Python implementation.
|
||||
- The integration tests use an on-disk temporary database, not `:memory:`,
|
||||
because §8's topology is one writer connection plus a read pool — in-memory
|
||||
SQLite is per-connection, so the readers would see an empty database.
|
||||
|
||||
## CI and container image
|
||||
|
||||
`.gitea/workflows/ci.yml` runs on Gitea Actions (and unmodified on GitHub
|
||||
Actions), gated fastest-first: `fmt` → `clippy` → `test` → `cargo deny` → static
|
||||
musl build. The dependency audit also runs weekly, since advisories appear without
|
||||
any code changing.
|
||||
|
||||
The `Dockerfile` is the optional convenience, not the intended deployment path —
|
||||
§8 is explicit that neither Docker nor Compose should be *required*. It builds
|
||||
against musl and runs from `scratch` as uid 65534: **8.7 MB**, no libc, no shell,
|
||||
no package manager. `rusqlite` bundles SQLite and `reqwest` uses rustls, so there
|
||||
is nothing left to link.
|
||||
|
||||
```sh
|
||||
docker build -t jray-server .
|
||||
docker run --rm -p 8080:8080 -v jray-data:/data -e JRAY_TMDB_API_KEY=... jray-server
|
||||
```
|
||||
|
||||
Two things that are easy to get wrong, so they are handled explicitly:
|
||||
|
||||
- **Static linkage is asserted with `file`, not `ldd`.** The musl target produces
|
||||
a *static-PIE*, and `ldd` prints the musl loader for one — an `ldd`-based check
|
||||
reports a perfectly static binary as dynamic.
|
||||
- **`/data` is staged with the runtime uid's ownership.** Docker seeds a named
|
||||
volume from the image directory, ownership included, so a non-root server can
|
||||
create the database on first run. A **bind mount is not seeded this way** — the
|
||||
host directory keeps its own ownership, so `chown 65534:65534` it first or the
|
||||
server exits with "unable to open database file".
|
||||
|
||||
Two tests are worth knowing about:
|
||||
|
||||
- `content_id::tests::golden_vector_hash_is_stable` locks the §9a canonical form.
|
||||
**The JRay plugin must reproduce it byte-identically**; §8 notes the extraction
|
||||
side is Python, so this can no longer be one shared implementation and must be
|
||||
cross-tested instead. `GOLDEN_VECTORS` is that fixture, and its hash was
|
||||
verified against an independent Python implementation.
|
||||
- The integration tests use an on-disk temporary database, not `:memory:`,
|
||||
because §8's topology is one writer connection plus a read pool — in-memory
|
||||
SQLite is per-connection, so the readers would see an empty database.
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **The §6 stage-3 thresholds are still §10's guesses.** §10 (5) is explicit that
|
||||
running the check over the 331 real corpus files would give the true
|
||||
distribution of honest-upload match ratios, and is "the single cheapest way to
|
||||
de-risk UR-3 and UR-5". The scoring logic is deliberately pure functions in
|
||||
`castcheck.rs` so that retuning is a test-data exercise, not a code change.
|
||||
- `POST /manifests/{id}/report` records reports but nothing consumes them yet.
|
||||
§5a's divergence detection and the operator kill switch are not implemented;
|
||||
delisting is currently a manual `UPDATE`, which §5a does note is the intended
|
||||
shape ("one UPDATE", not a moderation queue).
|
||||
- No admin surface. Revocation is automatic (§5a), but an operator has no
|
||||
endpoint for the deliberate kill-switch case.
|
||||
Reference in New Issue
Block a user