Initial implementation: core vertical slice
CI / fmt, clippy, test (push) Failing after 2m46s
CI / static musl binary (push) Has been skipped
CI / advisories and licences (push) Successful in 4m22s

Implements the core of SPEC.md — the manifest exchange, less audio-tier
matching (§3) and federation (§9a), both of which the spec sequences as
later work.

- §2 Jmanifest format and series bundles
- §3 cut matching: exact / runtime / loose tiers
- §4 API, less POST /manifests/search
- §5 rate limiting; §5a trust model, anonymous bearer tokens
- §6 upload validation, all four stages
- §7 relational storage, no JSON blob on the write path
- §8 Rust + Axum + SQLite, single serialized writer, in-process job queue
- §9a content addressing, computed on upload

Reconciled against the system spec:

- anneal_sec removed, withdrawn upstream by AR-012/AR-013. Presence follows
  track extent, so a track survives its own gaps and there is nothing to
  anneal. Its successor extinction_sec and the new gallery_scope are accepted
  and stored; scope enters the §7 ranking. A manifest still carrying
  anneal_sec is a hard 400, not silently ignored — it came from a pipeline
  whose window semantics differ from what this server assumes.
- Audio signature: media under 120 s now emits no signature at all, matching
  scene-actor-extraction IR-007. The earlier §3 draft allowed a shortened
  window under 150 s, which was the weaker rule — a caller-varying length is
  the property SR-004 forbids.
- UR IDs regularised to UR-nnn; docs/requirements.md registers 32
  requirements, each tracing to an SR-nnn or PR-nnn.

189 tests: unit, end-to-end through the real router, and an injection suite
covering SQL, JSON, header and Unicode payloads. Writing that suite found two
real gaps, both fixed here: compatibility homoglyphs passed the §5a character
class, and a one-frame audio signature was accepted on a feature-length item.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-30 18:14:02 +02:00
co-authored by Claude Opus 5
commit a848750a65
38 changed files with 13014 additions and 0 deletions
+187
View File
@@ -0,0 +1,187 @@
//! `GET /manifests/exists` and its batch form — UR-1.
//!
//! Deliberately a *separate, cheaper* endpoint from the fetch: it answers
//! "should I bother?" for a whole library sweep without transferring payloads,
//! and it is the endpoint a scheduled task will hammer. It is also the most
//! abuse-prone surface, since it doubles as an oracle for "does the community
//! have this title" — so it is rate-limited harder than the fetches and returns
//! no manifest content (§0).
use axum::extract::{Query, State};
use axum::http::HeaderMap;
use axum::response::{IntoResponse, Response};
use axum::Json;
use serde::{Deserialize, Serialize};
use super::LookupParams;
use crate::db::repo;
use crate::error::{ApiError, ApiResult};
use crate::matching::{self, StoredCut};
use crate::model::{IdentityType, MatchTier};
use crate::ratelimit::Surface;
use crate::state::{with_quota_headers, AppState};
/// §4: no manifest content, just availability and tier.
#[derive(Debug, Clone, Serialize)]
pub struct ExistsResponse {
pub exists: bool,
#[serde(skip_serializing_if = "Option::is_none")]
pub r#match: Option<&'static str>,
#[serde(skip_serializing_if = "Option::is_none")]
pub manifest_id: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub actor_count: Option<i64>,
}
impl ExistsResponse {
fn absent() -> Self {
Self { exists: false, r#match: None, manifest_id: None, actor_count: None }
}
}
/// §4 batch form: up to 100 items.
///
/// Exists specifically so the §5 rate limit can be generous per *request* while
/// staying strict per *item*, and so a 2000-item library sweep is 20 requests
/// rather than 2000.
pub const MAX_BATCH_ITEMS: usize = 100;
#[derive(Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct BatchRequest {
pub items: Vec<LookupParams>,
}
#[derive(Debug, Serialize)]
pub struct BatchResponse {
/// Positional, matching the request order (§4).
pub results: Vec<ExistsResponse>,
}
pub async fn exists(
State(state): State<AppState>,
peer: crate::state::PeerIp,
headers: HeaderMap,
Query(params): Query<LookupParams>,
) -> ApiResult<Response> {
let ip = state.client_ip(&headers, peer.0);
let quota = state.check_limit(&ip, Surface::ExistsSingle)?;
let body = lookup_one(&state, &params).await?;
Ok(with_quota_headers(Json(body).into_response(), quota))
}
pub async fn exists_batch(
State(state): State<AppState>,
peer: crate::state::PeerIp,
headers: HeaderMap,
super::json::Json(req): super::json::Json<BatchRequest>,
) -> ApiResult<Response> {
if req.items.len() > MAX_BATCH_ITEMS {
return Err(ApiError::BadRequest(format!(
"items: at most {MAX_BATCH_ITEMS} per request, got {}",
req.items.len()
)));
}
if req.items.is_empty() {
return Err(ApiError::BadRequest("items: must not be empty".into()));
}
let ip = state.client_ip(&headers, peer.0);
let quota = state.check_limit(&ip, Surface::ExistsBatch)?;
let mut results = Vec::with_capacity(req.items.len());
for item in &req.items {
// A malformed item yields "absent" rather than failing the whole batch —
// a sweep of 100 items should not be lost to one bad entry.
results.push(lookup_one(&state, item).await.unwrap_or_else(|_| ExistsResponse::absent()));
}
Ok(with_quota_headers(Json(BatchResponse { results }).into_response(), quota))
}
async fn lookup_one(state: &AppState, params: &LookupParams) -> ApiResult<ExistsResponse> {
let Some((kind, tmdb_id, imdb_id)) = resolve_kind(params) else {
return Err(ApiError::BadRequest(
"requires tmdb_id/imdb_id, or series_tmdb_id with season and episode".into(),
));
};
let (season, episode) = match kind {
IdentityType::Movie => (None, None),
IdentityType::Episode => (params.season, params.episode),
};
let client_cut = params.client_cut();
let found = state
.db
.read(move |conn| {
let Some(title) = repo::find_title(conn, kind, tmdb_id.as_deref(), imdb_id.as_deref())?
else {
return Ok(None);
};
let candidates = repo::candidates_for_title(conn, &title.id, season, episode)?;
if candidates.is_empty() {
return Ok(None);
}
let cuts: Vec<(String, StoredCut)> = candidates
.iter()
.map(|m| {
(
m.id.clone(),
StoredCut { runtime_sec: m.runtime_sec, video_hash: m.video_hash.clone() },
)
})
.collect();
let Some((id, m)) = matching::best_match(&client_cut, &cuts) else {
return Ok(None);
};
let actor_count = repo::manifest_actor_ids(conn, &id)?.len() as i64;
Ok(Some((id, m.tier, actor_count)))
})
.await
.map_err(ApiError::Internal)?;
// §4: `exists: false` is returned with `200`, not `404` — absence is a normal
// answer to this question, and `404` would conflate "no manifest" with "bad
// route" for the client.
Ok(match found {
Some((id, tier, actor_count)) => ExistsResponse {
exists: true,
// With no cut parameters the answer is "some manifest exists" with
// `"match": "unknown"`; the client must still fetch to find out
// whether a cut aligns. This is the mode a library sweep uses (§4).
r#match: Some(tier.as_str()),
manifest_id: Some(id),
actor_count: Some(actor_count),
},
None => ExistsResponse::absent(),
})
}
/// Determines whether these parameters address a movie or an episode.
pub fn resolve_kind(
params: &LookupParams,
) -> Option<(IdentityType, Option<String>, Option<String>)> {
if params.series_tmdb_id.is_some() || params.series_imdb_id.is_some() {
// Episode coordinates are required alongside series identity; without
// them the caller wants the series bundle endpoint instead.
params.season?;
params.episode?;
return Some((
IdentityType::Episode,
params.series_tmdb_id.clone(),
params.series_imdb_id.clone(),
));
}
if params.tmdb_id.is_some() || params.imdb_id.is_some() {
return Some((IdentityType::Movie, params.tmdb_id.clone(), params.imdb_id.clone()));
}
None
}
/// Exposed for tests asserting the documented tier string.
pub fn tier_str(t: MatchTier) -> &'static str {
t.as_str()
}
+351
View File
@@ -0,0 +1,351 @@
//! Manifest fetch endpoints (§4).
//!
//! §7: the submitted JSON was parsed, validated, resolved to TMDB person ids,
//! written as rows and discarded. Everything served here is **reconstructed**
//! from those rows, never echoed — which is what makes §5a's Threat 1 defence
//! structural rather than a promise.
use axum::extract::{Path, Query, State};
use axum::http::HeaderMap;
use axum::response::{IntoResponse, Response};
use axum::Json;
use serde::Serialize;
use super::LookupParams;
use crate::db::repo::{self, ManifestRow};
use crate::error::{ApiError, ApiResult};
use crate::matching::{self, StoredCut};
use crate::model::{
Actor, Coverage, Cut, Extraction, GalleryScope, Identity, IdentityType, Jmanifest, MatchTier,
SeriesBundle, SeriesRef, JMANIFEST_VERSION,
};
use crate::ratelimit::Surface;
use crate::state::{with_quota_headers, AppState};
#[derive(Debug, Serialize)]
pub struct FetchResponse {
pub r#match: &'static str,
/// Scene offset the client must add (§3). Zero for the tiers currently
/// served; present unconditionally so the plugin contract does not change
/// when `audio` is enabled.
pub offset_sec: f64,
pub manifest: Jmanifest,
}
#[derive(Debug, Serialize)]
pub struct StatusResponse {
pub status: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub reason: Option<String>,
}
pub async fn get_movie(
State(state): State<AppState>,
peer: crate::state::PeerIp,
headers: HeaderMap,
Query(params): Query<LookupParams>,
) -> ApiResult<Response> {
let ip = state.client_ip(&headers, peer.0);
let quota = state.check_limit(&ip, Surface::ManifestFetch)?;
if params.tmdb_id.is_none() && params.imdb_id.is_none() {
return Err(ApiError::BadRequest("requires tmdb_id or imdb_id".into()));
}
let body = fetch_best(&state, IdentityType::Movie, &params, None, None).await?;
Ok(with_quota_headers(Json(body).into_response(), quota))
}
pub async fn get_episode(
State(state): State<AppState>,
peer: crate::state::PeerIp,
headers: HeaderMap,
Query(params): Query<LookupParams>,
) -> ApiResult<Response> {
let ip = state.client_ip(&headers, peer.0);
let quota = state.check_limit(&ip, Surface::ManifestFetch)?;
if params.series_tmdb_id.is_none() && params.series_imdb_id.is_none() {
return Err(ApiError::BadRequest("requires series_tmdb_id or series_imdb_id".into()));
}
let (Some(season), Some(episode)) = (params.season, params.episode) else {
return Err(ApiError::BadRequest("requires season and episode".into()));
};
let body =
fetch_best(&state, IdentityType::Episode, &params, Some(season), Some(episode)).await?;
Ok(with_quota_headers(Json(body).into_response(), quota))
}
async fn fetch_best(
state: &AppState,
kind: IdentityType,
params: &LookupParams,
season: Option<i64>,
episode: Option<i64>,
) -> ApiResult<FetchResponse> {
let (tmdb_id, imdb_id) = match kind {
IdentityType::Movie => (params.tmdb_id.clone(), params.imdb_id.clone()),
IdentityType::Episode => (params.series_tmdb_id.clone(), params.series_imdb_id.clone()),
};
let client_cut = params.client_cut();
let found = state
.db
.read(move |conn| {
let Some(title) = repo::find_title(conn, kind, tmdb_id.as_deref(), imdb_id.as_deref())?
else {
return Ok(None);
};
let candidates = repo::candidates_for_title(conn, &title.id, season, episode)?;
let cuts: Vec<(ManifestRow, StoredCut)> = candidates
.into_iter()
.map(|m| {
let cut =
StoredCut { runtime_sec: m.runtime_sec, video_hash: m.video_hash.clone() };
(m, cut)
})
.collect();
let Some((row, m)) = matching::best_match(&client_cut, &cuts) else {
return Ok(None);
};
let manifest = reconstruct(conn, &row, &title, kind)?;
Ok(Some((m.tier, m.offset_sec, manifest)))
})
.await
.map_err(ApiError::Internal)?;
// §4: `404` if none clears `loose`.
let (tier, offset_sec, manifest) = found.ok_or(ApiError::NotFound)?;
Ok(FetchResponse { r#match: tier.as_str(), offset_sec, manifest })
}
/// `GET /manifests/series/{series_tmdb_id}?season=` (§4).
///
/// Returns whatever episodes the server holds. **Partial bundles are normal** — a
/// bundle with 9 of 13 episodes is a valid, useful response, not an error (§2).
/// Episode-level cut matching is done client-side against the returned bundle,
/// since a client pulling a whole series already knows its own runtimes.
pub async fn get_series(
State(state): State<AppState>,
peer: crate::state::PeerIp,
headers: HeaderMap,
Path(series_tmdb_id): Path<String>,
Query(params): Query<LookupParams>,
) -> ApiResult<Response> {
let ip = state.client_ip(&headers, peer.0);
let quota = state.check_limit(&ip, Surface::SeriesFetch)?;
let season = params.season;
let bundle = state
.db
.read(move |conn| {
let Some(title) =
repo::find_title(conn, IdentityType::Episode, Some(&series_tmdb_id), None)?
else {
return Ok(None);
};
let rows = repo::episodes_for_series(conn, &title.id, season)?;
// Multiple contributors may hold the same episode; `episodes_for_series`
// orders by rank, so keep the first per (season, episode).
let mut episodes: Vec<Jmanifest> = Vec::new();
let mut seen: Vec<(i64, i64)> = Vec::new();
let mut seasons: Vec<i64> = Vec::new();
for row in rows {
let key = (row.season.unwrap_or(-1), row.episode.unwrap_or(-1));
if seen.contains(&key) {
continue;
}
seen.push(key);
if !seasons.contains(&key.0) {
seasons.push(key.0);
}
episodes.push(reconstruct(conn, &row, &title, IdentityType::Episode)?);
}
seasons.sort_unstable();
Ok(Some(SeriesBundle {
jmanifest_version: JMANIFEST_VERSION,
series: SeriesRef {
series_tmdb_id: title.tmdb_id.clone(),
series_imdb_id: title.imdb_id.clone(),
title: title.name.clone(),
},
coverage: Some(Coverage { episodes_available: episodes.len(), seasons }),
episodes,
}))
})
.await
.map_err(ApiError::Internal)?;
let bundle = bundle.filter(|b| !b.episodes.is_empty()).ok_or(ApiError::NotFound)?;
Ok(with_quota_headers(Json(bundle).into_response(), quota))
}
/// `GET /manifests/{id}` — fetch a specific manifest by its server-assigned id,
/// for debugging and for the "report this manifest" flow (§4).
pub async fn get_by_id(
State(state): State<AppState>,
peer: crate::state::PeerIp,
headers: HeaderMap,
Path(id): Path<String>,
) -> ApiResult<Response> {
let ip = state.client_ip(&headers, peer.0);
let quota = state.check_limit(&ip, Surface::ManifestFetch)?;
let manifest = state
.db
.read(move |conn| {
let Some(row) = repo::manifest_by_id(conn, &id)? else { return Ok(None) };
// Unlisted manifests are not served to anyone (§6 stage 3).
if row.status != "listed" && row.status != "flagged" {
return Ok(None);
}
let title = title_of(conn, &row.title_id)?;
let kind =
if title.kind == "movie" { IdentityType::Movie } else { IdentityType::Episode };
Ok(Some(reconstruct(conn, &row, &title, kind)?))
})
.await
.map_err(ApiError::Internal)?;
let manifest = manifest.ok_or(ApiError::NotFound)?;
Ok(with_quota_headers(Json(manifest).into_response(), quota))
}
/// `GET /manifests/{id}/status` — poll the outcome of the asynchronous cast
/// check (§4).
pub async fn get_status(
State(state): State<AppState>,
Path(id): Path<String>,
) -> ApiResult<Json<StatusResponse>> {
let found = state
.db
.read(move |conn| repo::manifest_status(conn, &id))
.await
.map_err(ApiError::Internal)?;
match found {
Some((status, reason)) => Ok(Json(StatusResponse { status, reason })),
// §6 deletes rejected manifests, so a vanished id is reported as
// rejected rather than as a bad route.
None => Ok(Json(StatusResponse {
status: "rejected".into(),
reason: Some("not_found_or_rejected".into()),
})),
}
}
fn title_of(conn: &rusqlite::Connection, title_id: &str) -> anyhow::Result<repo::TitleRow> {
let row = conn.query_row(
"SELECT id, kind, tmdb_id, imdb_id, name, year, adult, certification
FROM titles WHERE id = ?1",
rusqlite::params![title_id],
|r| {
Ok(repo::TitleRow {
id: r.get(0)?,
kind: r.get(1)?,
tmdb_id: r.get(2)?,
imdb_id: r.get(3)?,
name: r.get(4)?,
year: r.get(5)?,
adult: r.get::<_, i64>(6)? != 0,
certification: r.get(7)?,
})
},
)?;
Ok(row)
}
/// Rebuilds a Jmanifest from stored rows.
///
/// Names come from `people` — populated from TMDB by the server — so `name` is
/// server-authoritative on download and a name a contributor invented does not
/// round-trip (§2, §5a).
pub fn reconstruct(
conn: &rusqlite::Connection,
row: &ManifestRow,
title: &repo::TitleRow,
kind: IdentityType,
) -> anyhow::Result<Jmanifest> {
let stored = repo::actors_for_manifest(conn, &row.id)?;
let actors = stored
.into_iter()
.map(|a| Actor {
name: a.name,
imdb_id: None,
tmdb_id: Some(a.tmdb_person_id.to_string()),
scenes: a
.scenes_cs
.into_iter()
.map(|(s, e)| [s as f64 / 100.0, e as f64 / 100.0])
.collect(),
})
.collect();
let identity = match kind {
IdentityType::Movie => Identity {
kind,
tmdb_id: title.tmdb_id.clone(),
imdb_id: title.imdb_id.clone(),
series_tmdb_id: None,
series_imdb_id: None,
season: None,
episode: None,
title: title.name.clone(),
year: title.year,
},
IdentityType::Episode => Identity {
kind,
tmdb_id: None,
imdb_id: None,
series_tmdb_id: title.tmdb_id.clone(),
series_imdb_id: title.imdb_id.clone(),
season: row.season,
episode: row.episode,
title: title.name.clone(),
year: title.year,
},
};
// An unrecognised stored scope is served as absent rather than guessed at:
// the column is written from a closed enum, so anything else means the row
// predates a schema change and its meaning is unknown (UR-014's spirit).
let gallery_scope = match row.gallery_scope.as_deref() {
Some("global") => Some(GalleryScope::Global),
Some("limited") => Some(GalleryScope::Limited),
_ => None,
};
let extraction = Extraction {
sample_fps: row.sample_fps,
extinction_sec: row.extinction_sec,
pipeline_version: row.pipeline_version.clone(),
gallery_size: None,
gallery_scope,
};
let has_extraction = extraction.sample_fps.is_some()
|| extraction.extinction_sec.is_some()
|| extraction.pipeline_version.is_some()
|| extraction.gallery_scope.is_some();
Ok(Jmanifest {
jmanifest_version: JMANIFEST_VERSION,
identity,
cut: Cut {
runtime_sec: row.runtime_sec,
container_duration_sec: None,
video_hash: row.video_hash.clone(),
audio_signature: None,
},
extraction: has_extraction.then_some(extraction),
actors,
})
}
/// Exposed so tests can assert the served tier strings.
pub fn tier_name(t: MatchTier) -> &'static str {
t.as_str()
}
+287
View File
@@ -0,0 +1,287 @@
//! A JSON extractor that fails with the status codes §4 specifies.
//!
//! Axum's own `Json` rejects a body that parses as JSON but does not match the
//! target type with **422 Unprocessable Entity**. §4 is explicit that this case
//! is **`400`** — "malformed, or contains an unrecognised or forbidden field" —
//! and that distinction is load-bearing: §6 requires that a client which forgets
//! to strip `movie` or `jellyfin_id` gets "a hard `400` naming the offending
//! field". A client checking for 400 would mishandle a 422.
//!
//! This wrapper also guarantees the field name reaches the caller, since serde's
//! `deny_unknown_fields` error text is what identifies the offending key.
use axum::extract::{FromRequest, Request};
use axum::http::header::CONTENT_TYPE;
use crate::error::ApiError;
/// Drop-in replacement for `axum::Json` on request bodies.
pub struct Json<T>(pub T);
impl<T, S> FromRequest<S> for Json<T>
where
T: serde::de::DeserializeOwned,
S: Send + Sync,
{
type Rejection = ApiError;
async fn from_request(req: Request, state: &S) -> Result<Self, Self::Rejection> {
// A wrong content type is the client's mistake, reported as such rather
// than as a parse failure.
let content_type =
req.headers().get(CONTENT_TYPE).and_then(|v| v.to_str().ok()).unwrap_or("").to_string();
let mime = content_type.split(';').next().unwrap_or("").trim().to_ascii_lowercase();
if !(mime == "application/json" || mime.ends_with("+json")) {
return Err(ApiError::BadRequest("expected content-type: application/json".into()));
}
// A declared charset other than UTF-8 is refused up front, so the client
// learns what is wrong rather than receiving a confusing parse error from
// deep inside the document. See `require_utf8` for why UTF-8 is the only
// accepted encoding.
if let Some(charset) =
content_type.split(';').skip(1).filter_map(|p| p.trim().strip_prefix("charset=")).next()
{
let charset = charset.trim().trim_matches('"').to_ascii_lowercase();
if !matches!(charset.as_str(), "utf-8" | "utf8") {
return Err(ApiError::BadRequest(format!(
"unsupported charset {charset:?}: JSON must be UTF-8 encoded (RFC 8259 §8.1)"
)));
}
}
let bytes = axum::body::Bytes::from_request(req, state).await.map_err(|e| {
// §6 stage 1: the body cap aborts mid-transfer, and that must surface
// as `413`, not as a generic parse error. Axum folds the length-limit
// case into `FailedToBufferBody`, so the status it chose is the
// reliable discriminator.
if e.status() == axum::http::StatusCode::PAYLOAD_TOO_LARGE {
ApiError::PayloadTooLarge("request body exceeds the limit for this route".into())
} else {
ApiError::BadRequest(format!("could not read request body: {e}"))
}
})?;
// Encoding is checked before parsing, so a mis-encoded body gets an
// actionable message instead of whatever the parser happens to trip over.
let text = require_utf8(&bytes)?;
serde_json::from_str(text)
.map(Json)
// serde's message names the offending field, which is exactly what §6
// requires the response to identify.
.map_err(|e| ApiError::BadRequest(e.to_string()))
}
}
/// Enforces that the body is UTF-8, naming the encoding it appears to be.
///
/// **UTF-8 is the only accepted encoding, deliberately.** RFC 8259 §8.1 requires
/// it for JSON exchanged outside a closed ecosystem, and this is a public,
/// federated API. Three further reasons make it the right call *here*
/// specifically, rather than merely conventional:
///
/// 1. **§9a content addressing hashes bytes.** `content_id` is a SHA-256 over the
/// canonical form, so the same manifest submitted in two encodings would
/// produce two different ids — silently defeating federation deduplication.
/// That is precisely the failure mode §9a quantises scene times to avoid, and
/// it would be reintroduced at the encoding layer.
/// 2. **UTF-16 admits lone surrogates**, which have no UTF-8 representation. A
/// field able to carry them is a channel for bytes that survive validation but
/// are not text — against §5a's premise that no field can carry a payload.
/// 3. **§5a's character class assumes well-formed Unicode scalar values.** NFC
/// normalisation and the category checks are defined over scalars, so admitting
/// an encoding that can express non-scalars would undermine both.
///
/// serde_json would reject non-UTF-8 anyway; the value added here is a diagnosable
/// error rather than a misleading one. A UTF-16 body otherwise fails with "key
/// must be a string", which points an operator at the wrong problem entirely.
fn require_utf8(bytes: &[u8]) -> Result<&str, ApiError> {
// A BOM is not valid JSON (RFC 8259 §8.1: "implementations MUST NOT add a
// byte order mark"), and it is the clearest signal of an encoding mistake, so
// it is named rather than left to the parser.
let encoding_hint = match bytes {
[0xEF, 0xBB, 0xBF, ..] => Some("UTF-8 with a byte order mark"),
[0xFF, 0xFE, 0x00, 0x00, ..] => Some("UTF-32LE"),
[0x00, 0x00, 0xFE, 0xFF, ..] => Some("UTF-32BE"),
[0xFF, 0xFE, ..] => Some("UTF-16LE"),
[0xFE, 0xFF, ..] => Some("UTF-16BE"),
// Unmarked UTF-16 is the common case, since encoders often omit the BOM.
// A JSON document always begins with an ASCII character, so an
// interleaved NUL in the first two bytes is conclusive.
[0x00, b, ..] if b.is_ascii_graphic() => Some("UTF-16BE (no BOM)"),
[b, 0x00, ..] if b.is_ascii_graphic() => Some("UTF-16LE (no BOM)"),
_ => None,
};
if let Some(encoding) = encoding_hint {
return Err(ApiError::BadRequest(format!(
"request body appears to be {encoding}: JSON must be UTF-8 encoded \
without a byte order mark (RFC 8259 §8.1)"
)));
}
std::str::from_utf8(bytes).map_err(|e| {
ApiError::BadRequest(format!(
"request body is not valid UTF-8 at byte {}: JSON must be UTF-8 encoded \
(RFC 8259 §8.1)",
e.valid_up_to()
))
})
}
#[cfg(test)]
mod tests {
use super::*;
use axum::http::StatusCode;
use axum::response::IntoResponse;
#[derive(serde::Deserialize)]
#[serde(deny_unknown_fields)]
struct Probe {
_wanted: i64,
}
async fn extract(body: &'static str, content_type: Option<&str>) -> StatusCode {
let mut builder = Request::builder().method("POST").uri("/");
if let Some(ct) = content_type {
builder = builder.header(CONTENT_TYPE, ct);
}
let req = builder.body(axum::body::Body::from(body)).unwrap();
match Json::<Probe>::from_request(req, &()).await {
Ok(_) => StatusCode::OK,
Err(e) => e.into_response().status(),
}
}
#[tokio::test]
async fn schema_mismatch_is_400_not_422() {
// The whole reason this extractor exists (§4, §6).
assert_eq!(
extract(r#"{"unexpected":1}"#, Some("application/json")).await,
StatusCode::BAD_REQUEST
);
}
#[tokio::test]
async fn malformed_json_is_400() {
assert_eq!(extract("{ nope", Some("application/json")).await, StatusCode::BAD_REQUEST);
}
#[tokio::test]
async fn missing_content_type_is_400() {
assert_eq!(extract(r#"{"_wanted":1}"#, None).await, StatusCode::BAD_REQUEST);
}
#[tokio::test]
async fn content_type_parameters_are_tolerated() {
assert_eq!(
extract(r#"{"_wanted":1}"#, Some("application/json; charset=utf-8")).await,
StatusCode::OK
);
}
#[tokio::test]
async fn valid_body_extracts() {
assert_eq!(extract(r#"{"_wanted":1}"#, Some("application/json")).await, StatusCode::OK);
}
#[tokio::test]
async fn an_explicit_utf8_charset_is_accepted() {
for ct in [
"application/json; charset=utf-8",
"application/json;charset=UTF-8",
"application/json; charset=\"utf-8\"",
"application/json; charset=utf8",
] {
assert_eq!(extract(r#"{"_wanted":1}"#, Some(ct)).await, StatusCode::OK, "{ct}");
}
}
#[tokio::test]
async fn a_non_utf8_charset_is_refused_by_name() {
for ct in [
"application/json; charset=utf-16",
"application/json; charset=iso-8859-1",
"application/json; charset=windows-1252",
] {
assert_eq!(
extract(r#"{"_wanted":1}"#, Some(ct)).await,
StatusCode::BAD_REQUEST,
"{ct}"
);
}
}
/// Builds a request from raw bytes, since these bodies are not valid `&str`.
async fn extract_bytes(body: Vec<u8>) -> Result<(), ApiError> {
let req = Request::builder()
.method("POST")
.uri("/")
.header(CONTENT_TYPE, "application/json")
.body(axum::body::Body::from(body))
.unwrap();
Json::<Probe>::from_request(req, &()).await.map(|_| ())
}
#[tokio::test]
async fn utf16_bodies_are_rejected_with_an_actionable_message() {
// The reason this check exists: serde_json rejects UTF-16 anyway, but with
// "key must be a string", which points an operator at the wrong problem.
let doc = r#"{"_wanted":1}"#;
let le: Vec<u8> = doc.encode_utf16().flat_map(|u| u.to_le_bytes()).collect();
let err = extract_bytes(le).await.unwrap_err().to_string();
assert!(err.contains("UTF-16LE"), "should name the encoding: {err}");
assert!(err.contains("UTF-8"), "should say what is required: {err}");
let be: Vec<u8> = doc.encode_utf16().flat_map(|u| u.to_be_bytes()).collect();
let err = extract_bytes(be).await.unwrap_err().to_string();
assert!(err.contains("UTF-16BE"), "should name the encoding: {err}");
// With BOMs.
let mut le_bom = vec![0xFF, 0xFE];
le_bom.extend(doc.encode_utf16().flat_map(|u| u.to_le_bytes()));
assert!(extract_bytes(le_bom).await.is_err());
let mut be_bom = vec![0xFE, 0xFF];
be_bom.extend(doc.encode_utf16().flat_map(|u| u.to_be_bytes()));
assert!(extract_bytes(be_bom).await.is_err());
}
#[tokio::test]
async fn a_utf8_bom_is_rejected() {
// RFC 8259 §8.1: implementations MUST NOT add a byte order mark.
let mut body = vec![0xEF, 0xBB, 0xBF];
body.extend_from_slice(br#"{"_wanted":1}"#);
let err = extract_bytes(body).await.unwrap_err().to_string();
assert!(err.contains("byte order mark"), "{err}");
}
#[tokio::test]
async fn invalid_utf8_is_rejected_with_the_offending_offset() {
// A truncated multi-byte sequence inside an otherwise well-formed document.
let body = b"{\"_wanted\":\"\xC3\x28\"}".to_vec();
let err = extract_bytes(body).await.unwrap_err().to_string();
assert!(err.contains("not valid UTF-8"), "{err}");
assert!(err.contains("byte 12"), "should locate the failure: {err}");
}
#[tokio::test]
async fn valid_multibyte_utf8_is_accepted() {
// The check must not reject legitimate non-ASCII content — actor names are
// routinely non-Latin (§5a accepts any Unicode letter).
// Rejected for the unknown `_note` field, not for its encoding — which is
// the distinction being asserted.
let body = r#"{"_wanted":1,"_note":"宮崎 駿 Renée"}"#.as_bytes().to_vec();
let err = extract_bytes(body).await.unwrap_err().to_string();
assert!(err.contains("_note"), "should fail on the schema, not the encoding: {err}");
}
#[tokio::test]
async fn an_empty_body_is_not_mistaken_for_an_encoding_problem() {
let err = extract_bytes(Vec::new()).await.unwrap_err().to_string();
assert!(!err.contains("UTF-16"), "empty body is a parse error, not an encoding one: {err}");
}
}
+35
View File
@@ -0,0 +1,35 @@
//! HTTP surface (§4). Base path `/api/v1`, JSON throughout.
pub mod exists;
pub mod fetch;
pub mod json;
pub mod report;
pub mod upload;
use serde::Deserialize;
use crate::matching::ClientCut;
/// Identity + cut query parameters, shared by the read endpoints (§4).
#[derive(Debug, Clone, Default, Deserialize)]
pub struct LookupParams {
pub tmdb_id: Option<String>,
pub imdb_id: Option<String>,
pub series_tmdb_id: Option<String>,
pub series_imdb_id: Option<String>,
pub season: Option<i64>,
pub episode: Option<i64>,
pub runtime_sec: Option<f64>,
pub video_hash: Option<String>,
}
impl LookupParams {
pub fn client_cut(&self) -> ClientCut {
ClientCut {
// A non-finite or non-positive runtime is not a usable signal; treat
// it as absent rather than letting it drive a match.
runtime_sec: self.runtime_sec.filter(|r| r.is_finite() && *r > 0.0),
video_hash: self.video_hash.clone(),
}
}
}
+141
View File
@@ -0,0 +1,141 @@
//! `POST /manifests/{id}/report` (§4), and `GET /health`.
//!
//! Reports are a moderation lever and cheap to abuse, hence the tight §5 limit.
//! A report never changes `status` by itself: §5a keeps delisting an operator
//! action, because automatic delisting on report would hand any client a remote
//! delete primitive.
use axum::extract::{Path, State};
use axum::http::HeaderMap;
use axum::response::{IntoResponse, Response};
use axum::Json;
use serde::{Deserialize, Serialize};
use crate::db::repo;
use crate::error::{ApiError, ApiResult};
use crate::ratelimit::Surface;
use crate::state::{with_quota_headers, AppState};
use crate::worker::now_iso;
/// §4: `{ "reason": "misaligned" | "wrong_actors" | "spam", "note": "..." }`.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum ReportReason {
Misaligned,
WrongActors,
Spam,
}
impl ReportReason {
fn as_str(self) -> &'static str {
match self {
ReportReason::Misaligned => "misaligned",
ReportReason::WrongActors => "wrong_actors",
ReportReason::Spam => "spam",
}
}
}
#[derive(Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ReportRequest {
pub reason: ReportReason,
#[serde(default)]
pub note: Option<String>,
}
/// §5a: `note` is free text from an anonymous caller, so it is capped hard. It is
/// never served back to clients — only the operator reads it.
const MAX_NOTE_CHARS: usize = 500;
#[derive(Debug, Serialize)]
pub struct ReportAccepted {
pub report_id: String,
}
pub async fn post_report(
State(state): State<AppState>,
peer: crate::state::PeerIp,
headers: HeaderMap,
Path(manifest_id): Path<String>,
super::json::Json(req): super::json::Json<ReportRequest>,
) -> ApiResult<Response> {
let ip = state.client_ip(&headers, peer.0);
let quota = state.check_limit(&ip, Surface::Report)?;
let note = match req.note {
Some(n) if n.chars().count() > MAX_NOTE_CHARS => {
return Err(ApiError::BadRequest(format!(
"note: longer than {MAX_NOTE_CHARS} characters"
)))
}
// Strip control characters; the note is operator-facing text, not markup.
Some(n) => Some(n.chars().filter(|c| !c.is_control()).collect::<String>()),
None => None,
};
let ip_hash = crate::auth::hash_ip(&ip, &state.config.server_id);
let reason = req.reason.as_str();
let now = now_iso();
let id_for_check = manifest_id.clone();
let exists = state
.db
.read(move |c| Ok(repo::manifest_by_id(c, &id_for_check)?.is_some()))
.await
.map_err(ApiError::Internal)?;
if !exists {
return Err(ApiError::NotFound);
}
let report_id = state
.db
.write(move |tx| {
repo::insert_report(tx, &manifest_id, reason, note.as_deref(), &ip_hash, &now)
})
.await
.map_err(ApiError::Internal)?;
Ok(with_quota_headers(Json(ReportAccepted { report_id }).into_response(), quota))
}
#[derive(Debug, Serialize)]
pub struct Health {
pub status: &'static str,
pub version: &'static str,
}
/// `GET /health` — liveness, unauthenticated and unlimited (§4, §5).
pub async fn health() -> Json<Health> {
Json(Health { status: "ok", version: env!("CARGO_PKG_VERSION") })
}
#[derive(Debug, Serialize)]
pub struct Readiness {
pub status: &'static str,
pub database: &'static str,
/// §8: TMDB is a hard dependency for UR-3. If it is unconfigured, uploads
/// accumulate in `pending` rather than being listed unverified — worth
/// surfacing rather than failing silently.
pub tmdb_configured: bool,
}
/// Readiness check verifying the database opens and migrations are current (§8).
pub async fn ready(State(state): State<AppState>) -> ApiResult<Json<Readiness>> {
let ok = state
.db
.read(|conn| {
// Any query against a schema table proves both that the file opens
// and that migrations have been applied.
let n: i64 = conn.query_row("SELECT COUNT(*) FROM manifests", [], |r| r.get(0))?;
Ok(n >= 0)
})
.await
.map_err(ApiError::Internal)?;
Ok(Json(Readiness {
status: if ok { "ready" } else { "degraded" },
database: "ok",
tmdb_configured: state.tmdb.is_configured(),
}))
}
+241
View File
@@ -0,0 +1,241 @@
//! Contribution endpoints (§4) — UR-2 and UR-6.
//!
//! Both require a token (§5). Both return `202`: the upload has passed size and
//! schema validation and is held unlisted pending the asynchronous TMDB cast
//! check (§6 stage 3).
use axum::extract::State;
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::Json;
use serde::Serialize;
use crate::db::repo;
use crate::error::{ApiError, ApiResult};
use crate::ingest::{self, IngestOutcome};
use crate::model::{IdentityType, Jmanifest, SeriesBundle};
use crate::ratelimit::Surface;
use crate::state::{with_quota_headers, AppState};
use crate::validate::{self, limits};
use crate::worker::now_iso;
#[derive(Debug, Serialize)]
pub struct UploadAccepted {
pub manifest_id: String,
pub status: &'static str,
}
/// `POST /manifests` — UR-2.
pub async fn post_manifest(
State(state): State<AppState>,
headers: HeaderMap,
super::json::Json(manifest): super::json::Json<Jmanifest>,
) -> ApiResult<Response> {
let contributor = state.require_contributor(&headers).await?;
// §5: limits are per token where one is present.
let quota = state.check_limit(&contributor.id, Surface::ManifestUpload)?;
// §6 stage 2. A rejection names the offending field, so a client that forgets
// to strip `movie`/`jellyfin_id` gets a diagnosable `400`.
let valid =
validate::validate_manifest(manifest).map_err(|e| ApiError::BadRequest(e.to_string()))?;
let origin = state.config.server_id.clone();
let contributor_id = contributor.id.clone();
let now = now_iso();
let outcome = state
.db
.write(move |tx| ingest::persist(tx, &valid, Some(&contributor_id), &origin, None, &now))
.await
.map_err(ApiError::Internal)?;
let resp = match outcome {
IngestOutcome::Pending { manifest_id } => {
(StatusCode::ACCEPTED, Json(UploadAccepted { manifest_id, status: "pending" }))
.into_response()
}
// §4 `409` — an identical `(identity, cut)` manifest already exists from
// this contributor.
IngestOutcome::DuplicateFromContributor { manifest_id } => {
return Err(ApiError::Conflict(format!(
"an identical manifest already exists from this contributor: {manifest_id}"
)))
}
// §9a: identical content already held, from any source. Not an error —
// the contributor's work is simply already represented.
IngestOutcome::DuplicateContent { manifest_id } => {
(StatusCode::OK, Json(UploadAccepted { manifest_id, status: "already_present" }))
.into_response()
}
};
Ok(with_quota_headers(resp, quota))
}
#[derive(Debug, Serialize)]
pub struct BundleResult {
pub season: Option<i64>,
pub episode: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub manifest_id: Option<String>,
pub status: &'static str,
#[serde(skip_serializing_if = "Option::is_none")]
pub reason: Option<String>,
}
#[derive(Debug, Serialize)]
pub struct BundleAccepted {
pub results: Vec<BundleResult>,
}
/// `POST /manifests/bundle` — UR-6.
///
/// **Per-episode validation, not atomic**: valid episodes are accepted and
/// invalid ones rejected, with a per-episode result list. All-or-nothing would let
/// one bad episode discard an entire season's compute (§2).
///
/// **One rate-limit unit**, so contributing a season is not punished relative to
/// contributing a film (§2, §5).
pub async fn post_bundle(
State(state): State<AppState>,
headers: HeaderMap,
super::json::Json(bundle): super::json::Json<SeriesBundle>,
) -> ApiResult<Response> {
let contributor = state.require_contributor(&headers).await?;
let quota = state.check_limit(&contributor.id, Surface::BundleUpload)?;
// §4: `413` for exceeding the episode cap, distinct from a malformed envelope.
if bundle.episodes.len() > limits::MAX_BUNDLE_EPISODES {
return Err(ApiError::PayloadTooLarge(format!(
"bundle carries {} episodes, limit is {}",
bundle.episodes.len(),
limits::MAX_BUNDLE_EPISODES
)));
}
// §4: `400` only for the envelope itself; individual bad episodes are
// reported in the results list, not as a whole-request error.
validate::validate_bundle_envelope(&bundle).map_err(|e| ApiError::BadRequest(e.to_string()))?;
let series_tmdb = bundle.series.series_tmdb_id.clone();
let mut results = Vec::with_capacity(bundle.episodes.len());
for episode in bundle.episodes {
let coords = (episode.identity.season, episode.identity.episode);
// An episode whose identity contradicts the envelope is rejected on its
// own rather than being silently reattributed to the bundle's series.
if episode.identity.kind != IdentityType::Episode {
results.push(BundleResult {
season: coords.0,
episode: coords.1,
manifest_id: None,
status: "rejected",
reason: Some("identity.type must be 'episode' within a bundle".into()),
});
continue;
}
if let (Some(envelope), Some(ep)) = (&series_tmdb, &episode.identity.series_tmdb_id) {
if envelope != ep {
results.push(BundleResult {
season: coords.0,
episode: coords.1,
manifest_id: None,
status: "rejected",
reason: Some("series_tmdb_id does not match the bundle envelope".into()),
});
continue;
}
}
let valid = match validate::validate_manifest(episode) {
Ok(v) => v,
Err(e) => {
results.push(BundleResult {
season: coords.0,
episode: coords.1,
manifest_id: None,
status: "rejected",
reason: Some(e.to_string()),
});
continue;
}
};
let origin = state.config.server_id.clone();
let contributor_id = contributor.id.clone();
let now = now_iso();
// One transaction per episode, so a bundle never holds the write lock for
// the whole request (§8 chunked ingest reasoning).
let outcome = state
.db
.write(move |tx| {
ingest::persist(tx, &valid, Some(&contributor_id), &origin, None, &now)
})
.await;
results.push(match outcome {
Ok(IngestOutcome::Pending { manifest_id }) => BundleResult {
season: coords.0,
episode: coords.1,
manifest_id: Some(manifest_id),
status: "pending",
reason: None,
},
Ok(IngestOutcome::DuplicateFromContributor { manifest_id })
| Ok(IngestOutcome::DuplicateContent { manifest_id }) => BundleResult {
season: coords.0,
episode: coords.1,
manifest_id: Some(manifest_id),
status: "already_present",
reason: None,
},
Err(e) => {
tracing::error!(error = ?e, "bundle episode failed to persist");
BundleResult {
season: coords.0,
episode: coords.1,
manifest_id: None,
status: "rejected",
reason: Some("internal error".into()),
}
}
});
}
let resp = (StatusCode::ACCEPTED, Json(BundleAccepted { results })).into_response();
Ok(with_quota_headers(resp, quota))
}
#[derive(Debug, Serialize)]
pub struct TokenIssued {
pub token: String,
}
/// Issues an anonymous bearer capability (§5a).
///
/// Self-issued on request: no email, no verification, no personal data. Stored
/// only as a hash, so the server cannot enumerate who holds tokens. Discarding a
/// token and requesting another is trivially easy — and that is fine, because the
/// token is not the defence; the content checks are.
pub async fn post_token(
State(state): State<AppState>,
peer: crate::state::PeerIp,
headers: HeaderMap,
) -> ApiResult<Json<TokenIssued>> {
let ip = state.client_ip(&headers, peer.0);
// Reuse the report budget: issuing tokens is cheap but should not be a free
// unbounded write.
state.check_limit(&ip, Surface::Report)?;
let token = crate::auth::generate_token();
let hash = crate::auth::hash_token(&token);
let now = now_iso();
state
.db
.write(move |tx| repo::insert_contributor(tx, &hash, &now))
.await
.map_err(ApiError::Internal)?;
Ok(Json(TokenIssued { token }))
}