dtourolle/jRay
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 6 Wiki Activity

feat(audio): v1 signature producer, bit-exact with the pipeline

Browse Source 
`AudioSignature` is the DSP — band table, periodic Hann, radix-2 FFT,
band-mean peak, energy class, packing — and `AudioSignatureService` the
decode, running the FFmpeg binary `IMediaEncoder.EncoderPath` names. The
plugin gained no dependency.

The server specification's prose does not determine a byte stream, so the
parameters it leaves open are pinned by the fixture shared with the
extraction repo and restated at the top of `AudioSignature`: double
throughout, whole frames only, periodic Hann, unnormalised FFT, band mean
rather than sum, argmax ties to the lowest index.

`fixtures/audio/` holds the extraction repo's three files byte-identically
and the computed signature equals the recorded vector exactly. The binding
check regenerates the fixture PCM from `make_fixture.py`'s arithmetic and
verifies it against the recorded decode checksums, so it runs on a host
with no codec at all and a decode divergence stays distinguishable from a
DSP one; the two tests that drive real FFmpeg self-skip without a binary.

The workflow named "Test Plugin" until now only compiled one. A test that
is built and never run is not evidence, and a golden vector shared across
two repos exists precisely so CI fails when they drift.

TRACES: JR-042, JR-043 | SR-003
This commit is contained in:
dtourolle
2026-07-31 16:24:11 +02:00
parent 19aecee646
commit 17be9bcc4e
9 changed files with 1359 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/test.yaml
+10
View File
@@ -42,6 +42,16 @@ jobs:
working-directory: test-${{ github.run_id }} working-directory: test-${{ github.run_id }}
run: dotnet build Jellyfin.Plugin.JRay.sln --configuration Debug --no-restore --no-self-contained /m:1 run: dotnet build Jellyfin.Plugin.JRay.sln --configuration Debug --no-restore --no-self-contained /m:1
# The workflow is named "Test Plugin" and until now only compiled one. A
# test that is built but never run is not evidence, and JR-043 is the case
# that makes it matter: the point of a golden vector shared with the
# extraction repo is that CI fails when the two implementations drift.
# T1 needs no ASP.NET runtime and no FFmpeg — the audio golden check
# regenerates its own fixture PCM.
- name: Run tests
working-directory: test-${{ github.run_id }}
run: dotnet test Jellyfin.Plugin.JRay.sln --configuration Debug --no-restore --no-build
- name: Cleanup - name: Cleanup
if: always() if: always()
run: rm -rf test-${{ github.run_id }} run: rm -rf test-${{ github.run_id }}
Jellyfin.Plugin.JRay.Tests/AudioSignatureTests.cs
+366
View File
@@ -0,0 +1,366 @@
using System;
using System.Buffers.Binary;
using System.Globalization;
using System.IO;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using Jellyfin.Plugin.JRay.Services;
using Microsoft.Extensions.Logging.Abstractions;
using Xunit;
namespace Jellyfin.Plugin.JRay.Tests;
/// <summary>
/// JR-042 (the signature is computed exactly per server specification §3) and
/// JR-043 (the golden-vector fixture shared with the extraction repo).
///
/// The headline claim is cross-repo: the C++ pipeline and this plugin are two
/// independent implementations of one fingerprint, and two fingerprints that
/// differ in any parameter simply do not match. That claim is only worth
/// anything if it is checked, so `fixtures/audio/` holds the same three files
/// the extraction repo holds — byte-identical — and these tests assert against
/// the values recorded in them, never against each other.
///
/// The binding check needs no FFmpeg. `make_fixture.py` generates the fixture
/// media from plain arithmetic, so UT-038 regenerates that PCM here and proves
/// it is byte-identical to what the pipeline decoded, using the checksums the
/// fixture records. Everything after that is pure DSP, which is what lets this
/// run on any CI host — the extraction repo's counterpart (UT-101) drives the
/// same vector through libavcodec, and both must land on the same string.
///
/// TRACES: UT-038, UT-039, UT-040, UT-041, UT-042, UT-043, UT-044 | JR-042, JR-043
/// </summary>
public class AudioSignatureTests
{
private static readonly string FixtureDir =
Path.Combine(AppContext.BaseDirectory, "fixtures", "audio");
private static readonly Lazy<JsonDocument> GoldenDoc = new(() =>
JsonDocument.Parse(File.ReadAllText(Path.Combine(FixtureDir, "jray_audio_v1_golden.json"))));
private static readonly Lazy<short[]> FixturePcm = new(GenerateFixturePcm);
private static readonly Lazy<float[]> FixtureSamples = new(() =>
{
var pcm = FixturePcm.Value;
var samples = new float[pcm.Length];
for (var i = 0; i < pcm.Length; i++)
{
// FFmpeg's native s16 -> flt conversion. 1/32768 is a power of two,
// so this is exact rather than merely close.
samples[i] = pcm[i] / 32768f;
}
return samples;
});
private static JsonElement Golden => GoldenDoc.Value.RootElement;
private static string GoldenSignature => Golden.GetProperty("signature").GetString()!;
// UT-038
[Fact]
public void RegeneratedFixturePcm_MatchesTheRecordedDecodedWindow()
{
// Checked before the signature, and separately from it, so a mismatch
// is diagnosable: if this passes and UT-039 fails, the DSP diverged; if
// this fails, the input did, and the signature comparison would only
// have told you "different" without saying where.
var pcm = FixturePcm.Value;
var decoded = Golden.GetProperty("decoded_window");
Assert.Equal(decoded.GetProperty("samples").GetInt32(), pcm.Length);
var s16 = new byte[pcm.Length * sizeof(short)];
for (var i = 0; i < pcm.Length; i++)
{
BinaryPrimitives.WriteInt16LittleEndian(s16.AsSpan(i * sizeof(short)), pcm[i]);
}
Assert.Equal(Hex64(decoded.GetProperty("s16le_fnv1a64").GetString()!), Fnv1a64(s16));
var f32 = new byte[FixtureSamples.Value.Length * sizeof(float)];
for (var i = 0; i < FixtureSamples.Value.Length; i++)
{
BinaryPrimitives.WriteSingleLittleEndian(f32.AsSpan(i * sizeof(float)), FixtureSamples.Value[i]);
}
Assert.Equal(Hex64(decoded.GetProperty("f32le_fnv1a64").GetString()!), Fnv1a64(f32));
}
// UT-039
[Fact]
public void Signature_OfTheGoldenFixture_MatchesTheRecordedValueExactly()
{
// The cross-repo check. Not "close", not "matches to within a tier" —
// the same string the C++ producer emits for the same audio.
Assert.Equal(GoldenSignature, AudioSignature.FromMonoSamples(FixtureSamples.Value));
}
// UT-040
[Fact]
public void BandTable_MatchesTheRecordedOne_AndTilesTheRangeExactly()
{
// The band-to-FFT-bin table is the part of the construction most likely
// to drift between two implementations — an off-by-one in a ceiling, a
// half-open range read as closed — so it is pinned independently of the
// signature it produces.
var table = AudioSignature.BandFftBins();
var want = Golden.GetProperty("band_fft_bins");
Assert.Equal(want.GetArrayLength(), table.Count);
for (var b = 0; b < table.Count; b++)
{
Assert.Equal(want[b][0].GetInt32(), table[b].Low);
Assert.Equal(want[b][1].GetInt32(), table[b].High);
Assert.True(table[b].High > table[b].Low, $"band {b} is empty");
if (b > 0)
{
// Contiguous, so the frame energy really is the sum of the band
// sums — no gap, no bin counted twice.
Assert.Equal(table[b - 1].High, table[b].Low);
}
}
}
// UT-041
[Fact]
public void Signature_IsWellFormed_PrefixFrameCountAndStructuralBytes()
{
var signature = AudioSignature.FromMonoSamples(FixtureSamples.Value);
Assert.NotNull(signature);
// JR-045 — the signature carries its own version, separate from
// schema_version, so a future DSP change is detectable rather than
// silently producing signatures that no longer match.
Assert.StartsWith(AudioSignature.VersionPrefix, signature, StringComparison.Ordinal);
var bytes = Convert.FromBase64String(signature[AudioSignature.VersionPrefix.Length..]);
Assert.Equal(Golden.GetProperty("frame_count").GetInt32(), bytes.Length);
Assert.Equal(AudioSignature.ExpectedFrames, bytes.Length);
// The server validates this structure on upload: each byte is a 5-bit
// band index plus a 2-bit energy class, so bit 7 is always clear and an
// arbitrary byte is not a valid signature. That is what keeps the field
// from being usable as a payload channel.
var bandsSeen = new bool[AudioSignature.NumBands];
var classesSeen = new bool[4];
foreach (var b in bytes)
{
Assert.Equal(0, b & 0x80);
bandsSeen[(b >> 2) & 0x1F] = true;
classesSeen[b & 0x03] = true;
}
// The fixture is built to exercise the whole output alphabet. If it ever
// stops doing so, the golden vector has become a weaker check than it
// looks — so that property is asserted rather than assumed.
Assert.All(bandsSeen, Assert.True);
Assert.All(classesSeen, Assert.True);
}
// UT-042
[Fact]
public void PackFrames_UsesWholeFramesOnly()
{
Assert.Empty(AudioSignature.PackFrames(new float[AudioSignature.FrameSize - 1]));
Assert.Single(AudioSignature.PackFrames(new float[AudioSignature.FrameSize]));
Assert.Single(AudioSignature.PackFrames(
new float[AudioSignature.FrameSize + AudioSignature.HopSize - 1]));
Assert.Equal(2, AudioSignature.PackFrames(
new float[AudioSignature.FrameSize + AudioSignature.HopSize].AsSpan()).Length);
// A partial frame is not a signature: below one frame there is nothing
// to emit, and emitting a padded frame would be a different fingerprint
// from the pipeline's.
Assert.Null(AudioSignature.FromMonoSamples(new float[AudioSignature.FrameSize - 1]));
// The full window is 1288 frames — asserted as a constant rather than by
// running the DSP over 1.3M zeros, which is the same claim for free.
Assert.Equal(1323000, AudioSignature.WindowSamples);
Assert.Equal(
1 + ((AudioSignature.WindowSamples - AudioSignature.FrameSize) / AudioSignature.HopSize),
AudioSignature.ExpectedFrames);
}
// UT-043
[Fact]
public async Task Decode_ThroughFfmpeg_ReproducesTheGoldenSignature()
{
// The one test that exercises the real decode — the command line, the
// stream selection, the downmix and resample — rather than the DSP
// alone. It needs an FFmpeg binary, which the plugin gets from Jellyfin
// at run time and which a bare CI container may not have; UT-038 and
// UT-039 are what make the cross-repo claim binding without one.
var ffmpeg = FindFfmpeg();
if (ffmpeg is null)
{
return;
}
var signature = await AudioSignatureService.ComputeWithEncoderAsync(
ffmpeg,
Path.Combine(FixtureDir, "jray_audio_v1_tone.flac"),
AudioSignature.WindowSec,
NullLogger.Instance,
CancellationToken.None).ConfigureAwait(true);
Assert.Equal(GoldenSignature, signature);
}
// UT-044
[Fact]
public async Task Decode_TakesTheWindowFromTheCentre_NotTheHead()
{
// Sampling from the centre is the whole reason the construction avoids
// the head and tail — logos and cold opens at one end, credits at the
// other — so it needs its own check. Nothing else here pins the seek: a
// head-anchored window passes every other test in this file.
var ffmpeg = FindFfmpeg();
if (ffmpeg is null)
{
return;
}
var pad = 90 * AudioSignature.SampleRate;
var padded = new short[(pad * 2) + FixturePcm.Value.Length];
FixturePcm.Value.CopyTo(padded, pad);
var wav = Path.Combine(Path.GetTempPath(), $"jray_audio_centre_{Environment.ProcessId}.wav");
try
{
WriteWav(wav, padded);
var signature = await AudioSignatureService.ComputeWithEncoderAsync(
ffmpeg,
wav,
padded.Length / (double)AudioSignature.SampleRate,
NullLogger.Instance,
CancellationToken.None).ConfigureAwait(true);
Assert.Equal(GoldenSignature, signature);
}
finally
{
File.Delete(wav);
}
}
/// <summary>
/// Regenerates the fixture media's PCM, as <c>make_fixture.py</c> defines
/// it: 120 s of tones stepping through all 32 log-bands, amplitudes walking
/// a golden-ratio sequence so all four energy classes appear, over a quiet
/// constant 777 Hz bed so no frame is degenerate.
/// </summary>
/// <remarks>
/// A port, not a re-derivation — the point is that a repo can rebuild the
/// input from scratch and check the result against the recorded checksums,
/// which is what UT-038 does. The closest sample to a quantisation boundary
/// sits 3.7e-7 away from one, so the result does not depend on which libm
/// rounds the sine.
/// </remarks>
private static short[] GenerateFixturePcm()
{
const int Segment = 32768;
const int BandStride = 7;
const double AmpLogMin = -1.55;
const double AmpLogSpan = 1.53;
const double PhiFrac = 0.6180339887498949;
const double BackgroundHz = 777.0;
const double BackgroundAmp = 0.004;
var n = (int)Math.Round(AudioSignature.SampleRate * AudioSignature.WindowSec);
var samples = new short[n];
var twoPi = 2.0 * Math.PI;
var phase = 0.0;
for (var start = 0; start < n; start += Segment)
{
var s = start / Segment;
var end = Math.Min(n, start + Segment);
var band = (s * BandStride) % AudioSignature.NumBands;
var freq = AudioSignature.BandLoHz * Math.Pow(
AudioSignature.BandHiHz / AudioSignature.BandLoHz,
(band + 0.5) / AudioSignature.NumBands);
var amp = Math.Pow(10.0, AmpLogMin + (AmpLogSpan * ((s * PhiFrac) % 1.0)));
var step = twoPi * freq / AudioSignature.SampleRate;
for (var k = 0; k < end - start; k++)
{
var i = start + k;
var x = amp * Math.Sin(phase + (step * k));
x += BackgroundAmp * Math.Sin(twoPi * BackgroundHz * i / AudioSignature.SampleRate);
x = Math.Clamp(x, -1.0, 1.0);
samples[i] = (short)Math.Floor((x * 32767.0) + 0.5);
}
phase = (phase + (step * (end - start))) % twoPi;
}
return samples;
}
private static void WriteWav(string path, short[] samples)
{
var dataBytes = samples.Length * sizeof(short);
using var stream = File.Create(path);
using var writer = new BinaryWriter(stream);
writer.Write("RIFF"u8);
writer.Write(36 + dataBytes);
writer.Write("WAVE"u8);
writer.Write("fmt "u8);
writer.Write(16); // PCM header size
writer.Write((short)1); // PCM
writer.Write((short)1); // mono
writer.Write(AudioSignature.SampleRate);
writer.Write(AudioSignature.SampleRate * sizeof(short));
writer.Write((short)sizeof(short)); // block align
writer.Write((short)16); // bits per sample
writer.Write("data"u8);
writer.Write(dataBytes);
foreach (var sample in samples)
{
writer.Write(sample);
}
}
private static string? FindFfmpeg()
{
var configured = Environment.GetEnvironmentVariable("JRAY_TEST_FFMPEG");
if (!string.IsNullOrEmpty(configured))
{
return File.Exists(configured) ? configured : null;
}
var name = OperatingSystem.IsWindows() ? "ffmpeg.exe" : "ffmpeg";
foreach (var dir in (Environment.GetEnvironmentVariable("PATH") ?? string.Empty)
.Split(Path.PathSeparator, StringSplitOptions.RemoveEmptyEntries))
{
var candidate = Path.Combine(dir, name);
if (File.Exists(candidate))
{
return candidate;
}
}
return null;
}
private static ulong Fnv1a64(ReadOnlySpan<byte> data)
{
var hash = 0xcbf29ce484222325UL;
foreach (var b in data)
{
hash ^= b;
hash *= 0x100000001b3UL;
}
return hash;
}
private static ulong Hex64(string value)
{
var digits = value.StartsWith("0x", StringComparison.OrdinalIgnoreCase) ? value[2..] : value;
return ulong.Parse(digits, NumberStyles.HexNumber, CultureInfo.InvariantCulture);
}
}
Jellyfin.Plugin.JRay.Tests/Jellyfin.Plugin.JRay.Tests.csproj
+12
View File
@@ -53,4 +53,16 @@
<ProjectReference Include="..\Jellyfin.Plugin.JRay\Jellyfin.Plugin.JRay.csproj" /> <ProjectReference Include="..\Jellyfin.Plugin.JRay\Jellyfin.Plugin.JRay.csproj" />
</ItemGroup> </ItemGroup>
<ItemGroup>
<!--
The audio-signature golden vector, shared verbatim with the extraction
repo (JR-043): the same three files, byte for byte, in both repos. Two
independent implementations of one fingerprint are only useful if they
agree exactly, and this is what makes that a checked claim.
-->
<Content Include="fixtures\**\*">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Project> </Project>
Jellyfin.Plugin.JRay.Tests/fixtures/audio/jray_audio_v1_golden.json
+194
View File
@@ -0,0 +1,194 @@
{
"_": "Golden vector for the JRay v1 audio signature (JRay-public-server SPEC.md \u00a73). Shared verbatim between scene-actor-extraction (C++) and the jRay Jellyfin plugin (C#) so the two implementations can be proven bit-identical. IR-004, IR-005, IR-007, IR-008.",
"version": "v1",
"signature": "v1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAeHx8eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh8fHzk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5V1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dycnJycnJycnJycnJycnJycnJycnJycnJycnJycnMPDgwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKytFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRWNjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2Njfn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5/GxoZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGTc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3UlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU1JsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbAoLCwoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCwsLJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSVDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ15eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eX19eeXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXkXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFzExMTExMTExMTExMTExMTExMTExMTExMTExMTExT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09qampqampqampqampqampqampqampqampqampqamsHBwUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyM+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4/PlhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYd3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3ExMRERERERERERERERERERERERERERERERERERERES8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpLS0plZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZQMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0eHh44ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4OFdXV1ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWV1dXcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXEPDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDyoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKysqRERERERERERERERERERERERERERERERERERERERjY2NjY2NjY2NjYw==",
"frame_count": 1288,
"media": {
"file": "jray_audio_v1_tone.flac",
"generator": "make_fixture.py",
"container": "FLAC (lossless \u2014 decodes to exactly the PCM make_fixture.py emits)",
"duration_sec": 120.0,
"sample_rate": 11025,
"channels": 1,
"sample_format": "s16",
"sha256": "912ecd426cd426dccb37753e0249694227619c701cb9f533502b37da0fbe8096",
"bytes": 585142
},
"decoded_window": {
"_": "Checksums of the 120 s centre window after downmix to mono and resample to 11025 Hz, i.e. exactly the stream `ffmpeg -ss <mid-60> -t 120 -i <file> -vn -ac 1 -ar 11025 -f f32le -` produces. Check these first: a mismatch here is a decode problem, not a DSP one.",
"samples": 1323000,
"f32le_fnv1a64": "0x1ef7899cd4d12662",
"s16le_fnv1a64": "0xf824fa56f125c0dc"
},
"params": {
"window_sec": 120.0,
"window_centre": "runtime/2, i.e. samples from runtime/2 - 60 s; truncated to exactly 1323000 samples",
"min_duration_sec": 120.0,
"min_duration_rule": "IR-007 \u2014 below this emit NO signature and apply no sync offset",
"sample_rate": 11025,
"channels": 1,
"arithmetic": "IEEE-754 double throughout; float32 is not sufficient",
"sample_scale": "s16 * (1/32768), FFmpeg's native s16->flt",
"frame_size": 4096,
"hop_size": 1024,
"frame_count_rule": "1 + (n_samples - 4096) / 1024, integer division; whole frames only",
"window_fn": "Hann, PERIODIC: w[n] = 0.5 * (1 - cos(2*pi*n/4096))",
"transform": "radix-2 DIT complex FFT over the 4096 real samples (imag=0), no normalisation",
"magnitude": "sqrt(re^2 + im^2), linear",
"band_lo_hz": 300.0,
"band_hi_hz": 3000.0,
"num_bands": 32,
"band_edges": "edge[b] = 300 * (3000/300)^(b/32), b = 0..32",
"band_bins": "band b owns FFT bins [k_lo[b], k_lo[b+1]) with k_lo[b] = ceil(edge[b] * 4096 / 11025); see band_fft_bins",
"band_value": "MEAN of the linear magnitudes in the band (not sum, not max)",
"peak_bin": "argmax over the 32 band values; ties resolve to the LOWEST index",
"energy_metric": "E = mean magnitude over all FFT bins 112..1114, i.e. the whole 300-3000 Hz band",
"energy_reference": "upper median of E over all frames: sorted[n/2], no averaging of the two middle values",
"energy_ratio": "r = log10((E + 1e-12) / (E_ref + 1e-12))",
"energy_class_edges": [
-0.6,
-0.2,
0.2
],
"energy_class": "0 if r < -0.6, 1 if r < -0.2, 2 if r < 0.2, else 3",
"byte_layout": "bit7 = 0 (reserved), bits6..2 = 5-bit band index, bits1..0 = 2-bit energy class; byte = (band << 2) | class",
"base64": "standard alphabet A-Za-z0-9+/ with '=' padding",
"prefix": "v1:"
},
"band_fft_bins": [
[
112,
120
],
[
120,
129
],
[
129,
139
],
[
139,
149
],
[
149,
160
],
[
160,
172
],
[
172,
185
],
[
185,
199
],
[
199,
213
],
[
213,
229
],
[
229,
246
],
[
246,
265
],
[
265,
285
],
[
285,
306
],
[
306,
328
],
[
328,
353
],
[
353,
379
],
[
379,
408
],
[
408,
438
],
[
438,
471
],
[
471,
506
],
[
506,
543
],
[
543,
584
],
[
584,
627
],
[
627,
674
],
[
674,
724
],
[
724,
778
],
[
778,
836
],
[
836,
899
],
[
899,
966
],
[
966,
1038
],
[
1038,
1115
]
],
"notes": [
"The server spec fixes the window, rate, STFT geometry, band and the 5+2 bit packing. Everything under params beyond that (Hann periodicity, band aggregation, the energy-class definition, tie-breaking, base64 alphabet) is pinned HERE for v1 \u2014 the spec does not constrain it, and two implementations that guess differently produce non-matching signatures.",
"Decision margins on this fixture: the two strongest bands are within 1.3% on the closest frame, and the closest frame to an energy-class edge is 3.6e-3 away in log10. Both are many orders of magnitude above double-precision FFT differences, so any two correct double- precision implementations agree; a float32 implementation is not guaranteed to.",
"Coverage: all 32 bands and all 4 energy classes appear in the golden signature.",
"Robustness observed on this fixture: identical peak-bin sequence after a stereo/44100 Hz round trip and after AAC 128 kbit/s re-encoding."
]
}
Jellyfin.Plugin.JRay.Tests/fixtures/audio/jray_audio_v1_tone.flac
BIN
View File
Binary file not shown.
Jellyfin.Plugin.JRay.Tests/fixtures/audio/make_fixture.py
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""Regenerate the JRay audio-signature golden fixture.
python3 make_fixture.py # writes jray_audio_v1_tone.flac here
This is the *source of truth* for the fixture media: `jray_audio_v1_tone.flac`
is a lossless FLAC encoding of exactly the PCM this script emits, so any repo
that wants to check its own audio-signature implementation against the golden
vector in `jray_audio_v1_golden.json` can regenerate the input from scratch and
confirm it is byte-identical (the golden file records `pcm_fnv1a64`, a hash of
the decoded 16-bit samples).
Deliberately dependency-free (no numpy) and written in plain arithmetic so it
ports to any language in ~20 lines.
Signal — 120.000 s, mono, 11025 Hz, 16-bit signed PCM:
* split into segments of 32768 samples (~2.97 s), 40.4 segments in total;
* segment `s` carries one sine at the geometric centre of log-band
`(s * 7) mod 32` of the 300-3000 Hz band, so all 32 bands are exercised;
* its amplitude walks a golden-ratio low-discrepancy sequence over
[10^-1.55, 10^-0.02] so frame energies spread continuously across ~1.5
decades and all four energy classes are exercised, without a dense cluster
of frames sitting on a class boundary;
* phase is carried across segment boundaries (no clicks);
* a constant, far quieter 777 Hz tone sits underneath so no frame is
degenerate;
* samples are quantised with floor(x * 32767 + 0.5).
Why FLAC and not WAV: 120 s of 11025 Hz 16-bit PCM is 2.6 MB and does not
compress in git. FLAC is lossless — FFmpeg decodes it to exactly the PCM
written here — and is ~3.5x smaller. `--wav` writes the uncompressed original
if you want to diff it.
"""
import math
import struct
import subprocess
import sys
import os
SAMPLE_RATE = 11025
DURATION_SEC = 120.0
SEGMENT = 32768 # samples per tone segment
BAND_STRIDE = 7 # coprime with 32 -> visits every band
BAND_LO_HZ = 300.0
BAND_HI_HZ = 3000.0
NUM_BANDS = 32
AMP_LOG_MIN = -1.55 # 10^-1.55 ~= 0.028
AMP_LOG_SPAN = 1.53 # up to 10^-0.02 ~= 0.955
PHI_FRAC = 0.6180339887498949
BG_HZ = 777.0
BG_AMP = 0.004
OUT_FLAC = "jray_audio_v1_tone.flac"
OUT_WAV = "jray_audio_v1_tone.wav"
def generate():
"""Return the 120 s signal as a list of int16 sample values."""
n = int(round(SAMPLE_RATE * DURATION_SEC))
out = [0] * n
phase = 0.0
two_pi = 2.0 * math.pi
for start in range(0, n, SEGMENT):
s = start // SEGMENT
end = min(n, start + SEGMENT)
band = (s * BAND_STRIDE) % NUM_BANDS
# geometric centre of log-band `band`
freq = BAND_LO_HZ * (BAND_HI_HZ / BAND_LO_HZ) ** ((band + 0.5) / NUM_BANDS)
amp = 10.0 ** (AMP_LOG_MIN + AMP_LOG_SPAN * ((s * PHI_FRAC) % 1.0))
step = two_pi * freq / SAMPLE_RATE
for k in range(end - start):
i = start + k
x = amp * math.sin(phase + step * k)
x += BG_AMP * math.sin(two_pi * BG_HZ * i / SAMPLE_RATE)
if x > 1.0:
x = 1.0
elif x < -1.0:
x = -1.0
out[i] = int(math.floor(x * 32767.0 + 0.5))
phase = (phase + step * (end - start)) % two_pi
return out
def write_wav(path, samples):
data = struct.pack("<%dh" % len(samples), *samples)
hdr = b"RIFF" + struct.pack("<I", 36 + len(data)) + b"WAVE"
hdr += b"fmt " + struct.pack("<IHHIIHH", 16, 1, 1, SAMPLE_RATE,
SAMPLE_RATE * 2, 2, 16)
hdr += b"data" + struct.pack("<I", len(data))
with open(path, "wb") as fh:
fh.write(hdr + data)
def main():
here = os.path.dirname(os.path.abspath(__file__))
samples = generate()
wav = os.path.join(here, OUT_WAV)
write_wav(wav, samples)
if "--wav" in sys.argv:
print("wrote", wav)
return
flac = os.path.join(here, OUT_FLAC)
# -compression_level 12 is deterministic for a given libFLAC/ffmpeg build;
# only the container bytes vary, never the decoded PCM.
subprocess.run(["ffmpeg", "-nostdin", "-v", "error", "-y", "-i", wav,
"-c:a", "flac", "-compression_level", "12", flac],
check=True)
os.remove(wav)
print("wrote", flac)
if __name__ == "__main__":
main()
Jellyfin.Plugin.JRay/ServiceRegistrator.cs
+5
View File
@@ -22,5 +22,10 @@ public class ServiceRegistrator : IPluginServiceRegistrator
// server that is down should be skipped for the whole sweep, not // server that is down should be skipped for the whole sweep, not
// retried once per item (JR-037). // retried once per item (JR-037).
serviceCollection.AddSingleton<IManifestExchangeClient, ManifestExchangeClient>(); serviceCollection.AddSingleton<IManifestExchangeClient, ManifestExchangeClient>();
// Registered concretely: the signature is computed, not yet consumed, so
// there is no second implementation for an interface to abstract over
// and nothing to gain from inventing one (JR-042).
serviceCollection.AddSingleton<AudioSignatureService>();
} }
} }
Jellyfin.Plugin.JRay/Services/AudioSignature.cs
+373
View File
@@ -0,0 +1,373 @@
using System;
using System.Collections.Generic;
namespace Jellyfin.Plugin.JRay.Services;
/// <summary>
/// The JRay v1 content-derived audio signature: spectral peak bins taken from
/// the centre of a media file, so a truth file is self-identifying.
/// </summary>
/// <remarks>
/// The construction is owned by the public server specification §3 and is
/// implemented a second time, in C++, by the extraction pipeline
/// (<c>src/audio_signature.*</c>, extraction <c>IR-004</c>). <b>The two must
/// agree byte for byte</b> — a signature that differs in any parameter simply
/// does not match, which defeats the entire point of having one. Every constant
/// below is therefore load-bearing, and any change to one is a change to the
/// <c>v1:</c> prefix as well.
/// <para>
/// The server specification's prose is not sufficient to reproduce a byte
/// stream, so the details it leaves open are pinned by the golden fixture
/// shared with the extraction repo (JR-043,
/// <c>fixtures/audio/jray_audio_v1_golden.json</c>), and restated here:
/// </para>
/// <list type="bullet">
/// <item>Arithmetic is IEEE-754 <c>double</c> throughout. <c>float</c> is not
/// sufficient: the fixture has frames whose two strongest bands are within 1.3%
/// of each other.</item>
/// <item>Samples arrive as FFmpeg's native <c>s16 -&gt; flt</c> conversion,
/// <c>x * (1/32768)</c>, widened to double here.</item>
/// <item>Whole frames only:
/// <c>n_frames = 1 + (n_samples - 4096) / 1024</c>, integer division.</item>
/// <item>Hann window, <b>periodic</b>: <c>0.5 * (1 - cos(2*pi*n/4096))</c>, not
/// the symmetric <c>N-1</c> variant.</item>
/// <item>Plain radix-2 FFT, no normalisation; magnitude is
/// <c>sqrt(re^2 + im^2)</c>.</item>
/// <item>A band's value is the <b>mean</b> of the linear magnitudes in it, so a
/// wide high band is not favoured over a narrow low one.</item>
/// <item>The peak is the <c>argmax</c> over the 32 bands, ties to the lowest
/// index. The specification's log is a monotone squash and so cannot change an
/// argmax; it is applied only where it is observable, in the energy class.</item>
/// </list>
/// <para>
/// The FFT is written out here rather than taken from a library for the same
/// reason the pipeline writes its own: it is a fixed, fully specified transform,
/// and a dependency whose version could change the numerics is a liability when
/// the output has to be identical across two languages.
/// </para>
/// </remarks>
// TRACES: JR-042 | SR-003
public static class AudioSignature
{
/// <summary>Sample rate the signature is computed at, in Hz.</summary>
public const int SampleRate = 11025;
/// <summary>STFT frame size, in samples.</summary>
public const int FrameSize = 4096;
/// <summary>STFT hop size, in samples (~93 ms).</summary>
public const int HopSize = 1024;
/// <summary>Number of logarithmically spaced bands.</summary>
public const int NumBands = 32;
/// <summary>Low edge of the analysed band, in Hz.</summary>
public const double BandLoHz = 300.0;
/// <summary>High edge of the analysed band, in Hz.</summary>
public const double BandHiHz = 3000.0;
/// <summary>Length of the analysed window, in seconds.</summary>
public const double WindowSec = 120.0;
/// <summary>
/// Length of the analysed window, in samples (120.000 s at 11025 Hz).
/// </summary>
/// <remarks>
/// The decoded window is truncated to exactly this, so the frame count is
/// the same for every input and does not wobble with seek granularity or a
/// resampler tail.
/// </remarks>
public const int WindowSamples = 1323000;
/// <summary>Frames a full window yields: <c>1 + (1323000 - 4096) / 1024</c>.</summary>
/// <remarks>
/// The server specification says "~1290" and accepts a tolerance; the exact
/// count follows from the framing rule and is 1288.
/// </remarks>
public const int ExpectedFrames = 1288;
/// <summary>Guard added to both sides of the energy ratio.</summary>
public const double EnergyEps = 1e-12;
/// <summary>
/// The signature's own version prefix, separate from <c>schema_version</c>.
/// </summary>
/// <remarks>
/// A future change to the DSP chain must be <i>detectable</i> rather than
/// silently producing signatures that no longer match (JR-045).
/// </remarks>
public const string VersionPrefix = "v1:";
// Class thresholds on log10(E_frame / E_median). They deliberately straddle
// r = 0 rather than sit on it, so the median frame itself is never on a
// boundary.
private static readonly double[] EnergyClassEdges = [-0.6, -0.2, 0.2];
private static readonly (int Low, int High)[] Bands = BuildBandTable();
private static readonly double[] Window = BuildHannWindow();
private static readonly int[] BitReversal = BuildBitReversal();
private static readonly double[][] TwiddleReal = BuildTwiddles(cosine: true);
private static readonly double[][] TwiddleImag = BuildTwiddles(cosine: false);
/// <summary>
/// Gets the half-open FFT bin range <c>[Low, High)</c> owned by each of the
/// 32 log-spaced bands.
/// </summary>
/// <remarks>
/// Exposed so the golden fixture can assert the table itself rather than
/// only the signature it produces: the band table is the part of the
/// construction most likely to drift between two implementations.
/// </remarks>
/// <returns>One range per band, contiguous and non-overlapping.</returns>
public static IReadOnlyList<(int Low, int High)> BandFftBins() => Bands;
/// <summary>
/// Packs one byte per whole STFT frame: a 5-bit peak band index and a 2-bit
/// energy class.
/// </summary>
/// <remarks>
/// The byte layout is <c>(band &lt;&lt; 2) | class</c>, so bit 7 is always
/// clear and an arbitrary byte is not a valid signature. That structural
/// constraint is what the server validates on upload, and what keeps the
/// field from being usable as a payload channel.
/// </remarks>
/// <param name="mono">Mono samples at <see cref="SampleRate"/>, in [-1, 1).</param>
/// <returns>One byte per frame; empty when not even one frame fits.</returns>
public static byte[] PackFrames(ReadOnlySpan<float> mono)
{
if (mono.Length < FrameSize)
{
return [];
}
var frames = 1 + ((mono.Length - FrameSize) / HopSize);
var binLow = Bands[0].Low;
var binHigh = Bands[NumBands - 1].High; // exclusive
double binCount = binHigh - binLow;
var re = new double[FrameSize];
var im = new double[FrameSize];
var peak = new byte[frames];
var energy = new double[frames];
for (var f = 0; f < frames; f++)
{
var src = mono.Slice(f * HopSize, FrameSize);
for (var n = 0; n < FrameSize; n++)
{
re[n] = src[n] * Window[n];
im[n] = 0.0;
}
Fft(re, im);
// Per-band mean magnitude. The bands tile 300-3000 Hz with no gaps
// and no overlaps, so the frame's band-limited energy is the sum of
// the band sums — accumulated in band order, because the order of a
// floating-point summation is part of the contract.
var best = -1.0;
var bestBand = 0;
var total = 0.0;
for (var b = 0; b < NumBands; b++)
{
var (low, high) = Bands[b];
var sum = 0.0;
for (var k = low; k < high; k++)
{
sum += Math.Sqrt((re[k] * re[k]) + (im[k] * im[k]));
}
total += sum;
var mean = sum / (high - low);
if (mean > best)
{
best = mean; // ties -> lowest index
bestBand = b;
}
}
peak[f] = (byte)bestBand;
energy[f] = total / binCount;
}
// The reference is the upper median of the frame energies: an actually
// observed value rather than the average of the two middle ones, so it
// is bit-reproducible. It is also gain-invariant — loudness
// normalisation must not change a signature — and barely moves when the
// window is trimmed.
var sorted = (double[])energy.Clone();
Array.Sort(sorted);
var reference = sorted[sorted.Length / 2];
var packed = new byte[frames];
for (var f = 0; f < frames; f++)
{
var r = Math.Log10((energy[f] + EnergyEps) / (reference + EnergyEps));
packed[f] = (byte)(((peak[f] & 0x1F) << 2) | (EnergyClass(r) & 0x03));
}
return packed;
}
/// <summary>
/// Computes the full signature string for a decoded centre window.
/// </summary>
/// <param name="mono">Mono samples at <see cref="SampleRate"/>, in [-1, 1).</param>
/// <returns>
/// <c>v1:</c> followed by the base64 of <see cref="PackFrames"/>, or
/// <c>null</c> when not even one frame fits.
/// </returns>
public static string? FromMonoSamples(ReadOnlySpan<float> mono)
{
var packed = PackFrames(mono);
if (packed.Length == 0)
{
return null;
}
// Standard alphabet with '=' padding, which is what the server and the
// pipeline both emit.
return VersionPrefix + Convert.ToBase64String(packed);
}
private static int EnergyClass(double ratio)
{
if (ratio < EnergyClassEdges[0])
{
return 0;
}
if (ratio < EnergyClassEdges[1])
{
return 1;
}
if (ratio < EnergyClassEdges[2])
{
return 2;
}
return 3;
}
// edge[b] = 300 * (3000/300)^(b/32); band b owns FFT bins
// [k_lo[b], k_lo[b+1]) with k_lo[b] = ceil(edge[b] / hz_per_bin). Taking the
// ceiling once, into an integer table, means membership is never decided by
// a float comparison per bin per frame — which is where two implementations
// would otherwise be free to disagree.
private static (int Low, int High)[] BuildBandTable()
{
var hzPerBin = (double)SampleRate / FrameSize;
var edges = new int[NumBands + 1];
for (var b = 0; b <= NumBands; b++)
{
var hz = BandLoHz * Math.Pow(BandHiHz / BandLoHz, (double)b / NumBands);
edges[b] = (int)Math.Ceiling(hz / hzPerBin);
}
var table = new (int Low, int High)[NumBands];
for (var b = 0; b < NumBands; b++)
{
table[b] = (edges[b], edges[b + 1]);
}
return table;
}
private static double[] BuildHannWindow()
{
var w = new double[FrameSize];
for (var n = 0; n < FrameSize; n++)
{
w[n] = 0.5 * (1.0 - Math.Cos(2.0 * Math.PI * n / FrameSize));
}
return w;
}
private static int[] BuildBitReversal()
{
var bits = 0;
while ((1 << bits) < FrameSize)
{
bits++;
}
var rev = new int[FrameSize];
for (var i = 0; i < FrameSize; i++)
{
var r = 0;
for (var b = 0; b < bits; b++)
{
if ((i & (1 << b)) != 0)
{
r |= 1 << (bits - 1 - b);
}
}
rev[i] = r;
}
return rev;
}
// Twiddles are precomputed per stage from cos/sin of -2*pi*j/len, so the
// angle is an exactly reproducible double in either language and only the
// library's own rounding of cos/sin (<= 1 ulp) can differ — orders of
// magnitude below the decision margins the golden fixture records.
private static double[][] BuildTwiddles(bool cosine)
{
var stages = new List<double[]>();
for (var len = 2; len <= FrameSize; len <<= 1)
{
var half = len / 2;
var stage = new double[half];
for (var j = 0; j < half; j++)
{
var angle = -2.0 * Math.PI * j / len;
stage[j] = cosine ? Math.Cos(angle) : Math.Sin(angle);
}
stages.Add(stage);
}
return [.. stages];
}
// Radix-2 decimation-in-time complex FFT, in place, no normalisation.
private static void Fft(double[] re, double[] im)
{
for (var i = 0; i < FrameSize; i++)
{
var j = BitReversal[i];
if (i < j)
{
(re[i], re[j]) = (re[j], re[i]);
(im[i], im[j]) = (im[j], im[i]);
}
}
var stage = 0;
for (var len = 2; len <= FrameSize; len <<= 1, stage++)
{
var half = len / 2;
var wr = TwiddleReal[stage];
var wi = TwiddleImag[stage];
for (var start = 0; start < FrameSize; start += len)
{
for (var j = 0; j < half; j++)
{
var a = start + j;
var b = a + half;
var tr = (re[b] * wr[j]) - (im[b] * wi[j]);
var ti = (re[b] * wi[j]) + (im[b] * wr[j]);
re[b] = re[a] - tr;
im[b] = im[a] - ti;
re[a] += tr;
im[a] += ti;
}
}
}
}
}
Jellyfin.Plugin.JRay/Services/AudioSignatureService.cs
+285
View File
@@ -0,0 +1,285 @@
using System;
using System.Buffers.Binary;
using System.Diagnostics;
using System.Globalization;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using MediaBrowser.Controller.MediaEncoding;
using Microsoft.Extensions.Logging;
namespace Jellyfin.Plugin.JRay.Services;
/// <summary>
/// Computes the v1 audio signature for a media file, decoding its centre window
/// with the FFmpeg binary Jellyfin already ships.
/// </summary>
/// <remarks>
/// <b>No new dependency.</b> FFmpeg performs decode, downmix and resample; the
/// plugin adds only the fixed FFT and bin-peak extraction in
/// <see cref="AudioSignature"/>. The binary is reached through
/// <see cref="IMediaEncoder.EncoderPath"/>, so an installation that can
/// transcode can compute signatures, with nothing further to install and no
/// second copy of FFmpeg to keep in step.
/// <para>
/// The pipeline computes the same signature for files it processes locally
/// (extraction <c>IR-004</c>); this exists for the files it never sees. Both
/// producers must therefore agree exactly, including on the decode: the command
/// below is the CLI spelling of what the pipeline asks libswresample for — best
/// audio stream, mono, 11025 Hz, 32-bit float — and the golden fixture pins the
/// decoded PCM as well as the signature, so a codec-level divergence is
/// distinguishable from a DSP-level one (JR-043).
/// </para>
/// <para>
/// <b>Every failure degrades to no signature rather than to an error.</b> A
/// signature is an enhancement to cut matching; a missing one costs a tier, and
/// must never be able to break a fetch.
/// </para>
/// </remarks>
// TRACES: JR-042 | SR-003
public class AudioSignatureService
{
private readonly IMediaEncoder _encoder;
private readonly ILogger<AudioSignatureService> _logger;
/// <summary>
/// Initializes a new instance of the <see cref="AudioSignatureService"/> class.
/// </summary>
/// <param name="encoder">Supplies the path of the FFmpeg binary Jellyfin ships.</param>
/// <param name="logger">Logger.</param>
public AudioSignatureService(IMediaEncoder encoder, ILogger<AudioSignatureService> logger)
{
_encoder = encoder;
_logger = logger;
}
/// <summary>
/// Computes the signature of the 120 s window centred on the media's
/// midpoint.
/// </summary>
/// <remarks>
/// The centre is used because the head and tail are the least
/// content-specific parts of a release: logos and cold opens at one end,
/// credits at the other.
/// <para>
/// Media shorter than <see cref="AudioSignature.WindowSec"/> yields
/// <c>null</c>: the window underflows, so there is no signature — the
/// identical rule the extraction producer applies, since diverging here
/// would break exactly the short items most likely to be misidentified
/// (JR-044, whose remaining half — applying no sync offset — belongs with
/// signature matching).
/// </para>
/// </remarks>
/// <param name="path">Path of the media file.</param>
/// <param name="runtimeSeconds">The item's runtime, as Jellyfin knows it.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>
/// The <c>v1:</c>-prefixed signature, or <c>null</c> for short media, media
/// with no usable audio, and any decode failure.
/// </returns>
public Task<string?> ComputeAsync(
string path,
double runtimeSeconds,
CancellationToken cancellationToken)
=> ComputeWithEncoderAsync(_encoder?.EncoderPath, path, runtimeSeconds, _logger, cancellationToken);
/// <summary>
/// <see cref="ComputeAsync"/> with the FFmpeg binary named explicitly.
/// </summary>
/// <remarks>
/// Internal so the golden-fixture test can drive the real decode path with
/// whatever FFmpeg the machine has, rather than standing up a fake
/// <see cref="IMediaEncoder"/> — a stub of a thirty-member interface would
/// be the larger risk of the two, and it is the decode that is under test.
/// </remarks>
/// <param name="encoderPath">Path of the FFmpeg binary to run.</param>
/// <param name="path">Path of the media file.</param>
/// <param name="runtimeSeconds">The item's runtime, as Jellyfin knows it.</param>
/// <param name="logger">Logger.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The <c>v1:</c>-prefixed signature, or <c>null</c>.</returns>
internal static async Task<string?> ComputeWithEncoderAsync(
string? encoderPath,
string path,
double runtimeSeconds,
ILogger logger,
CancellationToken cancellationToken)
{
if (string.IsNullOrEmpty(path) || runtimeSeconds < AudioSignature.WindowSec)
{
return null;
}
if (string.IsNullOrEmpty(encoderPath))
{
logger.LogDebug("No FFmpeg binary available; skipping the audio signature for {Path}", path);
return null;
}
try
{
var samples = await DecodeCentreWindowAsync(encoderPath, path, runtimeSeconds, logger, cancellationToken)
.ConfigureAwait(false);
if (samples is null)
{
return null;
}
return AudioSignature.FromMonoSamples(samples);
}
catch (OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
logger.LogDebug(ex, "Audio signature failed for {Path}; continuing without one", path);
return null;
}
}
/// <summary>
/// Decodes the centre window as mono 32-bit float PCM at 11025 Hz.
/// </summary>
/// <remarks>
/// The stream is truncated to exactly
/// <see cref="AudioSignature.WindowSamples"/> samples, so the frame count is
/// the same for every input rather than wobbling with seek granularity or a
/// resampler tail.
/// <para>
/// No <c>-map</c> is given: FFmpeg's default audio selection is the same
/// "best stream" choice the pipeline makes with
/// <c>av_find_best_stream</c>, and naming <c>0:a:0</c> instead would pick a
/// different track from the pipeline's on any file whose first audio stream
/// is not its main one — a commentary track, say.
/// </para>
/// </remarks>
private static async Task<float[]?> DecodeCentreWindowAsync(
string encoderPath,
string path,
double runtimeSeconds,
ILogger logger,
CancellationToken cancellationToken)
{
var start = (runtimeSeconds / 2.0) - (AudioSignature.WindowSec / 2.0);
if (start < 0.0)
{
start = 0.0;
}
var startArgument = start.ToString("0.000", CultureInfo.InvariantCulture);
var windowArgument = AudioSignature.WindowSec.ToString("0.000", CultureInfo.InvariantCulture);
var rateArgument = AudioSignature.SampleRate.ToString(CultureInfo.InvariantCulture);
var startInfo = new ProcessStartInfo
{
FileName = encoderPath,
RedirectStandardOutput = true,
RedirectStandardError = true,
RedirectStandardInput = false,
UseShellExecute = false,
CreateNoWindow = true,
};
startInfo.ArgumentList.Add("-nostdin");
startInfo.ArgumentList.Add("-v");
startInfo.ArgumentList.Add("error");
// Input seeking, so FFmpeg does not decode the whole file to reach the
// middle of it. Accurate by default: it seeks to the preceding keyframe
// and discards the excess, which is what the pipeline does by hand.
startInfo.ArgumentList.Add("-ss");
startInfo.ArgumentList.Add(startArgument);
startInfo.ArgumentList.Add("-i");
startInfo.ArgumentList.Add(path);
startInfo.ArgumentList.Add("-t");
startInfo.ArgumentList.Add(windowArgument);
startInfo.ArgumentList.Add("-vn");
startInfo.ArgumentList.Add("-sn");
startInfo.ArgumentList.Add("-dn");
startInfo.ArgumentList.Add("-ac");
startInfo.ArgumentList.Add("1");
startInfo.ArgumentList.Add("-ar");
startInfo.ArgumentList.Add(rateArgument);
startInfo.ArgumentList.Add("-f");
startInfo.ArgumentList.Add("f32le");
startInfo.ArgumentList.Add("-");
using var process = new Process { StartInfo = startInfo };
if (!process.Start())
{
return null;
}
try
{
// Read stderr concurrently: it is redirected, so leaving it unread
// would deadlock the moment FFmpeg filled the pipe.
var errorTask = process.StandardError.ReadToEndAsync(cancellationToken);
var bytes = await ReadWindowAsync(process.StandardOutput.BaseStream, cancellationToken)
.ConfigureAwait(false);
var error = await errorTask.ConfigureAwait(false);
if (bytes.Length < AudioSignature.FrameSize * sizeof(float))
{
// No audio stream, an unreadable file, or a runtime Jellyfin
// knows but the container does not support seeking into.
logger.LogDebug(
"FFmpeg returned {Bytes} bytes of audio for {Path}: {Error}",
bytes.Length,
path,
error);
return null;
}
var samples = new float[bytes.Length / sizeof(float)];
for (var i = 0; i < samples.Length; i++)
{
samples[i] = BinaryPrimitives.ReadSingleLittleEndian(bytes.AsSpan(i * sizeof(float)));
}
return samples;
}
finally
{
if (!process.HasExited)
{
try
{
process.Kill(entireProcessTree: true);
}
catch (InvalidOperationException)
{
// Exited between the check and the kill.
}
}
}
}
private static async Task<byte[]> ReadWindowAsync(Stream stream, CancellationToken cancellationToken)
{
var wanted = AudioSignature.WindowSamples * sizeof(float);
var buffer = new byte[wanted];
var filled = 0;
while (filled < wanted)
{
var read = await stream.ReadAsync(buffer.AsMemory(filled, wanted - filled), cancellationToken)
.ConfigureAwait(false);
if (read == 0)
{
break;
}
filled += read;
}
if (filled == wanted)
{
return buffer;
}
var truncated = new byte[filled];
Array.Copy(buffer, truncated, filled);
return truncated;
}
}