dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: deploy from 0bd2747

Browse Source
This commit is contained in:
dtourolle
2026-07-21 08:56:42 +02:00
parent 36f75ba199
commit 34b1f2c58f
24 changed files with 2066 additions and 1897 deletions
Download Patch File Download Diff File Expand all files Collapse all files
index.html
+110 -108
View File
@@ -14,7 +14,7 @@
<link rel="next" href="best-model/">
<link rel="next" href="methodology/">
@@ -243,6 +243,25 @@
<li class="md-tabs__item">
<a href="methodology/" class="md-tabs__link">
How We Score Against X-Ray
</a>
</li>
<li class="md-tabs__item">
@@ -270,26 +289,7 @@
Model Bake-off & Re-tune (full log)
</a>
</li>
<li class="md-tabs__item">
<a href="optimizer-experiments/" class="md-tabs__link">
Optimizer Experiments (prior round)
Full Experiment Log
</a>
</li>
@@ -427,10 +427,10 @@
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#start-here-four-questions-this-bake-off-answers" class="md-nav__link">
<a href="#findings" class="md-nav__link">
<span class="md-ellipsis">
Start here — four questions this bake-off answers
Findings
</span>
</a>
@@ -438,10 +438,10 @@
</li>
<li class="md-nav__item">
<a href="#the-full-technical-log" class="md-nav__link">
<a href="#full-experiment-log" class="md-nav__link">
<span class="md-ellipsis">
The full technical log
Full experiment log
</span>
</a>
@@ -473,6 +473,33 @@
<li class="md-nav__item">
<a href="methodology/" class="md-nav__link">
<span class="md-ellipsis">
How We Score Against X-Ray
</span>
</a>
</li>
@@ -487,10 +514,10 @@
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" >
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
@@ -508,8 +535,8 @@
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
@@ -652,34 +679,7 @@
<span class="md-ellipsis">
Model Bake-off & Re-tune (full log)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="optimizer-experiments/" class="md-nav__link">
<span class="md-ellipsis">
Optimizer Experiments (prior round)
Full Experiment Log
@@ -746,10 +746,10 @@
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#start-here-four-questions-this-bake-off-answers" class="md-nav__link">
<a href="#findings" class="md-nav__link">
<span class="md-ellipsis">
Start here — four questions this bake-off answers
Findings
</span>
</a>
@@ -757,10 +757,10 @@
</li>
<li class="md-nav__item">
<a href="#the-full-technical-log" class="md-nav__link">
<a href="#full-experiment-log" class="md-nav__link">
<span class="md-ellipsis">
The full technical log
Full experiment log
</span>
</a>
@@ -796,79 +796,81 @@
<h1 id="scene-actor-extraction">scene-actor-extraction<a class="headerlink" href="#scene-actor-extraction" title="Permanent link">&para;</a></h1>
<p>A face-recognition pipeline that finds when each actor appears on screen in a
film or TV episode — built on <a href="https://gitea.tourolle.paris/dtourolle/KPN">KPN++</a>
(a C++20 Kahn Process Network library) for the detect → track → match → scene
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.</p>
<p>This is a perfect X-Ray second, on a film the optimizer never saw:</p>
<p>A face-recognition pipeline that finds when each actor appears on screen in
a film or TV episode, built on <a href="https://gitea.tourolle.paris/dtourolle/KPN">KPN++</a>
(a C++20 Kahn Process Network library) for the detect, track, match, and
scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated
optimizer.</p>
<p>This is a correctly scored second from a held-out film, one the optimizer
never saw during tuning:</p>
<p><img alt="A perfect X-Ray second: three faces named at 100%, two more correctly carried off-screen" src="assets/images/lovelace_perfect_second.jpg" /></p>
<p>Every visible face named at 100% — Chris Noth, Hank Azaria, Bobby Cannavale —
the background extra honestly left unnamed, and the two credited cast without
a visible face correctly carried as present off-screen by the tracker's
presence windows. That's the pipeline exactly reproducing Amazon X-Ray's
record for this second.</p>
<p>It doesn't always go like that: the hardest held-out film scores 46% F1, and
the report is honest about <em>why</em> — one tunable trade (extinction bridging at
hard cuts), one structural ceiling (X-Ray credits people whose faces never
appear), and a few cases where the pipeline is right and X-Ray is wrong. The
evidence for all of it is in the pages below.</p>
<h2 id="start-here-four-questions-this-bake-off-answers">Start here — four questions this bake-off answers<a class="headerlink" href="#start-here-four-questions-this-bake-off-answers" title="Permanent link">&para;</a></h2>
<p>Every visible face is named at 100% confidence (Chris Noth, Hank Azaria,
Bobby Cannavale), the background extra is correctly left unnamed, and the
two credited cast members without a visible face are correctly reported
present but not visible. This matches Amazon X-Ray's own record for this
second exactly.</p>
<p>Results are not uniform across films. The hardest held-out film scores 46%
F1. This report documents why: one tunable trade (extinction bridging at
hard cuts), one structural limit (X-Ray credits people whose faces never
appear on screen), and a small number of cases where the pipeline is
correct and X-Ray's ground truth is not. Read
<a href="methodology/">how we score against X-Ray</a> first. X-Ray's ground truth is
scene-level; the pipeline's output is per-second. That difference shapes
every finding below.</p>
<h2 id="findings">Findings<a class="headerlink" href="#findings" title="Permanent link">&para;</a></h2>
<div class="grid cards">
<ul>
<li>
<p><span class="twemoji lg middle"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M18 2c-.9 0-2 1-2 2H8c0-1-1.1-2-2-2H2v9c0 1 1 2 2 2h2.2c.4 2 1.7 3.7 4.8 4v2.08C8 19.54 8 22 8 22h8s0-2.46-3-2.92V17c3.1-.3 4.4-2 4.8-4H20c1 0 2-1 2-2V2zM6 11H4V4h2zm14 0h-2V4h2z"/></svg></span> <strong><a href="best-model/">Which model is best?</a></strong></p>
<hr />
<p>Calibration curves first (discriminative power, independent of any
threshold), then F1 on the actual benchmark. LVFace-B Glint360K wins
both.</p>
<p>Calibration curves first, independent of any threshold, then held-out
F1 across three models. LVFace-B Glint360K wins both, and wins on every
held-out film.</p>
</li>
<li>
<p><span class="twemoji lg middle"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M14 12v7.88c.04.3-.06.62-.29.83a.996.996 0 0 1-1.41 0l-2.01-2.01a.99.99 0 0 1-.29-.83V12h-.03L4.21 4.62a1 1 0 0 1 .17-1.4c.19-.14.4-.22.62-.22h14c.22 0 .43.08.62.22a1 1 0 0 1 .17 1.4L14.03 12z"/></svg></span> <strong><a href="gallery-scope/">Whole vs. cast-restricted gallery</a></strong></p>
<hr />
<p>Restricting the matcher to a film's credited cast is a clean win on
every axis (+3.3pp F1, less than a third the misIDs) — but isn't a
shipped runtime feature yet.</p>
<p>Restricting the matcher to a film's credited cast improves F1,
recall, and misID rate at once, but is not a shipped runtime feature
yet.</p>
</li>
<li>
<p><span class="twemoji lg middle"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m12 0-.66.03 3.81 3.81L16.5 2.5c3.25 1.57 5.59 4.74 5.95 8.5h1.5C23.44 4.84 18.29 0 12 0m0 4c-1.93 0-3.5 1.57-3.5 3.5S10.07 11 12 11s3.5-1.57 3.5-3.5S13.93 4 12 4M.05 13C.56 19.16 5.71 24 12 24l.66-.03-3.81-3.81L7.5 21.5c-3.25-1.56-5.59-4.74-5.95-8.5zM12 13c-3.87 0-7 1.57-7 3.5V18h14v-1.5c0-1.93-3.13-3.5-7-3.5"/></svg></span> <strong><a href="pose-expansion/">Does pose expansion help?</a></strong></p>
<hr />
<p>A convincing training-set effect that didn't reproduce on 5 held-out
films once two methodology bugs were caught and fixed. An honest null
result, not a forced narrative.</p>
<p>A training-set effect that did not reproduce on 5 held-out films once
two methodology bugs in the comparison harness were found and fixed.</p>
</li>
<li>
<p><span class="twemoji lg middle"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M18 16h-.58l-.81-.81A7.07 7.07 0 0 0 18 11c0-3.87-3.13-7-7-7-1.5 0-3 .5-4.21 1.4-3.09 2.32-3.72 6.71-1.4 9.8s6.71 3.72 9.8 1.4l.81.81V18l5 5 2-2zm-7 0c-2.76 0-5-2.24-5-5s2.24-5 5-5 5 2.24 5 5-2.24 5-5 5M3 6 1 8V1h7L6 3H3zm18-5v7l-2-2V3h-3l-2-2zM6 19l2 2H1v-7l2 2v3z"/></svg></span> <strong><a href="lvface-deep-dive/">Deep dive: LVFace-B Glint360K</a></strong></p>
<hr />
<p>The held-out generalization gap, how the error budget decomposes
(extinction bridging at hard cuts, X-Ray's scene-membership vs.
on-screen-face ceiling), and the frames where the pipeline is right
and the ground truth is wrong.</p>
<p>The held-out generalization gap, the two mechanisms behind its errors,
and every distinct case where it names someone outside the film's
credited cast.</p>
</li>
</ul>
</div>
<h2 id="the-full-technical-log">The full technical log<a class="headerlink" href="#the-full-technical-log" title="Permanent link">&para;</a></h2>
<h2 id="full-experiment-log">Full experiment log<a class="headerlink" href="#full-experiment-log" title="Permanent link">&para;</a></h2>
<ul>
<li><strong><a href="model-bakeoff/">Model bake-off + threshold re-tune</a></strong> —
the complete experiment log behind the four pages above: the ROCm teardown
deadlock root cause and fix, DE concurrency tuning, the full 16-combo
results table, and every caveat. This is where the shipped
<a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/4b5557974bef8783bacc375c0869e8f589d1b0a3/src/config.hpp"><code>src/config.hpp</code></a> defaults come from.</li>
<li><strong><a href="optimizer-experiments/">Optimizer experiments (prior round)</a></strong> — the
earlier scene-union-metric tuning pass, superseded by the per-second metric
used in the bake-off but kept for the ground-truth/architecture background.</li>
<li><strong><a href="service-conversion/">Service conversion (proposal)</a></strong> — design sketch
for a native idle-GPU worker gated on screen lock, not yet built.</li>
<li><strong><a href="model-bakeoff/">Full experiment log</a></strong>: the complete log behind the
four pages above, including how replaying against cached embeddings
inside the same KPN network makes a full model and configuration
comparison practical, the full results table, and every caveat. This is
where the shipped <a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/0bd27470698c45cab21935d636a04512360e1008/src/config.hpp"><code>src/config.hpp</code></a>
defaults come from.</li>
<li><strong><a href="service-conversion/">Service conversion (proposal)</a></strong>: design
sketch for a native idle-GPU worker gated on screen lock, not yet built.</li>
</ul>
<h2 id="reproducing-the-benchmarks">Reproducing the benchmarks<a class="headerlink" href="#reproducing-the-benchmarks" title="Permanent link">&para;</a></h2>
<p>Gallery <code>.h5</code> files, embedding dumps, the X-Ray corpus, montage frame images,
and DE trajectories are not committed to this repository — they're pushed to
the Gitea package registry and pulled on demand:</p>
<p>Gallery <code>.h5</code> files, embedding dumps, the X-Ray corpus, montage frame
images, and DE trajectories are not committed to this repository. They are
pushed to the Gitea package registry and pulled on demand:</p>
<div class="language-bash highlight"><pre><span></span><code><span id="__span-0-1"><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a>scripts/artifacts/pull_artifacts.sh<span class="w"> </span>galleries
</span><span id="__span-0-2"><a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a>scripts/artifacts/pull_artifacts.sh<span class="w"> </span>experiment-data
</span><span id="__span-0-3"><a id="__codelineno-0-3" name="__codelineno-0-3" href="#__codelineno-0-3"></a>scripts/artifacts/pull_artifacts.sh<span class="w"> </span>montage-frames<span class="w"> </span>&lt;film-slug&gt;
</span></code></pre></div>
<p>See <a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/4b5557974bef8783bacc375c0869e8f589d1b0a3/scripts/artifacts/push_artifacts.sh"><code>scripts/artifacts/push_artifacts.sh</code></a>
for the upload side (requires a <code>GITEA_TOKEN</code> with package write scope).</p>
<p>See <a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/0bd27470698c45cab21935d636a04512360e1008/scripts/artifacts/push_artifacts.sh"><code>scripts/artifacts/push_artifacts.sh</code></a>
for the upload side, which requires a <code>GITEA_TOKEN</code> with package write
scope.</p>
@@ -905,13 +907,13 @@ for the upload side (requires a <code>GITEA_TOKEN</code> with package write scop
<a href="best-model/" class="md-footer__link md-footer__link--next" aria-label="Next: Best Model">
<a href="methodology/" class="md-footer__link md-footer__link--next" aria-label="Next: How We Score Against X-Ray">
<div class="md-footer__title">
<span class="md-footer__direction">
Next
</span>
<div class="md-ellipsis">
Best Model
How We Score Against X-Ray
</div>
</div>
<div class="md-footer__button md-icon">