docs: deploy from 0bd2747
This commit is contained in:
+236
-118
@@ -13,7 +13,7 @@
|
||||
<link rel="canonical" href="https://pages.tourolle.paris/dtourolle/scene-actor-extraction/best-model/">
|
||||
|
||||
|
||||
<link rel="prev" href="..">
|
||||
<link rel="prev" href="../methodology/">
|
||||
|
||||
|
||||
<link rel="next" href="../gallery-scope/">
|
||||
@@ -242,6 +242,25 @@
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-tabs__item">
|
||||
<a href="../methodology/" class="md-tabs__link">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
How We Score Against X-Ray
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -272,26 +291,7 @@
|
||||
|
||||
|
||||
|
||||
Model Bake-off & Re-tune (full log)
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-tabs__item">
|
||||
<a href="../optimizer-experiments/" class="md-tabs__link">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Optimizer Experiments (prior round)
|
||||
Full Experiment Log
|
||||
|
||||
</a>
|
||||
</li>
|
||||
@@ -393,6 +393,33 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../methodology/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
How We Score Against X-Ray
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -414,10 +441,10 @@
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" checked>
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="">
|
||||
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="">
|
||||
|
||||
|
||||
|
||||
@@ -435,8 +462,8 @@
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="true">
|
||||
<label class="md-nav__title" for="__nav_2">
|
||||
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
|
||||
<label class="md-nav__title" for="__nav_3">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
@@ -524,10 +551,10 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#second-signal-f1-on-the-actual-benchmark" class="md-nav__link">
|
||||
<a href="#second-signal-held-out-f1" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Second signal: F1 on the actual benchmark
|
||||
Second signal: held-out F1
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -535,10 +562,21 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#caveat-model-choice-is-an-operational-change" class="md-nav__link">
|
||||
<a href="#full-training-matrix-picture" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Caveat: model choice is an operational change
|
||||
Full training-matrix picture
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#operational-note" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Operational note
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -659,34 +697,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Model Bake-off & Re-tune (full log)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../optimizer-experiments/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Optimizer Experiments (prior round)
|
||||
Full Experiment Log
|
||||
|
||||
|
||||
|
||||
@@ -764,10 +775,10 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#second-signal-f1-on-the-actual-benchmark" class="md-nav__link">
|
||||
<a href="#second-signal-held-out-f1" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Second signal: F1 on the actual benchmark
|
||||
Second signal: held-out F1
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -775,10 +786,21 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#caveat-model-choice-is-an-operational-change" class="md-nav__link">
|
||||
<a href="#full-training-matrix-picture" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Caveat: model choice is an operational change
|
||||
Full training-matrix picture
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#operational-note" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Operational note
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -803,31 +825,35 @@
|
||||
|
||||
|
||||
<h1 id="which-embedding-model-is-best">Which embedding model is best?<a class="headerlink" href="#which-embedding-model-is-best" title="Permanent link">¶</a></h1>
|
||||
<p>Four candidates went into the bake-off: three ArcFace variants (w600k-R50,
|
||||
R18, w600k-MBF) and LVFace-B (Glint360K), a Vision-Transformer embedder that's
|
||||
a drop-in replacement for ArcFace's <code>[N,3,112,112]</code> input / 512-d output. The
|
||||
open question: is LVFace (455MB) actually better, or just the biggest?</p>
|
||||
<p>Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
|
||||
455MB) were compared. r50 is excluded from the training/held-out comparison
|
||||
below; its gallery has roughly 30% fewer reference images per actor than the
|
||||
other three on the identical source photos, which confounds a direct score
|
||||
comparison (see <a href="../model-bakeoff/">the full experiment log</a> for detail). It
|
||||
remains in the calibration comparison, which does not depend on the gallery
|
||||
image count.</p>
|
||||
<h2 id="first-signal-calibration-curves">First signal: calibration curves<a class="headerlink" href="#first-signal-calibration-curves" title="Permanent link">¶</a></h2>
|
||||
<p>Each gallery carries a fitted Platt sigmoid <code>P(match | cosine similarity) =
|
||||
σ(a·sim + b)</code>, embedded directly in the gallery's HDF5 file
|
||||
(<a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/4b5557974bef8783bacc375c0869e8f589d1b0a3/src/gallery/gallery_calibration.hpp"><code>src/gallery/gallery_calibration.hpp</code></a>). This is a property of the embedding
|
||||
space alone — computed from intra/inter-actor reference-image pairs, no
|
||||
tracking or scene logic involved — so it's a clean first read on discriminative
|
||||
power before running a single benchmark.</p>
|
||||
σ(a·sim + b)</code>, stored directly in the gallery HDF5
|
||||
(<a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/0bd27470698c45cab21935d636a04512360e1008/src/gallery/gallery_calibration.hpp"><code>src/gallery/gallery_calibration.hpp</code></a>).
|
||||
This is a property of the embedding space alone, computed from intra- and
|
||||
inter-actor reference-image pairs with no tracking or scene logic involved,
|
||||
so it is a clean first read on discriminative power before running a
|
||||
benchmark.</p>
|
||||
<p><img alt="Calibrated P(match|similarity) for all four models" src="../assets/images/calibration_curves.png" /></p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>model</th>
|
||||
<th><code>a</code> (steepness)</th>
|
||||
<th>a (steepness)</th>
|
||||
<th>boundary at P=0.5</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><strong>LVFace-B Glint360K</strong></td>
|
||||
<td><strong>17.7</strong></td>
|
||||
<td><strong>sim 0.228</strong></td>
|
||||
<td>LVFace-B Glint360K</td>
|
||||
<td>17.7</td>
|
||||
<td>sim 0.228</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>ArcFace w600k-MBF</td>
|
||||
@@ -846,13 +872,123 @@ power before running a single benchmark.</p>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>LVFace has both the steepest transition and the lowest decision boundary — it
|
||||
separates same-actor from different-actor reference pairs more confidently, at
|
||||
a <em>lower</em> similarity threshold, than any ArcFace variant. That's a genuine
|
||||
head start before the tracking/scoring pipeline is even involved.</p>
|
||||
<h2 id="second-signal-f1-on-the-actual-benchmark">Second signal: F1 on the actual benchmark<a class="headerlink" href="#second-signal-f1-on-the-actual-benchmark" title="Permanent link">¶</a></h2>
|
||||
<p>Best full-gallery (no cast-restriction) result per model, from the 16-combo
|
||||
bake-off matrix (<a href="../model-bakeoff/">full experiment log</a>):</p>
|
||||
<p>LVFace has both the steepest transition and the lowest decision boundary,
|
||||
separating same-actor from different-actor reference pairs more confidently
|
||||
at a lower similarity than any ArcFace variant.</p>
|
||||
<h2 id="second-signal-held-out-f1">Second signal: held-out F1<a class="headerlink" href="#second-signal-held-out-f1" title="Permanent link">¶</a></h2>
|
||||
<p>Each model's own tuned <code>full_exp</code> config, replayed against the 5 films the
|
||||
optimizer never saw and scored the same way:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>film</th>
|
||||
<th>LVFace F1</th>
|
||||
<th>mbf F1</th>
|
||||
<th>r18 F1</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Benny & Joon</td>
|
||||
<td>83.0%</td>
|
||||
<td>78.5%</td>
|
||||
<td>77.1%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Lovelace</td>
|
||||
<td>77.5%</td>
|
||||
<td>73.7%</td>
|
||||
<td>72.2%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Valerian and the City of a Thousand Planets</td>
|
||||
<td>74.1%</td>
|
||||
<td>70.2%</td>
|
||||
<td>71.0%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Downton Abbey: A New Era</td>
|
||||
<td>56.2%</td>
|
||||
<td>55.0%</td>
|
||||
<td>53.0%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>The Many Saints of Newark</td>
|
||||
<td>46.3%</td>
|
||||
<td>44.5%</td>
|
||||
<td>42.1%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>macro average</strong></td>
|
||||
<td><strong>67.4%</strong></td>
|
||||
<td><strong>64.4%</strong></td>
|
||||
<td><strong>63.1%</strong></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>LVFace scores highest on all 5 held-out films; the ranking never flips
|
||||
between models. Total misID count across the 5 films: LVFace 1032, mbf
|
||||
2197, r18 1224. LVFace has less than half mbf's misID total and still
|
||||
scores higher on every film.</p>
|
||||
<p>Held-out results are stronger evidence than training results, because
|
||||
training numbers can reflect what the optimizer was tuned to fit rather
|
||||
than general performance. On training data, the ordering is not as clean:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>film</th>
|
||||
<th>LVFace F1</th>
|
||||
<th>mbf F1</th>
|
||||
<th>r18 F1</th>
|
||||
<th>best</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Café Society</td>
|
||||
<td>68.1%</td>
|
||||
<td>62.2%</td>
|
||||
<td>60.1%</td>
|
||||
<td>LVFace</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Lord of War</td>
|
||||
<td>75.6%</td>
|
||||
<td>77.2%</td>
|
||||
<td>75.6%</td>
|
||||
<td>mbf</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Scarface</td>
|
||||
<td>71.5%</td>
|
||||
<td>68.6%</td>
|
||||
<td>64.1%</td>
|
||||
<td>LVFace</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Sound of Metal</td>
|
||||
<td>78.8%</td>
|
||||
<td>76.5%</td>
|
||||
<td>71.6%</td>
|
||||
<td>LVFace</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
|
||||
table where LVFace does not score highest. LVFace's training-set macro
|
||||
average (75.3%, see <a href="../model-bakeoff/">the full experiment log</a>) is not a
|
||||
uniform win across every film it contributes to; the held-out result, where
|
||||
LVFace wins all 5 films outright, is the stronger claim.</p>
|
||||
<p>This reverses an earlier, superseded benchmarking pass that used a
|
||||
scene-union metric and found the three models statistically
|
||||
indistinguishable (around 85% each), concluding LVFace was not worth its
|
||||
size. That metric masked out-of-cast false positives behind a
|
||||
gallery-intersect-cast recall filter; the per-second metric used here does
|
||||
not.</p>
|
||||
<h2 id="full-training-matrix-picture">Full training-matrix picture<a class="headerlink" href="#full-training-matrix-picture" title="Permanent link">¶</a></h2>
|
||||
<p><img alt="All 12 combos ranked by training-set F1" src="../assets/images/rep4_matrix_f1.png" /></p>
|
||||
<p>Best full-gallery combo per model (all three are <code>full_exp</code>), from the
|
||||
training matrix in <a href="../model-bakeoff/">the full experiment log</a>:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
@@ -865,18 +1001,18 @@ bake-off matrix (<a href="../model-bakeoff/">full experiment log</a>):</p>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><strong>LVFace-B Glint360K</strong></td>
|
||||
<td><strong>75.3%</strong></td>
|
||||
<td>LVFace-B Glint360K</td>
|
||||
<td>75.3%</td>
|
||||
<td>89.7%</td>
|
||||
<td><strong>65.4%</strong></td>
|
||||
<td>65.4%</td>
|
||||
<td>232</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>ArcFace w600k-MBF</td>
|
||||
<td>74.2%</td>
|
||||
<td>87.4%</td>
|
||||
<td>64.4%</td>
|
||||
<td>57</td>
|
||||
<td>72.0%</td>
|
||||
<td>87.7%</td>
|
||||
<td>61.4%</td>
|
||||
<td>240</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>ArcFace R18</td>
|
||||
@@ -885,36 +1021,18 @@ bake-off matrix (<a href="../model-bakeoff/">full experiment log</a>):</p>
|
||||
<td>57.7%</td>
|
||||
<td>242</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>ArcFace w600k-R50</td>
|
||||
<td>68.5%</td>
|
||||
<td>94.0%</td>
|
||||
<td>54.1%</td>
|
||||
<td>150</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>The full 16-combo picture makes the model ordering visible at a glance — LVFace
|
||||
(yellow) tops both the restricted and full columns, and R18 (green) props up
|
||||
the bottom of the full-gallery ranking:</p>
|
||||
<p><img alt="All 16 bake-off combos ranked by training-set F1" src="../assets/images/rep4_matrix_f1.png" /></p>
|
||||
<p>LVFace wins outright, with the highest recall of any full-mode combo. This
|
||||
reverses an earlier conclusion from a prior (superseded) benchmarking pass
|
||||
using a scene-union metric, which found the three models statistically
|
||||
indistinguishable (~85% each) and concluded LVFace wasn't worth its size — that
|
||||
metric hid out-of-cast false positives behind a gallery∩cast recall mask (see
|
||||
<a href="../optimizer-experiments/">the prior optimizer round</a>); the per-second metric
|
||||
used here does not.</p>
|
||||
<p>Held-out validation (5 films never seen by the optimizer) confirms LVFace's
|
||||
lead holds up out of sample — see the
|
||||
<a href="../lvface-deep-dive/">LVFace deep dive</a> for the full breakdown, including
|
||||
where it fails.</p>
|
||||
<h2 id="caveat-model-choice-is-an-operational-change">Caveat: model choice is an operational change<a class="headerlink" href="#caveat-model-choice-is-an-operational-change" title="Permanent link">¶</a></h2>
|
||||
<p>Switching the default embedder isn't just flipping a config value — the
|
||||
gallery itself is model-specific (embeddings from different models aren't
|
||||
comparable), so any existing gallery built against ArcFace w600k-R50 needs to
|
||||
be rebuilt from source images against LVFace before the new default takes
|
||||
effect. <a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/4b5557974bef8783bacc375c0869e8f589d1b0a3/scripts/optimizer/reembed_gallery.py"><code>scripts/optimizer/reembed_gallery.py</code></a>
|
||||
<p>LVFace leads within both the restricted and full gallery modes, visible
|
||||
directly in the chart above without reading the table. The three models'
|
||||
misID counts on the full gallery are nearly identical (232/240/242); LVFace's
|
||||
lead here is a precision-and-recall lead, not a misID one.</p>
|
||||
<h2 id="operational-note">Operational note<a class="headerlink" href="#operational-note" title="Permanent link">¶</a></h2>
|
||||
<p>Switching the default embedder is not a config change alone; the gallery
|
||||
is model-specific, since embeddings from different models are not
|
||||
comparable. Any existing gallery built against a different model must be
|
||||
rebuilt from source images before the new default takes effect.
|
||||
<a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/0bd27470698c45cab21935d636a04512360e1008/scripts/optimizer/reembed_gallery.py"><code>scripts/optimizer/reembed_gallery.py</code></a>
|
||||
does this from a reference gallery's cached source images without
|
||||
re-downloading anything.</p>
|
||||
|
||||
@@ -952,7 +1070,7 @@ re-downloading anything.</p>
|
||||
<nav class="md-footer__inner md-grid" aria-label="Footer" >
|
||||
|
||||
|
||||
<a href=".." class="md-footer__link md-footer__link--prev" aria-label="Previous: Home">
|
||||
<a href="../methodology/" class="md-footer__link md-footer__link--prev" aria-label="Previous: How We Score Against X-Ray">
|
||||
<div class="md-footer__button md-icon">
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11z"/></svg>
|
||||
@@ -962,7 +1080,7 @@ re-downloading anything.</p>
|
||||
Previous
|
||||
</span>
|
||||
<div class="md-ellipsis">
|
||||
Home
|
||||
How We Score Against X-Ray
|
||||
</div>
|
||||
</div>
|
||||
</a>
|
||||
|
||||
Reference in New Issue
Block a user