docs: deploy from 0bd2747
This commit is contained in:
+164
-154
@@ -80,7 +80,7 @@
|
||||
<div data-md-component="skip">
|
||||
|
||||
|
||||
<a href="#pose-expansion-does-learning-new-poses-mid-film-help" class="md-skip">
|
||||
<a href="#pose-expansion-does-promoting-new-poses-mid-film-help" class="md-skip">
|
||||
Skip to content
|
||||
</a>
|
||||
|
||||
@@ -242,6 +242,25 @@
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-tabs__item">
|
||||
<a href="../methodology/" class="md-tabs__link">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
How We Score Against X-Ray
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -272,26 +291,7 @@
|
||||
|
||||
|
||||
|
||||
Model Bake-off & Re-tune (full log)
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-tabs__item">
|
||||
<a href="../optimizer-experiments/" class="md-tabs__link">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Optimizer Experiments (prior round)
|
||||
Full Experiment Log
|
||||
|
||||
</a>
|
||||
</li>
|
||||
@@ -393,6 +393,33 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../methodology/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
How We Score Against X-Ray
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -414,10 +441,10 @@
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_2" checked>
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="">
|
||||
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="">
|
||||
|
||||
|
||||
|
||||
@@ -435,8 +462,8 @@
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="true">
|
||||
<label class="md-nav__title" for="__nav_2">
|
||||
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
|
||||
<label class="md-nav__title" for="__nav_3">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
@@ -569,10 +596,10 @@
|
||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-training-set-signal" class="md-nav__link">
|
||||
<a href="#training-set-signal" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The training-set signal
|
||||
Training-set signal
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -580,10 +607,10 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#held-out-test-does-it-reproduce" class="md-nav__link">
|
||||
<a href="#held-out-test" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Held-out test: does it reproduce?
|
||||
Held-out test
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -591,10 +618,10 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#two-bugs-this-required-catching-this-sections-own-methodology" class="md-nav__link">
|
||||
<a href="#two-methodology-bugs-caught-during-this-check" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Two bugs this required catching (this section's own methodology)
|
||||
Two methodology bugs caught during this check
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -602,10 +629,10 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#what-this-means" class="md-nav__link">
|
||||
<a href="#conclusion" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
What this means
|
||||
Conclusion
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -670,34 +697,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Model Bake-off & Re-tune (full log)
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../optimizer-experiments/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Optimizer Experiments (prior round)
|
||||
Full Experiment Log
|
||||
|
||||
|
||||
|
||||
@@ -764,10 +764,10 @@
|
||||
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-training-set-signal" class="md-nav__link">
|
||||
<a href="#training-set-signal" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The training-set signal
|
||||
Training-set signal
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -775,10 +775,10 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#held-out-test-does-it-reproduce" class="md-nav__link">
|
||||
<a href="#held-out-test" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Held-out test: does it reproduce?
|
||||
Held-out test
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -786,10 +786,10 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#two-bugs-this-required-catching-this-sections-own-methodology" class="md-nav__link">
|
||||
<a href="#two-methodology-bugs-caught-during-this-check" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Two bugs this required catching (this section's own methodology)
|
||||
Two methodology bugs caught during this check
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -797,10 +797,10 @@
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#what-this-means" class="md-nav__link">
|
||||
<a href="#conclusion" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
What this means
|
||||
Conclusion
|
||||
|
||||
</span>
|
||||
</a>
|
||||
@@ -824,16 +824,20 @@
|
||||
|
||||
|
||||
|
||||
<h1 id="pose-expansion-does-learning-new-poses-mid-film-help">Pose expansion: does "learning" new poses mid-film help?<a class="headerlink" href="#pose-expansion-does-learning-new-poses-mid-film-help" title="Permanent link">¶</a></h1>
|
||||
<p><code>expand_gallery</code> (<a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/4b5557974bef8783bacc375c0869e8f589d1b0a3/src/gallery/track_gallery.hpp"><code>src/gallery/track_gallery.hpp</code></a>)
|
||||
promotes a confidently-identified
|
||||
track's novel-pose reference views into a per-film, in-memory gallery annex — the
|
||||
idea being that once the pipeline is sure who someone is, a pose it hasn't seen
|
||||
before (turned head, different lighting) becomes a free extra reference for
|
||||
recognising that actor again later in the same film, without touching the baked
|
||||
gallery.</p>
|
||||
<h2 id="the-training-set-signal">The training-set signal<a class="headerlink" href="#the-training-set-signal" title="Permanent link">¶</a></h2>
|
||||
<p>Averaged across all 4 models, on the 4 films used for optimization:</p>
|
||||
<h1 id="pose-expansion-does-promoting-new-poses-mid-film-help">Pose expansion: does promoting new poses mid-film help?<a class="headerlink" href="#pose-expansion-does-promoting-new-poses-mid-film-help" title="Permanent link">¶</a></h1>
|
||||
<p><code>expand_gallery</code>
|
||||
(<a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/0bd27470698c45cab21935d636a04512360e1008/src/gallery/track_gallery.hpp"><code>src/gallery/track_gallery.hpp</code></a>)
|
||||
promotes a confidently identified track's novel-pose reference views into a
|
||||
per-film, in-memory gallery annex. The idea: once the pipeline is confident
|
||||
about an identity, a pose it has not seen before (turned head, different
|
||||
lighting) becomes an extra reference for recognizing that actor again later
|
||||
in the same film, without touching the baked gallery.</p>
|
||||
<h2 id="training-set-signal">Training-set signal<a class="headerlink" href="#training-set-signal" title="Permanent link">¶</a></h2>
|
||||
<p>Averaged across the 3 compared models (r50 excluded), on the 4 films used
|
||||
for optimization. These are the corrected, full-coverage figures, see the
|
||||
<a href="../model-bakeoff/#a-scoring-bug-worth-recording-dropped-film-evaluations">dropped-film note</a>
|
||||
in the experiment log for why an earlier version of this table overstated the
|
||||
full-mode misID jump (209 → 864) that was itself partly a truncation artifact:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
@@ -848,54 +852,54 @@ gallery.</p>
|
||||
<tr>
|
||||
<td>full</td>
|
||||
<td>off</td>
|
||||
<td>71.2%</td>
|
||||
<td>58.3%</td>
|
||||
<td>209</td>
|
||||
<td>70.0%</td>
|
||||
<td>57.6%</td>
|
||||
<td>407</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>full</td>
|
||||
<td><strong>on</strong></td>
|
||||
<td>71.2%</td>
|
||||
<td>59.7%</td>
|
||||
<td><strong>864</strong></td>
|
||||
<td>on</td>
|
||||
<td>72.1%</td>
|
||||
<td>61.5%</td>
|
||||
<td>714</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>restricted</td>
|
||||
<td>off</td>
|
||||
<td>73.6%</td>
|
||||
<td>61.3%</td>
|
||||
<td>194</td>
|
||||
<td>75.1%</td>
|
||||
<td>63.9%</td>
|
||||
<td>179</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>restricted</td>
|
||||
<td><strong>on</strong></td>
|
||||
<td><strong>75.4%</strong></td>
|
||||
<td><strong>64.5%</strong></td>
|
||||
<td>135</td>
|
||||
<td>on</td>
|
||||
<td>76.7%</td>
|
||||
<td>67.2%</td>
|
||||
<td>120</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>In <code>restricted</code> mode (matcher's candidate set capped to the film's own credited
|
||||
cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually
|
||||
lower. In <code>full</code> mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but
|
||||
misID roughly quadrupled (209 → 864) — see the
|
||||
<a href="../model-bakeoff/">bake-off experiment log</a> for the per-model breakdown. That's the number that motivated this page: <strong>does turning
|
||||
expansion on actually change what gets recognised, frame by frame, or is the
|
||||
aggregate F1 shift something else?</strong></p>
|
||||
<h2 id="held-out-test-does-it-reproduce">Held-out test: does it reproduce?<a class="headerlink" href="#held-out-test-does-it-reproduce" title="Permanent link">¶</a></h2>
|
||||
<p>Same model + same tuned config, <code>expand_gallery</code> toggled on vs. off, nothing else
|
||||
changed — full gallery mode, per-second scoring against X-Ray. This isolates
|
||||
expansion from every other variable (config, model, threshold) that differs
|
||||
between the training-set <code>exp</code>/<code>noexp</code> rows above.</p>
|
||||
<p><strong>LVFace-B Glint360K, all 5 held-out films</strong> (films never seen by the optimizer):</p>
|
||||
<p>In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp
|
||||
recall, lower misID. In full mode it looks like a recall-for-misID trade:
|
||||
+2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See
|
||||
<a href="../model-bakeoff/">the full experiment log</a> for the per-model breakdown.
|
||||
This asymmetry motivated the question below: does turning expansion on
|
||||
change what gets recognized frame by frame, or is the aggregate F1 shift
|
||||
coming from something else.</p>
|
||||
<h2 id="held-out-test">Held-out test<a class="headerlink" href="#held-out-test" title="Permanent link">¶</a></h2>
|
||||
<p>Same model, same tuned config, <code>expand_gallery</code> toggled on vs. off, nothing
|
||||
else changed, full gallery mode, per-second scoring against X-Ray. This
|
||||
isolates expansion from every other variable that differs between the
|
||||
training-set rows above.</p>
|
||||
<p>LVFace-B Glint360K, all 5 held-out films:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>film</th>
|
||||
<th>F1 (exp)</th>
|
||||
<th>F1 (noexp)</th>
|
||||
<th>TPI Δ</th>
|
||||
<th>FN Δ</th>
|
||||
<th>TPI delta</th>
|
||||
<th>FN delta</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
@@ -936,54 +940,60 @@ between the training-set <code>exp</code>/<code>noexp</code> rows above.</p>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong>ArcFace R18</strong> (Benny & Joon, r18's own tuned config): F1 77.1% for both, TPI/FN
|
||||
identical, FPI differs by 2 (noise).</p>
|
||||
<p><strong>Every film, both models tested: F1 within 0.1–0.2pp, TPI/FN swings in the tens
|
||||
out of tens of thousands.</strong> That's noise, not a signal — expansion made no
|
||||
measurable difference to per-second onscreen identification anywhere it was
|
||||
tested on unseen data.</p>
|
||||
<h2 id="two-bugs-this-required-catching-this-sections-own-methodology">Two bugs this required catching (this section's own methodology)<a class="headerlink" href="#two-bugs-this-required-catching-this-sections-own-methodology" title="Permanent link">¶</a></h2>
|
||||
<p>Getting to the clean table above took two wrong turns, both worth recording
|
||||
since they're exactly the kind of error that produces a false positive "look,
|
||||
expansion helped!" finding:</p>
|
||||
<p>ArcFace R18, Benny & Joon, r18's own tuned config: F1 77.1% for both, TPI
|
||||
and FN identical, FPI differs by 2.</p>
|
||||
<p>Every film, both models tested: F1 differs by 0.1-0.2pp, TPI/FN swings are
|
||||
in the tens out of tens of thousands. This is noise, not a signal.
|
||||
Expansion made no measurable difference to per-second on-screen
|
||||
identification on any held-out film tested.</p>
|
||||
<h2 id="two-methodology-bugs-caught-during-this-check">Two methodology bugs caught during this check<a class="headerlink" href="#two-methodology-bugs-caught-during-this-check" title="Permanent link">¶</a></h2>
|
||||
<p>Getting to the table above required catching two wrong turns, both worth
|
||||
recording because they are exactly the kind of error that produces a false
|
||||
positive "expansion helped" finding.</p>
|
||||
<ol>
|
||||
<li><strong>Timeout truncation.</strong> The first Downton Abbey <code>exp</code> replay was cut off by a
|
||||
60s subprocess timeout at ~76% through the film (5589 of 7368 expected
|
||||
seconds) — a genuinely large, silent data loss that showed up as a large,
|
||||
convincing-looking TPI gap (47938 vs 52032) purely because one run had a
|
||||
quarter of the film missing. Caught by comparing <code>n_seconds</code> between runs
|
||||
before trusting any score delta; fixed by re-running with a longer timeout.</li>
|
||||
<li><strong>Bbox-matching bug.</strong> An early per-second raw-annotation diff matched each
|
||||
<code>exp</code> detection to the <em>first</em> <code>noexp</code> detection with IoU > 0.5, not the
|
||||
<em>best</em>-overlapping one. With 3 faces close together in frame, this produced
|
||||
spurious "disagreements" (e.g. "exp says Aidan Quinn, noexp says Johnny
|
||||
Depp" at the same seconds) that vanished entirely once the match picked the
|
||||
true best-IoU candidate — both configs had actually output the exact same
|
||||
three names at the exact same three boxes.</li>
|
||||
<li><strong>Timeout truncation.</strong> The first Downton Abbey <code>exp</code> replay was cut off
|
||||
by a 60-second subprocess timeout at about 76% through the film (5589 of
|
||||
7368 expected seconds). This silent data loss produced a large,
|
||||
convincing-looking TPI gap (47938 vs 52032) purely because one run was
|
||||
missing a quarter of the film. Caught by comparing <code>n_seconds</code> between
|
||||
runs before trusting any score delta; fixed by re-running with a longer
|
||||
timeout.</li>
|
||||
<li><strong>Bbox-matching bug.</strong> An early per-second raw-annotation diff matched
|
||||
each <code>exp</code> detection to the first <code>noexp</code> detection with IoU above 0.5,
|
||||
not the best-overlapping one. With 3 faces close together in frame, this
|
||||
produced spurious disagreements (for example "exp says Aidan Quinn,
|
||||
noexp says Johnny Depp" at the same second) that vanished once the match
|
||||
used the best-IoU candidate instead of the first one. Both configs had
|
||||
actually output the same three names at the same three boxes.</li>
|
||||
</ol>
|
||||
<p>Both bugs independently pointed toward "expansion is doing something," and both
|
||||
were artifacts of the comparison harness, not the pipeline. Worth remembering
|
||||
when a before/after diff looks dramatic: check that the two runs actually cover
|
||||
the same seconds, and match entities by best overlap, not first-found.</p>
|
||||
<h2 id="what-this-means">What this means<a class="headerlink" href="#what-this-means" title="Permanent link">¶</a></h2>
|
||||
<p>The training-set aggregate effect (particularly the ~4x misID increase in full
|
||||
mode) doesn't reproduce on held-out data — at minimum it's far smaller than the
|
||||
training-set numbers suggested, and plausibly it's sampling variation from only
|
||||
4 training films rather than a real, generalizable mechanism. This doesn't mean
|
||||
<code>expand_gallery</code> never does anything (the mechanism is real — see
|
||||
<a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/4b5557974bef8783bacc375c0869e8f589d1b0a3/src/gallery/track_gallery.hpp"><code>track_gallery.hpp</code></a>'s
|
||||
promotion logging: tracks <em>do</em> get confirmed and views <em>do</em>
|
||||
get promoted into the annex on every film tested), only that <strong>whatever effect
|
||||
it has on final per-second identification was too small to detect against 5
|
||||
held-out films</strong> with this scoring method. A cleaner test would need either many
|
||||
more held-out films or a metric that can see the annex's direct contribution
|
||||
(e.g. tagging which reference embedding won each match), neither of which this
|
||||
pass had budget for.</p>
|
||||
<p><strong>Practical takeaway</strong>: don't treat the training-set <code>exp</code> vs <code>noexp</code> numbers in
|
||||
the <a href="../model-bakeoff/">bake-off experiment log</a> as proof that expansion
|
||||
changes real-world behavior
|
||||
in either direction — on the evidence gathered so far, it doesn't move the
|
||||
needle enough to see.</p>
|
||||
<p>Both bugs independently pointed toward "expansion is doing something," and
|
||||
both were artifacts of the comparison harness, not the pipeline. Before
|
||||
trusting a dramatic before/after diff, check that both runs cover the same
|
||||
seconds and that entities are matched by best overlap, not first found.</p>
|
||||
<h2 id="conclusion">Conclusion<a class="headerlink" href="#conclusion" title="Permanent link">¶</a></h2>
|
||||
<p>The training-set aggregate effect, particularly the full-mode misID
|
||||
increase, does not reproduce on held-out data. At minimum it
|
||||
is far smaller than the training-set numbers suggested; it may be sampling
|
||||
variation from only 4 training films rather than a generalizable
|
||||
mechanism. Note the same <em>class</em> of harness bug appears twice in this
|
||||
investigation, the timeout truncation in bug #1 above, and the dropped-film
|
||||
aggregation that inflated the raw training-set misID figures. Both make an
|
||||
inert config look consequential; both are reasons to distrust a dramatic
|
||||
training-set delta until it survives on held-out films, which this one did
|
||||
not. This does not mean <code>expand_gallery</code> never does anything: the
|
||||
mechanism is real, and
|
||||
<a href="https://gitea.tourolle.paris/dtourolle/scene-actor-extraction/raw/commit/0bd27470698c45cab21935d636a04512360e1008/src/gallery/track_gallery.hpp"><code>track_gallery.hpp</code></a>'s
|
||||
promotion logging confirms tracks get confirmed and views get promoted
|
||||
into the annex on every film tested. It means whatever effect expansion
|
||||
has on final per-second identification was too small to detect against 5
|
||||
held-out films with this scoring method. A cleaner test would need either
|
||||
more held-out films or a metric that can see the annex's direct
|
||||
contribution, such as tagging which reference embedding won each match;
|
||||
neither was in scope for this pass.</p>
|
||||
<p>Do not treat the training-set exp/noexp numbers in
|
||||
<a href="../model-bakeoff/">the full experiment log</a> as proof that expansion changes
|
||||
real-world behavior in either direction. On the evidence gathered so far,
|
||||
it does not move the needle enough to see.</p>
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user