docs: montage-renderer imagery, visual polish, README screenshots
- switch report frames to the scene best/worst montage renderer (Onscreen/Offscreen panels + TPI/FPI/FN legend): perfect-second hero, wedding couple, funeral 19-of-20, polygraph bridging, crew-scene FN ceiling, Robert Patrick ground-truth gap, rapid-cut double label, Herbie Hancock on an in-fiction screen - deep dive restructured: extinction bridging framed as designed behavior with a measurable cost (debug overlay draws the boxes; the shipped output is presence windows), plus the face-vs-presence ceiling and two X-Ray-is-wrong exhibits - Material polish: light/dark palette toggle, landing-page grid cards, figure/caption CSS, how-to-read admonition; site_url set so 404 links resolve under the Pages subpath - README: perfect-second and screen-call frames committed (gitignore exceptions), readme_example.jpg retired - build_site.sh: stage_frame helper downscales montage frames to 1920px and pulls any missing montage-frames packages
This commit is contained in:
+3
-2
@@ -85,9 +85,10 @@ docs_site/
|
|||||||
# ignore docs/assets/ itself — only its contents, minus the one exception.)
|
# ignore docs/assets/ itself — only its contents, minus the one exception.)
|
||||||
docs/assets/images/*
|
docs/assets/images/*
|
||||||
!docs/assets/images/pipeline_topology.svg
|
!docs/assets/images/pipeline_topology.svg
|
||||||
# readme_example.jpg is referenced directly by README.md, which renders on the
|
# These frames are referenced directly by README.md, which renders on the
|
||||||
# plain Gitea repo view — committed for the same reason as the SVG above.
|
# plain Gitea repo view — committed for the same reason as the SVG above.
|
||||||
!docs/assets/images/readme_example.jpg
|
!docs/assets/images/lovelace_perfect_second.jpg
|
||||||
|
!docs/assets/images/valerian_screen_call.jpg
|
||||||
|
|
||||||
# Python
|
# Python
|
||||||
__pycache__/
|
__pycache__/
|
||||||
|
|||||||
@@ -8,9 +8,18 @@ discussion in the [deep dive](https://pages.tourolle.paris/dtourolle/scene-actor
|
|||||||
Full benchmark write-up, model comparison, and failure-mode analysis:
|
Full benchmark write-up, model comparison, and failure-mode analysis:
|
||||||
**https://pages.tourolle.paris/dtourolle/scene-actor-extraction/**
|
**https://pages.tourolle.paris/dtourolle/scene-actor-extraction/**
|
||||||
|
|
||||||

|

|
||||||
*A held-out film (never used for threshold tuning) — three actors correctly
|
*A perfect X-Ray second on a held-out film (never used for threshold tuning):
|
||||||
identified with calibrated confidence scores.*
|
every visible face named at 100%, the background extra honestly left unnamed,
|
||||||
|
and the two credited cast without a visible face correctly carried as present
|
||||||
|
off-screen. Bottom panels show the per-second verdict against Amazon X-Ray
|
||||||
|
(green = correct, orange = wrong, blue = missed).*
|
||||||
|
|
||||||
|
It also doesn't care whether the face is in the room:
|
||||||
|
|
||||||
|

|
||||||
|
*Herbie Hancock at 98% — as a face on a screen inside the movie, under a
|
||||||
|
sci-fi HUD overlay.*
|
||||||
|
|
||||||
## How it works
|
## How it works
|
||||||
|
|
||||||
|
|||||||
Binary file not shown.
|
After Width: | Height: | Size: 239 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 72 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 165 KiB |
+50
-22
@@ -5,32 +5,60 @@ film or TV episode — built on [KPN++](https://gitea.tourolle.paris/dtourolle/K
|
|||||||
(a C++20 Kahn Process Network library) for the detect → track → match → scene
|
(a C++20 Kahn Process Network library) for the detect → track → match → scene
|
||||||
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
|
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
|
||||||
|
|
||||||
This is what a good second looks like — one sampled frame from a held-out film,
|
This is a perfect X-Ray second, on a film the optimizer never saw:
|
||||||
19 faces named, all 19 correct, the rest honestly declared unknown:
|
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
And this is why the work isn't done: on this same film the same config misses
|
Every visible face named at 100% — Chris Noth, Hank Azaria, Bobby Cannavale —
|
||||||
6 in 10 of the actor-seconds X-Ray says are present, and on the worst held-out
|
the background extra honestly left unnamed, and the two credited cast without
|
||||||
film it reports ghost actors over empty walls — at 100% confidence. Both
|
a visible face correctly carried as present off-screen by the tracker's
|
||||||
stories, with the evidence, are in the pages below.
|
presence windows. That's the pipeline exactly reproducing Amazon X-Ray's
|
||||||
|
record for this second.
|
||||||
|
|
||||||
|
It doesn't always go like that: the hardest held-out film scores 46% F1, and
|
||||||
|
the report is honest about *why* — one tunable trade (extinction bridging at
|
||||||
|
hard cuts), one structural ceiling (X-Ray credits people whose faces never
|
||||||
|
appear), and a few cases where the pipeline is right and X-Ray is wrong. The
|
||||||
|
evidence for all of it is in the pages below.
|
||||||
|
|
||||||
## Start here — four questions this bake-off answers
|
## Start here — four questions this bake-off answers
|
||||||
|
|
||||||
- **[Which model is best?](best-model.md)** — calibration curves first
|
<div class="grid cards" markdown>
|
||||||
(discriminative power, independent of any threshold), then F1 on the actual
|
|
||||||
benchmark. LVFace-B Glint360K wins both.
|
- :material-trophy:{ .lg .middle } **[Which model is best?](best-model.md)**
|
||||||
- **[Whole gallery vs. limited (cast-restricted) gallery](gallery-scope.md)** —
|
|
||||||
restricting the matcher to a film's credited cast is a clean win on every
|
---
|
||||||
axis (+3.3pp F1, less than a third the misIDs), but isn't a shipped runtime
|
|
||||||
feature yet.
|
Calibration curves first (discriminative power, independent of any
|
||||||
- **[Does pose expansion help?](pose-expansion.md)** — a real training-set
|
threshold), then F1 on the actual benchmark. LVFace-B Glint360K wins
|
||||||
effect that didn't reproduce on 5 held-out films once two methodology bugs
|
both.
|
||||||
were caught and fixed. An honest null result, not a forced narrative.
|
|
||||||
- **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)** — the winning
|
- :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)**
|
||||||
model's held-out generalization gap, its two real failure modes (frozen-bbox
|
|
||||||
"ghost tracks"), and one case where it correctly identified an actor that
|
---
|
||||||
the X-Ray ground truth itself failed to credit.
|
|
||||||
|
Restricting the matcher to a film's credited cast is a clean win on
|
||||||
|
every axis (+3.3pp F1, less than a third the misIDs) — but isn't a
|
||||||
|
shipped runtime feature yet.
|
||||||
|
|
||||||
|
- :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
A convincing training-set effect that didn't reproduce on 5 held-out
|
||||||
|
films once two methodology bugs were caught and fixed. An honest null
|
||||||
|
result, not a forced narrative.
|
||||||
|
|
||||||
|
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
The held-out generalization gap, how the error budget decomposes
|
||||||
|
(extinction bridging at hard cuts, X-Ray's scene-membership vs.
|
||||||
|
on-screen-face ceiling), and the frames where the pipeline is right
|
||||||
|
and the ground truth is wrong.
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
## The full technical log
|
## The full technical log
|
||||||
|
|
||||||
@@ -43,7 +71,7 @@ stories, with the evidence, are in the pages below.
|
|||||||
earlier scene-union-metric tuning pass, superseded by the per-second metric
|
earlier scene-union-metric tuning pass, superseded by the per-second metric
|
||||||
used in the bake-off but kept for the ground-truth/architecture background.
|
used in the bake-off but kept for the ground-truth/architecture background.
|
||||||
- **[Service conversion (proposal)](service-conversion.md)** — design sketch
|
- **[Service conversion (proposal)](service-conversion.md)** — design sketch
|
||||||
for an idle-GPU Docker worker, not yet built.
|
for a native idle-GPU worker gated on screen lock, not yet built.
|
||||||
|
|
||||||
## Reproducing the benchmarks
|
## Reproducing the benchmarks
|
||||||
|
|
||||||
|
|||||||
+139
-84
@@ -2,27 +2,45 @@
|
|||||||
|
|
||||||
LVFace won the model bake-off (see [Which model is best?](best-model.md)) and is
|
LVFace won the model bake-off (see [Which model is best?](best-model.md)) and is
|
||||||
the shipped default embedder. This page is the honest accounting of how it
|
the shipped default embedder. This page is the honest accounting of how it
|
||||||
actually performs — what a good second looks like, where it's wrong and *why*,
|
actually performs — what a good second looks like, where the errors actually
|
||||||
and one case where the ground truth itself is wrong and LVFace is right.
|
come from, and two cases where the ground truth itself is wrong and LVFace is
|
||||||
|
right.
|
||||||
|
|
||||||
|
!!! note "How to read the frames on this page"
|
||||||
|
|
||||||
|
The top is the film frame, with a box and name on every face the pipeline
|
||||||
|
identified. The bottom panels are the per-second verdict against X-Ray:
|
||||||
|
**Onscreen** lists faces named in the frame, **Offscreen** lists cast
|
||||||
|
X-Ray marks present in the scene without a visible face — presence
|
||||||
|
carried by the tracker's windows, not by a detection. Colors are the
|
||||||
|
score: <span style="color:#0ca30c">**green**</span> = correct (TPI),
|
||||||
|
<span style="color:#eb6834">**orange**</span> = wrong (FPI),
|
||||||
|
<span style="color:#3987e5">**blue**</span> = missed (FN).
|
||||||
|
|
||||||
## What good looks like
|
## What good looks like
|
||||||
|
|
||||||
Before the failure analysis, the ceiling. This is a single sampled second from
|

|
||||||
Downton Abbey's wedding scene — a packed, hat-heavy, period-costume group shot,
|
|
||||||
about as hostile as ensemble framing gets:
|
|
||||||
|
|
||||||

|
Six faces on screen, all six named correctly — including Penelope Wilton at the
|
||||||
*Frame `downton_abbey/best/best_t00127.jpg` from the `montage-frames` artifact
|
edge of the pews and a half-occluded Michelle Dockery — while thirteen more
|
||||||
package (`scripts/artifacts/pull_artifacts.sh montage-frames
|
cast members X-Ray marks present in the scene are correctly carried as
|
||||||
Downton_Abbey__A_New_Era`) — green = identified, blue = detected but unknown.*
|
"Offscreen" by their presence windows. One miss in the whole frame: Maggie
|
||||||
|
Smith (blue). Score for this second: 0.86.
|
||||||
|
|
||||||
**Nineteen named faces in one frame, all nineteen correct** — Jim Carter half
|

|
||||||
behind a flower arrangement, Penelope Wilton at a three-quarter turn, Lesley
|
|
||||||
Nicol under a hat brim. The blue "unknown" boxes are the honest cases: faces the
|
The same film's funeral gathering: mourning dress, hats, half the faces turned.
|
||||||
detector found but the matcher declined to name rather than guess. The one miss
|
**Nineteen of the twenty cast X-Ray lists for this scene are scored correctly**
|
||||||
at this second is Maggie Smith — not on screen in this framing, but X-Ray marks
|
— seven named on screen at up to 100% confidence, twelve more correctly held
|
||||||
her present for the scene. That distinction (on-screen face vs. scene-level
|
as present off-screen.
|
||||||
ground truth) sets up everything below.
|
|
||||||
|
And the pipeline doesn't need the face to be *real*:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
That's Herbie Hancock at 98% — as a face on a *screen inside the movie*, over a
|
||||||
|
sci-fi HUD overlay, during a video call in Valerian. A face is a face, whether
|
||||||
|
it's in the room or on the bridge's comms display.
|
||||||
|
|
||||||
## Training vs. held-out: the generalization gap
|
## Training vs. held-out: the generalization gap
|
||||||
|
|
||||||
@@ -43,97 +61,134 @@ against the 5 films the optimizer never saw:
|
|||||||
|
|
||||||
**67.4% held-out vs. 75.3% on training** — an ~8pp drop, and a **37pp spread
|
**67.4% held-out vs. 75.3% on training** — an ~8pp drop, and a **37pp spread
|
||||||
between the best and worst held-out film**. The config does not generalize
|
between the best and worst held-out film**. The config does not generalize
|
||||||
uniformly; two films are outright failure cases, for reasons that turn out to
|
uniformly, and the spread traces to two mechanisms, both visible frame by
|
||||||
be one mechanism.
|
frame below.
|
||||||
|
|
||||||
## The failure mode: frozen-bbox "ghost tracks"
|
## Mechanism 1: extinction bridging — usually right, wrong at hard cuts
|
||||||
|
|
||||||
|
The extinction window keeps an identity alive through seconds where no face is
|
||||||
|
detectable. **Most of the time this is exactly what you want**, and it's where
|
||||||
|
a lot of the TPI count comes from:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Lovelace's polygraph scene: only Eric Roberts and Amanda Seyfried have visible
|
||||||
|
faces, but X-Ray lists eight cast present — and all eight score green, the
|
||||||
|
other six correctly carried by presence windows through a scene where the
|
||||||
|
camera never shows them. A perfect second, and the extinction/anneal machinery
|
||||||
|
is *why*.
|
||||||
|
|
||||||
|
The same mechanism has a failure case: a hard cut into long faceless footage.
|
||||||
Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst
|
Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst
|
||||||
recall of the five) trace to the same root cause, verified directly against
|
recall of the five) are dominated by it — verified directly against the raw
|
||||||
the raw per-frame stream and the HDF5 dump's own detection counts — not
|
per-frame stream and the HDF5 dump's own detection counts, not inferred from
|
||||||
inferred from the score alone.
|
the score alone. **This is not a malfunction**: the tracker is doing exactly
|
||||||
|
what its window is for; the footage just stops cooperating. In the debug
|
||||||
|
overlay (which draws a bridged identity's last-known bbox, unlike the shipped
|
||||||
|
output, which emits presence windows and no boxes at all) the bridged state is
|
||||||
|
visible spatially:
|
||||||
|
|
||||||

|

|
||||||
|
*Debug-overlay rendering (`dump_error_frames.py --raw`): "Jon Bernthal", "Joey
|
||||||
|
Diaz" and "Billy Magnussen" are extinction-bridged identities from the previous
|
||||||
|
shot, drawn frozen over the wall and the hanging plates. Frame
|
||||||
|
`many_saints/fpi/fpi_t03543.jpg`, `montage-frames` artifact package.*
|
||||||
|
|
||||||
At this second, three of the four labeled boxes ("Jon Bernthal", "Joey Diaz",
|
The cost is measurable, not just visible. Downton Abbey's hard cut into its
|
||||||
"Billy Magnussen") sit over empty background — a blurred wall, hanging
|
closing credits, plotting the dump's own per-second `face_count` (detector
|
||||||
plates — with no face in them. The real face in frame carries a second,
|
output, independent of the tracker) against what the tracker reports:
|
||||||
colliding label from another frozen box. And it isn't an isolated second — the
|
|
||||||
same signature recurs throughout the film:
|
|
||||||
|
|
||||||

|
|
||||||
*Frame `many_saints_intervals/w002_worst_t01382.jpg`, same artifact package —
|
|
||||||
one frame, three distinct error classes.*
|
|
||||||
|
|
||||||
This frame is worth reading closely, because it separates three things that a
|
|
||||||
single aggregate F1 number smears together. The two green labels ("Jon Bernthal
|
|
||||||
100%", "Michela De Rossi 100%") float over a staircase and a policeman's back —
|
|
||||||
frozen boxes from a previous shot, reported at full confidence. Meanwhile the
|
|
||||||
two *real* frontal faces in frame get honest blue "unknown 0%" boxes (they're
|
|
||||||
uncredited day-players with no gallery reference — the
|
|
||||||
[gallery coverage gap](gallery-scope.md)), and three more people simply face
|
|
||||||
away from camera, invisible to any face detector but still "present" in X-Ray's
|
|
||||||
scene-level ground truth. Precision failure, gallery-coverage failure, and the
|
|
||||||
face-vs-presence ceiling — one frame.
|
|
||||||
|
|
||||||
### The mechanism, measured
|
|
||||||
|
|
||||||
The starkest case is Downton Abbey's hard cut from a packed group shot into a
|
|
||||||
long blank credits sequence. Plotting the detector's per-second `face_count`
|
|
||||||
(from the dump HDF5, independent of the tracker) against what the tracker
|
|
||||||
reports makes the failure legible at a glance:
|
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
From the cut onward the detector sees **zero faces for nearly a minute** — and
|
From the cut onward the detector sees **zero faces for nearly a minute** — and
|
||||||
the tracker keeps reporting the last group shot's 15 identities the entire
|
the tracker keeps reporting the last shot's 15 identities the whole time
|
||||||
time, each with the *exact same bounding box, unchanged to the pixel* (verified
|
(verified for Hugh Bonneville: bbox `(1743.2, 0.0, 171.3, 317.8)`, unchanged to
|
||||||
for Hugh Bonneville: `(1743.2, 0.0, 171.3, 317.8)` at every sampled second for
|
the pixel, at every sampled second for 57+ seconds). The staircase at the right
|
||||||
57+ seconds). The staircase decay at the right edge is the extinction window
|
edge is the extinction window expiring actor by actor. That plateau is
|
||||||
finally expiring, actor by actor.
|
`SceneTrackerFunc::active_[actor_idx].last_bbox`
|
||||||
|
|
||||||
This is `SceneTrackerFunc::active_[actor_idx].last_bbox`
|
|
||||||
([`src/nodes/scene_tracker_node.hpp`](https://REPOLINK/src/nodes/scene_tracker_node.hpp))
|
([`src/nodes/scene_tracker_node.hpp`](https://REPOLINK/src/nodes/scene_tracker_node.hpp))
|
||||||
being re-emitted unchanged — the
|
re-emitted as designed: `extinction_sec=57.4` was tuned long because bridging
|
||||||
extinction state machine working exactly as coded, not a bug. `extinction_sec`
|
wins on most footage (see the polygraph frame above) — the training films just
|
||||||
was tuned to 57.4s specifically because long windows bridge real gaps
|
never contained a faceless stretch long enough to show the cost side, and the
|
||||||
(occlusion, a turned face) in most footage; a hard cut into long faceless
|
held-out set did.
|
||||||
footage is the one case where that same bridging manufactures ghosts, and the
|
|
||||||
training films never contained one long enough to punish it. The optimizer
|
The same track-continuation machinery has one milder spatial artifact, worth
|
||||||
"discovered" the plateau at the top of its search range for a reason that only
|
knowing when reading these frames:
|
||||||
generalizes to films that never go faceless for a minute.
|
|
||||||
|

|
||||||
|
*Café Society (a training film), a shot/reverse-shot dialog: that is Steve
|
||||||
|
Carell wearing both his own label and Jesse Eisenberg's.*
|
||||||
|
|
||||||
|
At a rapid cut, the previous shot's track can linger for a beat at nearly the
|
||||||
|
same screen position the new face occupies — here Jesse Eisenberg's box from
|
||||||
|
the counter-shot lands on Steve Carell. Note what the score panel says,
|
||||||
|
though: both actors are green, because both *are* present in this dialog
|
||||||
|
scene per X-Ray. The spatial label is briefly wrong; the per-second presence
|
||||||
|
claim — the thing the pipeline actually ships — is right. It's the same trade
|
||||||
|
as the extinction window: track continuation smooths over cuts, and 1 fps
|
||||||
|
sampling occasionally catches the seam.
|
||||||
|
|
||||||
|
## Mechanism 2: the face-vs-presence ceiling
|
||||||
|
|
||||||
|
Downton Abbey's recall didn't collapse because faces were misread — it
|
||||||
|
collapsed because for most of its 80084 FN-seconds there was **no face to
|
||||||
|
read**:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
A newsreel crew hauls equipment through the hall: X-Ray credits 22 cast as
|
||||||
|
present in this scene; not one face looks at the camera. Eight are still
|
||||||
|
scored green (windows bridging from adjacent shots) — the other fourteen are
|
||||||
|
blue FNs that no face-recognition pipeline could ever recover. X-Ray encodes
|
||||||
|
*scene membership*; the pipeline measures *on-screen faces*. In ensemble films
|
||||||
|
those two definitions diverge massively, and that gap — not identification
|
||||||
|
error — is most of what the FN column counts.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Same ceiling from the other side: Michela De Rossi in frame but turned away,
|
||||||
|
five cast correctly bridged as offscreen (green), four blue FNs — and one
|
||||||
|
orange we'll come back to below.
|
||||||
|
|
||||||
## Where LVFace beat X-Ray
|
## Where LVFace beat X-Ray
|
||||||
|
|
||||||
Not every flagged "misID" is actually wrong.
|
Not every orange in these frames is actually wrong.
|
||||||
[`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)
|
[`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)
|
||||||
counts a name as a true out-of-cast misID whenever the named actor isn't in
|
scores strictly against X-Ray — but X-Ray itself has holes, and the pipeline
|
||||||
X-Ray's credited cast list for the film at all — but X-Ray's cast list is
|
found two kinds.
|
||||||
itself incomplete.
|
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
Germar Terrell Gardner — a real, clean, high-confidence detection — is counted
|
Germar Terrell Gardner — a real, clean, high-confidence detection — is counted
|
||||||
as a misID here because he doesn't appear in X-Ray's `people.csv` for The Many
|
as an out-of-cast misID because he doesn't appear in X-Ray's `people.csv` for
|
||||||
Saints of Newark at all. But Jellyfin's independent cast metadata *does* credit
|
The Many Saints of Newark at all. But Jellyfin's independent cast metadata
|
||||||
him for this exact film (cross-checked via
|
*does* credit him for this exact film (cross-checked via
|
||||||
`experiments/manifests/jellyfin_casts.json` from the `experiment-data` artifact
|
`experiments/manifests/jellyfin_casts.json` from the `experiment-data` artifact
|
||||||
package, a completely separate data source from X-Ray). This isn't a lookalike error or a gallery mixup — it's the
|
package, a completely separate data source from X-Ray). That's also him in
|
||||||
pipeline correctly recognising a real cast member that one ground-truth source
|
orange in the frame above — every one of those "errors" is the pipeline being
|
||||||
happened to omit.
|
right about a person X-Ray forgot.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
And it isn't only uncredited bit-parts. That is **Robert Patrick** — top-billed
|
||||||
|
in Lovelace, unmistakably on screen, reading his newspaper, identified at
|
||||||
|
100% — scored orange because X-Ray's people-in-scene list for *this scene*
|
||||||
|
doesn't include him. The identification is flawless; the ground truth missed
|
||||||
|
an actor sitting in the middle of the frame.
|
||||||
|
|
||||||
This doesn't mean every flagged misID is secretly correct — Many Saints'
|
This doesn't mean every flagged misID is secretly correct — Many Saints'
|
||||||
974-count total is still overwhelmingly the frozen-bbox failure mode above,
|
974-count total is still overwhelmingly extinction bridging at cuts, not
|
||||||
not uncredited-but-real cameos. But it's a reminder that the X-Ray corpus is a
|
uncredited cameos. But the X-Ray corpus is a convenient, large-scale ground
|
||||||
convenient, large-scale ground truth, not a perfect one, and the "misID" number
|
truth, not a perfect one, and the misID/FPI numbers in these tables carry an
|
||||||
in any of these tables has some irreducible noise floor from ground-truth gaps
|
irreducible noise floor from ground-truth gaps in both directions.
|
||||||
in the other direction too.
|
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
LVFace is the right default: it wins the model comparison outright, it can name
|
LVFace is the right default: it wins the model comparison outright, it names
|
||||||
19 faces correctly in a single hostile group shot, and its failures are
|
19 of 20 correctly across a hat-heavy funeral crowd, and it recognises a face
|
||||||
traceable, understood, and mostly attributable to one tunable knob
|
on a screen inside the movie. Its error budget decomposes into two understood
|
||||||
(`extinction_sec`) rather than the embedder itself. The held-out
|
mechanisms — extinction bridging at hard cuts (a tunable trade, not a bug) and
|
||||||
|
the face-vs-presence ceiling baked into X-Ray's semantics — plus a nonzero
|
||||||
|
slice where the pipeline is right and the ground truth is wrong. The held-out
|
||||||
generalization gap (75.3% → 67.4%) is real and should be treated as the honest
|
generalization gap (75.3% → 67.4%) is real and should be treated as the honest
|
||||||
expected performance, not the training-set number.
|
expected performance, not the training-set number.
|
||||||
|
|||||||
@@ -0,0 +1,24 @@
|
|||||||
|
/* Frames and charts presented as cards */
|
||||||
|
.md-typeset img {
|
||||||
|
border-radius: 6px;
|
||||||
|
}
|
||||||
|
.md-typeset p > img:only-child {
|
||||||
|
box-shadow: 0 2px 12px rgba(0, 0, 0, 0.35);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Captions: an italic-only paragraph immediately after an image reads as a
|
||||||
|
figure caption — centered, small, muted. Falls back to plain italics in
|
||||||
|
browsers without :has(). */
|
||||||
|
.md-typeset p:has(> img:only-child) + p > em:only-child {
|
||||||
|
display: block;
|
||||||
|
text-align: center;
|
||||||
|
font-size: 0.72rem;
|
||||||
|
line-height: 1.5;
|
||||||
|
color: var(--md-default-fg-color--light);
|
||||||
|
margin-top: -0.4rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Slightly tighter hero image spacing on the landing page */
|
||||||
|
.md-typeset h1 + p + p > img:only-child {
|
||||||
|
margin-top: 0.2rem;
|
||||||
|
}
|
||||||
+20
-2
@@ -7,16 +7,31 @@ repo_name: dtourolle/scene-actor-extraction
|
|||||||
theme:
|
theme:
|
||||||
name: material
|
name: material
|
||||||
palette:
|
palette:
|
||||||
- scheme: slate
|
- media: "(prefers-color-scheme: dark)"
|
||||||
primary: indigo
|
scheme: slate
|
||||||
|
primary: black
|
||||||
|
accent: amber
|
||||||
|
toggle:
|
||||||
|
icon: material/weather-sunny
|
||||||
|
name: Switch to light mode
|
||||||
|
- media: "(prefers-color-scheme: light)"
|
||||||
|
scheme: default
|
||||||
|
primary: black
|
||||||
accent: indigo
|
accent: indigo
|
||||||
|
toggle:
|
||||||
|
icon: material/weather-night
|
||||||
|
name: Switch to dark mode
|
||||||
features:
|
features:
|
||||||
- navigation.tabs
|
- navigation.tabs
|
||||||
- navigation.sections
|
- navigation.sections
|
||||||
- navigation.top
|
- navigation.top
|
||||||
|
- navigation.footer
|
||||||
- content.code.copy
|
- content.code.copy
|
||||||
- content.code.annotate
|
- content.code.annotate
|
||||||
|
|
||||||
|
extra_css:
|
||||||
|
- stylesheets/extra.css
|
||||||
|
|
||||||
nav:
|
nav:
|
||||||
- Home: index.md
|
- Home: index.md
|
||||||
- Findings:
|
- Findings:
|
||||||
@@ -44,5 +59,8 @@ markdown_extensions:
|
|||||||
base_path: ['.']
|
base_path: ['.']
|
||||||
check_paths: true
|
check_paths: true
|
||||||
- pymdownx.details
|
- pymdownx.details
|
||||||
|
- pymdownx.emoji:
|
||||||
|
emoji_index: !!python/name:material.extensions.emoji.twemoji
|
||||||
|
emoji_generator: !!python/name:material.extensions.emoji.to_svg
|
||||||
- attr_list
|
- attr_list
|
||||||
- md_in_html
|
- md_in_html
|
||||||
|
|||||||
+50
-23
@@ -11,38 +11,65 @@ cd "$REPO_ROOT"
|
|||||||
ASSETS_DIR="docs/assets/images"
|
ASSETS_DIR="docs/assets/images"
|
||||||
mkdir -p "$ASSETS_DIR"
|
mkdir -p "$ASSETS_DIR"
|
||||||
|
|
||||||
# Frames referenced by docs/model-bakeoff.md. Pull the film's montage
|
# Frames referenced by the docs. The scene best/worst montages (proper
|
||||||
# frames from the registry if this machine doesn't already have them locally.
|
# renderer: Onscreen/Offscreen panel + TPI/FPI/FN legend) live in the
|
||||||
FRAMES_ROOT="experiments/results/holdout/frames"
|
# montage-frames registry packages, one per film; pull any film that's missing.
|
||||||
if [ ! -d "$FRAMES_ROOT/many_saints" ] || [ ! -d "$FRAMES_ROOT/downton_abbey" ]; then
|
MONTAGE_ROOT="experiments/results/holdout/montage_bestworst"
|
||||||
echo "==> pulling montage frames (not found locally)..."
|
for film in Downton_Abbey__A_New_Era Lovelace Café_Society \
|
||||||
scripts/artifacts/pull_artifacts.sh montage-frames Many_Saints_of_Newark || true
|
Valerian_and_the_City_of_a_Thousand_Plan The_Many_Saints_of_Newark; do
|
||||||
scripts/artifacts/pull_artifacts.sh montage-frames Downton_Abbey__A_New_Era || true
|
if [ ! -d "${MONTAGE_ROOT}/${film}" ]; then
|
||||||
|
echo "==> pulling montage-frames/${film} (not found locally)..."
|
||||||
|
scripts/artifacts/pull_artifacts.sh montage-frames "$film" || true
|
||||||
fi
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# stage_frame <src> <dest-basename>: downscale to <=1920px wide for the site
|
||||||
|
stage_frame() {
|
||||||
|
if [ ! -f "$1" ]; then
|
||||||
|
echo "WARN: $1 not present; keeping existing ${ASSETS_DIR}/$2 (if any)"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
python3 - "$1" "${ASSETS_DIR}/$2" <<'PY'
|
||||||
|
import sys
|
||||||
|
from PIL import Image
|
||||||
|
src, dst = sys.argv[1], sys.argv[2]
|
||||||
|
im = Image.open(src)
|
||||||
|
if im.width > 1920:
|
||||||
|
im = im.resize((1920, round(im.height * 1920 / im.width)), Image.LANCZOS)
|
||||||
|
im.save(dst, quality=88)
|
||||||
|
print(f"staged {dst} ({im.width}x{im.height})")
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
echo "==> staging referenced frames into ${ASSETS_DIR}"
|
echo "==> staging referenced frames into ${ASSETS_DIR}"
|
||||||
cp -v "${FRAMES_ROOT}/many_saints/fpi/fpi_t03543.jpg" \
|
stage_frame "${MONTAGE_ROOT}/Downton_Abbey__A_New_Era/scene_4/4_best_t000128.jpg" \
|
||||||
"${ASSETS_DIR}/many_saints_ghost_fpi.jpg"
|
downton_wedding_couple.jpg
|
||||||
cp -v "${FRAMES_ROOT}/downton_abbey/best/best_t00127.jpg" \
|
stage_frame "${MONTAGE_ROOT}/Downton_Abbey__A_New_Era/scene_16/16_best_t007100.jpg" \
|
||||||
"${ASSETS_DIR}/downton_wedding_19_correct.jpg"
|
downton_funeral_19of20.jpg
|
||||||
if [ -f "${FRAMES_ROOT}/many_saints_intervals/w002_worst/w002_worst_t01382.jpg" ]; then
|
stage_frame "${MONTAGE_ROOT}/Downton_Abbey__A_New_Era/scene_7/7_worst_t001754.jpg" \
|
||||||
cp -v "${FRAMES_ROOT}/many_saints_intervals/w002_worst/w002_worst_t01382.jpg" \
|
downton_crew_fn.jpg
|
||||||
"${ASSETS_DIR}/many_saints_ghosts_vs_unknowns.jpg"
|
stage_frame "${MONTAGE_ROOT}/Lovelace/scene_24/24_best_t003607.jpg" \
|
||||||
else
|
lovelace_polygraph_bridged.jpg
|
||||||
echo "WARN: many_saints_intervals frames not present; keeping existing" \
|
stage_frame "${MONTAGE_ROOT}/Lovelace/scene_6/6_worst_t000421.jpg" \
|
||||||
"${ASSETS_DIR}/many_saints_ghosts_vs_unknowns.jpg (if any)"
|
lovelace_robert_patrick_fpi.jpg
|
||||||
fi
|
stage_frame "${MONTAGE_ROOT}/Lovelace/scene_15/15_best_t002005.jpg" \
|
||||||
|
lovelace_perfect_second.jpg
|
||||||
|
stage_frame "${MONTAGE_ROOT}/Café_Society/scene_20/20_best_t002430.jpg" \
|
||||||
|
cafe_society_rapid_cut.jpg
|
||||||
|
stage_frame "${MONTAGE_ROOT}/Valerian_and_the_City_of_a_Thousand_Plan/scene_4/4_best_t001027.jpg" \
|
||||||
|
valerian_screen_call.jpg
|
||||||
|
stage_frame "${MONTAGE_ROOT}/The_Many_Saints_of_Newark/out_of_cast_fpi/4_worst_t000871.jpg" \
|
||||||
|
many_saints_outofcast_fpi.jpg
|
||||||
|
# debug-overlay example (extinction state drawn as frozen boxes) — from the
|
||||||
|
# dump_error_frames output, not the montage package
|
||||||
|
stage_frame "experiments/results/holdout/frames/many_saints/fpi/fpi_t03543.jpg" \
|
||||||
|
many_saints_ghost_fpi.jpg
|
||||||
|
|
||||||
if [ ! -f "${ASSETS_DIR}/germar_beats_xray.jpg" ]; then
|
if [ ! -f "${ASSETS_DIR}/germar_beats_xray.jpg" ]; then
|
||||||
echo "==> pulling report-highlights/germar_beats_xray.jpg..."
|
echo "==> pulling report-highlights/germar_beats_xray.jpg..."
|
||||||
scripts/artifacts/pull_artifacts.sh report-highlights germar_beats_xray.jpg
|
scripts/artifacts/pull_artifacts.sh report-highlights germar_beats_xray.jpg
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [ ! -f "${ASSETS_DIR}/readme_example.jpg" ]; then
|
|
||||||
echo "==> pulling report-highlights/readme_example.jpg..."
|
|
||||||
scripts/artifacts/pull_artifacts.sh report-highlights readme_example.jpg
|
|
||||||
fi
|
|
||||||
|
|
||||||
# pipeline_topology.svg is small and hand-authored (not pulled from anywhere) —
|
# pipeline_topology.svg is small and hand-authored (not pulled from anywhere) —
|
||||||
# committed directly at docs/assets/images/, not staged from the registry.
|
# committed directly at docs/assets/images/, not staged from the registry.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user