Matches the convention used by pyPhotoAlbum and the other projects here:
runs-on: linux/amd64 with a container image from the Gitea registry,
instead of setup-python plus an ad-hoc `pip install pytest pytest-cov
flake8 coverage-badge interrogate` on a self-hosted runner.
pyWebLayout is a library, so the image carries all four interpreters
pyproject.toml claims to support - 3.10, 3.11, 3.12, 3.13 - each in its
own venv at /opt/py<version> with every dependency pre-installed. The
matrix picks one per job. A CI run now downloads nothing, and coverage
widens from 3.10/3.12/3.13 to the full declared range.
Ubuntu marks its system Python externally-managed, so per-interpreter
venvs are used rather than --break-system-packages; that also keeps the
four dependency sets isolated.
Two defects in the existing workflow are fixed while rewriting it:
- pytest runs under continue-on-error so the badge steps still execute,
but nothing afterwards checked its outcome - the job reported green on
a red suite. An explicit gate now fails the job.
- Every matrix leg ran the badge steps and force-pushed the badges
branch, so three jobs raced to publish. Badges and artifacts are now
produced by the 3.13 leg only.
setuptools is pinned below 81 in the image: that release dropped
pkg_resources, which coverage-badge imports at startup, and without the
pin the badge step dies with ModuleNotFoundError. Found by running the
workflow's own commands in the image rather than assuming they work.
Also raises the test Flask server's readiness budget from 5s to 30s.
Making that check raise instead of silently falling through (737cf07)
turned runner load into a hard failure; it showed up as 17 spurious
errors in one containerised run and did not reproduce in three repeats.
The loop still exits as soon as the server answers.
Verified locally: image builds, and tests/ passes 916 on each of 3.10,
3.11, 3.12 and 3.13 inside it. The publishing leg was run end to end -
clean-install dependency check, pytest with coverage, both badges,
coverage summary at 81.5%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
concrete/interaction_handler.py was 310 lines reachable only from
examples/07_pressed_state_demo.py - no library code, no tests. Press and
hover feedback existed but could not be used through the library's own
interface.
Adds to the manager:
handle_hover(point) -> frame if hover changed, else None
handle_touch_down(point) -> frame showing the pressed state
handle_touch_up(point) -> (frame, callback result)
reset_interaction_state()
Returning None when nothing changed visually lets a UI skip a redraw it
does not need - which matters on e-ink.
Press state belongs to one rendered page, so the InteractionStateManager
is bound lazily and rebound whenever the displayed page changes, resetting
the outgoing one so a press cannot survive a page turn.
Wiring it up immediately surfaced a real bug it had been hiding.
LinkText.render passed [origin, origin + size] - a list of two numpy
arrays - to PIL's draw.rectangle, which needs a flat four-scalar box.
Rendering any hovered or pressed link raised
TypeError: coordinate list must contain exactly 2 coordinates
so the entire feature was broken on this PIL version. Fixed by building
the box explicitly, and the two branches now share it instead of
duplicating the call.
Tests cover hover/press/release, no-op paths, that an unchanged hover
reports no change, state rebinding across navigation, reset, and
regressions for the rectangle crash.
916 passed. examples/07_pressed_state_demo.py still runs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
core/highlight.py was 249 lines of fully implemented, fully tested code
that nothing could reach. EreaderLayoutManager had no highlight API, so
highlighting was not available through the library's own interface - it
was tested in isolation and otherwise dead.
Adds to the manager:
highlight_point(point, color, note, tags) tap to highlight a word
highlight_range(start, end, ...) drag to highlight a span
remove_highlight(id) / clear_highlights()
list_highlights() / get_highlights_for_current_page()
Highlights default to the bookmarks directory, so a document's reading
state lives in one place rather than two.
A Highlight carried only pixel bounds, which describe the one rendering
they were taken from: change the font scale or page size and they no
longer point at anything. Highlight now also records the
RenderingPosition of the page it was made on, and page association goes
through that instead of through bounds overlap. The field is optional and
read with .get, so existing stores load unchanged - they simply never
match a page, which is the honest answer for a highlight whose only
anchor is stale pixels.
Also removes the persistence duplication the review called out.
BookmarkManager and HighlightManager each had their own copy of "make the
directory, read the file, swallow and print on failure". Both now use
core/persistence.py, which logs with exc_info instead of printing and
catches specific exceptions rather than bare Exception. File names and
formats are unchanged, so nothing needs migrating.
Tests cover point and range highlighting, colour/note/tag round trips,
misses returning None, page scoping across navigation, persistence across
a restart, and that a corrupt highlight file does not stop a book from
opening.
902 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
S16 replaced backward pagination's estimate-render-compare-adjust loop
with anchor replay, but left _estimate_page_start and
_adjust_start_estimate behind. Nothing in the library calls them; only
their own tests did.
_estimate_page_start guessed max(1, int(10 / font_scale)) blocks per
page - a constant with no relationship to page size, block length or
font metrics - and _adjust_start_estimate halved the error each round to
converge on it. Anchor replay makes both meaningless: it walks forward
from a known page boundary instead of guessing at one.
Removes their five tests with them rather than leaving tests pinning
behaviour nothing depends on.
889 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ereader_layout.py defined _add_page_methods() and called it at import
time, attaching can_fit_line and available_width to the Page class if
they were absent. Page defines both, so the patch never fired - but the
two definitions of can_fit_line disagreed:
Page can_fit_line(baseline_spacing, ascent=0, descent=0)
monkey patch can_fit_line(line_height)
The patched version had no way to express descent, which is exactly the
clipping bug S2 fixed. Had Page.can_fit_line ever been renamed or moved,
this would have silently reinstated pre-S2 behaviour as a side effect of
importing a module in a different package.
Import-time patching of another module's class has no place here. If the
layout engine needs something from Page, it belongs on Page.
Tests pin the outcome: the module exposes no patcher, Page owns both
attributes, and can_fit_line still rejects a line whose descender would
hang past the content box.
894 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_scale_block_fonts rebuilt a block by constructing a plain
Word(word.text, scaled_style) for every word. LinkedWord is a Word
subclass, so the reconstruction downgraded it and dropped the link
target: every hyperlink in the document disappeared the moment the reader
changed font size. Nothing caught it because the function returns the
block unchanged at scale 1.0 with no family override, which is what the
tests exercised.
Add Word.with_style(), overridden by LinkedWord to carry location, link
type, callback, params and title across. Putting the copy behaviour on
the word class means any future Word subclass either inherits a correct
copy or overrides one, rather than being silently flattened by a
constructor call in the layout engine.
Also extend coverage beyond Paragraph and Heading. Quote, HList and Table
were returned unscaled, so a font-size change left quoted text, list
items and table cells at their original size while the surrounding text
reflowed. Table rows are re-added to the section they came from, so a
<thead> row does not become a body row. Image, HorizontalRule, PageBreak
and CodeBlock still pass through: they carry no styled words.
Scaled blocks are now memoised per (block, scale) for the life of the
layouter. Previously a fresh Paragraph and Word were allocated for every
word on every page render at any scale != 1.0, on the hot path, against
the caching work in concrete/text.py.
Tests cover with_style on both word classes, link survival and target
preservation across all four container types, per-container scaling,
table section preservation, memoisation, and that originals are never
mutated. End-to-end: links remain tappable at 0.8x, 1.5x and 2.0x.
Note: query_point's hit region is offset from LinkText.origin by roughly
the ascent, so probing a link's own centre reports "empty". That
reproduces identically at scale 1.0, predates this change, and is tracked
separately as R9 - the end-to-end test scans instead of probing.
891 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PageBuffer started a ProcessPoolExecutor(max_workers=4) and submitted page
renders to it. Every job failed. _render_page_worker returned
pickle.dumps(page), and a Page holds a live PIL canvas, which is not
picklable, so check_completed_renders swallowed a TypeError into a bare
print and cached nothing. The cost was paid in full for zero benefit:
four interpreter copies plus the whole block list shipped per job (~880KB
for a 200-block document).
Three further defects would have had to be fixed before it could ever
have worked: the worker built its BidirectionalLayouter without page_size
so it silently used the (800, 600) default; check_completed_renders
cached every result with is_backward=False, so backward renders landed in
the forward buffer; and both _queue_*_renders broke at the end of their
first loop body, queueing one page each despite looping buffer_size
times.
Two live defects go with it:
R1 - on Python 3.14 the default start method became forkserver, so
submit() reaches _check_not_importing_main() and raises unless the
caller sits inside an `if __name__ == "__main__"` guard.
EreaderLayoutManager.get_current_page() raised outright from ordinary
module-level script code.
R2 - PageBuffer.__del__ called executor.shutdown(wait=True). Blocking
on a process pool from a finaliser at interpreter teardown deadlocked;
the test suite finished in 11.5s and then never exited.
S12's measurement gate, on the tests/data Wikipedia fixture (411 blocks)
with text caches warm:
800x600 p50 8.8 ms p95 15.4 ms
1072x1448 p50 13.8 ms p95 56.1 ms
A page turn is cheaper than the IPC meant to hide it, so the gate says
delete rather than replace. The LRU buffers, position maps and
invalidation logic are kept unchanged; only the executor, worker,
pickling, prefetch queueing and the lock guarding the pending-render dict
are removed. If a slower device ever changes the numbers, the fallback is
a synchronous readahead() method or a single worker thread, not
processes.
EreaderLayoutManager.shutdown() becomes idempotent and its __del__ no
longer propagates exceptions - it was doing bookmark file I/O during
interpreter teardown.
Adds tests/layout/test_page_buffer.py, which the module had none of:
LRU eviction and position-map cleanup, cache hits, font-scale
invalidation, backward round-trip, and subprocess regressions for R1
(no __main__ guard) and R2 (exit without explicit shutdown).
870 passed, and the suite now exits in 12s wall instead of hanging.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
render_page_backward searched for the previous page's start: estimate a block
index, lay out forward, bisect on the block difference, up to ten times. Both the
estimator and the adjuster pinned word_index to 0 and moved only block_index -
but pages routinely start mid-block, so the answer was not in the search space.
The loop could never match, exhausted its iterations and fell through to a
fallback that jumped to the start of the document.
Measured on a document with one 1200-word paragraph, whose page starts are
(0,0), (2,208), (2,494), (2,780): three of four backward calls returned (0,0),
each after ten full page layouts. The bisection was unsound in its own space too
- 40 small paragraphs, every page starting on a block boundary, also failed.
Pagination is a pure function, so the page before P is the q with next(q) == P,
and it is found by replaying the chain forward rather than guessing q. Three
sources: the chain recorded as pages are laid out forward (exact, one layout,
covers paging back and forth); replay from the block containing P and then from
earlier blocks (exact when P lies on that chain); and failing that, the last
start before P, which overlaps slightly rather than skipping content.
warm: 4/4 exact, 1 layout each
cold, fresh layouter per call: 12/13 exact, worst 17 layouts
cold, repeated back presses: ~4 layouts per turn
Each step returns a page ending exactly where the reader is, so paging back never
skips or repeats content. That chain can differ from the one seen reading forward
from page one if the reader arrived by a jump - pagination from a different start
is a different chain, and nothing can recover the original without replaying the
whole document.
Complementary to S11 rather than caused by it: before S11 forward pagination
dead-ended at the first page-spanning block, so mid-block starts never arose and
the block-granular search looked adequate.
The URL-image tests hard-coded port 5555 and never shut the server down,
so a leftover or concurrent run left the port occupied and the readiness
loop fell through silently — the tests then ran against whatever was
listening, or against nothing.
Use werkzeug's make_server on port 0, record the assigned port, and shut
the server down in tearDownClass. The readiness loop now raises instead
of falling through when the server never comes up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FormFieldText treats its origin as the control's top-left - size and in_object
both measure down from it - but drew the label by calling Text.render at that
origin, and Text anchors on the baseline. The label's glyphs therefore landed
above the origin, outside the box the control claims, on top of whatever was
there. In a stacked form that is the preceding field's input box, which is what
example_10_forms.png showed: every label but the first crowding the box above it.
The label is now offset down by its ascent, so it occupies the space the control
accounts for. Height derives from the label's ink height rather than the nominal
font size, which had also eaten into the 5px gap between label and box.
LABEL_GAP names that gap and field_area_offset gives the distance from the origin
to the top of the input box; render, handle_click and the height calculation now
share it instead of each recomputing font_size + 5.
Also recorded under S12: the broken process pool is not merely wasted work. It
forks from a process that already has threads, and
tests/layout/test_ereader_image_rendering.py hangs at interpreter exit roughly
one run in four - every test passes, then the process never returns.
Both renderers placed the baseline at box_top + height/2 + descent/2. Centring
glyphs of visual height ascent+descent in a box of height H puts the baseline at
box_top + H/2 + (ascent-descent)/2; the two agree only when ascent is exactly
twice descent. DejaVu is nearer 4:1, so labels sat high against the top edge -
measured at 5px above and 11px below for a 14px button.
ButtonText also sized itself from the nominal font size, which is smaller than
the text's visual height (17px of ink for a 14px DejaVu font), leaving the
button too short to centre its label in. It now measures ascent+descent, with a
fallback for font objects that cannot report metrics.
docs/images/example_07_pressed_state.png was stale - no example writes it, the
demo emits demo_07_pressed.png at the repo root and the docs copy had been
placed by hand in November. Refreshed here; the demo should write straight to
docs/images/ so it cannot drift again.
add_child invalidated the canvas but left _draw bound to it, and the draw
property only rebuilt when _draw was None. Callers therefore got a context
pointing at a discarded image while page._canvas stayed None. table_layouter
reads page._canvas directly, so every image inside a table cell laid out after
any other content silently degraded to a grey [Image: WxH] placeholder.
The property now rebuilds when either half is missing. On its own that would
make layout allocate a full-page RGBA canvas per line, because layout measures
text through the page - so measurement moves to page.measurement_draw, a 1x1
scratch context that is never invalidated. Its mode matches the render canvas
so that Text's width cache does not hold two entries per word.
Children built against the scratch context are re-bound to the live canvas by
render_children, which already synchronised _draw and _canvas; that behaviour
was incidental and is now load-bearing and documented as such.
Regenerating the examples shows table images rendering as images rather than
placeholders. The empty header row in the second table of example 05 is
unrelated and pre-existing - row height ignores cell padding, so text is clipped
as padding grows - recorded as evidence under S6.
Inline tags map to ignore_handler because they are meant to be consumed by
extract_text_content, but only paragraph_handler and heading_handler ever called
it. Every other container walked its children calling process_element, so inline
tags returned None and their text was dropped:
<p>hello <b>world</b> again</p> -> hello world again (correct)
<div>hello <b>world</b> again</div> -> nothing at all
<li>hello <b>world</b> again</li> -> nothing at all
<td>hello <b>world</b> again</td> -> nothing at all
<td><a href=u>link</a> text</td> -> text (link discarded)
div_handler ignored bare text nodes outright, so a div containing text produced
no blocks whatsoever - which for real HTML and EPUB is most of the document.
Where text did survive, in cells and list items, each text node became its own
paragraph, so "a <b>b</b> c" fragmented onto separate lines.
process_block_children now walks a container's children once, gathering runs of
inline content into a single paragraph and letting block children through to
their own handlers, preserving document order. div, li, td, th and blockquote
all delegate to it, so they gain nested blocks, links and mixed content
together. <br> ends the current run rather than being a no-op.
extract_text_content is split so the run-level logic can be reused without
building a synthetic element: extract_words_from_nodes takes the nodes directly,
and skips comments, which previously had their text extracted as content.
paragraph_handler keeps its own image-splitting path for now; folding it into
process_block_children would also fix the ordering of text around images in a
paragraph, but it carries the EPUB cover-detection behaviour and is left alone.
LeftAlignmentHandler spread each line's residual space across its word gaps,
clamped to max_spacing. A line whose residual divided to under max_spacing was
stretched flush, one that exceeded it was not, so left-aligned text was
justified sometimes, by a different amount per line - which reads as a wobbling
right edge rather than as ragged-right. Centre/right did the same, and computed
their start position from a different spacing than the one they returned, so
centred lines were not centred.
Ragged alignments now use a constant word space - the font's own space advance,
clamped to the style's bounds - and report overflow instead of tightening, so
line breaking decides what fits rather than rendering squeezing it.
Justification kept two further defects:
- the final line of a paragraph was stretched across the measure, so a
three-word tail was spread edge to edge. Line now carries is_paragraph_end,
set on the line holding the last word, and renders flush left. A paragraph
continued on the next page is not marked, so it stays justified.
- gaps were floored per gap with a truncated remainder, discarding the
fractional part of both. Lines stopped one or two pixels short, differently
each time. Distributing by cumulative rounding makes the gaps sum to the
residual exactly; advance ends now land identically on every line.
Alignment is configurable rather than hardcoded: PageStyle.default_alignment,
defaulting to JUSTIFY for body text. text_align on abstract and concrete styles
defaults to None meaning "unspecified", so HTML without text-align inherits the
page default while explicit CSS still wins. Headings are never justified.
paragraph_layouter placed lines at page.border_size while sizing them to
available_width, which subtracts both paddings. Text therefore started flush
against the left border and the entire padding budget accumulated on the right,
so lines broke well short of the right border.
Page now describes its content box directly - content_origin, content_rect and
remaining_height - and the layouters use it instead of each recomputing the
geometry from border_size. The four block layouters had all been computing
remaining space as size[1] - y_offset - border_size, subtracting the border but
not the bottom padding, so every block type could be placed into the bottom
padding; remaining_height fixes that too.
Page also gains an origin, defaulting to (0, 0). That is inert for a top-level
page but lets a page be positioned inside another surface, which table cells
need in order to be laid out by the normal engine.
Golden images regenerated: content now sits inside the padding on all sides.
render_page_forward discarded new_pos on the failure path, but a block that
only partially fitted has still advanced the position: paragraph_layouter
reports the word it stopped at, and _layout_paragraph_on_page packs it into
new_pos. Dropping it told the caller no progress was made, so navigation
dead-ended on any paragraph larger than a single page - the reader saw "end of
document" mid-book.
A 2877-word paragraph at 800x600 rendered 26 lines and reported the start
position back; it now paginates across 12 pages.
Also guard the navigation loop: EreaderManager.next_page treats no-progress as
end-of-document, which is only correct at the actual end. Anywhere else it now
logs the offending block index and skips that block, so a future layout bug
costs one block rather than the rest of the book.
Rendering a page re-measured and re-rasterised the same words constantly: at
1404x1872 a page issued ~2800 textlength calls and ~2500 draw.text calls for
fewer than 1000 distinct (font, string) pairs. Profiling a page turn showed
FreeType glyph rendering at 57% of total time and width measurement at 38% of
layout.
Both are now cached. Measured on Crime and Punishment at 1404x1872, one page:
layout 35ms -> 19ms
render 84ms -> 41ms
total 120ms -> 60ms
Eviction ranks by use count rather than recency. Word frequency in prose is
Zipfian and stationary, so the words worth keeping are the ones used most, and
unlike recency this lets a document's own frequencies be seeded up front --
see prewarm_caches(). Two details keep the policy from costing more than it
saves, since get() runs once per word drawn:
- counting is O(1) with no reordering, because structures that reorder on
every hit measured 3-5ms/page slower than the hit rate they bought;
- eviction samples 8 entries and drops the least used of those, rather than
maintaining a global order.
Aging (halving all counts periodically) is on by default. Without it a font
size change drove the hit rate to 0% on a real access trace: every key was new
and the previous size's entries held counts nothing could beat.
Both caches are bounded, since the glyph bitmaps reach ~19MB over a long
session and the target is a 512MB Pi Zero 2. Defaults are 4MB of bitmaps and
8192 widths; configure_text_caches() tunes them. Cache size barely affects
speed (2MB is within 13% of unbounded) because a miss costs only one ~44us
rasterisation, so the bound can be set for memory, not throughput.
EreaderLayoutManager.prewarm_caches() counts the book's word frequencies and
preloads the most common ones, seeding each with its document frequency. This
moves that rasterisation to open time and cut misses by 27%, for ~15% faster
page turns at a one-off ~300ms cost. It is opt-in; nothing calls it yet.
Rendering is no longer bit-identical. PIL positions text at sub-pixel offsets,
so the cache buckets that phase, defaulting to 2 buckets per axis. Total ink
per page is unchanged and the mean pixel difference is 3.6/255 -- a fifth of
one step of a 16-level e-ink panel. subpixel_steps=4 halves that if wanted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dtourolleandClaude Opus 5