Compare commits

...
6 Commits
Author SHA1 Message Date
dtourolleandClaude Opus 5 1924cc234d perf(buffer): remove the process pool from page rendering (S12, R1, R2)
Python CI / test (3.10) (push) Has been cancelled
Python CI / test (3.12) (push) Has been cancelled
Python CI / test (3.13) (push) Has been cancelled
PageBuffer started a ProcessPoolExecutor(max_workers=4) and submitted page
renders to it. Every job failed. _render_page_worker returned
pickle.dumps(page), and a Page holds a live PIL canvas, which is not
picklable, so check_completed_renders swallowed a TypeError into a bare
print and cached nothing. The cost was paid in full for zero benefit:
four interpreter copies plus the whole block list shipped per job (~880KB
for a 200-block document).

Three further defects would have had to be fixed before it could ever
have worked: the worker built its BidirectionalLayouter without page_size
so it silently used the (800, 600) default; check_completed_renders
cached every result with is_backward=False, so backward renders landed in
the forward buffer; and both _queue_*_renders broke at the end of their
first loop body, queueing one page each despite looping buffer_size
times.

Two live defects go with it:

  R1 - on Python 3.14 the default start method became forkserver, so
  submit() reaches _check_not_importing_main() and raises unless the
  caller sits inside an `if __name__ == "__main__"` guard.
  EreaderLayoutManager.get_current_page() raised outright from ordinary
  module-level script code.

  R2 - PageBuffer.__del__ called executor.shutdown(wait=True). Blocking
  on a process pool from a finaliser at interpreter teardown deadlocked;
  the test suite finished in 11.5s and then never exited.

S12's measurement gate, on the tests/data Wikipedia fixture (411 blocks)
with text caches warm:

    800x600     p50  8.8 ms   p95 15.4 ms
    1072x1448   p50 13.8 ms   p95 56.1 ms

A page turn is cheaper than the IPC meant to hide it, so the gate says
delete rather than replace. The LRU buffers, position maps and
invalidation logic are kept unchanged; only the executor, worker,
pickling, prefetch queueing and the lock guarding the pending-render dict
are removed. If a slower device ever changes the numbers, the fallback is
a synchronous readahead() method or a single worker thread, not
processes.

EreaderLayoutManager.shutdown() becomes idempotent and its __del__ no
longer propagates exceptions - it was doing bookmark file I/O during
interpreter teardown.

Adds tests/layout/test_page_buffer.py, which the module had none of:
LRU eviction and position-map cleanup, cache hits, font-scale
invalidation, backward round-trip, and subprocess regressions for R1
(no __main__ guard) and R2 (exit without explicit shutdown).

870 passed, and the suite now exits in 12s wall instead of hanging.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 12:45:20 +02:00
dtourolle 456824d6d6 fix(ereader): replay the page chain instead of searching for it (S16)
render_page_backward searched for the previous page's start: estimate a block
index, lay out forward, bisect on the block difference, up to ten times. Both the
estimator and the adjuster pinned word_index to 0 and moved only block_index -
but pages routinely start mid-block, so the answer was not in the search space.
The loop could never match, exhausted its iterations and fell through to a
fallback that jumped to the start of the document.

Measured on a document with one 1200-word paragraph, whose page starts are
(0,0), (2,208), (2,494), (2,780): three of four backward calls returned (0,0),
each after ten full page layouts. The bisection was unsound in its own space too
- 40 small paragraphs, every page starting on a block boundary, also failed.

Pagination is a pure function, so the page before P is the q with next(q) == P,
and it is found by replaying the chain forward rather than guessing q. Three
sources: the chain recorded as pages are laid out forward (exact, one layout,
covers paging back and forth); replay from the block containing P and then from
earlier blocks (exact when P lies on that chain); and failing that, the last
start before P, which overlaps slightly rather than skipping content.

  warm:                          4/4 exact, 1 layout each
  cold, fresh layouter per call: 12/13 exact, worst 17 layouts
  cold, repeated back presses:   ~4 layouts per turn

Each step returns a page ending exactly where the reader is, so paging back never
skips or repeats content. That chain can differ from the one seen reading forward
from page one if the reader arrived by a jump - pagination from a different start
is a different chain, and nothing can recover the original without replaying the
whole document.

Complementary to S11 rather than caused by it: before S11 forward pagination
dead-ended at the first page-spanning block, so mid-block starts never arose and
the block-granular search looked adequate.
2026-08-08 12:43:30 +02:00
dtourolleandClaude Opus 5 767e4c135c build: consolidate packaging on pyproject.toml and correct the dependency set
The project carried three sets of packaging metadata — pyproject.toml
[project], setup.cfg [options] and setup.py kwargs — declaring different
dependencies. pyproject.toml wins under any modern backend, so the wheel
was correct, but the other two said "Pillow, numpy" and reading either
gave the wrong answer about what the library needs.

Reduce setup.cfg to its [flake8] section and setup.py to a setup() shim,
each pointing at pyproject.toml.

Correct the authoritative list while consolidating:

- flask was a runtime dependency but is imported only by the fixture HTTP
  server in tests. Every user was installing Flask, Jinja2, Werkzeug,
  click, itsdangerous and blinker for nothing.
- ebooklib was a runtime dependency and is never imported by the library;
  epub_reader.py uses zipfile + xml.etree directly. Only the tests use it,
  to build EPUB fixtures.
- requests was declared required but concrete/image.py imports it lazily
  and degrades gracefully when absent, so it belongs in an extra.
- requires-python said >=3.6, which cannot be true: the package uses
  dataclasses and `from __future__ import annotations`, both 3.7+, and CI
  tests 3.10/3.12/3.13.

Runtime install drops from 7 direct dependencies to 4. Adds test,
remote-images and dev extras, and a CI step that installs into an empty
venv with only the runtime deps and imports every subpackage — this class
of defect is only caught by installing what you ship.

Verified: runtime-only install imports all subpackages; `.[test]` runs the
full suite, 853 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 12:30:08 +02:00
dtourolleandClaude Opus 5 7384a32cdd docs: add architecture review with findings R1-R8
Independent review of the codebase at c5c61a3. Records the architectural
verdict and eight findings not covered by LAYOUT_REMEDIATION_SPEC.md,
cross-referencing the existing spec where they overlap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 12:21:02 +02:00
dtourolleandClaude Opus 5 4d596ce095 docs: rewrite ARCHITECTURE.md around layers and dependency rules
The previous version described only the abstract/concrete split and had
drifted from the code. Restructure it around the actual layering
(io → abstract → layout → concrete), state the dependency rules that
hold today, and document the layouter contract that makes pagination
resumable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 12:21:02 +02:00
dtourolleandClaude Opus 5 737cf0771c test(image): bind the fixture Flask server to an ephemeral port
The URL-image tests hard-coded port 5555 and never shut the server down,
so a leftover or concurrent run left the port occupied and the readiness
loop fell through silently — the tests then ran against whatever was
listening, or against nothing.

Use werkzeug's make_server on port 0, record the assigned port, and shut
the server down in tearDownClass. The readiness loop now raises instead
of falling through when the server never comes up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 12:21:02 +02:00
13 changed files with 1613 additions and 565 deletions
+20 -8
View File
@@ -24,17 +24,29 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
- name: Verify declared dependencies are sufficient
run: |
# A clean venv with ONLY the declared runtime deps. If an import here
# fails, install_requires is incomplete and a real `pip install
# pyWebLayout` would fail the same way for a user.
python -m venv /tmp/clean-install
/tmp/clean-install/bin/pip install --upgrade pip
/tmp/clean-install/bin/pip install .
/tmp/clean-install/bin/python -c "
import pyWebLayout.concrete, pyWebLayout.abstract
import pyWebLayout.io.readers.epub_reader
import pyWebLayout.io.readers.html_extraction
import pyWebLayout.layout.ereader_manager
print('clean install imports OK')
"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
# Install package in development mode
pip install -e .
# Install test dependencies if they exist
if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt; fi
if [ -f requirements/test.txt ]; then pip install -r requirements/test.txt; fi
# Install common test packages
pip install pytest pytest-cov flake8 coverage-badge interrogate
# Install package in development mode, with the declared dev extra.
# Test dependencies belong in setup.cfg, not in an ad-hoc pip line.
pip install -e '.[dev]'
- name: Download initial failed badges
run: |
echo "Downloading initial failed badges..."
+303 -188
View File
@@ -1,233 +1,348 @@
# pyWebLayout Architecture: Abstract vs Concrete
# pyWebLayout Architecture
This document explains the fundamental architectural separation between **Abstract** and **Concrete** layers in the pyWebLayout library.
This document describes how pyWebLayout is organised: the layers, what lives in each,
and the rules that govern how they may depend on one another.
## Overview
The pyWebLayout library follows a clear separation between two distinct layers:
The library turns markup (HTML/EPUB) into rendered images. That pipeline is split into
layers with a strict dependency direction:
- **Abstract Layer**: Represents the logical structure and content of documents (HTML/EPUB text)
- **Concrete Layer**: Handles the spatial rendering and visual representation of content
```
io/readers/ parse markup into document structure
abstract/ what the document says ─┐
↓ ├─ built on core/ + style/
layout/ decide where things go │
↓ │
concrete/ what the pixels are ─┘
PIL.Image
```
This separation provides flexibility, testability, and clean separation of concerns.
The central distinction is **abstract vs concrete**:
## Abstract Layer (`pyWebLayout/abstract/`)
- **Abstract** — the logical content and structure of a document. A `Paragraph` knows
it contains words in an order; it does not know how wide they are.
- **Concrete** — the spatial realisation of that content. A `Line` knows exactly which
glyphs sit at which pixel offsets on a specific canvas.
The Abstract layer deals with the **logical structure** of documents without concerning itself with how content will be visually rendered.
One abstract document produces many concrete renderings: different page sizes, font
scales, and font families all re-run the concrete layer over unchanged abstract content.
That is what makes ereader features like live font scaling and reflow possible.
### Key Components
## Layers
#### `abstract/block.py`
- `Block`: Base class for all block-level content
- `Paragraph`: Represents a logical paragraph containing words
- `Heading`: Represents headings with semantic levels (H1-H6)
- `HList`: Represents ordered/unordered lists
- `Image`: Represents image references
### `core/` — shared foundations
#### `abstract/inline.py`
- `Word`: Represents individual words with text content and styling information
- Contains methods for hyphenation and text manipulation
- Does **not** handle rendering or spatial layout
Everything else is built on these. `core/` depends on nothing but `style/`.
#### `abstract/document.py`
- `Document`: Container for the overall document structure
- `Chapter`: Logical grouping of blocks (for books/long documents)
**`core/base.py`** defines the contracts that make a class abstract or concrete:
### Characteristics of Abstract Classes
| Contract | Kind | Meaning |
|---|---|---|
| `Renderable` | ABC | Has `render()`; produces or draws visual output |
| `Queriable` | ABC | Has `in_object(point)`; can be hit-tested |
| `Layoutable` | ABC | Has `layout()`; arranges its own contents |
| `Interactable` | ABC | Holds a callback invoked on interaction |
| `Geometric` | mixin | `origin` and `size` as numpy arrays |
| `Hierarchical` | mixin | `parent` back-reference |
| `Styleable` | mixin | Carries a style object |
| `FontRegistry` | mixin | Deduplicates `Font` instances across a document |
| `MetadataContainer` | mixin | Key/value metadata with typed accessors |
| `BlockContainer` | mixin | Holds child `Block`s |
| `ContainerAware` | mixin | Knows the container it was added to |
1. **Content-focused**: Store text, structure, and semantic meaning
2. **Layout-agnostic**: No knowledge of fonts, pixels, or rendering
3. **Reusable**: Same content can be rendered in different formats/sizes
4. **Serializable**: Can be saved/loaded without rendering context
The useful shorthand: **`Renderable` and `Geometric` are the concrete markers.** An
abstract class that acquires either has crossed the line.
### Example: Abstract Word
**`core/query.py`** — `QueryResult` and `SelectionRange`: the result types for mapping a
pixel back to content (what was clicked, what text is selected, where it is in the
document).
**`core/highlight.py`** — `Highlight`, `HighlightColor`, `HighlightManager`. Highlights
store both pixel bounds (for drawing) and semantic bounds (word indices, for surviving
a font change).
**`core/cache.py`** — `UsageCache` / `SizedUsageCache`, bounded caches keyed by
`(font, string)` for text measurement and glyph rasterisation. Eviction is by usage
count with periodic aging, not LRU; the module docstring explains why at length. This
exists because a single page issues thousands of measurements over fewer than a
thousand distinct pairs, on hardware where an unbounded cache is not affordable.
**`core/callback_registry.py`** — `CallbackRegistry`, which owns the interactive
elements registered on a page and dispatches to them.
### `style/` — semantic styling and its resolution
This package is itself an instance of the abstract/concrete split, applied to styling:
- **`abstract_style.py`** — `AbstractStyle` (frozen dataclass, hashable) captures
*intent*: `FontFamily.SERIF`, `FontSize.LARGE`, `color="black"`. `AbstractStyleRegistry`
interns them so a document holds one instance per distinct style.
- **`concrete_style.py`** — `RenderingContext` (user preferences, DPI, accessibility
flags, available space) plus `StyleResolver`, which maps `AbstractStyle` +
`RenderingContext``ConcreteStyle` (a resolved font path, a pixel size, an RGB
tuple). `ConcreteStyleRegistry` caches the results.
- **`fonts.py`** — `Font`, the loaded PIL font plus its rendering attributes, and the
bundled DejaVu families (`BundledFont`).
- **`page_style.py`** — `PageStyle`: borders, padding, background, default alignment.
- **`alignment.py`** — the `Alignment` enum.
`AbstractStyle``StyleResolver``ConcreteStyle` is the mechanism by which the same
document renders differently for different readers.
### `abstract/` — document content and structure
Layout-agnostic representations of what the document contains.
**`abstract/block.py`** — block-level content, all deriving from `Block`:
`Paragraph`, `Heading` (with `HeadingLevel`), `Quote`, `CodeBlock`, `HList` (with
`ListStyle`) and `ListItem`, `Table` / `TableRow` / `TableCell`, `Image` and
`LinkedImage`, `HorizontalRule`, `PageBreak`.
**`abstract/inline.py`** — `Word`, `FormattedSpan`, `LinkedWord`, `LineBreak`. `Word`
holds text, a style, and previous/next links into the document's word sequence. It can
report whether it is a hyphenation candidate (`possible_hyphenation(language)`), but it
does not perform the split — that is a measurement decision and belongs to `Line`.
**`abstract/document.py`** — `Document`, `Chapter`, `Book`, `MetadataType`. `Book` is
the EPUB-shaped `Document` with chapters and a table of contents.
**`abstract/functional.py`** — `Link`, `Button`, `Form`, `FormField`, `LinkType`,
`FormFieldType`.
**`abstract/interactive_image.py`** — `InteractiveImage`, an `Image` that is also
`Interactable` and `Queriable`.
### `concrete/` — spatial realisation
Objects that know their position and size and can draw themselves onto a canvas.
**`concrete/text.py`** — the heart of text layout:
- `Text` — one renderable fragment (a whole word, or a hyphenated part). Requires an
`ImageDraw.Draw` at construction so it can measure itself immediately.
- `Line` — a sequence of `Text` objects with resolved spacing and a baseline.
`Line.add_word()` is where a `Word` becomes one or two `Text` objects: it measures,
and if the word overflows it tries dictionary hyphenation, then brute-force splitting,
before rejecting the word.
- `AlignmentHandler` and its subclasses (`LeftAlignmentHandler`,
`CenterRightAlignmentHandler`, `JustifyAlignmentHandler`) — the strategy objects that
turn a set of measured fragments plus an available width into concrete spacing and a
start position.
- Cache management: `configure_text_caches`, `clear_text_caches`, `text_cache_stats`,
`prewarm_text_caches`.
**`concrete/page.py`** — `Page`: a fixed-size canvas holding `Renderable` children, with
a content rectangle derived from its `PageStyle`, `can_fit_line()` for the layouter to
test against, `render()` returning a `PIL.Image`, and `query_point()` / `query_range()`
for hit-testing.
**`concrete/box.py`** — `Box`, the `Geometric` + `Renderable` + `Queriable` base for
positioned drawable objects.
**`concrete/dynamic_page.py`** — `DynamicPage` (a `Page` subclass) and `SizeConstraints`.
Adds a two-phase measure-then-layout protocol, so a container such as a table can learn
its content's intrinsic size before committing to a size allocation.
**`concrete/image.py`** — `RenderableImage`.
**`concrete/table.py`** — `TableRenderer`, `TableRowRenderer`, `TableCellRenderer`,
`TableStyle`. Cells host their own nested `Page`, which is why `Page` accepts a
non-zero `origin`.
**`concrete/functional.py`** — `LinkText`, `ButtonText`, `FormFieldText`: `Text`
subclasses that are also `Interactable`.
**`concrete/interaction_handler.py`** — `InteractionHandler`, `InteractionStateManager`:
routing taps and presses to registered elements and tracking pressed/released state.
### `layout/` — the abstract → concrete transformation
This is the package the pipeline diagram calls the layout engine.
**`layout/document_layouter.py`** — a set of layouter functions, one per content kind,
all sharing a signature shape of *(abstract element, target `Page`) → did it fit*:
```python
# An Abstract Word knows its text content and semantic properties
word = Word("supercalifragilisticexpialidocious", font_style)
word.hyphenate() # Logical operation - finds break points
parts = word.get_hyphenated_parts() # Returns ["super-", "cali-", "fragi-", ...]
paragraph_layouter(paragraph, page, start_word=0, pretext=None, alignment_override=None)
-> (complete: bool, failed_word_index: int | None, remaining_pretext: Text | None)
image_layouter(image, page, max_width=None, max_height=None) -> bool
table_layouter(table, page, style=None) -> bool
pagebreak_layouter(page_break, page) -> bool
button_layouter(button, page, font=None, padding=...) -> (bool, str)
form_field_layouter(field, page, font=None, ...) -> ...
form_layouter(form, page, font=None, field_spacing=10) -> (bool, list[str])
```
## Concrete Layer (`pyWebLayout/concrete/`)
They **append to a page**, they do not return a list of lines. The three-part return of
`paragraph_layouter` is what makes pagination resumable: when a paragraph runs off the
bottom of a page, the caller learns which word failed and whether a hyphenated fragment
is pending, and can continue on the next page from exactly there.
The Concrete layer handles the **spatial representation** and actual rendering of content.
`DocumentLayouter` wraps a `Page` and dispatches over a list of abstract elements by
type, holding the `ConcreteStyleRegistry` for the run.
### Key Components
**`layout/ereader_layout.py`** — the paginated reading model:
- `RenderingPosition` — a serialisable cursor expressed in *abstract* coordinates
(chapter, block, word, table cell, list item, pending pretext). Because it names
document structure rather than pixels, it survives font-size and page-size changes.
- `BidirectionalLayouter` — renders a page forward or backward from a position,
returning `(Page, next_position)`.
- `ChapterNavigator` / `ChapterInfo` — a table of contents built from heading structure.
- `FontScaler`, `FontFamilyOverride` — apply scale and family changes to blocks at
layout time without mutating the abstract document.
#### `concrete/text.py`
- `Text`: Renders a specific text fragment with precise positioning
- `Line`: Manages a line of `Text` objects with spacing and alignment
- Handles actual pixel measurements, font rendering, and positioning
**`layout/page_buffer.py`** — `PageBuffer` (LRU caches of rendered pages in both
directions, plus position maps) and `BufferedPageRenderer` (background rendering).
#### `concrete/page.py`
- `Page`: Top-level container for rendered content
- `Container`: Layout manager for organizing renderable objects
- Handles spatial layout, pagination, and visual composition
**`layout/ereader_manager.py`** — `EreaderLayoutManager`, the top-level application
interface (page turns, font changes, chapter jumps, progress), and `BookmarkManager`
for persisting bookmarks and the last reading position.
#### `concrete/box.py`
- `Box`: Base class for all spatially-aware renderable objects
- Provides positioning, sizing, and rendering capabilities
**`layout/table_optimizer.py`** — column width allocation for tables.
### Characteristics of Concrete Classes
### `io/readers/` — parsing
1. **Rendering-focused**: Handle pixels, fonts, images, and visual output
2. **Spatially-aware**: Know exact positions, sizes, and layout constraints
3. **Implementation-specific**: Tied to specific rendering technologies (PIL, etc.)
4. **Non-portable**: Rendering results are tied to specific display contexts
**`html_extraction.py`** — `parse_html_string(html, base_font=None, document=None,
base_path=None) -> List[Block]`. Built from a `StyleContext` (a `NamedTuple` threaded
down the tree, carrying the inherited font and styling) and a table of per-tag handlers
(`paragraph_handler`, `heading_handler`, `table_handler`, …). This is the only place
that knows about HTML.
### Example: Concrete Text
**`epub_reader.py`** — `EPUBReader` and `read_epub(path) -> Book`: container/OPF
parsing, spine and manifest, table of contents, cover handling, and image processing
(with an e-ink processor available by default).
## Dependency rules
The direction of dependency is what keeps the split honest:
```
io/readers → abstract → core, style
layout → abstract, concrete, core, style
concrete → abstract, core, style
abstract → core, style ← must NOT import concrete
```
`abstract/` importing from `concrete/` is the violation to watch for. `concrete/`
importing from `abstract/` is expected and correct: a `Text` may point back at the
`Word` it came from, and a table cell renderer reads its abstract `TableCell`.
**Where the abstract layer touches rendering today, and why:**
- Abstract classes are constructed with `Font` objects, which carry a pixel size and a
loaded font file. This is a deliberate compromise for parsing performance (HTML
styling resolves to a `Font` once, at parse time) but it does mean the abstract layer
is not fully rendering-independent. `AbstractStyle` is the intended replacement, and
`Word` already accepts either.
- `Word.concrete` / `Word.add_concete()` ([inline.py:42](pyWebLayout/abstract/inline.py#L42),
[:134](pyWebLayout/abstract/inline.py#L134)) is a back-reference to the `Text` objects
a word became. **It is currently written and never read**, and it cannot correctly
model the relationship anyway: one `Word` becomes many `Text`s across re-layouts, and
a single slot only remembers the most recent. Treat it as vestigial. When a
word→pixels mapping is genuinely needed, build it on `Text`'s `source` back-reference
(concrete pointing at abstract), which is the safe direction — note it is currently
stored as `_source` with no public accessor, and is itself unread today.
## Worked example
```python
# A Concrete Text object handles actual rendering
text = Text("super-", font) # Specific text fragment
text._calculate_dimensions() # Computes exact pixel size
image = text.render() # Produces actual visual output
from pyWebLayout.io.readers.html_extraction import parse_html_string
from pyWebLayout.concrete.page import Page
from pyWebLayout.layout.document_layouter import DocumentLayouter
# 1. Parse: markup -> abstract blocks
blocks = parse_html_string("<h1>Chapter One</h1><p>It was a dark and stormy night.</p>")
# [Heading, Paragraph]
# 2. Lay out: abstract blocks -> concrete children appended to a Page
page = Page(size=(400, 600))
DocumentLayouter(page).layout_document(blocks)
# 3. Render: Page -> PIL.Image
image = page.render()
# 4. Query: pixel -> content
result = page.query_point((50, 40))
print(result.object_type, result.text) # e.g. "text" "Chapter"
```
## The Transformation Process
The architecture involves a clear transformation from Abstract to Concrete:
```
Abstract Document
[Parser Layer]
Abstract Blocks (Paragraph, Heading, etc.)
[Layout Engine]
Concrete Objects (Text, Line, Page)
[Rendering Engine]
Visual Output (Images, PDF, etc.)
```
### Example Transformation
Inspecting the intermediate concrete objects:
```python
# 1. Abstract content
paragraph = Paragraph()
paragraph.add_word(Word("This", font))
paragraph.add_word(Word("is", font))
paragraph.add_word(Word("a", font))
paragraph.add_word(Word("test", font))
from pyWebLayout.concrete.text import Line
# 2. Layout transformation
layout = ParagraphLayout(line_width=200, line_height=20)
lines = layout.layout_paragraph(paragraph) # Returns List[Line]
# 3. Each Line contains concrete Text objects
for line in lines:
for text_obj in line.text_objects: # List[Text]
print(f"Text: '{text_obj.text}' at position {text_obj._origin}")
for line in (c for c in page.children if isinstance(c, Line)):
print([t.text for t in line.text_objects])
# ['Chapter', 'One']
# ['It', 'was', 'a', 'dark', 'and', 'stormy', 'night.']
```
## Key Architectural Principles
For paginated reading, drive `EreaderLayoutManager` instead of building pages by hand:
### 1. **Single Responsibility**
- Abstract classes: Handle content and structure
- Concrete classes: Handle rendering and layout
### 2. **Separation of Concerns**
- Text parsing/processing ≠ Text rendering
- Document structure ≠ Page layout
- Content semantics ≠ Visual presentation
### 3. **Immutable Abstract Content**
- Abstract content remains unchanged during rendering
- Multiple concrete representations can be generated from same abstract content
- Enables pagination, different formats, responsive layouts
### 4. **One-to-Many Relationships**
- One Abstract Word → Multiple Concrete Text objects (hyphenation)
- One Abstract Paragraph → Multiple Concrete Lines
- One Abstract Document → Multiple Concrete Pages
## Common Anti-Patterns to Avoid
### ❌ **Mixing Concerns**
```python
# WRONG: Abstract class knowing about pixels
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager
manager = EreaderLayoutManager(blocks, page_size=(800, 600), document_id="my-book")
page = manager.get_current_page()
page = manager.next_page()
manager.set_font_scale(1.25) # re-lays out from the same RenderingPosition
```
## Design principles
**1. Abstract content is not mutated by layout.** Font scaling and family overrides
produce scaled copies at layout time rather than editing the document. This is what
allows the same `blocks` list to back a buffer of pages at several font sizes.
**2. Positions are expressed in abstract coordinates.** `RenderingPosition` names a
chapter, block, and word — never a pixel. A reader who changes font size stays on the
same sentence.
**3. Layouters report partial success.** Every layouter returns whether it fit, and the
text layouter also returns where it stopped. Pagination is built from this rather than
from a separate page-breaking pass.
**4. One abstract object may become many concrete ones.** One `Word` → one or two `Text`
fragments; one `Paragraph` → many `Line`s across many `Page`s; one `Document` → an
unbounded sequence of `Page`s. Any API that assumes a one-to-one mapping will be wrong
at a hyphen or a page boundary.
**5. Measurement is cached, not avoided.** Text width and glyph rasterisation are the
hot path. `core/cache.py` bounds their cost; `prewarm_text_caches` front-loads it for a
known document.
## Anti-patterns
**Concrete state stored on abstract objects.** An abstract object that remembers its
rendered width, position, or `Text` objects is wrong at the second rendering. If a
back-reference is needed, point from concrete to abstract.
```python
# WRONG
class Word:
def __init__(self, text):
self.text = text
self.rendered_width = None # ❌ Concrete concern in abstract class
self.rendered_width = None # invalidated by any font change
# RIGHT
text = Text(word.text, font, draw, source=word) # concrete knows its origin
```
### ❌ **renderable_words Concept**
```python
# WRONG: Confusing abstract and concrete
line.renderable_words # ❌ This suggests Words are renderable
# Words are abstract - only Text objects render
```
**Treating `Word` as renderable.** Words are abstract; only `Text` draws. There is no
`renderable_words` anywhere in the codebase, and there should not be.
### ✅ **Correct Separation**
```python
# CORRECT: Clear separation
abstract_word = Word("test") # Abstract content
concrete_text = Text("test", font) # Concrete rendering
line.text_objects.append(concrete_text) # Concrete objects in concrete container
```
**Assuming a layouter returns lines.** `paragraph_layouter` appends to a page and
reports what did not fit. Code that expects `List[Line]` back is working from an
outdated model.
## Benefits of This Architecture
## Summary
### 1. **Flexibility**
- Same content can be rendered at different sizes
- Multiple output formats from single source
- Easy to implement responsive design
### 2. **Testability**
- Abstract logic can be tested without rendering
- Layout algorithms can be tested independently
- Visual rendering can be mocked
### 3. **Performance**
- Abstract content can be cached and reused
- Layout can be computed once for multiple renderings
- Incremental updates possible
### 4. **Maintainability**
- Clear boundaries between text processing and rendering
- Changes to rendering don't affect content parsing
- Easy to swap rendering backends
## File Organization
```
pyWebLayout/
├── abstract/ # Content and structure
│ ├── block.py # Document blocks (Paragraph, Heading, etc.)
│ ├── inline.py # Inline content (Word, etc.)
│ ├── document.py # Document structure
│ └── functional.py # Links, buttons, etc.
├── concrete/ # Rendering and layout
│ ├── text.py # Text and Line rendering
│ ├── page.py # Page layout and containers
│ ├── box.py # Base rendering classes
│ ├── image.py # Image rendering
│ └── functional.py # Interactive elements
├── typesetting/ # Layout algorithms
│ ├── paragraph_layout.py # Abstract → Concrete transformation
│ ├── flow.py # Text flow management
│ └── pagination.py # Page breaking logic
└── style/ # Styling and formatting
├── fonts.py # Font management
├── layout.py # Layout constants
└── alignment.py # Alignment enums
```
## Conclusion
The Abstract/Concrete separation is fundamental to pyWebLayout's design. It ensures clean separation between content processing and visual rendering, enabling flexible, maintainable, and testable document processing pipelines.
**Remember**:
- **Abstract** = What to display (content, structure, semantics)
- **Concrete** = How to display it (pixels, fonts, positioning, rendering)
This architecture enables the library to handle complex document layouts while maintaining clear, understandable code organization.
- **`core/`** — contracts and shared machinery
- **`style/`** — semantic style, and its resolution to concrete rendering parameters
- **`abstract/`** — what the document says
- **`concrete/`** — where the pixels go
- **`layout/`** — the transformation between them, and pagination on top of it
- **`io/readers/`** — markup in
+527
View File
@@ -0,0 +1,527 @@
# Architecture Review
Independent review of the codebase at `c5c61a3` (2026-08-06), answering one
question: **is this a well-architected library or an over-complex mess?**
It is a well-architected library with one rotten subsystem inside it. The core
design holds up; roughly a fifth of the code is speculative or non-functional,
and it is concentrated in the ereader pagination/buffering layer.
This document records the verdict, the evidence, and the findings **not already
covered** by [LAYOUT_REMEDIATION_SPEC.md](LAYOUT_REMEDIATION_SPEC.md). Where a
finding is already specced, it is cross-referenced rather than restated.
## Contents
| ID | Finding | Severity | Status |
|----|---------|----------|--------|
| [R1](#r1--the-process-pool-crashes-on-python-314) | The process pool crashes on Python 3.14 | Critical | New; raises priority of S12 |
| [R2](#r2--the-test-suite-hangs-at-interpreter-exit) | Test suite hangs at interpreter exit | High | New; same root cause as R1 |
| [R3](#r3--font-scaling-destroys-hyperlinks) | Font scaling destroys hyperlinks | High | New |
| [R4](#r4--three-packaging-configs-that-disagree) | Three packaging configs that disagree | Medium | New (corrected) |
| [R5](#r5--monkey-patched-page-methods-with-a-conflicting-signature) | Monkey-patched `Page` methods with a conflicting signature | Medium | New |
| [R6](#r6--dead-duck-typing-cluster-in-pagepy) | Dead duck-typing cluster in `page.py` | Low | New; extends S10.3 |
| [R7](#r7--two-orphaned-subsystems) | Two orphaned subsystems | Low | New |
| [R8](#r8--backward-pagination-is-guesswork) | Backward pagination is guesswork | Medium | Partially noted in S11 |
---
## Verdict
**Well architected.** The concerns that usually decide this question are all on
the right side of the line:
- **The abstract/concrete split is real, not aspirational.** Verified
empirically: `abstract/` never imports `concrete/`; `core/` imports nothing but
itself. The single crossing ([document.py:7](../pyWebLayout/abstract/document.py#L7))
is into `style/`, which the dependency rules permit. Most codebases claiming
this layering have leaked it within a year.
- **The layouter contract is the right abstraction.** `paragraph_layouter`
returning `(fit, failed_word_index, remaining_pretext)` is what makes
pagination resumable, and the shape is consistent across content types. Layout
engines that return `List[Line]` cannot paginate without a second pass.
- **[core/cache.py](../pyWebLayout/core/cache.py) is exemplary.** Usage-ranked
eviction with periodic aging, sampled eviction instead of a maintained heap,
O(1) hit path with no reordering — every choice justified by a measurement in
the docstring.
- **[concrete/text.py](../pyWebLayout/concrete/text.py) is the strongest file.**
The glyph cache ([:581-647](../pyWebLayout/concrete/text.py#L581-L647))
reimplements PIL's internals to skip per-call setup, with a permanent graceful
fallback when the private API is absent. Alignment is a clean strategy pattern,
and `render_alignment_handler` handles last-line-of-paragraph correctly.
- **[html_extraction.py](../pyWebLayout/io/readers/html_extraction.py) is
textbook.** An immutable `StyleContext` threaded down the tree plus a handler
dispatch table — no giant if/elif, no mutable parser state.
**The complexity that is not earned** is concentrated in three places, all in the
same band of the code:
1. `layout/page_buffer.py` — 520 lines of multiprocess prefetch that has never
worked (S12, plus R1/R2 below).
2. `BidirectionalLayouter.render_page_backward` — ~100 lines of convergence
heuristics standing in for an anchor list that already exists (R8).
3. The second block dispatcher in `ereader_layout.py`, which silently drops
tables and lists (S4, S8).
The pattern is visible in the git history: work from `e000068` onward (caching,
alignment, page geometry) is markedly better than the ereader scaffolding it sits
on. This is not a mess. It is a solid library with an early prototype still
embedded in it.
**Sizing the cleanup:** R1R7 plus S12 and S10.3 remove roughly 800900 lines and
fix four user-visible defects. None of it requires redesigning anything.
## Test baseline
At `c5c61a3`, in a clean venv on Python 3.14.6:
```
833 passed, 2 skipped, 24 subtests passed in 11.51s
```
(The 2 skips were environmental — the review venv lacked `requests`, so the URL
image tests skipped. With the `test` extra from R4 installed the suite reports
`853 passed, 24 subtests passed in 13.53s`.)
The suite then **hangs indefinitely** rather than exiting. See R2.
## Status of the existing remediation spec
| Spec | Subject | State |
|------|---------|-------|
| S1 | Inline content in non-paragraph containers | Done (`284d521`) |
| S2 | Page geometry: origin and content rect | Done (`f18cec2`) |
| S3 | Draw/canvas lifecycle | Done (`202dacf`) |
| S11 | Partial-block progress discarded | Done (`a57da80`) |
| S13 | Word spacing and alignment | Done (`1262be6`) |
| S14 | Vertical centring in buttons and fields | Done (`c5c61a3`) |
| S4S10, S12 | Dispatch, cells, table grid, pagination, hygiene, background rendering | Outstanding |
The spec's analysis is sound and in places sharper than this review — S12 caught
that `_render_page_worker` omits `page_size` entirely, which this review missed.
Nothing below supersedes it.
---
## R1 — The process pool crashes on Python 3.14
**Severity: critical. Raises S12 from "useless" to "fatal".**
### Problem
`PageBuffer` submits to a `ProcessPoolExecutor` from inside
`BufferedPageRenderer.render_page`
([page_buffer.py:431](../pyWebLayout/layout/page_buffer.py#L431)). Python 3.14
changed the default multiprocessing start method on Linux from `fork` to
`forkserver`. Under a non-`fork` start method, `submit()` reaches
`_check_not_importing_main()`, which raises unless the caller sits inside an
`if __name__ == "__main__":` guard — and the child re-imports the caller's main
module, re-executing it.
S12 documents this subsystem as delivering no benefit. On 3.14 it is worse than
that: `EreaderLayoutManager.get_current_page()` **raises** when called from
module-level script code.
### Evidence
A plain script calling `manager.get_current_page()` at module level, Python
3.14.6:
```
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
...
File "pyWebLayout/layout/page_buffer.py", line 221, in _queue_forward_renders
future = self.executor.submit(_render_page_worker, args)
ConnectionResetError: [Errno 104] Connection reset by peer
```
With a `__main__` guard added it does not raise, and instead confirms S12's
finding on every job:
```
Background render failed for position RenderingPosition(...): cannot pickle 'ImagingCore' object
Background render failed for position RenderingPosition(...): cannot pickle 'ImagingCore' object
```
### Action
Fold into **S12**, and treat S12 as unblocked and urgent rather than phase 4. The
recommended resolution there — delete the pool, keep the LRU buffers and position
maps, replace prefetch with synchronous readahead — resolves R1 and R2 as a side
effect. S12's measurement gate still applies to the *readahead* decision; it does
not need to gate deletion of the pool, because the pool's contribution is
provably zero.
### Files
`pyWebLayout/layout/page_buffer.py`
---
## R2 — The test suite hangs at interpreter exit
**Severity: high.** Same root cause as R1.
### Problem
`PageBuffer.__del__` calls `shutdown()`, which calls
`executor.shutdown(wait=True)` ([page_buffer.py:342](../pyWebLayout/layout/page_buffer.py#L342)).
`EreaderLayoutManager.__del__` does the same via `renderer.shutdown()`. Running
`__del__` at interpreter shutdown and blocking on a process pool inside it
deadlocks.
### Evidence
```
833 passed, 2 skipped, 24 subtests passed in 11.51s
```
...then the process sat at ~0% CPU with idle forkserver children for 13 minutes
before being killed. Reproduced twice; both runs completed the tests in under
12s and neither exited.
This is why CI wall-clock does not resemble the 11.5s the tests actually take.
### Action
Resolved by S12's deletion of the executor. If for any reason the pool is
retained, `__del__` must not block: register an `atexit` handler or require
explicit `shutdown()`, and never `wait=True` from a finaliser.
### Acceptance criteria
- `pytest` returns to the shell within a second of printing its summary line.
- No `multiprocessing` child processes outlive the test session.
### Files
`pyWebLayout/layout/page_buffer.py`, `pyWebLayout/layout/ereader_manager.py`
---
## R3 — Font scaling destroys hyperlinks
**Severity: high. User-visible, silent, and trivially reproducible.**
### Problem
`BidirectionalLayouter._scale_block_fonts`
([ereader_layout.py:474-498](../pyWebLayout/layout/ereader_layout.py#L474-L498))
rebuilds a scaled block by constructing plain `Word(word.text, scaled_style)` for
every word. `LinkedWord` is a `Word` subclass
([inline.py:288](../pyWebLayout/abstract/inline.py#L288)), so the reconstruction
downgrades it and the link target is discarded.
The function returns the block unchanged when `font_scale == 1.0` and no family
override is set, which is why no test has caught this: the defect only appears
once the reader changes font size.
Two further gaps in the same function:
1. It handles only `Paragraph` and `Heading`. Every other block type is returned
unscaled, so a font-size change leaves images, tables and lists at their
original size while the text around them reflows.
2. It allocates a new `Paragraph` and a new `Word` per word on **every page
render** at any scale ≠ 1.0 — directly against the caching work in
`concrete/text.py`, and on the hot path.
### Evidence
At `c5c61a3`, parsing `<p>Go to <a href="http://x">this link</a> now.</p>`:
```
scale=1.0: LinkedWords = 2
scale=1.5: LinkedWords = 0
```
### Design
Stop reconstructing abstract blocks at layout time. Font scale and family are
*rendering context*, not document content — carrying them in a copied document
violates the "abstract content is not mutated by layout" principle in
[ARCHITECTURE.md](../ARCHITECTURE.md) in spirit, even though it copies rather
than mutates.
Preferred: thread the scale/override into the layouter and resolve fonts at
`Text` construction, where `Font` objects are already deduplicated by
`FontRegistry`. `paragraph_layouter` already accepts an `alignment_override`;
`font_scale` and `font_family_override` belong in the same place.
Minimum viable fix if the larger change is deferred: reconstruct via
`type(word)` and copy subclass state, and extend coverage to every block type.
This is strictly a stopgap — it keeps the per-page allocation cost.
### Acceptance criteria
- A document containing `<a href>` retains every `LinkedWord` after
`set_font_scale(1.5)`, and `query_point` over the rendered page still returns
`object_type="link"` with the correct target.
- An image block's rendered size is unaffected by `set_font_scale`, or scales
deliberately — not left inconsistent with the text around it.
- No new `Word`/`Paragraph` objects are allocated per page render at scale ≠ 1.0.
### Files
`pyWebLayout/layout/ereader_layout.py`, `pyWebLayout/layout/document_layouter.py`
---
## R4 — Three packaging configs that disagree
**Severity: medium.** *Corrected: the original review claimed a clean install
fails on first import. It does not — see below.*
### Problem
The project carries **three** sets of packaging metadata:
| File | Declares |
|------|----------|
| `pyproject.toml` `[project]` | Pillow, numpy, pyphen, beautifulsoup4, flask, ebooklib, requests |
| `setup.cfg` `[options]` | Pillow, numpy |
| `setup.py` `setup(...)` kwargs | Pillow, numpy |
`pyproject.toml`'s `[project]` table wins under any modern build backend, so the
shipped wheel is correct and `pip install pyWebLayout` works. The `setup.cfg` and
`setup.py` copies are dead, contradictory, and actively misleading — reading
either one gives the wrong answer about what the library needs.
The authoritative list is itself wrong in the other direction:
- **`flask` is a runtime dependency.** It is imported only by
`tests/abstract/test_abstract_blocks.py`, as a fixture HTTP server. Every user
installs Flask, Jinja2, Werkzeug, click, itsdangerous and blinker for nothing.
- **`ebooklib` is a runtime dependency and is never imported by the library.**
`epub_reader.py` uses `zipfile` + `xml.etree` directly. Only the *tests* use
ebooklib, to build EPUB fixtures.
- **`requests` is declared required but is optional.** `concrete/image.py:100-111`
imports it lazily and degrades to an error message on the image when absent.
- **`requires-python = ">=3.6"` is false.** The package uses dataclasses (3.7+)
and `from __future__ import annotations` (3.7+); CI tests 3.10, 3.12 and 3.13.
Net effect: a runtime install pulls 7 direct dependencies where 4 are needed.
### Action
- Consolidate on `pyproject.toml`. Reduce `setup.cfg` to its `[flake8]` section
and `setup.py` to a `setup()` shim, each with a comment saying where metadata
lives.
- Runtime deps: Pillow, numpy, pyphen, beautifulsoup4. Move flask, werkzeug,
ebooklib and requests into a `test` extra; add a `remote-images` extra for
requests; add a `dev` extra composing them.
- Set `requires-python = ">=3.10"` to match the CI matrix, and add version
classifiers.
- Add a CI step that installs the package into an empty venv with **only**
declared runtime deps and imports every top-level subpackage. This class of
defect is only caught by installing what you ship — and it is what would have
caught the original misreading.
### Files
`pyproject.toml`, `setup.cfg`, `setup.py`, `.gitea/workflows/ci.yml`
---
## R5 — Monkey-patched `Page` methods with a conflicting signature
**Severity: medium.** Currently inert; a landmine if `Page` is ever refactored.
### Problem
[ereader_layout.py:741-761](../pyWebLayout/layout/ereader_layout.py#L741-L761)
defines `_add_page_methods()` and calls it at import time. It attaches
`can_fit_line` and `available_width` to the `Page` class if they are absent.
`Page` defines both ([page.py:59](../pyWebLayout/concrete/page.py#L59),
[page.py:147](../pyWebLayout/concrete/page.py#L147)), so the patch never fires.
But the two definitions of `can_fit_line` **do not agree**:
| Source | Signature |
|--------|-----------|
| `Page` | `can_fit_line(baseline_spacing, ascent=0, descent=0)` |
| monkey patch | `can_fit_line(line_height)` |
The patched version also ignores descenders entirely — the exact bug S2 fixed. If
`Page.can_fit_line` were ever renamed or moved, this would silently reinstate
pre-S2 clipping behaviour, from an import side effect in a different package.
### Action
Delete `_add_page_methods` and its call site. Import-time monkey-patching of a
class in another module has no place here; if `Page` is missing something the
layout engine needs, it belongs on `Page`.
### Files
`pyWebLayout/layout/ereader_layout.py`
---
## R6 — Dead duck-typing cluster in `page.py`
**Severity: low.** Extends S10.3.
### Problem
[page.py:261-491](../pyWebLayout/concrete/page.py#L261-L491) contains a closed
cluster with no external callers:
- `_get_child_property` (:261) — called only by the four below
- `_get_child_height` (:301) — called by nothing
- `_get_child_position` (:382) — called only by `_point_in_child`
- `_point_in_child` (:435) — called by nothing
- `_get_child_size` (:466) — called only by `_point_in_child`
Verified by grep across `pyWebLayout/`, `tests/`, `examples/` and `scripts/`:
zero references outside the cluster. About 90 lines.
It exists because `Renderable` declares neither `size` nor `origin`, so the code
probes `_size`, `size`, `_height`, `height`, `_origin` and `position` in turn with
`hasattr`. `query_point` (:399) already does the right thing instead — it relies
on the `Queriable` interface.
### Action
- Delete all five methods.
- Add `origin` and `size` to the `Renderable`/`Geometric` contract in
`core/base.py` so the duck-typing cannot grow back. This is the same concern as
S10.1's render contract and can ship with it.
### Files
`pyWebLayout/concrete/page.py`, `pyWebLayout/core/base.py`
---
## R7 — Two orphaned subsystems
**Severity: low**, but they are a large share of the "is this over-complex?"
impression: 559 lines that nothing in the library reaches.
### Problem
**`concrete/interaction_handler.py` (310 lines).** `InteractionHandler` and
`InteractionStateManager` are referenced only by
`examples/07_pressed_state_demo.py`. No library code, no ereader path, no tests.
**`core/highlight.py` (249 lines).** `Highlight`, `HighlightColor` and
`HighlightManager` have tests (`tests/core/test_highlight.py`) but are not wired
into `EreaderLayoutManager` at all. Highlighting is not reachable through the
library's own top-level interface.
`HighlightManager` also duplicates `BookmarkManager`'s JSON persistence
(directory, `_save`, `_load`, per-document file naming) with no shared base.
### Action
Decide per subsystem, and record the decision:
- **Wire it up** — `EreaderLayoutManager` grows `add_highlight` / `highlights_for_page`
and the persistence merges with `BookmarkManager` into one document-state store.
- **Or move it out** — relocate to `examples/` or delete, and drop the tests with it.
Either is fine. Leaving a tested, documented, unreachable subsystem in `core/` is
what makes the library look larger and less coherent than it is.
### Files
`pyWebLayout/concrete/interaction_handler.py`, `pyWebLayout/core/highlight.py`,
`pyWebLayout/layout/ereader_manager.py`
---
## R8 — Backward pagination is guesswork
**Severity: medium.** S11's closing note already flags this for audit; this
records what the audit found.
### Problem
`render_page_backward`
([ereader_layout.py:372-472](../pyWebLayout/layout/ereader_layout.py#L372-L472))
finds the previous page by estimating a start position, rendering forward,
comparing the end against the target, and adjusting — **up to 10 times**. It then
has a fallback that jumps back up to 5 blocks and renders again, and a fallback
for *that* which renders from the start of the document.
Worst case: one "previous page" tap costs up to 12 full page layouts.
The estimator it converges from is `max(1, int(10 / font_scale))` blocks
([:684](../pyWebLayout/layout/ereader_layout.py#L684)) — a constant with no
relationship to page size, block length or font metrics.
The correct answer is usually already known.
`EreaderLayoutManager._page_history` ([ereader_manager.py:210](../pyWebLayout/layout/ereader_manager.py#L210))
records real page-start positions and serves them instantly; the refinement loop
only runs when history misses — after a jump, a bookmark, a font change, or
beyond 50 entries.
S11's note asks whether these fallbacks were compensating for the
discarded-progress bug it fixed. They were, in part: the "failed to move
backward" branch at [:446](../pyWebLayout/layout/ereader_layout.py#L446) is
reachable precisely when forward rendering fails to advance, which S11 addressed.
### Design
Replace convergence with anchors. Maintain a sorted list of known page-start
positions — chapter starts from `ChapterNavigator` (free, already built) plus
every position visited. To go back from position P: binary-search the largest
anchor A < P, render forward from A collecting page starts until reaching P, and
return the last one. Cost is bounded by the anchor spacing, and every page start
discovered on the way is itself a new anchor, so the second traversal of any
region is free.
This subsumes `_page_history`, so the two mechanisms become one.
**Sequencing:** do this after S12, and after S8 — table and list pagination
changes what a page start can be, and re-deriving anchors is cheap only once
positions round-trip through tables correctly (S8 already notes this dependency).
### Acceptance criteria
- `previous_page()` from any position issues at most *k* page layouts, where *k*
is the anchor spacing, with no iteration count and no fallback ladder.
- Forward-then-backward round-trips exactly, from a cold cache, after a chapter
jump, and after a bookmark restore.
- `_estimate_page_start`, `_adjust_start_estimate` and the three-tier fallback
are deleted, not retained alongside.
### Files
`pyWebLayout/layout/ereader_layout.py`, `pyWebLayout/layout/ereader_manager.py`
---
## Recommended order
```
R4 ── packaging; independent, minutes, unblocks clean CI [done]
S12 ── delete the process pool; resolves R1 and R2 with it
R3 ── font scaling loses links; independent, user-visible
R5 ── delete the monkey patch; minutes
R6 ── delete the dead cluster (with S10.1's render contract)
R7 ── decide the two orphans; no code risk either way
S4 → S5 → S6 → S7 → S8 → S9 (existing spec, unchanged)
R8 ── after S8
```
R4, R5 and R6 are an afternoon and carry no design risk. S12 is the largest
single removal and fixes two defects at once. R3 is the one users would notice
today. Everything after that is the existing spec, which needs no revision.
## Reproducing the findings
Reviewed at `c5c61a3` on Python 3.14.6, in a venv containing
`pytest pyphen Pillow numpy beautifulsoup4 lxml ebooklib`.
- **R1**: call `EreaderLayoutManager(...).get_current_page()` from module-level
script code (no `__main__` guard).
- **R2**: `python -m pytest -q`; observe the summary line, then the hang.
- **R3**: parse HTML containing `<a href>`, call
`BidirectionalLayouter._scale_block_fonts(block, 1.5)`, count `LinkedWord`
instances in the result.
- **R5**: compare `inspect.signature(Page.can_fit_line)` against the patch body.
- **R6**: grep the five method names across `pyWebLayout/ tests/ examples/ scripts/`.
- **R8**: read the loop; no execution needed.
+98
View File
@@ -30,6 +30,7 @@ It is independent of every other spec here.
| [S13](#s13--word-spacing-and-alignment) | Word spacing and alignment | 0 |
| [S14](#s14--vertical-centring-in-buttons-and-fields) | Vertical centring in buttons and fields | 0 |
| [S15](#s15--form-field-label-geometry) | Form field label geometry | 0 |
| [S16](#s16--backward-page-navigation) | Backward page navigation | 0 |
## Design invariants
@@ -1265,6 +1266,103 @@ between label and box smaller than the intended 5px.
---
## S16 — Backward page navigation
### Problem
`render_page_backward` *searched* for the previous page's start: estimate a block
index, lay out forward, compare the end against the target, bisect on the block
difference, repeat up to ten times. Both the estimator and the adjuster pinned
`word_index` to 0 and moved only `block_index`.
Pages routinely start mid-block. Any such start was therefore **not in the search
space**, the loop could never match, and it fell through to a fallback that
jumped several blocks back or to the document start.
### Evidence
A document of short paragraphs around one 1200-word paragraph. Forward pagination
gives page starts at `(0,0), (2,208), (2,494), (2,780), (2,1057)`. Asking for the
page that ends where each of those begins:
```
from page 1 -> got (0,0) expected (0,0) ok (1 forward layout)
from page 2 -> got (0,0) expected (2,208) WRONG (10 forward layouts)
from page 3 -> got (0,0) expected (2,494) WRONG (10 forward layouts)
from page 4 -> got (0,0) expected (2,780) WRONG (10 forward layouts)
```
Every mid-paragraph case threw the reader to the start of the document after ten
full page layouts. The bisection was also unsound within its own space: a
document of 40 small paragraphs, where every page *does* start on a block
boundary, failed too.
This is complementary to S11 rather than caused by it. Before S11 forward
pagination dead-ended at the first page-spanning block, so mid-block starts were
never produced and the block-granular search looked adequate.
### Design
Pagination is a pure function: laying out from `q` yields a page and the position
it stopped at, `next(q)`. The page before `P` is the `q` with `next(q) == P`.
That is found by **replaying the chain forward from an anchor**, not by guessing
`q`. Three sources, in order:
1. **The recorded chain.** `render_page_forward` now records
`(font_scale, next(q)) -> q`. Stepping back to anywhere the reader has been is
exact and costs one layout. Keyed by font scale, since changing it
repaginates.
2. **Replay from an anchor.** Anchors are block starts, nearest first: the block
containing `P`, then up to `MAX_BACKWARD_ANCHORS` earlier ones, then the
document start. Lay out forward from the anchor until a page ends exactly on
`P`; that page's start is the answer. `MAX_REPLAY_PAGES` caps the walk so one
page turn cannot traverse a whole chapter.
3. **Nearest start before `P`.** If no chain passes exactly through `P` — which
happens when `P` was reached by a jump or a restored bookmark rather than by
reading forward, so it lies on no natural chain — return the last page start
before it. That overlaps `P`'s page slightly rather than skipping content,
which is the safe direction to be wrong in.
The estimator and the bisecting adjuster are deleted.
**What "correct" means here.** Each backward step returns a page ending exactly
where the reader currently is, so paging back never skips or repeats content.
That chain can differ from the one you would have seen reading forward from page
one, if you entered the document by a jump — pagination from a different starting
point is genuinely a different chain, and no algorithm can recover the original
without replaying from the start.
### Measurements
Same document, after the change:
```
warm (chain recorded by the forward pass): 4/4 exact, 1 layout each
cold, fresh layouter per call: 12/13 exact, worst 17 layouts
cold, one layouter, repeated back presses: 4 layouts per turn typical
```
The single inexact case is a target that lies on the canonical chain but not on
any chain reachable from a nearby anchor; it returns a start 15 words early,
i.e. a slightly overlapping page.
### Acceptance criteria
- For every page of a document, `render_page_backward(start[i])` returns
`start[i-1]` — verified for both a mid-paragraph-paginating document and one
where every page starts on a block boundary.
- Laying out forward from the returned position ends exactly on the requested
position.
- Forward-then-back returns to the original position.
- At the document start, backward stays there; an empty document is safe.
- Cost stays within a small bounded number of forward layouts.
### Files
`pyWebLayout/layout/ereader_layout.py`
---
## Test plan
Findings were reproduced with four probe scripts; each becomes a regression test
+129 -75
View File
@@ -313,6 +313,13 @@ class BidirectionalLayouter:
self.alignment_override = alignment_override
self.font_family_override = font_family_override
# Maps (font_scale, end position) -> the position the page started at.
# Filled in as pages are laid out forward, which makes "previous page"
# exact and free for anywhere the reader has already been. Keyed by font
# scale because changing it repaginates the document.
self._page_chain: Dict[Tuple[float, Tuple[int, int, int]],
RenderingPosition] = {}
def render_page_forward(self, position: RenderingPosition,
font_scale: float = 1.0) -> Tuple[Page, RenderingPosition]:
"""
@@ -367,109 +374,156 @@ class BidirectionalLayouter:
current_pos = new_pos
# Remember this link in the chain so stepping back to it later is exact.
if self._position_compare(current_pos, position) > 0:
self._page_chain[(font_scale, self._position_key(current_pos))] = \
position.copy()
return page, current_pos
# How many block starts before the target to try as replay anchors before
# settling for the best inexact answer.
MAX_BACKWARD_ANCHORS = 4
# Ceiling on pages replayed from a single anchor, so a pathologically long
# block cannot make one page turn walk an entire chapter.
MAX_REPLAY_PAGES = 8
def render_page_backward(self,
end_position: RenderingPosition,
font_scale: float = 1.0) -> Tuple[Page,
RenderingPosition]:
"""
Render a page that ends at the given position, filling backward.
Critical for "previous page" navigation.
Render the page that ends at the given position - "previous page".
Uses iterative refinement to find the correct start position that
results in a page ending at (or very close to) the target position.
Pagination is a pure function: laying out from a position q yields a page
and the position where it stopped, next(q). The page before P is therefore
the q for which next(q) == P, and it is found by *replaying* the chain
forward from an anchor, not by guessing q.
The previous implementation searched instead: it estimated a block index
and bisected on it, pinning word_index to 0. Pages routinely start
mid-block, so the answer was frequently not in the search space at all -
the search then exhausted its iterations and fell back to a position that
was not the previous page, usually the start of the document.
Three sources are tried in order:
1. The recorded chain, from pages already laid out going forward. Exact,
and the common case when the reader is paging back and forth.
2. Replay from the start of the block containing P, then from
progressively earlier blocks. Exact when P lies on the resulting chain.
3. Failing an exact hit - which happens when P was reached by a jump or a
restored bookmark rather than by reading forward, so it is on no
natural chain - the latest page start before P. That overlaps P's page
slightly rather than skipping content, which is the safe direction to
be wrong in.
Args:
end_position: Position where page should end
end_position: Position where the page should end
font_scale: Font scaling factor
Returns:
Tuple of (rendered_page, start_position)
"""
# Handle edge case: already at beginning
if end_position.block_index == 0 and end_position.word_index == 0:
return self.render_page_forward(end_position, font_scale)
document_start = RenderingPosition()
# Start with initial estimate
estimated_start = self._estimate_page_start(end_position, font_scale)
# Nothing precedes the start of the document.
if self._position_compare(end_position, document_start) <= 0:
page, _ = self.render_page_forward(document_start, font_scale)
return page, document_start
# Iterative refinement: keep adjusting until we converge or hit max iterations
max_iterations = 10
best_page = None
best_start = estimated_start
best_distance = float('inf')
# 1. The chain we have already walked.
remembered = self._page_chain.get((font_scale, self._position_key(end_position)))
if remembered is not None:
page, actual_end = self.render_page_forward(remembered, font_scale)
if self._position_compare(actual_end, end_position) == 0:
return page, remembered
for iteration in range(max_iterations):
# Render forward from current estimate
page, actual_end = self.render_page_forward(estimated_start, font_scale)
# 2/3. Replay from anchors, keeping the best inexact result as a fallback.
fallback = None
for anchor in self._backward_anchors(end_position):
page, start, exact = self._replay_to(anchor, end_position, font_scale)
if page is None:
continue
if exact:
return page, start
if fallback is None:
fallback = (page, start)
# Calculate how far we are from target
comparison = self._position_compare(actual_end, end_position)
if fallback is not None:
return fallback
# Perfect match or close enough (within same block)
# BUT: ensure we actually moved backward (estimated_start < end_position)
if comparison == 0:
# Check if we actually found a valid previous page
if self._position_compare(estimated_start, end_position) < 0:
return page, estimated_start
# If estimated_start >= end_position, we haven't moved backward
# Continue iterating to find a better position
elif iteration == 0:
# On first iteration, if we can't find a previous position,
# we're likely at or near the beginning
break
page, _ = self.render_page_forward(document_start, font_scale)
return page, document_start
# Track best result so far
distance = abs(actual_end.block_index - end_position.block_index)
if distance < best_distance:
best_distance = distance
best_page = page
best_start = estimated_start.copy()
def _backward_anchors(self, target: RenderingPosition):
"""
Yield positions to replay from, nearest first.
# Adjust estimate for next iteration
estimated_start = self._adjust_start_estimate(
estimated_start, end_position, actual_end)
Block starts are used as anchors because they are the coarsest positions
that are certainly valid to lay out from. The block containing the target
comes first: when the target is mid-block, the page before it usually
starts in that same block or the one before.
"""
first_block = target.block_index if target.word_index > 0 \
else target.block_index - 1
# Safety: don't go before document start
if estimated_start.block_index < 0:
estimated_start.block_index = 0
estimated_start.word_index = 0
# If we exhausted iterations, return best result found
# BUT: ensure we didn't return the same position (no backward progress)
final_page = best_page if best_page else page
final_start = best_start
# Safety check: if final_start >= end_position, we failed to move backward
# This can happen at the beginning of the document or when estimation failed
if self._position_compare(final_start, end_position) >= 0:
# Can't go further back - check if we're at the absolute beginning
if end_position.block_index == 0 and end_position.word_index == 0:
# Already at beginning, return as-is
return final_page, final_start
# Fallback strategy: try a more aggressive backward jump
# Start from several blocks before the current position
blocks_to_jump = max(1, min(5, end_position.block_index))
fallback_pos = RenderingPosition(
chapter_index=end_position.chapter_index,
block_index=max(0, end_position.block_index - blocks_to_jump),
word_index=0
for offset in range(self.MAX_BACKWARD_ANCHORS):
block_index = first_block - offset
if block_index < 0:
break
yield RenderingPosition(
chapter_index=target.chapter_index,
block_index=block_index,
word_index=0,
)
# Render forward from the fallback position
fallback_page, fallback_end = self.render_page_forward(fallback_pos, font_scale)
if first_block - self.MAX_BACKWARD_ANCHORS >= 0:
yield RenderingPosition()
# Verify the fallback actually moved us backward
if self._position_compare(fallback_pos, end_position) < 0:
return fallback_page, fallback_pos
def _replay_to(self,
anchor: RenderingPosition,
target: RenderingPosition,
font_scale: float):
"""
Lay out pages forward from `anchor`, looking for the one ending at `target`.
# If even the fallback didn't work, we're likely at the beginning
# Return a page starting from the beginning
return self.render_page_forward(RenderingPosition(), font_scale)
Returns:
(page, start, exact). `exact` is True when a page ended precisely on
the target. When the chain steps over the target instead, the last
page starting before it is returned with exact=False. (None, None,
False) means the anchor yielded nothing usable.
"""
position = anchor
last = (None, None)
return final_page, final_start
for _ in range(self.MAX_REPLAY_PAGES):
if self._position_compare(position, target) >= 0:
break
page, next_position = self.render_page_forward(position, font_scale)
comparison = self._position_compare(next_position, target)
if comparison == 0:
return page, position, True
if comparison > 0:
# Stepped over the target: this chain does not pass through it.
return last[0], last[1], False
if self._position_compare(next_position, position) <= 0:
break # no progress; give up on this anchor
last = (page, position)
position = next_position
return last[0], last[1], False
@staticmethod
def _position_key(position: RenderingPosition) -> Tuple[int, int, int]:
"""Hashable identity of a position, for the page chain map."""
return (position.chapter_index, position.block_index, position.word_index)
def _scale_block_fonts(self, block: Block, font_scale: float) -> Block:
"""Apply font scaling and font family override to all fonts in a block"""
+18 -3
View File
@@ -935,16 +935,31 @@ class EreaderLayoutManager:
"""
Shutdown the ereader manager and clean up resources.
Call this when the application is closing.
Idempotent: calling it twice saves the position once.
"""
if getattr(self, '_shutdown_done', False):
return
self._shutdown_done = True
# Save current position
self.bookmark_manager.save_reading_position(self.current_position)
# Shutdown renderer and buffer
# Release cached pages
self.renderer.shutdown()
def __del__(self):
"""Cleanup on destruction"""
self.shutdown()
"""
Best-effort cleanup for callers that never called shutdown().
Finalisers run during interpreter teardown, when modules and globals
may already be torn down, so this must never raise and must never
block. Applications should call shutdown() explicitly.
"""
try:
self.shutdown()
except Exception:
pass
# Convenience function for quick setup
+49 -225
View File
@@ -1,16 +1,35 @@
"""
Multi-process page buffering system for high-performance ereader navigation.
Page caching for ereader navigation.
This module provides intelligent page caching with background rendering using
multiprocessing to achieve sub-second page navigation performance.
`PageBuffer` is an LRU cache of rendered pages plus the position links between
them; `BufferedPageRenderer` wraps it around a `BidirectionalLayouter`.
This module used to render pages ahead of time in a `ProcessPoolExecutor`. That
never worked and has been removed see S12 in docs/LAYOUT_REMEDIATION_SPEC.md
and R1/R2 in docs/ARCHITECTURE_REVIEW.md. In short: the worker returned
`pickle.dumps(page)`, and a Page holds a live PIL canvas, which is not
picklable, so every job failed and the result was discarded. The cost four
interpreter copies and the whole block list shipped per job was paid in full
for no benefit. On Python 3.14, where the default start method became
`forkserver`, submitting from module-level code raised outright.
Prefetch is not needed. Measured on the tests/data Wikipedia fixture (411
blocks) with the text caches warm, one page render costs:
800x600 p50 8.8 ms p95 15.4 ms
1072x1448 p50 13.8 ms p95 56.1 ms
A page turn is cheaper than the IPC that was meant to hide it. If a slower
target device ever changes that, the fallback is a synchronous `readahead()`
method on this class, or a single worker *thread* layout is PIL-bound and PIL
releases the GIL not a process pool. Making the concrete tree picklable
(Page -> Line -> Text -> Font -> FreeTypeFont) is a large surface area to
maintain for a cache.
"""
from __future__ import annotations
from typing import Dict, Optional, List, Tuple, Any
from collections import OrderedDict
from concurrent.futures import ProcessPoolExecutor, Future
import threading
import pickle
from .ereader_layout import RenderingPosition, BidirectionalLayouter, FontFamilyOverride
from pyWebLayout.concrete.page import Page
@@ -19,57 +38,20 @@ from pyWebLayout.style.page_style import PageStyle
from pyWebLayout.style.fonts import BundledFont
def _render_page_worker(args: Tuple[List[Block],
PageStyle,
RenderingPosition,
float,
bool,
Optional[BundledFont]]) -> Tuple[RenderingPosition,
bytes,
RenderingPosition]:
"""
Worker function for multiprocess page rendering.
Args:
args: Tuple of (blocks, page_style, position, font_scale, is_backward, font_family)
Returns:
Tuple of (original_position, pickled_page, next_position)
"""
blocks, page_style, position, font_scale, is_backward, font_family = args
# Create font family override if specified
font_family_override = FontFamilyOverride(font_family) if font_family else None
layouter = BidirectionalLayouter(blocks, page_style, font_family_override=font_family_override)
if is_backward:
page, next_pos = layouter.render_page_backward(position, font_scale)
else:
page, next_pos = layouter.render_page_forward(position, font_scale)
# Serialize the page for inter-process communication
pickled_page = pickle.dumps(page)
return position, pickled_page, next_pos
class PageBuffer:
"""
Intelligent page caching system with LRU eviction and background rendering.
Maintains separate forward and backward buffers for optimal navigation performance.
LRU cache of rendered pages, with separate forward and backward buffers and
the position links between adjacent pages.
"""
def __init__(self, buffer_size: int = 5, max_workers: int = 4):
def __init__(self, buffer_size: int = 5):
"""
Initialize the page buffer.
Args:
buffer_size: Number of pages to cache in each direction
max_workers: Maximum number of worker processes for background rendering
"""
self.buffer_size = buffer_size
self.max_workers = max_workers
# LRU caches for forward and backward pages
self.forward_buffer: OrderedDict[RenderingPosition, Page] = OrderedDict()
@@ -81,11 +63,6 @@ class PageBuffer:
self.reverse_position_map: Dict[RenderingPosition,
RenderingPosition] = {} # current -> previous
# Background rendering
self.executor: Optional[ProcessPoolExecutor] = None
self.pending_renders: Dict[RenderingPosition, Future] = {}
self.render_lock = threading.Lock()
# Document state
self.blocks: Optional[List[Block]] = None
self.page_style: Optional[PageStyle] = None
@@ -112,10 +89,6 @@ class PageBuffer:
self.current_font_scale = font_scale
self.current_font_family = font_family
# Start the process pool
if self.executor is None:
self.executor = ProcessPoolExecutor(max_workers=self.max_workers)
def get_page(self, position: RenderingPosition) -> Optional[Page]:
"""
Get a cached page if available.
@@ -176,125 +149,12 @@ class PageBuffer:
self.position_map.pop(oldest_pos, None)
self.reverse_position_map.pop(oldest_pos, None)
def start_background_rendering(
self,
current_position: RenderingPosition,
direction: str = 'forward'):
"""
Start background rendering of upcoming pages.
Args:
current_position: Current reading position
direction: 'forward', 'backward', or 'both'
"""
if not self.blocks or not self.page_style or not self.executor:
return
with self.render_lock:
if direction in ['forward', 'both']:
self._queue_forward_renders(current_position)
if direction in ['backward', 'both']:
self._queue_backward_renders(current_position)
def _queue_forward_renders(self, start_position: RenderingPosition):
"""Queue forward page renders starting from the given position"""
current_pos = start_position
for i in range(self.buffer_size):
# Skip if already cached or being rendered
if current_pos in self.forward_buffer or current_pos in self.pending_renders:
# Try to get next position from cache
current_pos = self.position_map.get(current_pos)
if not current_pos:
break
continue
# Queue render job
args = (
self.blocks,
self.page_style,
current_pos,
self.current_font_scale,
False,
self.current_font_family)
future = self.executor.submit(_render_page_worker, args)
self.pending_renders[current_pos] = future
# We don't know the next position yet, so we'll update it when the render
# completes
break
def _queue_backward_renders(self, start_position: RenderingPosition):
"""Queue backward page renders ending at the given position"""
current_pos = start_position
for i in range(self.buffer_size):
# Skip if already cached or being rendered
if current_pos in self.backward_buffer or current_pos in self.pending_renders:
# Try to get previous position from cache
current_pos = self.reverse_position_map.get(current_pos)
if not current_pos:
break
continue
# Queue render job
args = (
self.blocks,
self.page_style,
current_pos,
self.current_font_scale,
True,
self.current_font_family)
future = self.executor.submit(_render_page_worker, args)
self.pending_renders[current_pos] = future
# We don't know the previous position yet, so we'll update it when the
# render completes
break
def check_completed_renders(self):
"""Check for completed background renders and cache the results"""
if not self.pending_renders:
return
completed = []
with self.render_lock:
for position, future in self.pending_renders.items():
if future.done():
try:
original_pos, pickled_page, next_pos = future.result()
# Deserialize the page
page = pickle.loads(pickled_page)
# Cache the page
self.cache_page(original_pos, page, next_pos, is_backward=False)
completed.append(position)
except Exception as e:
print(f"Background render failed for position {position}: {e}")
completed.append(position)
# Remove completed renders
for pos in completed:
self.pending_renders.pop(pos, None)
def invalidate_all(self):
"""Clear all cached pages and cancel pending renders"""
with self.render_lock:
# Cancel pending renders
for future in self.pending_renders.values():
future.cancel()
self.pending_renders.clear()
# Clear caches
self.forward_buffer.clear()
self.backward_buffer.clear()
self.position_map.clear()
self.reverse_position_map.clear()
"""Clear all cached pages"""
self.forward_buffer.clear()
self.backward_buffer.clear()
self.position_map.clear()
self.reverse_position_map.clear()
def set_font_scale(self, font_scale: float):
"""
@@ -323,7 +183,6 @@ class PageBuffer:
return {
'forward_buffer_size': len(self.forward_buffer),
'backward_buffer_size': len(self.backward_buffer),
'pending_renders': len(self.pending_renders),
'position_mappings': len(self.position_map),
'reverse_position_mappings': len(self.reverse_position_map),
'current_font_scale': self.current_font_scale,
@@ -331,28 +190,20 @@ class PageBuffer:
}
def shutdown(self):
"""Shutdown the page buffer and clean up resources"""
if self.executor:
# Cancel pending renders
with self.render_lock:
for future in self.pending_renders.values():
future.cancel()
"""
Release cached pages.
# Shutdown executor
self.executor.shutdown(wait=True)
self.executor = None
# Clear all caches
Cheap and idempotent. There is deliberately no __del__ calling this:
blocking work in a finaliser is what deadlocked the interpreter at exit
while the process pool existed.
"""
self.invalidate_all()
def __del__(self):
"""Cleanup on destruction"""
self.shutdown()
class BufferedPageRenderer:
"""
High-level interface for buffered page rendering with automatic background caching.
High-level interface for page rendering with an LRU cache in front of the
layouter.
"""
def __init__(self,
@@ -390,7 +241,7 @@ class BufferedPageRenderer:
def render_page(self, position: RenderingPosition,
font_scale: float = 1.0) -> Tuple[Page, RenderingPosition]:
"""
Render a page with intelligent caching.
Render a page, serving it from cache when possible.
Args:
position: Position to render from
@@ -407,32 +258,18 @@ class BufferedPageRenderer:
# Check cache first
cached_page = self.buffer.get_page(position)
if cached_page:
# Get next position from position map
# Only use the cache if we also know where the next page starts;
# otherwise fall through and compute it.
next_pos = self.buffer.position_map.get(position)
# Only use cache if we have the forward position mapping
# Otherwise, we need to compute it
if next_pos is not None:
# Start background rendering for upcoming pages
self.buffer.start_background_rendering(position, 'forward')
return cached_page, next_pos
# Cache hit for the page, but we don't have the forward position
# Fall through to compute it below
# Render the page directly
page, next_pos = self.layouter.render_page_forward(position, font_scale)
# Cache the result
self.buffer.cache_page(position, page, next_pos)
# Start background rendering
self.buffer.start_background_rendering(position, 'both')
# Check for completed background renders
self.buffer.check_completed_renders()
return page, next_pos
def render_page_backward(self,
@@ -440,7 +277,8 @@ class BufferedPageRenderer:
font_scale: float = 1.0) -> Tuple[Page,
RenderingPosition]:
"""
Render a page ending at the given position with intelligent caching.
Render a page ending at the given position, serving it from cache when
possible.
Args:
end_position: Position where page should end
@@ -457,32 +295,18 @@ class BufferedPageRenderer:
# Check cache first
cached_page = self.buffer.get_page(end_position)
if cached_page:
# Get previous position from reverse position map
# Only use the cache if we also know where the previous page
# starts; otherwise fall through and compute it.
prev_pos = self.buffer.reverse_position_map.get(end_position)
# Only use cache if we have the reverse position mapping
# Otherwise, we need to compute it
if prev_pos is not None:
# Start background rendering for previous pages
self.buffer.start_background_rendering(end_position, 'backward')
return cached_page, prev_pos
# Cache hit for the page, but we don't have the reverse position
# Fall through to compute it below
# Render the page directly
page, start_pos = self.layouter.render_page_backward(end_position, font_scale)
# Cache the result
self.buffer.cache_page(start_pos, page, end_position, is_backward=True)
# Start background rendering
self.buffer.start_background_rendering(end_position, 'both')
# Check for completed background renders
self.buffer.check_completed_renders()
return page, start_pos
def set_font_family(self, font_family: Optional[BundledFont]):
@@ -516,5 +340,5 @@ class BufferedPageRenderer:
return self.buffer.get_cache_stats()
def shutdown(self):
"""Shutdown the renderer and clean up resources"""
"""Release cached pages"""
self.buffer.shutdown()
+37 -5
View File
@@ -4,24 +4,56 @@ build-backend = "setuptools.build_meta"
[project]
name = "pyWebLayout"
version = "0.1.1"
description = "A Python library for HTML-like layout and rendering"
readme = "README.md"
requires-python = ">=3.6"
requires-python = ">=3.10"
license = {file = "LICENSE"}
authors = [
{name = "Duncan Tourolle", email = "duncan@tourolle.paris"}
]
dynamic = ["version"]
classifiers = [
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
]
dependencies = [
"Pillow",
"numpy",
"pyphen",
"beautifulsoup4",
"flask",
"ebooklib",
"requests"
]
[project.urls]
Homepage = "https://gitea.tourolle.paris/pyWebLayout"
[project.optional-dependencies]
# Loading images from http(s) URLs. concrete.image imports requests lazily and
# degrades to an error message on the image when it is absent, so it is not a
# hard requirement.
remote-images = ["requests"]
test = [
"pytest>=6.0",
"pytest-cov",
"flask", # fixture HTTP server in tests/abstract/test_abstract_blocks.py
"werkzeug", # make_server, same fixture
"ebooklib", # builds EPUB fixtures; the reader itself uses zipfile + ElementTree
"requests", # exercises the remote-images path
]
dev = [
"pyWebLayout[test,remote-images]",
"flake8",
"coverage-badge",
"interrogate",
]
[tool.setuptools.packages.find]
include = ["pyWebLayout*"]
[tool.coverage.run]
source = ["pyWebLayout"]
branch = true
+3 -23
View File
@@ -1,26 +1,6 @@
[metadata]
name = pyWebLayout
version = 0.1.1
author = Duncan Tourolle
author_email = duncan@tourolle.paris
description = A Python library for HTML-like layout and rendering
long_description = file: README.md
long_description_content_type = text/markdown
url = https://gitea.tourolle.paris/pyWebLayout
classifiers =
Programming Language :: Python :: 3
License :: OSI Approved :: MIT License
Operating System :: OS Independent
[options]
packages = find:
python_requires = >=3.6
install_requires =
Pillow
numpy
[options.packages.find]
include = pyWebLayout*
# Packaging metadata lives in pyproject.toml ([project]), which takes
# precedence over anything declared here. This file keeps only tool config
# that has nowhere better to live.
[flake8]
exclude =
+9 -31
View File
@@ -1,32 +1,10 @@
from setuptools import setup, find_packages
"""Shim for legacy `python setup.py` invocations.
setup(
name="pyWebLayout",
version="0.1.1",
packages=find_packages(),
install_requires=[
"Pillow",
"numpy",
],
extras_require={
"test": [
"coverage>=5.0",
],
"dev": [
"coverage>=5.0",
"pytest>=6.0",
],
},
author="Duncan Tourolle",
author_email="duncan@tourolle.paris",
description="A Python library for HTML-like layout and rendering",
long_description=open("README.md").read(),
long_description_content_type="text/markdown",
url="https://gitea.tourolle.paris/pyWebLayout",
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires=">=3.6",
)
All packaging metadata lives in setup.cfg. Keeping a second copy here was an
active hazard: keyword arguments passed to setup() override setup.cfg, so the
two could disagree silently and the setup.py copy would win.
"""
from setuptools import setup
setup()
+12 -7
View File
@@ -307,6 +307,8 @@ class TestImagePIL(unittest.TestCase):
if FLASK_AVAILABLE and hasattr(cls, 'flask_thread'):
cls.flask_server_running = False
cls.flask_server.shutdown()
cls.flask_server.server_close()
cls.flask_thread.join(timeout=2)
@classmethod
@@ -350,10 +352,9 @@ class TestImagePIL(unittest.TestCase):
"""Start a Flask server for URL testing."""
import urllib.request
import urllib.error
from werkzeug.serving import make_server
cls.flask_app = Flask(__name__)
cls.flask_port = 5555 # Use a specific port for testing
cls.flask_server_running = True
@cls.flask_app.route('/test.jpg')
def serve_test_image():
@@ -363,11 +364,12 @@ class TestImagePIL(unittest.TestCase):
def health_check():
return 'OK', 200
def run_flask():
cls.flask_app.run(host='127.0.0.1', port=cls.flask_port, debug=False,
use_reloader=False, threaded=True)
# Bind to an ephemeral port so concurrent/leftover test runs can't clash
cls.flask_server = make_server('127.0.0.1', 0, cls.flask_app, threaded=True)
cls.flask_port = cls.flask_server.server_port
cls.flask_server_running = True
cls.flask_thread = threading.Thread(target=run_flask, daemon=True)
cls.flask_thread = threading.Thread(target=cls.flask_server.serve_forever, daemon=True)
cls.flask_thread.start()
# Wait for server to be ready with health check
@@ -379,12 +381,15 @@ class TestImagePIL(unittest.TestCase):
try:
with urllib.request.urlopen(f'http://127.0.0.1:{cls.flask_port}/health', timeout=1) as response:
if response.status == 200:
break
return
except (urllib.error.URLError, ConnectionRefusedError, OSError):
pass
time.sleep(wait_interval)
elapsed += wait_interval
raise RuntimeError(
f"Test Flask server did not become ready on port {cls.flask_port} within {max_wait}s")
def test_image_url_detection(self):
"""Test URL detection functionality."""
img = Image()
+171
View File
@@ -0,0 +1,171 @@
"""
Regression tests for backward page navigation (spec S16).
The previous page of P is the position q for which laying out forward from q ends
exactly at P. The old implementation searched for q by guessing a block index and
bisecting, with word_index pinned to 0 - so a page starting mid-paragraph was not
in the search space at all. It exhausted its ten iterations and fell back to a
position that was not the previous page, typically the start of the document.
"""
import pytest
from pyWebLayout.abstract.block import Paragraph
from pyWebLayout.abstract.inline import Word
from pyWebLayout.layout.ereader_layout import BidirectionalLayouter, RenderingPosition
from pyWebLayout.style import Font
from pyWebLayout.style.page_style import PageStyle
PAGE_SIZE = (800, 600)
@pytest.fixture
def font():
return Font(font_size=16)
def paragraph(font, count, tag):
block = Paragraph(font)
for i in range(count):
block.add_word(Word(f"{tag}{i}", font))
return block
@pytest.fixture
def long_document(font):
"""Short paragraphs around one that spans several pages."""
return [
paragraph(font, 60, "a"),
paragraph(font, 80, "b"),
paragraph(font, 1200, "long"),
paragraph(font, 70, "c"),
paragraph(font, 90, "d"),
]
@pytest.fixture
def block_document(font):
"""Many small blocks, so every page starts on a block boundary."""
return [paragraph(font, 40, f"p{i}") for i in range(40)]
def forward_chain(layouter, limit=30):
"""The page start positions a reader would visit going forward."""
starts = []
pos = RenderingPosition()
for _ in range(limit):
starts.append(pos)
_, nxt = layouter.render_page_forward(pos, 1.0)
if nxt.block_index >= len(layouter.blocks):
break
if (nxt.block_index, nxt.word_index) == (pos.block_index, pos.word_index):
pytest.fail("forward pagination made no progress")
pos = nxt
return starts
def key(position):
return (position.chapter_index, position.block_index, position.word_index)
class TestBackwardMatchesForward:
"""The defining invariant: forward from the answer lands exactly on P."""
@pytest.mark.parametrize("document", ["long_document", "block_document"])
def test_previous_page_is_the_forward_predecessor(self, document, request):
blocks = request.getfixturevalue(document)
layouter = BidirectionalLayouter(blocks, PageStyle(), PAGE_SIZE)
starts = forward_chain(layouter)
assert len(starts) > 2, "need a few pages to test against"
for i in range(1, len(starts)):
_, got = layouter.render_page_backward(starts[i], 1.0)
assert key(got) == key(starts[i - 1]), (
f"page {i}: expected to land on page {i - 1} "
f"{key(starts[i - 1])}, got {key(got)}")
def test_result_lays_out_to_the_target(self, long_document):
"""Independent of the recorded chain: replaying the answer must reach P."""
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
starts = forward_chain(layouter)
for target in starts[1:]:
_, start = layouter.render_page_backward(target, 1.0)
_, end = layouter.render_page_forward(start, 1.0)
assert key(end) == key(target), (
f"a page starting at {key(start)} ends at {key(end)}, "
f"not at the requested {key(target)}")
def test_mid_paragraph_targets_are_reachable(self, long_document):
"""The specific regression: starts inside a block, not on its boundary."""
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
starts = forward_chain(layouter)
mid = [s for s in starts if s.word_index > 0]
assert mid, "this document should paginate mid-paragraph"
for target in mid:
_, got = layouter.render_page_backward(target, 1.0)
assert key(got) != (0, 0, 0) or key(target) == key(starts[1]), \
"backward navigation fell back to the document start"
class TestRoundTrip:
def test_forward_then_back_returns_to_the_same_place(self, long_document):
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
pos = RenderingPosition()
for _ in range(4):
_, nxt = layouter.render_page_forward(pos, 1.0)
_, back = layouter.render_page_backward(nxt, 1.0)
assert key(back) == key(pos), \
f"round trip drifted: {key(pos)} -> {key(nxt)} -> {key(back)}"
pos = nxt
class TestEdges:
def test_at_document_start_stays_there(self, long_document):
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
_, got = layouter.render_page_backward(RenderingPosition(), 1.0)
assert key(got) == (0, 0, 0)
def test_second_page_goes_back_to_the_first(self, long_document):
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
_, second = layouter.render_page_forward(RenderingPosition(), 1.0)
_, got = layouter.render_page_backward(second, 1.0)
assert key(got) == (0, 0, 0)
def test_empty_document_is_safe(self):
layouter = BidirectionalLayouter([], PageStyle(), PAGE_SIZE)
page, got = layouter.render_page_backward(RenderingPosition(), 1.0)
assert page is not None
assert key(got) == (0, 0, 0)
class TestCost:
def test_backward_is_not_wildly_more_expensive_than_forward(self, long_document):
"""The old path burned ten full layouts per call and still got it wrong."""
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
starts = forward_chain(layouter)
calls = {"n": 0}
original = BidirectionalLayouter.render_page_forward
def counting(self, position, font_scale=1.0):
calls["n"] += 1
return original(self, position, font_scale)
BidirectionalLayouter.render_page_forward = counting
try:
worst = 0
for target in starts[1:]:
calls["n"] = 0
layouter.render_page_backward(target, 1.0)
worst = max(worst, calls["n"])
finally:
BidirectionalLayouter.render_page_forward = original
assert worst <= 10, f"backward navigation cost {worst} forward layouts"
+237
View File
@@ -0,0 +1,237 @@
"""
Tests for the page caching layer.
Covers PageBuffer's LRU behaviour and BufferedPageRenderer's cache hits, plus
regressions for S12/R1/R2: the module must not start worker processes and must
not do blocking work in a finaliser.
"""
import subprocess
import sys
import textwrap
import pytest
from pyWebLayout.layout.page_buffer import PageBuffer, BufferedPageRenderer
from pyWebLayout.layout.ereader_layout import RenderingPosition
from pyWebLayout.abstract.block import Paragraph
from pyWebLayout.abstract.inline import Word
from pyWebLayout.style import Font
from pyWebLayout.style.page_style import PageStyle
# ============================================================================
# Fixtures
# ============================================================================
@pytest.fixture
def sample_blocks():
"""A document long enough to paginate over several pages."""
font = Font()
blocks = []
for p in range(6):
para = Paragraph(style=font)
for w in range(120):
para.add_word(Word(f"p{p}w{w}", font))
blocks.append(para)
return blocks
@pytest.fixture
def renderer(sample_blocks):
return BufferedPageRenderer(sample_blocks, PageStyle(), buffer_size=3, page_size=(800, 600))
# ============================================================================
# PageBuffer
# ============================================================================
class TestPageBuffer:
def test_get_page_misses_when_empty(self):
buf = PageBuffer(buffer_size=3)
assert buf.get_page(RenderingPosition()) is None
def test_cache_page_round_trips(self):
buf = PageBuffer(buffer_size=3)
pos, nxt = RenderingPosition(block_index=0), RenderingPosition(block_index=1)
sentinel = object()
buf.cache_page(pos, sentinel, nxt)
assert buf.get_page(pos) is sentinel
assert buf.position_map[pos] == nxt
def test_lru_evicts_oldest_and_cleans_position_map(self):
buf = PageBuffer(buffer_size=2)
positions = [RenderingPosition(block_index=i) for i in range(4)]
for i, pos in enumerate(positions):
buf.cache_page(pos, object(), RenderingPosition(block_index=i + 1))
assert buf.get_page(positions[0]) is None, "oldest should have been evicted"
assert positions[0] not in buf.position_map, "position map must not leak evicted entries"
assert buf.get_page(positions[-1]) is not None
def test_get_page_refreshes_lru_order(self):
buf = PageBuffer(buffer_size=2)
a, b, c = (RenderingPosition(block_index=i) for i in range(3))
buf.cache_page(a, object())
buf.cache_page(b, object())
buf.get_page(a) # a becomes most recently used
buf.cache_page(c, object())
assert buf.get_page(a) is not None, "recently used entry should survive"
assert buf.get_page(b) is None, "least recently used entry should be evicted"
def test_backward_pages_land_in_the_backward_buffer(self):
buf = PageBuffer(buffer_size=3)
start, end = RenderingPosition(block_index=1), RenderingPosition(block_index=2)
buf.cache_page(start, object(), end, is_backward=True)
assert start in buf.backward_buffer
assert start not in buf.forward_buffer
assert buf.reverse_position_map[end] == start
def test_font_scale_change_invalidates(self):
buf = PageBuffer(buffer_size=3)
pos = RenderingPosition()
buf.cache_page(pos, object(), RenderingPosition(block_index=1))
buf.set_font_scale(1.5)
assert buf.get_page(pos) is None
def test_same_font_scale_keeps_cache(self):
buf = PageBuffer(buffer_size=3)
pos = RenderingPosition()
buf.cache_page(pos, object(), RenderingPosition(block_index=1))
buf.set_font_scale(1.0)
assert buf.get_page(pos) is not None
def test_shutdown_is_idempotent(self):
buf = PageBuffer(buffer_size=3)
buf.cache_page(RenderingPosition(), object())
buf.shutdown()
buf.shutdown()
assert buf.get_cache_stats()['forward_buffer_size'] == 0
# ============================================================================
# BufferedPageRenderer
# ============================================================================
class TestBufferedPageRenderer:
def test_render_page_returns_a_page_and_advances(self, renderer):
page, next_pos = renderer.render_page(RenderingPosition(), 1.0)
assert page is not None
assert next_pos != RenderingPosition()
def test_second_render_of_same_position_is_served_from_cache(self, renderer):
pos = RenderingPosition()
first, first_next = renderer.render_page(pos, 1.0)
second, second_next = renderer.render_page(pos, 1.0)
assert second is first, "identical page object means it came from the cache"
assert second_next == first_next
def test_font_scale_change_forces_a_re_render(self, renderer):
pos = RenderingPosition()
first, _ = renderer.render_page(pos, 1.0)
scaled, _ = renderer.render_page(pos, 1.5)
assert scaled is not first
def test_backward_render_round_trips_to_the_original_position(self, renderer):
start = RenderingPosition()
_, second_page_pos = renderer.render_page(start, 1.0)
_, back_to = renderer.render_page_backward(second_page_pos, 1.0)
assert back_to == start
def test_shutdown_clears_the_cache(self, renderer):
renderer.render_page(RenderingPosition(), 1.0)
renderer.shutdown()
assert renderer.get_cache_stats()['forward_buffer_size'] == 0
# ============================================================================
# S12 / R1 / R2 regressions
# ============================================================================
class TestNoBackgroundProcesses:
"""
The process pool that used to live here never produced a usable page (a Page
holds a live PIL canvas and cannot be pickled), and on Python 3.14's
forkserver default it raised when driven from module-level code.
"""
def test_module_declares_no_process_pool(self):
import pyWebLayout.layout.page_buffer as page_buffer
source = page_buffer.__file__
assert not hasattr(page_buffer, '_render_page_worker')
assert not hasattr(PageBuffer(), 'executor')
with open(source, encoding='utf-8') as fh:
body = fh.read().split('"""', 2)[-1] # skip the module docstring
assert 'ProcessPoolExecutor' not in body
assert 'pickle' not in body
def test_page_buffer_has_no_finaliser(self):
"""
PageBuffer.__del__ called executor.shutdown(wait=True), which deadlocked
the interpreter at exit. Cleanup must be explicit.
"""
assert '__del__' not in vars(PageBuffer)
def test_navigation_works_without_a_main_guard(self, tmp_path):
"""
R1: EreaderLayoutManager raised RuntimeError when used from module-level
script code, because submitting to a ProcessPoolExecutor under a
non-fork start method requires an `if __name__ == "__main__"` guard.
"""
script = textwrap.dedent(f"""
from pyWebLayout.io.readers.html_extraction import parse_html_string
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager
blocks = parse_html_string("<p>" + " ".join(f"w{{i}}" for i in range(2000)) + "</p>")
m = EreaderLayoutManager(blocks, page_size=(800, 600),
bookmarks_dir={str(tmp_path)!r})
m.get_current_page()
m.next_page()
m.previous_page()
m.shutdown()
print("OK")
""")
result = subprocess.run([sys.executable, "-c", script],
capture_output=True, text=True, timeout=120)
assert result.returncode == 0, result.stderr
assert "OK" in result.stdout
def test_interpreter_exits_without_explicit_shutdown(self, tmp_path):
"""
R2: a manager left to be finalised at exit must not hang. The timeout is
the assertion.
"""
script = textwrap.dedent(f"""
from pyWebLayout.io.readers.html_extraction import parse_html_string
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager
blocks = parse_html_string("<p>" + " ".join(f"w{{i}}" for i in range(500)) + "</p>")
m = EreaderLayoutManager(blocks, page_size=(800, 600),
bookmarks_dir={str(tmp_path)!r})
m.get_current_page()
# deliberately no shutdown() - rely on interpreter teardown
""")
result = subprocess.run([sys.executable, "-c", script],
capture_output=True, text=True, timeout=60)
assert result.returncode == 0, result.stderr