Compare commits
6
Commits
1985163827
...
1924cc234d
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1924cc234d | ||
|
|
456824d6d6 | ||
|
|
767e4c135c | ||
|
|
7384a32cdd | ||
|
|
4d596ce095 | ||
|
|
737cf0771c |
+20
-8
@@ -24,17 +24,29 @@ jobs:
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
|
||||
- name: Verify declared dependencies are sufficient
|
||||
run: |
|
||||
# A clean venv with ONLY the declared runtime deps. If an import here
|
||||
# fails, install_requires is incomplete and a real `pip install
|
||||
# pyWebLayout` would fail the same way for a user.
|
||||
python -m venv /tmp/clean-install
|
||||
/tmp/clean-install/bin/pip install --upgrade pip
|
||||
/tmp/clean-install/bin/pip install .
|
||||
/tmp/clean-install/bin/python -c "
|
||||
import pyWebLayout.concrete, pyWebLayout.abstract
|
||||
import pyWebLayout.io.readers.epub_reader
|
||||
import pyWebLayout.io.readers.html_extraction
|
||||
import pyWebLayout.layout.ereader_manager
|
||||
print('clean install imports OK')
|
||||
"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
# Install package in development mode
|
||||
pip install -e .
|
||||
# Install test dependencies if they exist
|
||||
if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt; fi
|
||||
if [ -f requirements/test.txt ]; then pip install -r requirements/test.txt; fi
|
||||
# Install common test packages
|
||||
pip install pytest pytest-cov flake8 coverage-badge interrogate
|
||||
|
||||
# Install package in development mode, with the declared dev extra.
|
||||
# Test dependencies belong in setup.cfg, not in an ad-hoc pip line.
|
||||
pip install -e '.[dev]'
|
||||
|
||||
- name: Download initial failed badges
|
||||
run: |
|
||||
echo "Downloading initial failed badges..."
|
||||
|
||||
+303
-188
@@ -1,233 +1,348 @@
|
||||
# pyWebLayout Architecture: Abstract vs Concrete
|
||||
# pyWebLayout Architecture
|
||||
|
||||
This document explains the fundamental architectural separation between **Abstract** and **Concrete** layers in the pyWebLayout library.
|
||||
This document describes how pyWebLayout is organised: the layers, what lives in each,
|
||||
and the rules that govern how they may depend on one another.
|
||||
|
||||
## Overview
|
||||
|
||||
The pyWebLayout library follows a clear separation between two distinct layers:
|
||||
The library turns markup (HTML/EPUB) into rendered images. That pipeline is split into
|
||||
layers with a strict dependency direction:
|
||||
|
||||
- **Abstract Layer**: Represents the logical structure and content of documents (HTML/EPUB text)
|
||||
- **Concrete Layer**: Handles the spatial rendering and visual representation of content
|
||||
```
|
||||
io/readers/ parse markup into document structure
|
||||
↓
|
||||
abstract/ what the document says ─┐
|
||||
↓ ├─ built on core/ + style/
|
||||
layout/ decide where things go │
|
||||
↓ │
|
||||
concrete/ what the pixels are ─┘
|
||||
↓
|
||||
PIL.Image
|
||||
```
|
||||
|
||||
This separation provides flexibility, testability, and clean separation of concerns.
|
||||
The central distinction is **abstract vs concrete**:
|
||||
|
||||
## Abstract Layer (`pyWebLayout/abstract/`)
|
||||
- **Abstract** — the logical content and structure of a document. A `Paragraph` knows
|
||||
it contains words in an order; it does not know how wide they are.
|
||||
- **Concrete** — the spatial realisation of that content. A `Line` knows exactly which
|
||||
glyphs sit at which pixel offsets on a specific canvas.
|
||||
|
||||
The Abstract layer deals with the **logical structure** of documents without concerning itself with how content will be visually rendered.
|
||||
One abstract document produces many concrete renderings: different page sizes, font
|
||||
scales, and font families all re-run the concrete layer over unchanged abstract content.
|
||||
That is what makes ereader features like live font scaling and reflow possible.
|
||||
|
||||
### Key Components
|
||||
## Layers
|
||||
|
||||
#### `abstract/block.py`
|
||||
- `Block`: Base class for all block-level content
|
||||
- `Paragraph`: Represents a logical paragraph containing words
|
||||
- `Heading`: Represents headings with semantic levels (H1-H6)
|
||||
- `HList`: Represents ordered/unordered lists
|
||||
- `Image`: Represents image references
|
||||
### `core/` — shared foundations
|
||||
|
||||
#### `abstract/inline.py`
|
||||
- `Word`: Represents individual words with text content and styling information
|
||||
- Contains methods for hyphenation and text manipulation
|
||||
- Does **not** handle rendering or spatial layout
|
||||
Everything else is built on these. `core/` depends on nothing but `style/`.
|
||||
|
||||
#### `abstract/document.py`
|
||||
- `Document`: Container for the overall document structure
|
||||
- `Chapter`: Logical grouping of blocks (for books/long documents)
|
||||
**`core/base.py`** defines the contracts that make a class abstract or concrete:
|
||||
|
||||
### Characteristics of Abstract Classes
|
||||
| Contract | Kind | Meaning |
|
||||
|---|---|---|
|
||||
| `Renderable` | ABC | Has `render()`; produces or draws visual output |
|
||||
| `Queriable` | ABC | Has `in_object(point)`; can be hit-tested |
|
||||
| `Layoutable` | ABC | Has `layout()`; arranges its own contents |
|
||||
| `Interactable` | ABC | Holds a callback invoked on interaction |
|
||||
| `Geometric` | mixin | `origin` and `size` as numpy arrays |
|
||||
| `Hierarchical` | mixin | `parent` back-reference |
|
||||
| `Styleable` | mixin | Carries a style object |
|
||||
| `FontRegistry` | mixin | Deduplicates `Font` instances across a document |
|
||||
| `MetadataContainer` | mixin | Key/value metadata with typed accessors |
|
||||
| `BlockContainer` | mixin | Holds child `Block`s |
|
||||
| `ContainerAware` | mixin | Knows the container it was added to |
|
||||
|
||||
1. **Content-focused**: Store text, structure, and semantic meaning
|
||||
2. **Layout-agnostic**: No knowledge of fonts, pixels, or rendering
|
||||
3. **Reusable**: Same content can be rendered in different formats/sizes
|
||||
4. **Serializable**: Can be saved/loaded without rendering context
|
||||
The useful shorthand: **`Renderable` and `Geometric` are the concrete markers.** An
|
||||
abstract class that acquires either has crossed the line.
|
||||
|
||||
### Example: Abstract Word
|
||||
**`core/query.py`** — `QueryResult` and `SelectionRange`: the result types for mapping a
|
||||
pixel back to content (what was clicked, what text is selected, where it is in the
|
||||
document).
|
||||
|
||||
**`core/highlight.py`** — `Highlight`, `HighlightColor`, `HighlightManager`. Highlights
|
||||
store both pixel bounds (for drawing) and semantic bounds (word indices, for surviving
|
||||
a font change).
|
||||
|
||||
**`core/cache.py`** — `UsageCache` / `SizedUsageCache`, bounded caches keyed by
|
||||
`(font, string)` for text measurement and glyph rasterisation. Eviction is by usage
|
||||
count with periodic aging, not LRU; the module docstring explains why at length. This
|
||||
exists because a single page issues thousands of measurements over fewer than a
|
||||
thousand distinct pairs, on hardware where an unbounded cache is not affordable.
|
||||
|
||||
**`core/callback_registry.py`** — `CallbackRegistry`, which owns the interactive
|
||||
elements registered on a page and dispatches to them.
|
||||
|
||||
### `style/` — semantic styling and its resolution
|
||||
|
||||
This package is itself an instance of the abstract/concrete split, applied to styling:
|
||||
|
||||
- **`abstract_style.py`** — `AbstractStyle` (frozen dataclass, hashable) captures
|
||||
*intent*: `FontFamily.SERIF`, `FontSize.LARGE`, `color="black"`. `AbstractStyleRegistry`
|
||||
interns them so a document holds one instance per distinct style.
|
||||
- **`concrete_style.py`** — `RenderingContext` (user preferences, DPI, accessibility
|
||||
flags, available space) plus `StyleResolver`, which maps `AbstractStyle` +
|
||||
`RenderingContext` → `ConcreteStyle` (a resolved font path, a pixel size, an RGB
|
||||
tuple). `ConcreteStyleRegistry` caches the results.
|
||||
- **`fonts.py`** — `Font`, the loaded PIL font plus its rendering attributes, and the
|
||||
bundled DejaVu families (`BundledFont`).
|
||||
- **`page_style.py`** — `PageStyle`: borders, padding, background, default alignment.
|
||||
- **`alignment.py`** — the `Alignment` enum.
|
||||
|
||||
`AbstractStyle` → `StyleResolver` → `ConcreteStyle` is the mechanism by which the same
|
||||
document renders differently for different readers.
|
||||
|
||||
### `abstract/` — document content and structure
|
||||
|
||||
Layout-agnostic representations of what the document contains.
|
||||
|
||||
**`abstract/block.py`** — block-level content, all deriving from `Block`:
|
||||
|
||||
`Paragraph`, `Heading` (with `HeadingLevel`), `Quote`, `CodeBlock`, `HList` (with
|
||||
`ListStyle`) and `ListItem`, `Table` / `TableRow` / `TableCell`, `Image` and
|
||||
`LinkedImage`, `HorizontalRule`, `PageBreak`.
|
||||
|
||||
**`abstract/inline.py`** — `Word`, `FormattedSpan`, `LinkedWord`, `LineBreak`. `Word`
|
||||
holds text, a style, and previous/next links into the document's word sequence. It can
|
||||
report whether it is a hyphenation candidate (`possible_hyphenation(language)`), but it
|
||||
does not perform the split — that is a measurement decision and belongs to `Line`.
|
||||
|
||||
**`abstract/document.py`** — `Document`, `Chapter`, `Book`, `MetadataType`. `Book` is
|
||||
the EPUB-shaped `Document` with chapters and a table of contents.
|
||||
|
||||
**`abstract/functional.py`** — `Link`, `Button`, `Form`, `FormField`, `LinkType`,
|
||||
`FormFieldType`.
|
||||
|
||||
**`abstract/interactive_image.py`** — `InteractiveImage`, an `Image` that is also
|
||||
`Interactable` and `Queriable`.
|
||||
|
||||
### `concrete/` — spatial realisation
|
||||
|
||||
Objects that know their position and size and can draw themselves onto a canvas.
|
||||
|
||||
**`concrete/text.py`** — the heart of text layout:
|
||||
- `Text` — one renderable fragment (a whole word, or a hyphenated part). Requires an
|
||||
`ImageDraw.Draw` at construction so it can measure itself immediately.
|
||||
- `Line` — a sequence of `Text` objects with resolved spacing and a baseline.
|
||||
`Line.add_word()` is where a `Word` becomes one or two `Text` objects: it measures,
|
||||
and if the word overflows it tries dictionary hyphenation, then brute-force splitting,
|
||||
before rejecting the word.
|
||||
- `AlignmentHandler` and its subclasses (`LeftAlignmentHandler`,
|
||||
`CenterRightAlignmentHandler`, `JustifyAlignmentHandler`) — the strategy objects that
|
||||
turn a set of measured fragments plus an available width into concrete spacing and a
|
||||
start position.
|
||||
- Cache management: `configure_text_caches`, `clear_text_caches`, `text_cache_stats`,
|
||||
`prewarm_text_caches`.
|
||||
|
||||
**`concrete/page.py`** — `Page`: a fixed-size canvas holding `Renderable` children, with
|
||||
a content rectangle derived from its `PageStyle`, `can_fit_line()` for the layouter to
|
||||
test against, `render()` returning a `PIL.Image`, and `query_point()` / `query_range()`
|
||||
for hit-testing.
|
||||
|
||||
**`concrete/box.py`** — `Box`, the `Geometric` + `Renderable` + `Queriable` base for
|
||||
positioned drawable objects.
|
||||
|
||||
**`concrete/dynamic_page.py`** — `DynamicPage` (a `Page` subclass) and `SizeConstraints`.
|
||||
Adds a two-phase measure-then-layout protocol, so a container such as a table can learn
|
||||
its content's intrinsic size before committing to a size allocation.
|
||||
|
||||
**`concrete/image.py`** — `RenderableImage`.
|
||||
|
||||
**`concrete/table.py`** — `TableRenderer`, `TableRowRenderer`, `TableCellRenderer`,
|
||||
`TableStyle`. Cells host their own nested `Page`, which is why `Page` accepts a
|
||||
non-zero `origin`.
|
||||
|
||||
**`concrete/functional.py`** — `LinkText`, `ButtonText`, `FormFieldText`: `Text`
|
||||
subclasses that are also `Interactable`.
|
||||
|
||||
**`concrete/interaction_handler.py`** — `InteractionHandler`, `InteractionStateManager`:
|
||||
routing taps and presses to registered elements and tracking pressed/released state.
|
||||
|
||||
### `layout/` — the abstract → concrete transformation
|
||||
|
||||
This is the package the pipeline diagram calls the layout engine.
|
||||
|
||||
**`layout/document_layouter.py`** — a set of layouter functions, one per content kind,
|
||||
all sharing a signature shape of *(abstract element, target `Page`) → did it fit*:
|
||||
|
||||
```python
|
||||
# An Abstract Word knows its text content and semantic properties
|
||||
word = Word("supercalifragilisticexpialidocious", font_style)
|
||||
word.hyphenate() # Logical operation - finds break points
|
||||
parts = word.get_hyphenated_parts() # Returns ["super-", "cali-", "fragi-", ...]
|
||||
paragraph_layouter(paragraph, page, start_word=0, pretext=None, alignment_override=None)
|
||||
-> (complete: bool, failed_word_index: int | None, remaining_pretext: Text | None)
|
||||
|
||||
image_layouter(image, page, max_width=None, max_height=None) -> bool
|
||||
table_layouter(table, page, style=None) -> bool
|
||||
pagebreak_layouter(page_break, page) -> bool
|
||||
button_layouter(button, page, font=None, padding=...) -> (bool, str)
|
||||
form_field_layouter(field, page, font=None, ...) -> ...
|
||||
form_layouter(form, page, font=None, field_spacing=10) -> (bool, list[str])
|
||||
```
|
||||
|
||||
## Concrete Layer (`pyWebLayout/concrete/`)
|
||||
They **append to a page**, they do not return a list of lines. The three-part return of
|
||||
`paragraph_layouter` is what makes pagination resumable: when a paragraph runs off the
|
||||
bottom of a page, the caller learns which word failed and whether a hyphenated fragment
|
||||
is pending, and can continue on the next page from exactly there.
|
||||
|
||||
The Concrete layer handles the **spatial representation** and actual rendering of content.
|
||||
`DocumentLayouter` wraps a `Page` and dispatches over a list of abstract elements by
|
||||
type, holding the `ConcreteStyleRegistry` for the run.
|
||||
|
||||
### Key Components
|
||||
**`layout/ereader_layout.py`** — the paginated reading model:
|
||||
- `RenderingPosition` — a serialisable cursor expressed in *abstract* coordinates
|
||||
(chapter, block, word, table cell, list item, pending pretext). Because it names
|
||||
document structure rather than pixels, it survives font-size and page-size changes.
|
||||
- `BidirectionalLayouter` — renders a page forward or backward from a position,
|
||||
returning `(Page, next_position)`.
|
||||
- `ChapterNavigator` / `ChapterInfo` — a table of contents built from heading structure.
|
||||
- `FontScaler`, `FontFamilyOverride` — apply scale and family changes to blocks at
|
||||
layout time without mutating the abstract document.
|
||||
|
||||
#### `concrete/text.py`
|
||||
- `Text`: Renders a specific text fragment with precise positioning
|
||||
- `Line`: Manages a line of `Text` objects with spacing and alignment
|
||||
- Handles actual pixel measurements, font rendering, and positioning
|
||||
**`layout/page_buffer.py`** — `PageBuffer` (LRU caches of rendered pages in both
|
||||
directions, plus position maps) and `BufferedPageRenderer` (background rendering).
|
||||
|
||||
#### `concrete/page.py`
|
||||
- `Page`: Top-level container for rendered content
|
||||
- `Container`: Layout manager for organizing renderable objects
|
||||
- Handles spatial layout, pagination, and visual composition
|
||||
**`layout/ereader_manager.py`** — `EreaderLayoutManager`, the top-level application
|
||||
interface (page turns, font changes, chapter jumps, progress), and `BookmarkManager`
|
||||
for persisting bookmarks and the last reading position.
|
||||
|
||||
#### `concrete/box.py`
|
||||
- `Box`: Base class for all spatially-aware renderable objects
|
||||
- Provides positioning, sizing, and rendering capabilities
|
||||
**`layout/table_optimizer.py`** — column width allocation for tables.
|
||||
|
||||
### Characteristics of Concrete Classes
|
||||
### `io/readers/` — parsing
|
||||
|
||||
1. **Rendering-focused**: Handle pixels, fonts, images, and visual output
|
||||
2. **Spatially-aware**: Know exact positions, sizes, and layout constraints
|
||||
3. **Implementation-specific**: Tied to specific rendering technologies (PIL, etc.)
|
||||
4. **Non-portable**: Rendering results are tied to specific display contexts
|
||||
**`html_extraction.py`** — `parse_html_string(html, base_font=None, document=None,
|
||||
base_path=None) -> List[Block]`. Built from a `StyleContext` (a `NamedTuple` threaded
|
||||
down the tree, carrying the inherited font and styling) and a table of per-tag handlers
|
||||
(`paragraph_handler`, `heading_handler`, `table_handler`, …). This is the only place
|
||||
that knows about HTML.
|
||||
|
||||
### Example: Concrete Text
|
||||
**`epub_reader.py`** — `EPUBReader` and `read_epub(path) -> Book`: container/OPF
|
||||
parsing, spine and manifest, table of contents, cover handling, and image processing
|
||||
(with an e-ink processor available by default).
|
||||
|
||||
## Dependency rules
|
||||
|
||||
The direction of dependency is what keeps the split honest:
|
||||
|
||||
```
|
||||
io/readers → abstract → core, style
|
||||
layout → abstract, concrete, core, style
|
||||
concrete → abstract, core, style
|
||||
abstract → core, style ← must NOT import concrete
|
||||
```
|
||||
|
||||
`abstract/` importing from `concrete/` is the violation to watch for. `concrete/`
|
||||
importing from `abstract/` is expected and correct: a `Text` may point back at the
|
||||
`Word` it came from, and a table cell renderer reads its abstract `TableCell`.
|
||||
|
||||
**Where the abstract layer touches rendering today, and why:**
|
||||
|
||||
- Abstract classes are constructed with `Font` objects, which carry a pixel size and a
|
||||
loaded font file. This is a deliberate compromise for parsing performance (HTML
|
||||
styling resolves to a `Font` once, at parse time) but it does mean the abstract layer
|
||||
is not fully rendering-independent. `AbstractStyle` is the intended replacement, and
|
||||
`Word` already accepts either.
|
||||
- `Word.concrete` / `Word.add_concete()` ([inline.py:42](pyWebLayout/abstract/inline.py#L42),
|
||||
[:134](pyWebLayout/abstract/inline.py#L134)) is a back-reference to the `Text` objects
|
||||
a word became. **It is currently written and never read**, and it cannot correctly
|
||||
model the relationship anyway: one `Word` becomes many `Text`s across re-layouts, and
|
||||
a single slot only remembers the most recent. Treat it as vestigial. When a
|
||||
word→pixels mapping is genuinely needed, build it on `Text`'s `source` back-reference
|
||||
(concrete pointing at abstract), which is the safe direction — note it is currently
|
||||
stored as `_source` with no public accessor, and is itself unread today.
|
||||
|
||||
## Worked example
|
||||
|
||||
```python
|
||||
# A Concrete Text object handles actual rendering
|
||||
text = Text("super-", font) # Specific text fragment
|
||||
text._calculate_dimensions() # Computes exact pixel size
|
||||
image = text.render() # Produces actual visual output
|
||||
from pyWebLayout.io.readers.html_extraction import parse_html_string
|
||||
from pyWebLayout.concrete.page import Page
|
||||
from pyWebLayout.layout.document_layouter import DocumentLayouter
|
||||
|
||||
# 1. Parse: markup -> abstract blocks
|
||||
blocks = parse_html_string("<h1>Chapter One</h1><p>It was a dark and stormy night.</p>")
|
||||
# [Heading, Paragraph]
|
||||
|
||||
# 2. Lay out: abstract blocks -> concrete children appended to a Page
|
||||
page = Page(size=(400, 600))
|
||||
DocumentLayouter(page).layout_document(blocks)
|
||||
|
||||
# 3. Render: Page -> PIL.Image
|
||||
image = page.render()
|
||||
|
||||
# 4. Query: pixel -> content
|
||||
result = page.query_point((50, 40))
|
||||
print(result.object_type, result.text) # e.g. "text" "Chapter"
|
||||
```
|
||||
|
||||
## The Transformation Process
|
||||
|
||||
The architecture involves a clear transformation from Abstract to Concrete:
|
||||
|
||||
```
|
||||
Abstract Document
|
||||
↓
|
||||
[Parser Layer]
|
||||
↓
|
||||
Abstract Blocks (Paragraph, Heading, etc.)
|
||||
↓
|
||||
[Layout Engine]
|
||||
↓
|
||||
Concrete Objects (Text, Line, Page)
|
||||
↓
|
||||
[Rendering Engine]
|
||||
↓
|
||||
Visual Output (Images, PDF, etc.)
|
||||
```
|
||||
|
||||
### Example Transformation
|
||||
Inspecting the intermediate concrete objects:
|
||||
|
||||
```python
|
||||
# 1. Abstract content
|
||||
paragraph = Paragraph()
|
||||
paragraph.add_word(Word("This", font))
|
||||
paragraph.add_word(Word("is", font))
|
||||
paragraph.add_word(Word("a", font))
|
||||
paragraph.add_word(Word("test", font))
|
||||
from pyWebLayout.concrete.text import Line
|
||||
|
||||
# 2. Layout transformation
|
||||
layout = ParagraphLayout(line_width=200, line_height=20)
|
||||
lines = layout.layout_paragraph(paragraph) # Returns List[Line]
|
||||
|
||||
# 3. Each Line contains concrete Text objects
|
||||
for line in lines:
|
||||
for text_obj in line.text_objects: # List[Text]
|
||||
print(f"Text: '{text_obj.text}' at position {text_obj._origin}")
|
||||
for line in (c for c in page.children if isinstance(c, Line)):
|
||||
print([t.text for t in line.text_objects])
|
||||
# ['Chapter', 'One']
|
||||
# ['It', 'was', 'a', 'dark', 'and', 'stormy', 'night.']
|
||||
```
|
||||
|
||||
## Key Architectural Principles
|
||||
For paginated reading, drive `EreaderLayoutManager` instead of building pages by hand:
|
||||
|
||||
### 1. **Single Responsibility**
|
||||
- Abstract classes: Handle content and structure
|
||||
- Concrete classes: Handle rendering and layout
|
||||
|
||||
### 2. **Separation of Concerns**
|
||||
- Text parsing/processing ≠ Text rendering
|
||||
- Document structure ≠ Page layout
|
||||
- Content semantics ≠ Visual presentation
|
||||
|
||||
### 3. **Immutable Abstract Content**
|
||||
- Abstract content remains unchanged during rendering
|
||||
- Multiple concrete representations can be generated from same abstract content
|
||||
- Enables pagination, different formats, responsive layouts
|
||||
|
||||
### 4. **One-to-Many Relationships**
|
||||
- One Abstract Word → Multiple Concrete Text objects (hyphenation)
|
||||
- One Abstract Paragraph → Multiple Concrete Lines
|
||||
- One Abstract Document → Multiple Concrete Pages
|
||||
|
||||
## Common Anti-Patterns to Avoid
|
||||
|
||||
### ❌ **Mixing Concerns**
|
||||
```python
|
||||
# WRONG: Abstract class knowing about pixels
|
||||
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager
|
||||
|
||||
manager = EreaderLayoutManager(blocks, page_size=(800, 600), document_id="my-book")
|
||||
page = manager.get_current_page()
|
||||
page = manager.next_page()
|
||||
manager.set_font_scale(1.25) # re-lays out from the same RenderingPosition
|
||||
```
|
||||
|
||||
## Design principles
|
||||
|
||||
**1. Abstract content is not mutated by layout.** Font scaling and family overrides
|
||||
produce scaled copies at layout time rather than editing the document. This is what
|
||||
allows the same `blocks` list to back a buffer of pages at several font sizes.
|
||||
|
||||
**2. Positions are expressed in abstract coordinates.** `RenderingPosition` names a
|
||||
chapter, block, and word — never a pixel. A reader who changes font size stays on the
|
||||
same sentence.
|
||||
|
||||
**3. Layouters report partial success.** Every layouter returns whether it fit, and the
|
||||
text layouter also returns where it stopped. Pagination is built from this rather than
|
||||
from a separate page-breaking pass.
|
||||
|
||||
**4. One abstract object may become many concrete ones.** One `Word` → one or two `Text`
|
||||
fragments; one `Paragraph` → many `Line`s across many `Page`s; one `Document` → an
|
||||
unbounded sequence of `Page`s. Any API that assumes a one-to-one mapping will be wrong
|
||||
at a hyphen or a page boundary.
|
||||
|
||||
**5. Measurement is cached, not avoided.** Text width and glyph rasterisation are the
|
||||
hot path. `core/cache.py` bounds their cost; `prewarm_text_caches` front-loads it for a
|
||||
known document.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
**Concrete state stored on abstract objects.** An abstract object that remembers its
|
||||
rendered width, position, or `Text` objects is wrong at the second rendering. If a
|
||||
back-reference is needed, point from concrete to abstract.
|
||||
|
||||
```python
|
||||
# WRONG
|
||||
class Word:
|
||||
def __init__(self, text):
|
||||
self.text = text
|
||||
self.rendered_width = None # ❌ Concrete concern in abstract class
|
||||
self.rendered_width = None # invalidated by any font change
|
||||
|
||||
# RIGHT
|
||||
text = Text(word.text, font, draw, source=word) # concrete knows its origin
|
||||
```
|
||||
|
||||
### ❌ **renderable_words Concept**
|
||||
```python
|
||||
# WRONG: Confusing abstract and concrete
|
||||
line.renderable_words # ❌ This suggests Words are renderable
|
||||
# Words are abstract - only Text objects render
|
||||
```
|
||||
**Treating `Word` as renderable.** Words are abstract; only `Text` draws. There is no
|
||||
`renderable_words` anywhere in the codebase, and there should not be.
|
||||
|
||||
### ✅ **Correct Separation**
|
||||
```python
|
||||
# CORRECT: Clear separation
|
||||
abstract_word = Word("test") # Abstract content
|
||||
concrete_text = Text("test", font) # Concrete rendering
|
||||
line.text_objects.append(concrete_text) # Concrete objects in concrete container
|
||||
```
|
||||
**Assuming a layouter returns lines.** `paragraph_layouter` appends to a page and
|
||||
reports what did not fit. Code that expects `List[Line]` back is working from an
|
||||
outdated model.
|
||||
|
||||
## Benefits of This Architecture
|
||||
## Summary
|
||||
|
||||
### 1. **Flexibility**
|
||||
- Same content can be rendered at different sizes
|
||||
- Multiple output formats from single source
|
||||
- Easy to implement responsive design
|
||||
|
||||
### 2. **Testability**
|
||||
- Abstract logic can be tested without rendering
|
||||
- Layout algorithms can be tested independently
|
||||
- Visual rendering can be mocked
|
||||
|
||||
### 3. **Performance**
|
||||
- Abstract content can be cached and reused
|
||||
- Layout can be computed once for multiple renderings
|
||||
- Incremental updates possible
|
||||
|
||||
### 4. **Maintainability**
|
||||
- Clear boundaries between text processing and rendering
|
||||
- Changes to rendering don't affect content parsing
|
||||
- Easy to swap rendering backends
|
||||
|
||||
## File Organization
|
||||
|
||||
```
|
||||
pyWebLayout/
|
||||
├── abstract/ # Content and structure
|
||||
│ ├── block.py # Document blocks (Paragraph, Heading, etc.)
|
||||
│ ├── inline.py # Inline content (Word, etc.)
|
||||
│ ├── document.py # Document structure
|
||||
│ └── functional.py # Links, buttons, etc.
|
||||
│
|
||||
├── concrete/ # Rendering and layout
|
||||
│ ├── text.py # Text and Line rendering
|
||||
│ ├── page.py # Page layout and containers
|
||||
│ ├── box.py # Base rendering classes
|
||||
│ ├── image.py # Image rendering
|
||||
│ └── functional.py # Interactive elements
|
||||
│
|
||||
├── typesetting/ # Layout algorithms
|
||||
│ ├── paragraph_layout.py # Abstract → Concrete transformation
|
||||
│ ├── flow.py # Text flow management
|
||||
│ └── pagination.py # Page breaking logic
|
||||
│
|
||||
└── style/ # Styling and formatting
|
||||
├── fonts.py # Font management
|
||||
├── layout.py # Layout constants
|
||||
└── alignment.py # Alignment enums
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
The Abstract/Concrete separation is fundamental to pyWebLayout's design. It ensures clean separation between content processing and visual rendering, enabling flexible, maintainable, and testable document processing pipelines.
|
||||
|
||||
**Remember**:
|
||||
- **Abstract** = What to display (content, structure, semantics)
|
||||
- **Concrete** = How to display it (pixels, fonts, positioning, rendering)
|
||||
|
||||
This architecture enables the library to handle complex document layouts while maintaining clear, understandable code organization.
|
||||
- **`core/`** — contracts and shared machinery
|
||||
- **`style/`** — semantic style, and its resolution to concrete rendering parameters
|
||||
- **`abstract/`** — what the document says
|
||||
- **`concrete/`** — where the pixels go
|
||||
- **`layout/`** — the transformation between them, and pagination on top of it
|
||||
- **`io/readers/`** — markup in
|
||||
|
||||
@@ -0,0 +1,527 @@
|
||||
# Architecture Review
|
||||
|
||||
Independent review of the codebase at `c5c61a3` (2026-08-06), answering one
|
||||
question: **is this a well-architected library or an over-complex mess?**
|
||||
|
||||
It is a well-architected library with one rotten subsystem inside it. The core
|
||||
design holds up; roughly a fifth of the code is speculative or non-functional,
|
||||
and it is concentrated in the ereader pagination/buffering layer.
|
||||
|
||||
This document records the verdict, the evidence, and the findings **not already
|
||||
covered** by [LAYOUT_REMEDIATION_SPEC.md](LAYOUT_REMEDIATION_SPEC.md). Where a
|
||||
finding is already specced, it is cross-referenced rather than restated.
|
||||
|
||||
## Contents
|
||||
|
||||
| ID | Finding | Severity | Status |
|
||||
|----|---------|----------|--------|
|
||||
| [R1](#r1--the-process-pool-crashes-on-python-314) | The process pool crashes on Python 3.14 | Critical | New; raises priority of S12 |
|
||||
| [R2](#r2--the-test-suite-hangs-at-interpreter-exit) | Test suite hangs at interpreter exit | High | New; same root cause as R1 |
|
||||
| [R3](#r3--font-scaling-destroys-hyperlinks) | Font scaling destroys hyperlinks | High | New |
|
||||
| [R4](#r4--three-packaging-configs-that-disagree) | Three packaging configs that disagree | Medium | New (corrected) |
|
||||
| [R5](#r5--monkey-patched-page-methods-with-a-conflicting-signature) | Monkey-patched `Page` methods with a conflicting signature | Medium | New |
|
||||
| [R6](#r6--dead-duck-typing-cluster-in-pagepy) | Dead duck-typing cluster in `page.py` | Low | New; extends S10.3 |
|
||||
| [R7](#r7--two-orphaned-subsystems) | Two orphaned subsystems | Low | New |
|
||||
| [R8](#r8--backward-pagination-is-guesswork) | Backward pagination is guesswork | Medium | Partially noted in S11 |
|
||||
|
||||
---
|
||||
|
||||
## Verdict
|
||||
|
||||
**Well architected.** The concerns that usually decide this question are all on
|
||||
the right side of the line:
|
||||
|
||||
- **The abstract/concrete split is real, not aspirational.** Verified
|
||||
empirically: `abstract/` never imports `concrete/`; `core/` imports nothing but
|
||||
itself. The single crossing ([document.py:7](../pyWebLayout/abstract/document.py#L7))
|
||||
is into `style/`, which the dependency rules permit. Most codebases claiming
|
||||
this layering have leaked it within a year.
|
||||
- **The layouter contract is the right abstraction.** `paragraph_layouter`
|
||||
returning `(fit, failed_word_index, remaining_pretext)` is what makes
|
||||
pagination resumable, and the shape is consistent across content types. Layout
|
||||
engines that return `List[Line]` cannot paginate without a second pass.
|
||||
- **[core/cache.py](../pyWebLayout/core/cache.py) is exemplary.** Usage-ranked
|
||||
eviction with periodic aging, sampled eviction instead of a maintained heap,
|
||||
O(1) hit path with no reordering — every choice justified by a measurement in
|
||||
the docstring.
|
||||
- **[concrete/text.py](../pyWebLayout/concrete/text.py) is the strongest file.**
|
||||
The glyph cache ([:581-647](../pyWebLayout/concrete/text.py#L581-L647))
|
||||
reimplements PIL's internals to skip per-call setup, with a permanent graceful
|
||||
fallback when the private API is absent. Alignment is a clean strategy pattern,
|
||||
and `render_alignment_handler` handles last-line-of-paragraph correctly.
|
||||
- **[html_extraction.py](../pyWebLayout/io/readers/html_extraction.py) is
|
||||
textbook.** An immutable `StyleContext` threaded down the tree plus a handler
|
||||
dispatch table — no giant if/elif, no mutable parser state.
|
||||
|
||||
**The complexity that is not earned** is concentrated in three places, all in the
|
||||
same band of the code:
|
||||
|
||||
1. `layout/page_buffer.py` — 520 lines of multiprocess prefetch that has never
|
||||
worked (S12, plus R1/R2 below).
|
||||
2. `BidirectionalLayouter.render_page_backward` — ~100 lines of convergence
|
||||
heuristics standing in for an anchor list that already exists (R8).
|
||||
3. The second block dispatcher in `ereader_layout.py`, which silently drops
|
||||
tables and lists (S4, S8).
|
||||
|
||||
The pattern is visible in the git history: work from `e000068` onward (caching,
|
||||
alignment, page geometry) is markedly better than the ereader scaffolding it sits
|
||||
on. This is not a mess. It is a solid library with an early prototype still
|
||||
embedded in it.
|
||||
|
||||
**Sizing the cleanup:** R1–R7 plus S12 and S10.3 remove roughly 800–900 lines and
|
||||
fix four user-visible defects. None of it requires redesigning anything.
|
||||
|
||||
## Test baseline
|
||||
|
||||
At `c5c61a3`, in a clean venv on Python 3.14.6:
|
||||
|
||||
```
|
||||
833 passed, 2 skipped, 24 subtests passed in 11.51s
|
||||
```
|
||||
|
||||
(The 2 skips were environmental — the review venv lacked `requests`, so the URL
|
||||
image tests skipped. With the `test` extra from R4 installed the suite reports
|
||||
`853 passed, 24 subtests passed in 13.53s`.)
|
||||
|
||||
The suite then **hangs indefinitely** rather than exiting. See R2.
|
||||
|
||||
## Status of the existing remediation spec
|
||||
|
||||
| Spec | Subject | State |
|
||||
|------|---------|-------|
|
||||
| S1 | Inline content in non-paragraph containers | Done (`284d521`) |
|
||||
| S2 | Page geometry: origin and content rect | Done (`f18cec2`) |
|
||||
| S3 | Draw/canvas lifecycle | Done (`202dacf`) |
|
||||
| S11 | Partial-block progress discarded | Done (`a57da80`) |
|
||||
| S13 | Word spacing and alignment | Done (`1262be6`) |
|
||||
| S14 | Vertical centring in buttons and fields | Done (`c5c61a3`) |
|
||||
| S4–S10, S12 | Dispatch, cells, table grid, pagination, hygiene, background rendering | Outstanding |
|
||||
|
||||
The spec's analysis is sound and in places sharper than this review — S12 caught
|
||||
that `_render_page_worker` omits `page_size` entirely, which this review missed.
|
||||
Nothing below supersedes it.
|
||||
|
||||
---
|
||||
|
||||
## R1 — The process pool crashes on Python 3.14
|
||||
|
||||
**Severity: critical. Raises S12 from "useless" to "fatal".**
|
||||
|
||||
### Problem
|
||||
|
||||
`PageBuffer` submits to a `ProcessPoolExecutor` from inside
|
||||
`BufferedPageRenderer.render_page`
|
||||
([page_buffer.py:431](../pyWebLayout/layout/page_buffer.py#L431)). Python 3.14
|
||||
changed the default multiprocessing start method on Linux from `fork` to
|
||||
`forkserver`. Under a non-`fork` start method, `submit()` reaches
|
||||
`_check_not_importing_main()`, which raises unless the caller sits inside an
|
||||
`if __name__ == "__main__":` guard — and the child re-imports the caller's main
|
||||
module, re-executing it.
|
||||
|
||||
S12 documents this subsystem as delivering no benefit. On 3.14 it is worse than
|
||||
that: `EreaderLayoutManager.get_current_page()` **raises** when called from
|
||||
module-level script code.
|
||||
|
||||
### Evidence
|
||||
|
||||
A plain script calling `manager.get_current_page()` at module level, Python
|
||||
3.14.6:
|
||||
|
||||
```
|
||||
RuntimeError:
|
||||
An attempt has been made to start a new process before the
|
||||
current process has finished its bootstrapping phase.
|
||||
...
|
||||
File "pyWebLayout/layout/page_buffer.py", line 221, in _queue_forward_renders
|
||||
future = self.executor.submit(_render_page_worker, args)
|
||||
ConnectionResetError: [Errno 104] Connection reset by peer
|
||||
```
|
||||
|
||||
With a `__main__` guard added it does not raise, and instead confirms S12's
|
||||
finding on every job:
|
||||
|
||||
```
|
||||
Background render failed for position RenderingPosition(...): cannot pickle 'ImagingCore' object
|
||||
Background render failed for position RenderingPosition(...): cannot pickle 'ImagingCore' object
|
||||
```
|
||||
|
||||
### Action
|
||||
|
||||
Fold into **S12**, and treat S12 as unblocked and urgent rather than phase 4. The
|
||||
recommended resolution there — delete the pool, keep the LRU buffers and position
|
||||
maps, replace prefetch with synchronous readahead — resolves R1 and R2 as a side
|
||||
effect. S12's measurement gate still applies to the *readahead* decision; it does
|
||||
not need to gate deletion of the pool, because the pool's contribution is
|
||||
provably zero.
|
||||
|
||||
### Files
|
||||
|
||||
`pyWebLayout/layout/page_buffer.py`
|
||||
|
||||
---
|
||||
|
||||
## R2 — The test suite hangs at interpreter exit
|
||||
|
||||
**Severity: high.** Same root cause as R1.
|
||||
|
||||
### Problem
|
||||
|
||||
`PageBuffer.__del__` calls `shutdown()`, which calls
|
||||
`executor.shutdown(wait=True)` ([page_buffer.py:342](../pyWebLayout/layout/page_buffer.py#L342)).
|
||||
`EreaderLayoutManager.__del__` does the same via `renderer.shutdown()`. Running
|
||||
`__del__` at interpreter shutdown and blocking on a process pool inside it
|
||||
deadlocks.
|
||||
|
||||
### Evidence
|
||||
|
||||
```
|
||||
833 passed, 2 skipped, 24 subtests passed in 11.51s
|
||||
```
|
||||
|
||||
...then the process sat at ~0% CPU with idle forkserver children for 13 minutes
|
||||
before being killed. Reproduced twice; both runs completed the tests in under
|
||||
12s and neither exited.
|
||||
|
||||
This is why CI wall-clock does not resemble the 11.5s the tests actually take.
|
||||
|
||||
### Action
|
||||
|
||||
Resolved by S12's deletion of the executor. If for any reason the pool is
|
||||
retained, `__del__` must not block: register an `atexit` handler or require
|
||||
explicit `shutdown()`, and never `wait=True` from a finaliser.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- `pytest` returns to the shell within a second of printing its summary line.
|
||||
- No `multiprocessing` child processes outlive the test session.
|
||||
|
||||
### Files
|
||||
|
||||
`pyWebLayout/layout/page_buffer.py`, `pyWebLayout/layout/ereader_manager.py`
|
||||
|
||||
---
|
||||
|
||||
## R3 — Font scaling destroys hyperlinks
|
||||
|
||||
**Severity: high. User-visible, silent, and trivially reproducible.**
|
||||
|
||||
### Problem
|
||||
|
||||
`BidirectionalLayouter._scale_block_fonts`
|
||||
([ereader_layout.py:474-498](../pyWebLayout/layout/ereader_layout.py#L474-L498))
|
||||
rebuilds a scaled block by constructing plain `Word(word.text, scaled_style)` for
|
||||
every word. `LinkedWord` is a `Word` subclass
|
||||
([inline.py:288](../pyWebLayout/abstract/inline.py#L288)), so the reconstruction
|
||||
downgrades it and the link target is discarded.
|
||||
|
||||
The function returns the block unchanged when `font_scale == 1.0` and no family
|
||||
override is set, which is why no test has caught this: the defect only appears
|
||||
once the reader changes font size.
|
||||
|
||||
Two further gaps in the same function:
|
||||
|
||||
1. It handles only `Paragraph` and `Heading`. Every other block type is returned
|
||||
unscaled, so a font-size change leaves images, tables and lists at their
|
||||
original size while the text around them reflows.
|
||||
2. It allocates a new `Paragraph` and a new `Word` per word on **every page
|
||||
render** at any scale ≠ 1.0 — directly against the caching work in
|
||||
`concrete/text.py`, and on the hot path.
|
||||
|
||||
### Evidence
|
||||
|
||||
At `c5c61a3`, parsing `<p>Go to <a href="http://x">this link</a> now.</p>`:
|
||||
|
||||
```
|
||||
scale=1.0: LinkedWords = 2
|
||||
scale=1.5: LinkedWords = 0
|
||||
```
|
||||
|
||||
### Design
|
||||
|
||||
Stop reconstructing abstract blocks at layout time. Font scale and family are
|
||||
*rendering context*, not document content — carrying them in a copied document
|
||||
violates the "abstract content is not mutated by layout" principle in
|
||||
[ARCHITECTURE.md](../ARCHITECTURE.md) in spirit, even though it copies rather
|
||||
than mutates.
|
||||
|
||||
Preferred: thread the scale/override into the layouter and resolve fonts at
|
||||
`Text` construction, where `Font` objects are already deduplicated by
|
||||
`FontRegistry`. `paragraph_layouter` already accepts an `alignment_override`;
|
||||
`font_scale` and `font_family_override` belong in the same place.
|
||||
|
||||
Minimum viable fix if the larger change is deferred: reconstruct via
|
||||
`type(word)` and copy subclass state, and extend coverage to every block type.
|
||||
This is strictly a stopgap — it keeps the per-page allocation cost.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- A document containing `<a href>` retains every `LinkedWord` after
|
||||
`set_font_scale(1.5)`, and `query_point` over the rendered page still returns
|
||||
`object_type="link"` with the correct target.
|
||||
- An image block's rendered size is unaffected by `set_font_scale`, or scales
|
||||
deliberately — not left inconsistent with the text around it.
|
||||
- No new `Word`/`Paragraph` objects are allocated per page render at scale ≠ 1.0.
|
||||
|
||||
### Files
|
||||
|
||||
`pyWebLayout/layout/ereader_layout.py`, `pyWebLayout/layout/document_layouter.py`
|
||||
|
||||
---
|
||||
|
||||
## R4 — Three packaging configs that disagree
|
||||
|
||||
**Severity: medium.** *Corrected: the original review claimed a clean install
|
||||
fails on first import. It does not — see below.*
|
||||
|
||||
### Problem
|
||||
|
||||
The project carries **three** sets of packaging metadata:
|
||||
|
||||
| File | Declares |
|
||||
|------|----------|
|
||||
| `pyproject.toml` `[project]` | Pillow, numpy, pyphen, beautifulsoup4, flask, ebooklib, requests |
|
||||
| `setup.cfg` `[options]` | Pillow, numpy |
|
||||
| `setup.py` `setup(...)` kwargs | Pillow, numpy |
|
||||
|
||||
`pyproject.toml`'s `[project]` table wins under any modern build backend, so the
|
||||
shipped wheel is correct and `pip install pyWebLayout` works. The `setup.cfg` and
|
||||
`setup.py` copies are dead, contradictory, and actively misleading — reading
|
||||
either one gives the wrong answer about what the library needs.
|
||||
|
||||
The authoritative list is itself wrong in the other direction:
|
||||
|
||||
- **`flask` is a runtime dependency.** It is imported only by
|
||||
`tests/abstract/test_abstract_blocks.py`, as a fixture HTTP server. Every user
|
||||
installs Flask, Jinja2, Werkzeug, click, itsdangerous and blinker for nothing.
|
||||
- **`ebooklib` is a runtime dependency and is never imported by the library.**
|
||||
`epub_reader.py` uses `zipfile` + `xml.etree` directly. Only the *tests* use
|
||||
ebooklib, to build EPUB fixtures.
|
||||
- **`requests` is declared required but is optional.** `concrete/image.py:100-111`
|
||||
imports it lazily and degrades to an error message on the image when absent.
|
||||
- **`requires-python = ">=3.6"` is false.** The package uses dataclasses (3.7+)
|
||||
and `from __future__ import annotations` (3.7+); CI tests 3.10, 3.12 and 3.13.
|
||||
|
||||
Net effect: a runtime install pulls 7 direct dependencies where 4 are needed.
|
||||
|
||||
### Action
|
||||
|
||||
- Consolidate on `pyproject.toml`. Reduce `setup.cfg` to its `[flake8]` section
|
||||
and `setup.py` to a `setup()` shim, each with a comment saying where metadata
|
||||
lives.
|
||||
- Runtime deps: Pillow, numpy, pyphen, beautifulsoup4. Move flask, werkzeug,
|
||||
ebooklib and requests into a `test` extra; add a `remote-images` extra for
|
||||
requests; add a `dev` extra composing them.
|
||||
- Set `requires-python = ">=3.10"` to match the CI matrix, and add version
|
||||
classifiers.
|
||||
- Add a CI step that installs the package into an empty venv with **only**
|
||||
declared runtime deps and imports every top-level subpackage. This class of
|
||||
defect is only caught by installing what you ship — and it is what would have
|
||||
caught the original misreading.
|
||||
|
||||
### Files
|
||||
|
||||
`pyproject.toml`, `setup.cfg`, `setup.py`, `.gitea/workflows/ci.yml`
|
||||
|
||||
---
|
||||
|
||||
## R5 — Monkey-patched `Page` methods with a conflicting signature
|
||||
|
||||
**Severity: medium.** Currently inert; a landmine if `Page` is ever refactored.
|
||||
|
||||
### Problem
|
||||
|
||||
[ereader_layout.py:741-761](../pyWebLayout/layout/ereader_layout.py#L741-L761)
|
||||
defines `_add_page_methods()` and calls it at import time. It attaches
|
||||
`can_fit_line` and `available_width` to the `Page` class if they are absent.
|
||||
|
||||
`Page` defines both ([page.py:59](../pyWebLayout/concrete/page.py#L59),
|
||||
[page.py:147](../pyWebLayout/concrete/page.py#L147)), so the patch never fires.
|
||||
But the two definitions of `can_fit_line` **do not agree**:
|
||||
|
||||
| Source | Signature |
|
||||
|--------|-----------|
|
||||
| `Page` | `can_fit_line(baseline_spacing, ascent=0, descent=0)` |
|
||||
| monkey patch | `can_fit_line(line_height)` |
|
||||
|
||||
The patched version also ignores descenders entirely — the exact bug S2 fixed. If
|
||||
`Page.can_fit_line` were ever renamed or moved, this would silently reinstate
|
||||
pre-S2 clipping behaviour, from an import side effect in a different package.
|
||||
|
||||
### Action
|
||||
|
||||
Delete `_add_page_methods` and its call site. Import-time monkey-patching of a
|
||||
class in another module has no place here; if `Page` is missing something the
|
||||
layout engine needs, it belongs on `Page`.
|
||||
|
||||
### Files
|
||||
|
||||
`pyWebLayout/layout/ereader_layout.py`
|
||||
|
||||
---
|
||||
|
||||
## R6 — Dead duck-typing cluster in `page.py`
|
||||
|
||||
**Severity: low.** Extends S10.3.
|
||||
|
||||
### Problem
|
||||
|
||||
[page.py:261-491](../pyWebLayout/concrete/page.py#L261-L491) contains a closed
|
||||
cluster with no external callers:
|
||||
|
||||
- `_get_child_property` (:261) — called only by the four below
|
||||
- `_get_child_height` (:301) — called by nothing
|
||||
- `_get_child_position` (:382) — called only by `_point_in_child`
|
||||
- `_point_in_child` (:435) — called by nothing
|
||||
- `_get_child_size` (:466) — called only by `_point_in_child`
|
||||
|
||||
Verified by grep across `pyWebLayout/`, `tests/`, `examples/` and `scripts/`:
|
||||
zero references outside the cluster. About 90 lines.
|
||||
|
||||
It exists because `Renderable` declares neither `size` nor `origin`, so the code
|
||||
probes `_size`, `size`, `_height`, `height`, `_origin` and `position` in turn with
|
||||
`hasattr`. `query_point` (:399) already does the right thing instead — it relies
|
||||
on the `Queriable` interface.
|
||||
|
||||
### Action
|
||||
|
||||
- Delete all five methods.
|
||||
- Add `origin` and `size` to the `Renderable`/`Geometric` contract in
|
||||
`core/base.py` so the duck-typing cannot grow back. This is the same concern as
|
||||
S10.1's render contract and can ship with it.
|
||||
|
||||
### Files
|
||||
|
||||
`pyWebLayout/concrete/page.py`, `pyWebLayout/core/base.py`
|
||||
|
||||
---
|
||||
|
||||
## R7 — Two orphaned subsystems
|
||||
|
||||
**Severity: low**, but they are a large share of the "is this over-complex?"
|
||||
impression: 559 lines that nothing in the library reaches.
|
||||
|
||||
### Problem
|
||||
|
||||
**`concrete/interaction_handler.py` (310 lines).** `InteractionHandler` and
|
||||
`InteractionStateManager` are referenced only by
|
||||
`examples/07_pressed_state_demo.py`. No library code, no ereader path, no tests.
|
||||
|
||||
**`core/highlight.py` (249 lines).** `Highlight`, `HighlightColor` and
|
||||
`HighlightManager` have tests (`tests/core/test_highlight.py`) but are not wired
|
||||
into `EreaderLayoutManager` at all. Highlighting is not reachable through the
|
||||
library's own top-level interface.
|
||||
|
||||
`HighlightManager` also duplicates `BookmarkManager`'s JSON persistence
|
||||
(directory, `_save`, `_load`, per-document file naming) with no shared base.
|
||||
|
||||
### Action
|
||||
|
||||
Decide per subsystem, and record the decision:
|
||||
|
||||
- **Wire it up** — `EreaderLayoutManager` grows `add_highlight` / `highlights_for_page`
|
||||
and the persistence merges with `BookmarkManager` into one document-state store.
|
||||
- **Or move it out** — relocate to `examples/` or delete, and drop the tests with it.
|
||||
|
||||
Either is fine. Leaving a tested, documented, unreachable subsystem in `core/` is
|
||||
what makes the library look larger and less coherent than it is.
|
||||
|
||||
### Files
|
||||
|
||||
`pyWebLayout/concrete/interaction_handler.py`, `pyWebLayout/core/highlight.py`,
|
||||
`pyWebLayout/layout/ereader_manager.py`
|
||||
|
||||
---
|
||||
|
||||
## R8 — Backward pagination is guesswork
|
||||
|
||||
**Severity: medium.** S11's closing note already flags this for audit; this
|
||||
records what the audit found.
|
||||
|
||||
### Problem
|
||||
|
||||
`render_page_backward`
|
||||
([ereader_layout.py:372-472](../pyWebLayout/layout/ereader_layout.py#L372-L472))
|
||||
finds the previous page by estimating a start position, rendering forward,
|
||||
comparing the end against the target, and adjusting — **up to 10 times**. It then
|
||||
has a fallback that jumps back up to 5 blocks and renders again, and a fallback
|
||||
for *that* which renders from the start of the document.
|
||||
|
||||
Worst case: one "previous page" tap costs up to 12 full page layouts.
|
||||
|
||||
The estimator it converges from is `max(1, int(10 / font_scale))` blocks
|
||||
([:684](../pyWebLayout/layout/ereader_layout.py#L684)) — a constant with no
|
||||
relationship to page size, block length or font metrics.
|
||||
|
||||
The correct answer is usually already known.
|
||||
`EreaderLayoutManager._page_history` ([ereader_manager.py:210](../pyWebLayout/layout/ereader_manager.py#L210))
|
||||
records real page-start positions and serves them instantly; the refinement loop
|
||||
only runs when history misses — after a jump, a bookmark, a font change, or
|
||||
beyond 50 entries.
|
||||
|
||||
S11's note asks whether these fallbacks were compensating for the
|
||||
discarded-progress bug it fixed. They were, in part: the "failed to move
|
||||
backward" branch at [:446](../pyWebLayout/layout/ereader_layout.py#L446) is
|
||||
reachable precisely when forward rendering fails to advance, which S11 addressed.
|
||||
|
||||
### Design
|
||||
|
||||
Replace convergence with anchors. Maintain a sorted list of known page-start
|
||||
positions — chapter starts from `ChapterNavigator` (free, already built) plus
|
||||
every position visited. To go back from position P: binary-search the largest
|
||||
anchor A < P, render forward from A collecting page starts until reaching P, and
|
||||
return the last one. Cost is bounded by the anchor spacing, and every page start
|
||||
discovered on the way is itself a new anchor, so the second traversal of any
|
||||
region is free.
|
||||
|
||||
This subsumes `_page_history`, so the two mechanisms become one.
|
||||
|
||||
**Sequencing:** do this after S12, and after S8 — table and list pagination
|
||||
changes what a page start can be, and re-deriving anchors is cheap only once
|
||||
positions round-trip through tables correctly (S8 already notes this dependency).
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- `previous_page()` from any position issues at most *k* page layouts, where *k*
|
||||
is the anchor spacing, with no iteration count and no fallback ladder.
|
||||
- Forward-then-backward round-trips exactly, from a cold cache, after a chapter
|
||||
jump, and after a bookmark restore.
|
||||
- `_estimate_page_start`, `_adjust_start_estimate` and the three-tier fallback
|
||||
are deleted, not retained alongside.
|
||||
|
||||
### Files
|
||||
|
||||
`pyWebLayout/layout/ereader_layout.py`, `pyWebLayout/layout/ereader_manager.py`
|
||||
|
||||
---
|
||||
|
||||
## Recommended order
|
||||
|
||||
```
|
||||
R4 ── packaging; independent, minutes, unblocks clean CI [done]
|
||||
S12 ── delete the process pool; resolves R1 and R2 with it
|
||||
R3 ── font scaling loses links; independent, user-visible
|
||||
R5 ── delete the monkey patch; minutes
|
||||
R6 ── delete the dead cluster (with S10.1's render contract)
|
||||
R7 ── decide the two orphans; no code risk either way
|
||||
S4 → S5 → S6 → S7 → S8 → S9 (existing spec, unchanged)
|
||||
R8 ── after S8
|
||||
```
|
||||
|
||||
R4, R5 and R6 are an afternoon and carry no design risk. S12 is the largest
|
||||
single removal and fixes two defects at once. R3 is the one users would notice
|
||||
today. Everything after that is the existing spec, which needs no revision.
|
||||
|
||||
## Reproducing the findings
|
||||
|
||||
Reviewed at `c5c61a3` on Python 3.14.6, in a venv containing
|
||||
`pytest pyphen Pillow numpy beautifulsoup4 lxml ebooklib`.
|
||||
|
||||
- **R1**: call `EreaderLayoutManager(...).get_current_page()` from module-level
|
||||
script code (no `__main__` guard).
|
||||
- **R2**: `python -m pytest -q`; observe the summary line, then the hang.
|
||||
- **R3**: parse HTML containing `<a href>`, call
|
||||
`BidirectionalLayouter._scale_block_fonts(block, 1.5)`, count `LinkedWord`
|
||||
instances in the result.
|
||||
- **R5**: compare `inspect.signature(Page.can_fit_line)` against the patch body.
|
||||
- **R6**: grep the five method names across `pyWebLayout/ tests/ examples/ scripts/`.
|
||||
- **R8**: read the loop; no execution needed.
|
||||
@@ -30,6 +30,7 @@ It is independent of every other spec here.
|
||||
| [S13](#s13--word-spacing-and-alignment) | Word spacing and alignment | 0 |
|
||||
| [S14](#s14--vertical-centring-in-buttons-and-fields) | Vertical centring in buttons and fields | 0 |
|
||||
| [S15](#s15--form-field-label-geometry) | Form field label geometry | 0 |
|
||||
| [S16](#s16--backward-page-navigation) | Backward page navigation | 0 |
|
||||
|
||||
## Design invariants
|
||||
|
||||
@@ -1265,6 +1266,103 @@ between label and box smaller than the intended 5px.
|
||||
|
||||
---
|
||||
|
||||
## S16 — Backward page navigation
|
||||
|
||||
### Problem
|
||||
|
||||
`render_page_backward` *searched* for the previous page's start: estimate a block
|
||||
index, lay out forward, compare the end against the target, bisect on the block
|
||||
difference, repeat up to ten times. Both the estimator and the adjuster pinned
|
||||
`word_index` to 0 and moved only `block_index`.
|
||||
|
||||
Pages routinely start mid-block. Any such start was therefore **not in the search
|
||||
space**, the loop could never match, and it fell through to a fallback that
|
||||
jumped several blocks back or to the document start.
|
||||
|
||||
### Evidence
|
||||
|
||||
A document of short paragraphs around one 1200-word paragraph. Forward pagination
|
||||
gives page starts at `(0,0), (2,208), (2,494), (2,780), (2,1057)`. Asking for the
|
||||
page that ends where each of those begins:
|
||||
|
||||
```
|
||||
from page 1 -> got (0,0) expected (0,0) ok (1 forward layout)
|
||||
from page 2 -> got (0,0) expected (2,208) WRONG (10 forward layouts)
|
||||
from page 3 -> got (0,0) expected (2,494) WRONG (10 forward layouts)
|
||||
from page 4 -> got (0,0) expected (2,780) WRONG (10 forward layouts)
|
||||
```
|
||||
|
||||
Every mid-paragraph case threw the reader to the start of the document after ten
|
||||
full page layouts. The bisection was also unsound within its own space: a
|
||||
document of 40 small paragraphs, where every page *does* start on a block
|
||||
boundary, failed too.
|
||||
|
||||
This is complementary to S11 rather than caused by it. Before S11 forward
|
||||
pagination dead-ended at the first page-spanning block, so mid-block starts were
|
||||
never produced and the block-granular search looked adequate.
|
||||
|
||||
### Design
|
||||
|
||||
Pagination is a pure function: laying out from `q` yields a page and the position
|
||||
it stopped at, `next(q)`. The page before `P` is the `q` with `next(q) == P`.
|
||||
That is found by **replaying the chain forward from an anchor**, not by guessing
|
||||
`q`. Three sources, in order:
|
||||
|
||||
1. **The recorded chain.** `render_page_forward` now records
|
||||
`(font_scale, next(q)) -> q`. Stepping back to anywhere the reader has been is
|
||||
exact and costs one layout. Keyed by font scale, since changing it
|
||||
repaginates.
|
||||
2. **Replay from an anchor.** Anchors are block starts, nearest first: the block
|
||||
containing `P`, then up to `MAX_BACKWARD_ANCHORS` earlier ones, then the
|
||||
document start. Lay out forward from the anchor until a page ends exactly on
|
||||
`P`; that page's start is the answer. `MAX_REPLAY_PAGES` caps the walk so one
|
||||
page turn cannot traverse a whole chapter.
|
||||
3. **Nearest start before `P`.** If no chain passes exactly through `P` — which
|
||||
happens when `P` was reached by a jump or a restored bookmark rather than by
|
||||
reading forward, so it lies on no natural chain — return the last page start
|
||||
before it. That overlaps `P`'s page slightly rather than skipping content,
|
||||
which is the safe direction to be wrong in.
|
||||
|
||||
The estimator and the bisecting adjuster are deleted.
|
||||
|
||||
**What "correct" means here.** Each backward step returns a page ending exactly
|
||||
where the reader currently is, so paging back never skips or repeats content.
|
||||
That chain can differ from the one you would have seen reading forward from page
|
||||
one, if you entered the document by a jump — pagination from a different starting
|
||||
point is genuinely a different chain, and no algorithm can recover the original
|
||||
without replaying from the start.
|
||||
|
||||
### Measurements
|
||||
|
||||
Same document, after the change:
|
||||
|
||||
```
|
||||
warm (chain recorded by the forward pass): 4/4 exact, 1 layout each
|
||||
cold, fresh layouter per call: 12/13 exact, worst 17 layouts
|
||||
cold, one layouter, repeated back presses: 4 layouts per turn typical
|
||||
```
|
||||
|
||||
The single inexact case is a target that lies on the canonical chain but not on
|
||||
any chain reachable from a nearby anchor; it returns a start 15 words early,
|
||||
i.e. a slightly overlapping page.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- For every page of a document, `render_page_backward(start[i])` returns
|
||||
`start[i-1]` — verified for both a mid-paragraph-paginating document and one
|
||||
where every page starts on a block boundary.
|
||||
- Laying out forward from the returned position ends exactly on the requested
|
||||
position.
|
||||
- Forward-then-back returns to the original position.
|
||||
- At the document start, backward stays there; an empty document is safe.
|
||||
- Cost stays within a small bounded number of forward layouts.
|
||||
|
||||
### Files
|
||||
|
||||
`pyWebLayout/layout/ereader_layout.py`
|
||||
|
||||
---
|
||||
|
||||
## Test plan
|
||||
|
||||
Findings were reproduced with four probe scripts; each becomes a regression test
|
||||
|
||||
@@ -313,6 +313,13 @@ class BidirectionalLayouter:
|
||||
self.alignment_override = alignment_override
|
||||
self.font_family_override = font_family_override
|
||||
|
||||
# Maps (font_scale, end position) -> the position the page started at.
|
||||
# Filled in as pages are laid out forward, which makes "previous page"
|
||||
# exact and free for anywhere the reader has already been. Keyed by font
|
||||
# scale because changing it repaginates the document.
|
||||
self._page_chain: Dict[Tuple[float, Tuple[int, int, int]],
|
||||
RenderingPosition] = {}
|
||||
|
||||
def render_page_forward(self, position: RenderingPosition,
|
||||
font_scale: float = 1.0) -> Tuple[Page, RenderingPosition]:
|
||||
"""
|
||||
@@ -367,109 +374,156 @@ class BidirectionalLayouter:
|
||||
|
||||
current_pos = new_pos
|
||||
|
||||
# Remember this link in the chain so stepping back to it later is exact.
|
||||
if self._position_compare(current_pos, position) > 0:
|
||||
self._page_chain[(font_scale, self._position_key(current_pos))] = \
|
||||
position.copy()
|
||||
|
||||
return page, current_pos
|
||||
|
||||
# How many block starts before the target to try as replay anchors before
|
||||
# settling for the best inexact answer.
|
||||
MAX_BACKWARD_ANCHORS = 4
|
||||
|
||||
# Ceiling on pages replayed from a single anchor, so a pathologically long
|
||||
# block cannot make one page turn walk an entire chapter.
|
||||
MAX_REPLAY_PAGES = 8
|
||||
|
||||
def render_page_backward(self,
|
||||
end_position: RenderingPosition,
|
||||
font_scale: float = 1.0) -> Tuple[Page,
|
||||
RenderingPosition]:
|
||||
"""
|
||||
Render a page that ends at the given position, filling backward.
|
||||
Critical for "previous page" navigation.
|
||||
Render the page that ends at the given position - "previous page".
|
||||
|
||||
Uses iterative refinement to find the correct start position that
|
||||
results in a page ending at (or very close to) the target position.
|
||||
Pagination is a pure function: laying out from a position q yields a page
|
||||
and the position where it stopped, next(q). The page before P is therefore
|
||||
the q for which next(q) == P, and it is found by *replaying* the chain
|
||||
forward from an anchor, not by guessing q.
|
||||
|
||||
The previous implementation searched instead: it estimated a block index
|
||||
and bisected on it, pinning word_index to 0. Pages routinely start
|
||||
mid-block, so the answer was frequently not in the search space at all -
|
||||
the search then exhausted its iterations and fell back to a position that
|
||||
was not the previous page, usually the start of the document.
|
||||
|
||||
Three sources are tried in order:
|
||||
|
||||
1. The recorded chain, from pages already laid out going forward. Exact,
|
||||
and the common case when the reader is paging back and forth.
|
||||
2. Replay from the start of the block containing P, then from
|
||||
progressively earlier blocks. Exact when P lies on the resulting chain.
|
||||
3. Failing an exact hit - which happens when P was reached by a jump or a
|
||||
restored bookmark rather than by reading forward, so it is on no
|
||||
natural chain - the latest page start before P. That overlaps P's page
|
||||
slightly rather than skipping content, which is the safe direction to
|
||||
be wrong in.
|
||||
|
||||
Args:
|
||||
end_position: Position where page should end
|
||||
end_position: Position where the page should end
|
||||
font_scale: Font scaling factor
|
||||
|
||||
Returns:
|
||||
Tuple of (rendered_page, start_position)
|
||||
"""
|
||||
# Handle edge case: already at beginning
|
||||
if end_position.block_index == 0 and end_position.word_index == 0:
|
||||
return self.render_page_forward(end_position, font_scale)
|
||||
document_start = RenderingPosition()
|
||||
|
||||
# Start with initial estimate
|
||||
estimated_start = self._estimate_page_start(end_position, font_scale)
|
||||
# Nothing precedes the start of the document.
|
||||
if self._position_compare(end_position, document_start) <= 0:
|
||||
page, _ = self.render_page_forward(document_start, font_scale)
|
||||
return page, document_start
|
||||
|
||||
# Iterative refinement: keep adjusting until we converge or hit max iterations
|
||||
max_iterations = 10
|
||||
best_page = None
|
||||
best_start = estimated_start
|
||||
best_distance = float('inf')
|
||||
# 1. The chain we have already walked.
|
||||
remembered = self._page_chain.get((font_scale, self._position_key(end_position)))
|
||||
if remembered is not None:
|
||||
page, actual_end = self.render_page_forward(remembered, font_scale)
|
||||
if self._position_compare(actual_end, end_position) == 0:
|
||||
return page, remembered
|
||||
|
||||
for iteration in range(max_iterations):
|
||||
# Render forward from current estimate
|
||||
page, actual_end = self.render_page_forward(estimated_start, font_scale)
|
||||
# 2/3. Replay from anchors, keeping the best inexact result as a fallback.
|
||||
fallback = None
|
||||
for anchor in self._backward_anchors(end_position):
|
||||
page, start, exact = self._replay_to(anchor, end_position, font_scale)
|
||||
if page is None:
|
||||
continue
|
||||
if exact:
|
||||
return page, start
|
||||
if fallback is None:
|
||||
fallback = (page, start)
|
||||
|
||||
# Calculate how far we are from target
|
||||
comparison = self._position_compare(actual_end, end_position)
|
||||
if fallback is not None:
|
||||
return fallback
|
||||
|
||||
# Perfect match or close enough (within same block)
|
||||
# BUT: ensure we actually moved backward (estimated_start < end_position)
|
||||
if comparison == 0:
|
||||
# Check if we actually found a valid previous page
|
||||
if self._position_compare(estimated_start, end_position) < 0:
|
||||
return page, estimated_start
|
||||
# If estimated_start >= end_position, we haven't moved backward
|
||||
# Continue iterating to find a better position
|
||||
elif iteration == 0:
|
||||
# On first iteration, if we can't find a previous position,
|
||||
# we're likely at or near the beginning
|
||||
break
|
||||
page, _ = self.render_page_forward(document_start, font_scale)
|
||||
return page, document_start
|
||||
|
||||
# Track best result so far
|
||||
distance = abs(actual_end.block_index - end_position.block_index)
|
||||
if distance < best_distance:
|
||||
best_distance = distance
|
||||
best_page = page
|
||||
best_start = estimated_start.copy()
|
||||
def _backward_anchors(self, target: RenderingPosition):
|
||||
"""
|
||||
Yield positions to replay from, nearest first.
|
||||
|
||||
# Adjust estimate for next iteration
|
||||
estimated_start = self._adjust_start_estimate(
|
||||
estimated_start, end_position, actual_end)
|
||||
Block starts are used as anchors because they are the coarsest positions
|
||||
that are certainly valid to lay out from. The block containing the target
|
||||
comes first: when the target is mid-block, the page before it usually
|
||||
starts in that same block or the one before.
|
||||
"""
|
||||
first_block = target.block_index if target.word_index > 0 \
|
||||
else target.block_index - 1
|
||||
|
||||
# Safety: don't go before document start
|
||||
if estimated_start.block_index < 0:
|
||||
estimated_start.block_index = 0
|
||||
estimated_start.word_index = 0
|
||||
|
||||
# If we exhausted iterations, return best result found
|
||||
# BUT: ensure we didn't return the same position (no backward progress)
|
||||
final_page = best_page if best_page else page
|
||||
final_start = best_start
|
||||
|
||||
# Safety check: if final_start >= end_position, we failed to move backward
|
||||
# This can happen at the beginning of the document or when estimation failed
|
||||
if self._position_compare(final_start, end_position) >= 0:
|
||||
# Can't go further back - check if we're at the absolute beginning
|
||||
if end_position.block_index == 0 and end_position.word_index == 0:
|
||||
# Already at beginning, return as-is
|
||||
return final_page, final_start
|
||||
|
||||
# Fallback strategy: try a more aggressive backward jump
|
||||
# Start from several blocks before the current position
|
||||
blocks_to_jump = max(1, min(5, end_position.block_index))
|
||||
fallback_pos = RenderingPosition(
|
||||
chapter_index=end_position.chapter_index,
|
||||
block_index=max(0, end_position.block_index - blocks_to_jump),
|
||||
word_index=0
|
||||
for offset in range(self.MAX_BACKWARD_ANCHORS):
|
||||
block_index = first_block - offset
|
||||
if block_index < 0:
|
||||
break
|
||||
yield RenderingPosition(
|
||||
chapter_index=target.chapter_index,
|
||||
block_index=block_index,
|
||||
word_index=0,
|
||||
)
|
||||
|
||||
# Render forward from the fallback position
|
||||
fallback_page, fallback_end = self.render_page_forward(fallback_pos, font_scale)
|
||||
if first_block - self.MAX_BACKWARD_ANCHORS >= 0:
|
||||
yield RenderingPosition()
|
||||
|
||||
# Verify the fallback actually moved us backward
|
||||
if self._position_compare(fallback_pos, end_position) < 0:
|
||||
return fallback_page, fallback_pos
|
||||
def _replay_to(self,
|
||||
anchor: RenderingPosition,
|
||||
target: RenderingPosition,
|
||||
font_scale: float):
|
||||
"""
|
||||
Lay out pages forward from `anchor`, looking for the one ending at `target`.
|
||||
|
||||
# If even the fallback didn't work, we're likely at the beginning
|
||||
# Return a page starting from the beginning
|
||||
return self.render_page_forward(RenderingPosition(), font_scale)
|
||||
Returns:
|
||||
(page, start, exact). `exact` is True when a page ended precisely on
|
||||
the target. When the chain steps over the target instead, the last
|
||||
page starting before it is returned with exact=False. (None, None,
|
||||
False) means the anchor yielded nothing usable.
|
||||
"""
|
||||
position = anchor
|
||||
last = (None, None)
|
||||
|
||||
return final_page, final_start
|
||||
for _ in range(self.MAX_REPLAY_PAGES):
|
||||
if self._position_compare(position, target) >= 0:
|
||||
break
|
||||
|
||||
page, next_position = self.render_page_forward(position, font_scale)
|
||||
comparison = self._position_compare(next_position, target)
|
||||
|
||||
if comparison == 0:
|
||||
return page, position, True
|
||||
|
||||
if comparison > 0:
|
||||
# Stepped over the target: this chain does not pass through it.
|
||||
return last[0], last[1], False
|
||||
|
||||
if self._position_compare(next_position, position) <= 0:
|
||||
break # no progress; give up on this anchor
|
||||
|
||||
last = (page, position)
|
||||
position = next_position
|
||||
|
||||
return last[0], last[1], False
|
||||
|
||||
@staticmethod
|
||||
def _position_key(position: RenderingPosition) -> Tuple[int, int, int]:
|
||||
"""Hashable identity of a position, for the page chain map."""
|
||||
return (position.chapter_index, position.block_index, position.word_index)
|
||||
|
||||
def _scale_block_fonts(self, block: Block, font_scale: float) -> Block:
|
||||
"""Apply font scaling and font family override to all fonts in a block"""
|
||||
|
||||
@@ -935,16 +935,31 @@ class EreaderLayoutManager:
|
||||
"""
|
||||
Shutdown the ereader manager and clean up resources.
|
||||
Call this when the application is closing.
|
||||
|
||||
Idempotent: calling it twice saves the position once.
|
||||
"""
|
||||
if getattr(self, '_shutdown_done', False):
|
||||
return
|
||||
self._shutdown_done = True
|
||||
|
||||
# Save current position
|
||||
self.bookmark_manager.save_reading_position(self.current_position)
|
||||
|
||||
# Shutdown renderer and buffer
|
||||
# Release cached pages
|
||||
self.renderer.shutdown()
|
||||
|
||||
def __del__(self):
|
||||
"""Cleanup on destruction"""
|
||||
self.shutdown()
|
||||
"""
|
||||
Best-effort cleanup for callers that never called shutdown().
|
||||
|
||||
Finalisers run during interpreter teardown, when modules and globals
|
||||
may already be torn down, so this must never raise and must never
|
||||
block. Applications should call shutdown() explicitly.
|
||||
"""
|
||||
try:
|
||||
self.shutdown()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# Convenience function for quick setup
|
||||
|
||||
@@ -1,16 +1,35 @@
|
||||
"""
|
||||
Multi-process page buffering system for high-performance ereader navigation.
|
||||
Page caching for ereader navigation.
|
||||
|
||||
This module provides intelligent page caching with background rendering using
|
||||
multiprocessing to achieve sub-second page navigation performance.
|
||||
`PageBuffer` is an LRU cache of rendered pages plus the position links between
|
||||
them; `BufferedPageRenderer` wraps it around a `BidirectionalLayouter`.
|
||||
|
||||
This module used to render pages ahead of time in a `ProcessPoolExecutor`. That
|
||||
never worked and has been removed — see S12 in docs/LAYOUT_REMEDIATION_SPEC.md
|
||||
and R1/R2 in docs/ARCHITECTURE_REVIEW.md. In short: the worker returned
|
||||
`pickle.dumps(page)`, and a Page holds a live PIL canvas, which is not
|
||||
picklable, so every job failed and the result was discarded. The cost — four
|
||||
interpreter copies and the whole block list shipped per job — was paid in full
|
||||
for no benefit. On Python 3.14, where the default start method became
|
||||
`forkserver`, submitting from module-level code raised outright.
|
||||
|
||||
Prefetch is not needed. Measured on the tests/data Wikipedia fixture (411
|
||||
blocks) with the text caches warm, one page render costs:
|
||||
|
||||
800x600 p50 8.8 ms p95 15.4 ms
|
||||
1072x1448 p50 13.8 ms p95 56.1 ms
|
||||
|
||||
A page turn is cheaper than the IPC that was meant to hide it. If a slower
|
||||
target device ever changes that, the fallback is a synchronous `readahead()`
|
||||
method on this class, or a single worker *thread* — layout is PIL-bound and PIL
|
||||
releases the GIL — not a process pool. Making the concrete tree picklable
|
||||
(Page -> Line -> Text -> Font -> FreeTypeFont) is a large surface area to
|
||||
maintain for a cache.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
from typing import Dict, Optional, List, Tuple, Any
|
||||
from collections import OrderedDict
|
||||
from concurrent.futures import ProcessPoolExecutor, Future
|
||||
import threading
|
||||
import pickle
|
||||
|
||||
from .ereader_layout import RenderingPosition, BidirectionalLayouter, FontFamilyOverride
|
||||
from pyWebLayout.concrete.page import Page
|
||||
@@ -19,57 +38,20 @@ from pyWebLayout.style.page_style import PageStyle
|
||||
from pyWebLayout.style.fonts import BundledFont
|
||||
|
||||
|
||||
def _render_page_worker(args: Tuple[List[Block],
|
||||
PageStyle,
|
||||
RenderingPosition,
|
||||
float,
|
||||
bool,
|
||||
Optional[BundledFont]]) -> Tuple[RenderingPosition,
|
||||
bytes,
|
||||
RenderingPosition]:
|
||||
"""
|
||||
Worker function for multiprocess page rendering.
|
||||
|
||||
Args:
|
||||
args: Tuple of (blocks, page_style, position, font_scale, is_backward, font_family)
|
||||
|
||||
Returns:
|
||||
Tuple of (original_position, pickled_page, next_position)
|
||||
"""
|
||||
blocks, page_style, position, font_scale, is_backward, font_family = args
|
||||
|
||||
# Create font family override if specified
|
||||
font_family_override = FontFamilyOverride(font_family) if font_family else None
|
||||
|
||||
layouter = BidirectionalLayouter(blocks, page_style, font_family_override=font_family_override)
|
||||
|
||||
if is_backward:
|
||||
page, next_pos = layouter.render_page_backward(position, font_scale)
|
||||
else:
|
||||
page, next_pos = layouter.render_page_forward(position, font_scale)
|
||||
|
||||
# Serialize the page for inter-process communication
|
||||
pickled_page = pickle.dumps(page)
|
||||
|
||||
return position, pickled_page, next_pos
|
||||
|
||||
|
||||
class PageBuffer:
|
||||
"""
|
||||
Intelligent page caching system with LRU eviction and background rendering.
|
||||
Maintains separate forward and backward buffers for optimal navigation performance.
|
||||
LRU cache of rendered pages, with separate forward and backward buffers and
|
||||
the position links between adjacent pages.
|
||||
"""
|
||||
|
||||
def __init__(self, buffer_size: int = 5, max_workers: int = 4):
|
||||
def __init__(self, buffer_size: int = 5):
|
||||
"""
|
||||
Initialize the page buffer.
|
||||
|
||||
Args:
|
||||
buffer_size: Number of pages to cache in each direction
|
||||
max_workers: Maximum number of worker processes for background rendering
|
||||
"""
|
||||
self.buffer_size = buffer_size
|
||||
self.max_workers = max_workers
|
||||
|
||||
# LRU caches for forward and backward pages
|
||||
self.forward_buffer: OrderedDict[RenderingPosition, Page] = OrderedDict()
|
||||
@@ -81,11 +63,6 @@ class PageBuffer:
|
||||
self.reverse_position_map: Dict[RenderingPosition,
|
||||
RenderingPosition] = {} # current -> previous
|
||||
|
||||
# Background rendering
|
||||
self.executor: Optional[ProcessPoolExecutor] = None
|
||||
self.pending_renders: Dict[RenderingPosition, Future] = {}
|
||||
self.render_lock = threading.Lock()
|
||||
|
||||
# Document state
|
||||
self.blocks: Optional[List[Block]] = None
|
||||
self.page_style: Optional[PageStyle] = None
|
||||
@@ -112,10 +89,6 @@ class PageBuffer:
|
||||
self.current_font_scale = font_scale
|
||||
self.current_font_family = font_family
|
||||
|
||||
# Start the process pool
|
||||
if self.executor is None:
|
||||
self.executor = ProcessPoolExecutor(max_workers=self.max_workers)
|
||||
|
||||
def get_page(self, position: RenderingPosition) -> Optional[Page]:
|
||||
"""
|
||||
Get a cached page if available.
|
||||
@@ -176,125 +149,12 @@ class PageBuffer:
|
||||
self.position_map.pop(oldest_pos, None)
|
||||
self.reverse_position_map.pop(oldest_pos, None)
|
||||
|
||||
def start_background_rendering(
|
||||
self,
|
||||
current_position: RenderingPosition,
|
||||
direction: str = 'forward'):
|
||||
"""
|
||||
Start background rendering of upcoming pages.
|
||||
|
||||
Args:
|
||||
current_position: Current reading position
|
||||
direction: 'forward', 'backward', or 'both'
|
||||
"""
|
||||
if not self.blocks or not self.page_style or not self.executor:
|
||||
return
|
||||
|
||||
with self.render_lock:
|
||||
if direction in ['forward', 'both']:
|
||||
self._queue_forward_renders(current_position)
|
||||
|
||||
if direction in ['backward', 'both']:
|
||||
self._queue_backward_renders(current_position)
|
||||
|
||||
def _queue_forward_renders(self, start_position: RenderingPosition):
|
||||
"""Queue forward page renders starting from the given position"""
|
||||
current_pos = start_position
|
||||
|
||||
for i in range(self.buffer_size):
|
||||
# Skip if already cached or being rendered
|
||||
if current_pos in self.forward_buffer or current_pos in self.pending_renders:
|
||||
# Try to get next position from cache
|
||||
current_pos = self.position_map.get(current_pos)
|
||||
if not current_pos:
|
||||
break
|
||||
continue
|
||||
|
||||
# Queue render job
|
||||
args = (
|
||||
self.blocks,
|
||||
self.page_style,
|
||||
current_pos,
|
||||
self.current_font_scale,
|
||||
False,
|
||||
self.current_font_family)
|
||||
future = self.executor.submit(_render_page_worker, args)
|
||||
self.pending_renders[current_pos] = future
|
||||
|
||||
# We don't know the next position yet, so we'll update it when the render
|
||||
# completes
|
||||
break
|
||||
|
||||
def _queue_backward_renders(self, start_position: RenderingPosition):
|
||||
"""Queue backward page renders ending at the given position"""
|
||||
current_pos = start_position
|
||||
|
||||
for i in range(self.buffer_size):
|
||||
# Skip if already cached or being rendered
|
||||
if current_pos in self.backward_buffer or current_pos in self.pending_renders:
|
||||
# Try to get previous position from cache
|
||||
current_pos = self.reverse_position_map.get(current_pos)
|
||||
if not current_pos:
|
||||
break
|
||||
continue
|
||||
|
||||
# Queue render job
|
||||
args = (
|
||||
self.blocks,
|
||||
self.page_style,
|
||||
current_pos,
|
||||
self.current_font_scale,
|
||||
True,
|
||||
self.current_font_family)
|
||||
future = self.executor.submit(_render_page_worker, args)
|
||||
self.pending_renders[current_pos] = future
|
||||
|
||||
# We don't know the previous position yet, so we'll update it when the
|
||||
# render completes
|
||||
break
|
||||
|
||||
def check_completed_renders(self):
|
||||
"""Check for completed background renders and cache the results"""
|
||||
if not self.pending_renders:
|
||||
return
|
||||
|
||||
completed = []
|
||||
|
||||
with self.render_lock:
|
||||
for position, future in self.pending_renders.items():
|
||||
if future.done():
|
||||
try:
|
||||
original_pos, pickled_page, next_pos = future.result()
|
||||
|
||||
# Deserialize the page
|
||||
page = pickle.loads(pickled_page)
|
||||
|
||||
# Cache the page
|
||||
self.cache_page(original_pos, page, next_pos, is_backward=False)
|
||||
|
||||
completed.append(position)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Background render failed for position {position}: {e}")
|
||||
completed.append(position)
|
||||
|
||||
# Remove completed renders
|
||||
for pos in completed:
|
||||
self.pending_renders.pop(pos, None)
|
||||
|
||||
def invalidate_all(self):
|
||||
"""Clear all cached pages and cancel pending renders"""
|
||||
with self.render_lock:
|
||||
# Cancel pending renders
|
||||
for future in self.pending_renders.values():
|
||||
future.cancel()
|
||||
self.pending_renders.clear()
|
||||
|
||||
# Clear caches
|
||||
self.forward_buffer.clear()
|
||||
self.backward_buffer.clear()
|
||||
self.position_map.clear()
|
||||
self.reverse_position_map.clear()
|
||||
"""Clear all cached pages"""
|
||||
self.forward_buffer.clear()
|
||||
self.backward_buffer.clear()
|
||||
self.position_map.clear()
|
||||
self.reverse_position_map.clear()
|
||||
|
||||
def set_font_scale(self, font_scale: float):
|
||||
"""
|
||||
@@ -323,7 +183,6 @@ class PageBuffer:
|
||||
return {
|
||||
'forward_buffer_size': len(self.forward_buffer),
|
||||
'backward_buffer_size': len(self.backward_buffer),
|
||||
'pending_renders': len(self.pending_renders),
|
||||
'position_mappings': len(self.position_map),
|
||||
'reverse_position_mappings': len(self.reverse_position_map),
|
||||
'current_font_scale': self.current_font_scale,
|
||||
@@ -331,28 +190,20 @@ class PageBuffer:
|
||||
}
|
||||
|
||||
def shutdown(self):
|
||||
"""Shutdown the page buffer and clean up resources"""
|
||||
if self.executor:
|
||||
# Cancel pending renders
|
||||
with self.render_lock:
|
||||
for future in self.pending_renders.values():
|
||||
future.cancel()
|
||||
"""
|
||||
Release cached pages.
|
||||
|
||||
# Shutdown executor
|
||||
self.executor.shutdown(wait=True)
|
||||
self.executor = None
|
||||
|
||||
# Clear all caches
|
||||
Cheap and idempotent. There is deliberately no __del__ calling this:
|
||||
blocking work in a finaliser is what deadlocked the interpreter at exit
|
||||
while the process pool existed.
|
||||
"""
|
||||
self.invalidate_all()
|
||||
|
||||
def __del__(self):
|
||||
"""Cleanup on destruction"""
|
||||
self.shutdown()
|
||||
|
||||
|
||||
class BufferedPageRenderer:
|
||||
"""
|
||||
High-level interface for buffered page rendering with automatic background caching.
|
||||
High-level interface for page rendering with an LRU cache in front of the
|
||||
layouter.
|
||||
"""
|
||||
|
||||
def __init__(self,
|
||||
@@ -390,7 +241,7 @@ class BufferedPageRenderer:
|
||||
def render_page(self, position: RenderingPosition,
|
||||
font_scale: float = 1.0) -> Tuple[Page, RenderingPosition]:
|
||||
"""
|
||||
Render a page with intelligent caching.
|
||||
Render a page, serving it from cache when possible.
|
||||
|
||||
Args:
|
||||
position: Position to render from
|
||||
@@ -407,32 +258,18 @@ class BufferedPageRenderer:
|
||||
# Check cache first
|
||||
cached_page = self.buffer.get_page(position)
|
||||
if cached_page:
|
||||
# Get next position from position map
|
||||
# Only use the cache if we also know where the next page starts;
|
||||
# otherwise fall through and compute it.
|
||||
next_pos = self.buffer.position_map.get(position)
|
||||
|
||||
# Only use cache if we have the forward position mapping
|
||||
# Otherwise, we need to compute it
|
||||
if next_pos is not None:
|
||||
# Start background rendering for upcoming pages
|
||||
self.buffer.start_background_rendering(position, 'forward')
|
||||
|
||||
return cached_page, next_pos
|
||||
|
||||
# Cache hit for the page, but we don't have the forward position
|
||||
# Fall through to compute it below
|
||||
|
||||
# Render the page directly
|
||||
page, next_pos = self.layouter.render_page_forward(position, font_scale)
|
||||
|
||||
# Cache the result
|
||||
self.buffer.cache_page(position, page, next_pos)
|
||||
|
||||
# Start background rendering
|
||||
self.buffer.start_background_rendering(position, 'both')
|
||||
|
||||
# Check for completed background renders
|
||||
self.buffer.check_completed_renders()
|
||||
|
||||
return page, next_pos
|
||||
|
||||
def render_page_backward(self,
|
||||
@@ -440,7 +277,8 @@ class BufferedPageRenderer:
|
||||
font_scale: float = 1.0) -> Tuple[Page,
|
||||
RenderingPosition]:
|
||||
"""
|
||||
Render a page ending at the given position with intelligent caching.
|
||||
Render a page ending at the given position, serving it from cache when
|
||||
possible.
|
||||
|
||||
Args:
|
||||
end_position: Position where page should end
|
||||
@@ -457,32 +295,18 @@ class BufferedPageRenderer:
|
||||
# Check cache first
|
||||
cached_page = self.buffer.get_page(end_position)
|
||||
if cached_page:
|
||||
# Get previous position from reverse position map
|
||||
# Only use the cache if we also know where the previous page
|
||||
# starts; otherwise fall through and compute it.
|
||||
prev_pos = self.buffer.reverse_position_map.get(end_position)
|
||||
|
||||
# Only use cache if we have the reverse position mapping
|
||||
# Otherwise, we need to compute it
|
||||
if prev_pos is not None:
|
||||
# Start background rendering for previous pages
|
||||
self.buffer.start_background_rendering(end_position, 'backward')
|
||||
|
||||
return cached_page, prev_pos
|
||||
|
||||
# Cache hit for the page, but we don't have the reverse position
|
||||
# Fall through to compute it below
|
||||
|
||||
# Render the page directly
|
||||
page, start_pos = self.layouter.render_page_backward(end_position, font_scale)
|
||||
|
||||
# Cache the result
|
||||
self.buffer.cache_page(start_pos, page, end_position, is_backward=True)
|
||||
|
||||
# Start background rendering
|
||||
self.buffer.start_background_rendering(end_position, 'both')
|
||||
|
||||
# Check for completed background renders
|
||||
self.buffer.check_completed_renders()
|
||||
|
||||
return page, start_pos
|
||||
|
||||
def set_font_family(self, font_family: Optional[BundledFont]):
|
||||
@@ -516,5 +340,5 @@ class BufferedPageRenderer:
|
||||
return self.buffer.get_cache_stats()
|
||||
|
||||
def shutdown(self):
|
||||
"""Shutdown the renderer and clean up resources"""
|
||||
"""Release cached pages"""
|
||||
self.buffer.shutdown()
|
||||
|
||||
+37
-5
@@ -4,24 +4,56 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "pyWebLayout"
|
||||
version = "0.1.1"
|
||||
description = "A Python library for HTML-like layout and rendering"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.6"
|
||||
requires-python = ">=3.10"
|
||||
license = {file = "LICENSE"}
|
||||
authors = [
|
||||
{name = "Duncan Tourolle", email = "duncan@tourolle.paris"}
|
||||
]
|
||||
dynamic = ["version"]
|
||||
classifiers = [
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Programming Language :: Python :: 3.13",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Operating System :: OS Independent",
|
||||
]
|
||||
dependencies = [
|
||||
"Pillow",
|
||||
"numpy",
|
||||
"pyphen",
|
||||
"beautifulsoup4",
|
||||
"flask",
|
||||
"ebooklib",
|
||||
"requests"
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://gitea.tourolle.paris/pyWebLayout"
|
||||
|
||||
[project.optional-dependencies]
|
||||
# Loading images from http(s) URLs. concrete.image imports requests lazily and
|
||||
# degrades to an error message on the image when it is absent, so it is not a
|
||||
# hard requirement.
|
||||
remote-images = ["requests"]
|
||||
test = [
|
||||
"pytest>=6.0",
|
||||
"pytest-cov",
|
||||
"flask", # fixture HTTP server in tests/abstract/test_abstract_blocks.py
|
||||
"werkzeug", # make_server, same fixture
|
||||
"ebooklib", # builds EPUB fixtures; the reader itself uses zipfile + ElementTree
|
||||
"requests", # exercises the remote-images path
|
||||
]
|
||||
dev = [
|
||||
"pyWebLayout[test,remote-images]",
|
||||
"flake8",
|
||||
"coverage-badge",
|
||||
"interrogate",
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
include = ["pyWebLayout*"]
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["pyWebLayout"]
|
||||
branch = true
|
||||
|
||||
@@ -1,26 +1,6 @@
|
||||
[metadata]
|
||||
name = pyWebLayout
|
||||
version = 0.1.1
|
||||
author = Duncan Tourolle
|
||||
author_email = duncan@tourolle.paris
|
||||
description = A Python library for HTML-like layout and rendering
|
||||
long_description = file: README.md
|
||||
long_description_content_type = text/markdown
|
||||
url = https://gitea.tourolle.paris/pyWebLayout
|
||||
classifiers =
|
||||
Programming Language :: Python :: 3
|
||||
License :: OSI Approved :: MIT License
|
||||
Operating System :: OS Independent
|
||||
|
||||
[options]
|
||||
packages = find:
|
||||
python_requires = >=3.6
|
||||
install_requires =
|
||||
Pillow
|
||||
numpy
|
||||
|
||||
[options.packages.find]
|
||||
include = pyWebLayout*
|
||||
# Packaging metadata lives in pyproject.toml ([project]), which takes
|
||||
# precedence over anything declared here. This file keeps only tool config
|
||||
# that has nowhere better to live.
|
||||
|
||||
[flake8]
|
||||
exclude =
|
||||
|
||||
@@ -1,32 +1,10 @@
|
||||
from setuptools import setup, find_packages
|
||||
"""Shim for legacy `python setup.py` invocations.
|
||||
|
||||
setup(
|
||||
name="pyWebLayout",
|
||||
version="0.1.1",
|
||||
packages=find_packages(),
|
||||
install_requires=[
|
||||
"Pillow",
|
||||
"numpy",
|
||||
],
|
||||
extras_require={
|
||||
"test": [
|
||||
"coverage>=5.0",
|
||||
],
|
||||
"dev": [
|
||||
"coverage>=5.0",
|
||||
"pytest>=6.0",
|
||||
],
|
||||
},
|
||||
author="Duncan Tourolle",
|
||||
author_email="duncan@tourolle.paris",
|
||||
description="A Python library for HTML-like layout and rendering",
|
||||
long_description=open("README.md").read(),
|
||||
long_description_content_type="text/markdown",
|
||||
url="https://gitea.tourolle.paris/pyWebLayout",
|
||||
classifiers=[
|
||||
"Programming Language :: Python :: 3",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Operating System :: OS Independent",
|
||||
],
|
||||
python_requires=">=3.6",
|
||||
)
|
||||
All packaging metadata lives in setup.cfg. Keeping a second copy here was an
|
||||
active hazard: keyword arguments passed to setup() override setup.cfg, so the
|
||||
two could disagree silently and the setup.py copy would win.
|
||||
"""
|
||||
|
||||
from setuptools import setup
|
||||
|
||||
setup()
|
||||
|
||||
@@ -307,6 +307,8 @@ class TestImagePIL(unittest.TestCase):
|
||||
|
||||
if FLASK_AVAILABLE and hasattr(cls, 'flask_thread'):
|
||||
cls.flask_server_running = False
|
||||
cls.flask_server.shutdown()
|
||||
cls.flask_server.server_close()
|
||||
cls.flask_thread.join(timeout=2)
|
||||
|
||||
@classmethod
|
||||
@@ -350,10 +352,9 @@ class TestImagePIL(unittest.TestCase):
|
||||
"""Start a Flask server for URL testing."""
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
from werkzeug.serving import make_server
|
||||
|
||||
cls.flask_app = Flask(__name__)
|
||||
cls.flask_port = 5555 # Use a specific port for testing
|
||||
cls.flask_server_running = True
|
||||
|
||||
@cls.flask_app.route('/test.jpg')
|
||||
def serve_test_image():
|
||||
@@ -363,11 +364,12 @@ class TestImagePIL(unittest.TestCase):
|
||||
def health_check():
|
||||
return 'OK', 200
|
||||
|
||||
def run_flask():
|
||||
cls.flask_app.run(host='127.0.0.1', port=cls.flask_port, debug=False,
|
||||
use_reloader=False, threaded=True)
|
||||
# Bind to an ephemeral port so concurrent/leftover test runs can't clash
|
||||
cls.flask_server = make_server('127.0.0.1', 0, cls.flask_app, threaded=True)
|
||||
cls.flask_port = cls.flask_server.server_port
|
||||
cls.flask_server_running = True
|
||||
|
||||
cls.flask_thread = threading.Thread(target=run_flask, daemon=True)
|
||||
cls.flask_thread = threading.Thread(target=cls.flask_server.serve_forever, daemon=True)
|
||||
cls.flask_thread.start()
|
||||
|
||||
# Wait for server to be ready with health check
|
||||
@@ -379,12 +381,15 @@ class TestImagePIL(unittest.TestCase):
|
||||
try:
|
||||
with urllib.request.urlopen(f'http://127.0.0.1:{cls.flask_port}/health', timeout=1) as response:
|
||||
if response.status == 200:
|
||||
break
|
||||
return
|
||||
except (urllib.error.URLError, ConnectionRefusedError, OSError):
|
||||
pass
|
||||
time.sleep(wait_interval)
|
||||
elapsed += wait_interval
|
||||
|
||||
raise RuntimeError(
|
||||
f"Test Flask server did not become ready on port {cls.flask_port} within {max_wait}s")
|
||||
|
||||
def test_image_url_detection(self):
|
||||
"""Test URL detection functionality."""
|
||||
img = Image()
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
"""
|
||||
Regression tests for backward page navigation (spec S16).
|
||||
|
||||
The previous page of P is the position q for which laying out forward from q ends
|
||||
exactly at P. The old implementation searched for q by guessing a block index and
|
||||
bisecting, with word_index pinned to 0 - so a page starting mid-paragraph was not
|
||||
in the search space at all. It exhausted its ten iterations and fell back to a
|
||||
position that was not the previous page, typically the start of the document.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from pyWebLayout.abstract.block import Paragraph
|
||||
from pyWebLayout.abstract.inline import Word
|
||||
from pyWebLayout.layout.ereader_layout import BidirectionalLayouter, RenderingPosition
|
||||
from pyWebLayout.style import Font
|
||||
from pyWebLayout.style.page_style import PageStyle
|
||||
|
||||
|
||||
PAGE_SIZE = (800, 600)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def font():
|
||||
return Font(font_size=16)
|
||||
|
||||
|
||||
def paragraph(font, count, tag):
|
||||
block = Paragraph(font)
|
||||
for i in range(count):
|
||||
block.add_word(Word(f"{tag}{i}", font))
|
||||
return block
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def long_document(font):
|
||||
"""Short paragraphs around one that spans several pages."""
|
||||
return [
|
||||
paragraph(font, 60, "a"),
|
||||
paragraph(font, 80, "b"),
|
||||
paragraph(font, 1200, "long"),
|
||||
paragraph(font, 70, "c"),
|
||||
paragraph(font, 90, "d"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def block_document(font):
|
||||
"""Many small blocks, so every page starts on a block boundary."""
|
||||
return [paragraph(font, 40, f"p{i}") for i in range(40)]
|
||||
|
||||
|
||||
def forward_chain(layouter, limit=30):
|
||||
"""The page start positions a reader would visit going forward."""
|
||||
starts = []
|
||||
pos = RenderingPosition()
|
||||
for _ in range(limit):
|
||||
starts.append(pos)
|
||||
_, nxt = layouter.render_page_forward(pos, 1.0)
|
||||
if nxt.block_index >= len(layouter.blocks):
|
||||
break
|
||||
if (nxt.block_index, nxt.word_index) == (pos.block_index, pos.word_index):
|
||||
pytest.fail("forward pagination made no progress")
|
||||
pos = nxt
|
||||
return starts
|
||||
|
||||
|
||||
def key(position):
|
||||
return (position.chapter_index, position.block_index, position.word_index)
|
||||
|
||||
|
||||
class TestBackwardMatchesForward:
|
||||
"""The defining invariant: forward from the answer lands exactly on P."""
|
||||
|
||||
@pytest.mark.parametrize("document", ["long_document", "block_document"])
|
||||
def test_previous_page_is_the_forward_predecessor(self, document, request):
|
||||
blocks = request.getfixturevalue(document)
|
||||
layouter = BidirectionalLayouter(blocks, PageStyle(), PAGE_SIZE)
|
||||
starts = forward_chain(layouter)
|
||||
assert len(starts) > 2, "need a few pages to test against"
|
||||
|
||||
for i in range(1, len(starts)):
|
||||
_, got = layouter.render_page_backward(starts[i], 1.0)
|
||||
assert key(got) == key(starts[i - 1]), (
|
||||
f"page {i}: expected to land on page {i - 1} "
|
||||
f"{key(starts[i - 1])}, got {key(got)}")
|
||||
|
||||
def test_result_lays_out_to_the_target(self, long_document):
|
||||
"""Independent of the recorded chain: replaying the answer must reach P."""
|
||||
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||
starts = forward_chain(layouter)
|
||||
|
||||
for target in starts[1:]:
|
||||
_, start = layouter.render_page_backward(target, 1.0)
|
||||
_, end = layouter.render_page_forward(start, 1.0)
|
||||
assert key(end) == key(target), (
|
||||
f"a page starting at {key(start)} ends at {key(end)}, "
|
||||
f"not at the requested {key(target)}")
|
||||
|
||||
def test_mid_paragraph_targets_are_reachable(self, long_document):
|
||||
"""The specific regression: starts inside a block, not on its boundary."""
|
||||
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||
starts = forward_chain(layouter)
|
||||
mid = [s for s in starts if s.word_index > 0]
|
||||
assert mid, "this document should paginate mid-paragraph"
|
||||
|
||||
for target in mid:
|
||||
_, got = layouter.render_page_backward(target, 1.0)
|
||||
assert key(got) != (0, 0, 0) or key(target) == key(starts[1]), \
|
||||
"backward navigation fell back to the document start"
|
||||
|
||||
|
||||
class TestRoundTrip:
|
||||
|
||||
def test_forward_then_back_returns_to_the_same_place(self, long_document):
|
||||
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||
pos = RenderingPosition()
|
||||
|
||||
for _ in range(4):
|
||||
_, nxt = layouter.render_page_forward(pos, 1.0)
|
||||
_, back = layouter.render_page_backward(nxt, 1.0)
|
||||
assert key(back) == key(pos), \
|
||||
f"round trip drifted: {key(pos)} -> {key(nxt)} -> {key(back)}"
|
||||
pos = nxt
|
||||
|
||||
|
||||
class TestEdges:
|
||||
|
||||
def test_at_document_start_stays_there(self, long_document):
|
||||
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||
_, got = layouter.render_page_backward(RenderingPosition(), 1.0)
|
||||
assert key(got) == (0, 0, 0)
|
||||
|
||||
def test_second_page_goes_back_to_the_first(self, long_document):
|
||||
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||
_, second = layouter.render_page_forward(RenderingPosition(), 1.0)
|
||||
_, got = layouter.render_page_backward(second, 1.0)
|
||||
assert key(got) == (0, 0, 0)
|
||||
|
||||
def test_empty_document_is_safe(self):
|
||||
layouter = BidirectionalLayouter([], PageStyle(), PAGE_SIZE)
|
||||
page, got = layouter.render_page_backward(RenderingPosition(), 1.0)
|
||||
assert page is not None
|
||||
assert key(got) == (0, 0, 0)
|
||||
|
||||
|
||||
class TestCost:
|
||||
|
||||
def test_backward_is_not_wildly_more_expensive_than_forward(self, long_document):
|
||||
"""The old path burned ten full layouts per call and still got it wrong."""
|
||||
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||
starts = forward_chain(layouter)
|
||||
|
||||
calls = {"n": 0}
|
||||
original = BidirectionalLayouter.render_page_forward
|
||||
|
||||
def counting(self, position, font_scale=1.0):
|
||||
calls["n"] += 1
|
||||
return original(self, position, font_scale)
|
||||
|
||||
BidirectionalLayouter.render_page_forward = counting
|
||||
try:
|
||||
worst = 0
|
||||
for target in starts[1:]:
|
||||
calls["n"] = 0
|
||||
layouter.render_page_backward(target, 1.0)
|
||||
worst = max(worst, calls["n"])
|
||||
finally:
|
||||
BidirectionalLayouter.render_page_forward = original
|
||||
|
||||
assert worst <= 10, f"backward navigation cost {worst} forward layouts"
|
||||
@@ -0,0 +1,237 @@
|
||||
"""
|
||||
Tests for the page caching layer.
|
||||
|
||||
Covers PageBuffer's LRU behaviour and BufferedPageRenderer's cache hits, plus
|
||||
regressions for S12/R1/R2: the module must not start worker processes and must
|
||||
not do blocking work in a finaliser.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import textwrap
|
||||
|
||||
import pytest
|
||||
|
||||
from pyWebLayout.layout.page_buffer import PageBuffer, BufferedPageRenderer
|
||||
from pyWebLayout.layout.ereader_layout import RenderingPosition
|
||||
from pyWebLayout.abstract.block import Paragraph
|
||||
from pyWebLayout.abstract.inline import Word
|
||||
from pyWebLayout.style import Font
|
||||
from pyWebLayout.style.page_style import PageStyle
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Fixtures
|
||||
# ============================================================================
|
||||
|
||||
@pytest.fixture
|
||||
def sample_blocks():
|
||||
"""A document long enough to paginate over several pages."""
|
||||
font = Font()
|
||||
blocks = []
|
||||
for p in range(6):
|
||||
para = Paragraph(style=font)
|
||||
for w in range(120):
|
||||
para.add_word(Word(f"p{p}w{w}", font))
|
||||
blocks.append(para)
|
||||
return blocks
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def renderer(sample_blocks):
|
||||
return BufferedPageRenderer(sample_blocks, PageStyle(), buffer_size=3, page_size=(800, 600))
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# PageBuffer
|
||||
# ============================================================================
|
||||
|
||||
class TestPageBuffer:
|
||||
def test_get_page_misses_when_empty(self):
|
||||
buf = PageBuffer(buffer_size=3)
|
||||
assert buf.get_page(RenderingPosition()) is None
|
||||
|
||||
def test_cache_page_round_trips(self):
|
||||
buf = PageBuffer(buffer_size=3)
|
||||
pos, nxt = RenderingPosition(block_index=0), RenderingPosition(block_index=1)
|
||||
sentinel = object()
|
||||
|
||||
buf.cache_page(pos, sentinel, nxt)
|
||||
|
||||
assert buf.get_page(pos) is sentinel
|
||||
assert buf.position_map[pos] == nxt
|
||||
|
||||
def test_lru_evicts_oldest_and_cleans_position_map(self):
|
||||
buf = PageBuffer(buffer_size=2)
|
||||
positions = [RenderingPosition(block_index=i) for i in range(4)]
|
||||
for i, pos in enumerate(positions):
|
||||
buf.cache_page(pos, object(), RenderingPosition(block_index=i + 1))
|
||||
|
||||
assert buf.get_page(positions[0]) is None, "oldest should have been evicted"
|
||||
assert positions[0] not in buf.position_map, "position map must not leak evicted entries"
|
||||
assert buf.get_page(positions[-1]) is not None
|
||||
|
||||
def test_get_page_refreshes_lru_order(self):
|
||||
buf = PageBuffer(buffer_size=2)
|
||||
a, b, c = (RenderingPosition(block_index=i) for i in range(3))
|
||||
buf.cache_page(a, object())
|
||||
buf.cache_page(b, object())
|
||||
|
||||
buf.get_page(a) # a becomes most recently used
|
||||
buf.cache_page(c, object())
|
||||
|
||||
assert buf.get_page(a) is not None, "recently used entry should survive"
|
||||
assert buf.get_page(b) is None, "least recently used entry should be evicted"
|
||||
|
||||
def test_backward_pages_land_in_the_backward_buffer(self):
|
||||
buf = PageBuffer(buffer_size=3)
|
||||
start, end = RenderingPosition(block_index=1), RenderingPosition(block_index=2)
|
||||
|
||||
buf.cache_page(start, object(), end, is_backward=True)
|
||||
|
||||
assert start in buf.backward_buffer
|
||||
assert start not in buf.forward_buffer
|
||||
assert buf.reverse_position_map[end] == start
|
||||
|
||||
def test_font_scale_change_invalidates(self):
|
||||
buf = PageBuffer(buffer_size=3)
|
||||
pos = RenderingPosition()
|
||||
buf.cache_page(pos, object(), RenderingPosition(block_index=1))
|
||||
|
||||
buf.set_font_scale(1.5)
|
||||
|
||||
assert buf.get_page(pos) is None
|
||||
|
||||
def test_same_font_scale_keeps_cache(self):
|
||||
buf = PageBuffer(buffer_size=3)
|
||||
pos = RenderingPosition()
|
||||
buf.cache_page(pos, object(), RenderingPosition(block_index=1))
|
||||
|
||||
buf.set_font_scale(1.0)
|
||||
|
||||
assert buf.get_page(pos) is not None
|
||||
|
||||
def test_shutdown_is_idempotent(self):
|
||||
buf = PageBuffer(buffer_size=3)
|
||||
buf.cache_page(RenderingPosition(), object())
|
||||
|
||||
buf.shutdown()
|
||||
buf.shutdown()
|
||||
|
||||
assert buf.get_cache_stats()['forward_buffer_size'] == 0
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# BufferedPageRenderer
|
||||
# ============================================================================
|
||||
|
||||
class TestBufferedPageRenderer:
|
||||
def test_render_page_returns_a_page_and_advances(self, renderer):
|
||||
page, next_pos = renderer.render_page(RenderingPosition(), 1.0)
|
||||
|
||||
assert page is not None
|
||||
assert next_pos != RenderingPosition()
|
||||
|
||||
def test_second_render_of_same_position_is_served_from_cache(self, renderer):
|
||||
pos = RenderingPosition()
|
||||
first, first_next = renderer.render_page(pos, 1.0)
|
||||
second, second_next = renderer.render_page(pos, 1.0)
|
||||
|
||||
assert second is first, "identical page object means it came from the cache"
|
||||
assert second_next == first_next
|
||||
|
||||
def test_font_scale_change_forces_a_re_render(self, renderer):
|
||||
pos = RenderingPosition()
|
||||
first, _ = renderer.render_page(pos, 1.0)
|
||||
scaled, _ = renderer.render_page(pos, 1.5)
|
||||
|
||||
assert scaled is not first
|
||||
|
||||
def test_backward_render_round_trips_to_the_original_position(self, renderer):
|
||||
start = RenderingPosition()
|
||||
_, second_page_pos = renderer.render_page(start, 1.0)
|
||||
|
||||
_, back_to = renderer.render_page_backward(second_page_pos, 1.0)
|
||||
|
||||
assert back_to == start
|
||||
|
||||
def test_shutdown_clears_the_cache(self, renderer):
|
||||
renderer.render_page(RenderingPosition(), 1.0)
|
||||
renderer.shutdown()
|
||||
|
||||
assert renderer.get_cache_stats()['forward_buffer_size'] == 0
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# S12 / R1 / R2 regressions
|
||||
# ============================================================================
|
||||
|
||||
class TestNoBackgroundProcesses:
|
||||
"""
|
||||
The process pool that used to live here never produced a usable page (a Page
|
||||
holds a live PIL canvas and cannot be pickled), and on Python 3.14's
|
||||
forkserver default it raised when driven from module-level code.
|
||||
"""
|
||||
|
||||
def test_module_declares_no_process_pool(self):
|
||||
import pyWebLayout.layout.page_buffer as page_buffer
|
||||
|
||||
source = page_buffer.__file__
|
||||
assert not hasattr(page_buffer, '_render_page_worker')
|
||||
assert not hasattr(PageBuffer(), 'executor')
|
||||
with open(source, encoding='utf-8') as fh:
|
||||
body = fh.read().split('"""', 2)[-1] # skip the module docstring
|
||||
assert 'ProcessPoolExecutor' not in body
|
||||
assert 'pickle' not in body
|
||||
|
||||
def test_page_buffer_has_no_finaliser(self):
|
||||
"""
|
||||
PageBuffer.__del__ called executor.shutdown(wait=True), which deadlocked
|
||||
the interpreter at exit. Cleanup must be explicit.
|
||||
"""
|
||||
assert '__del__' not in vars(PageBuffer)
|
||||
|
||||
def test_navigation_works_without_a_main_guard(self, tmp_path):
|
||||
"""
|
||||
R1: EreaderLayoutManager raised RuntimeError when used from module-level
|
||||
script code, because submitting to a ProcessPoolExecutor under a
|
||||
non-fork start method requires an `if __name__ == "__main__"` guard.
|
||||
"""
|
||||
script = textwrap.dedent(f"""
|
||||
from pyWebLayout.io.readers.html_extraction import parse_html_string
|
||||
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager
|
||||
|
||||
blocks = parse_html_string("<p>" + " ".join(f"w{{i}}" for i in range(2000)) + "</p>")
|
||||
m = EreaderLayoutManager(blocks, page_size=(800, 600),
|
||||
bookmarks_dir={str(tmp_path)!r})
|
||||
m.get_current_page()
|
||||
m.next_page()
|
||||
m.previous_page()
|
||||
m.shutdown()
|
||||
print("OK")
|
||||
""")
|
||||
result = subprocess.run([sys.executable, "-c", script],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "OK" in result.stdout
|
||||
|
||||
def test_interpreter_exits_without_explicit_shutdown(self, tmp_path):
|
||||
"""
|
||||
R2: a manager left to be finalised at exit must not hang. The timeout is
|
||||
the assertion.
|
||||
"""
|
||||
script = textwrap.dedent(f"""
|
||||
from pyWebLayout.io.readers.html_extraction import parse_html_string
|
||||
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager
|
||||
|
||||
blocks = parse_html_string("<p>" + " ".join(f"w{{i}}" for i in range(500)) + "</p>")
|
||||
m = EreaderLayoutManager(blocks, page_size=(800, 600),
|
||||
bookmarks_dir={str(tmp_path)!r})
|
||||
m.get_current_page()
|
||||
# deliberately no shutdown() - rely on interpreter teardown
|
||||
""")
|
||||
result = subprocess.run([sys.executable, "-c", script],
|
||||
capture_output=True, text=True, timeout=60)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
Reference in New Issue
Block a user