Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1924cc234d | ||
|
|
456824d6d6 | ||
|
|
767e4c135c | ||
|
|
7384a32cdd | ||
|
|
4d596ce095 | ||
|
|
737cf0771c |
+20
-8
@@ -24,17 +24,29 @@ jobs:
|
|||||||
with:
|
with:
|
||||||
python-version: ${{ matrix.python-version }}
|
python-version: ${{ matrix.python-version }}
|
||||||
|
|
||||||
|
- name: Verify declared dependencies are sufficient
|
||||||
|
run: |
|
||||||
|
# A clean venv with ONLY the declared runtime deps. If an import here
|
||||||
|
# fails, install_requires is incomplete and a real `pip install
|
||||||
|
# pyWebLayout` would fail the same way for a user.
|
||||||
|
python -m venv /tmp/clean-install
|
||||||
|
/tmp/clean-install/bin/pip install --upgrade pip
|
||||||
|
/tmp/clean-install/bin/pip install .
|
||||||
|
/tmp/clean-install/bin/python -c "
|
||||||
|
import pyWebLayout.concrete, pyWebLayout.abstract
|
||||||
|
import pyWebLayout.io.readers.epub_reader
|
||||||
|
import pyWebLayout.io.readers.html_extraction
|
||||||
|
import pyWebLayout.layout.ereader_manager
|
||||||
|
print('clean install imports OK')
|
||||||
|
"
|
||||||
|
|
||||||
- name: Install dependencies
|
- name: Install dependencies
|
||||||
run: |
|
run: |
|
||||||
python -m pip install --upgrade pip
|
python -m pip install --upgrade pip
|
||||||
# Install package in development mode
|
# Install package in development mode, with the declared dev extra.
|
||||||
pip install -e .
|
# Test dependencies belong in setup.cfg, not in an ad-hoc pip line.
|
||||||
# Install test dependencies if they exist
|
pip install -e '.[dev]'
|
||||||
if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt; fi
|
|
||||||
if [ -f requirements/test.txt ]; then pip install -r requirements/test.txt; fi
|
|
||||||
# Install common test packages
|
|
||||||
pip install pytest pytest-cov flake8 coverage-badge interrogate
|
|
||||||
|
|
||||||
- name: Download initial failed badges
|
- name: Download initial failed badges
|
||||||
run: |
|
run: |
|
||||||
echo "Downloading initial failed badges..."
|
echo "Downloading initial failed badges..."
|
||||||
|
|||||||
+303
-188
@@ -1,233 +1,348 @@
|
|||||||
# pyWebLayout Architecture: Abstract vs Concrete
|
# pyWebLayout Architecture
|
||||||
|
|
||||||
This document explains the fundamental architectural separation between **Abstract** and **Concrete** layers in the pyWebLayout library.
|
This document describes how pyWebLayout is organised: the layers, what lives in each,
|
||||||
|
and the rules that govern how they may depend on one another.
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
The pyWebLayout library follows a clear separation between two distinct layers:
|
The library turns markup (HTML/EPUB) into rendered images. That pipeline is split into
|
||||||
|
layers with a strict dependency direction:
|
||||||
|
|
||||||
- **Abstract Layer**: Represents the logical structure and content of documents (HTML/EPUB text)
|
```
|
||||||
- **Concrete Layer**: Handles the spatial rendering and visual representation of content
|
io/readers/ parse markup into document structure
|
||||||
|
↓
|
||||||
|
abstract/ what the document says ─┐
|
||||||
|
↓ ├─ built on core/ + style/
|
||||||
|
layout/ decide where things go │
|
||||||
|
↓ │
|
||||||
|
concrete/ what the pixels are ─┘
|
||||||
|
↓
|
||||||
|
PIL.Image
|
||||||
|
```
|
||||||
|
|
||||||
This separation provides flexibility, testability, and clean separation of concerns.
|
The central distinction is **abstract vs concrete**:
|
||||||
|
|
||||||
## Abstract Layer (`pyWebLayout/abstract/`)
|
- **Abstract** — the logical content and structure of a document. A `Paragraph` knows
|
||||||
|
it contains words in an order; it does not know how wide they are.
|
||||||
|
- **Concrete** — the spatial realisation of that content. A `Line` knows exactly which
|
||||||
|
glyphs sit at which pixel offsets on a specific canvas.
|
||||||
|
|
||||||
The Abstract layer deals with the **logical structure** of documents without concerning itself with how content will be visually rendered.
|
One abstract document produces many concrete renderings: different page sizes, font
|
||||||
|
scales, and font families all re-run the concrete layer over unchanged abstract content.
|
||||||
|
That is what makes ereader features like live font scaling and reflow possible.
|
||||||
|
|
||||||
### Key Components
|
## Layers
|
||||||
|
|
||||||
#### `abstract/block.py`
|
### `core/` — shared foundations
|
||||||
- `Block`: Base class for all block-level content
|
|
||||||
- `Paragraph`: Represents a logical paragraph containing words
|
|
||||||
- `Heading`: Represents headings with semantic levels (H1-H6)
|
|
||||||
- `HList`: Represents ordered/unordered lists
|
|
||||||
- `Image`: Represents image references
|
|
||||||
|
|
||||||
#### `abstract/inline.py`
|
Everything else is built on these. `core/` depends on nothing but `style/`.
|
||||||
- `Word`: Represents individual words with text content and styling information
|
|
||||||
- Contains methods for hyphenation and text manipulation
|
|
||||||
- Does **not** handle rendering or spatial layout
|
|
||||||
|
|
||||||
#### `abstract/document.py`
|
**`core/base.py`** defines the contracts that make a class abstract or concrete:
|
||||||
- `Document`: Container for the overall document structure
|
|
||||||
- `Chapter`: Logical grouping of blocks (for books/long documents)
|
|
||||||
|
|
||||||
### Characteristics of Abstract Classes
|
| Contract | Kind | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `Renderable` | ABC | Has `render()`; produces or draws visual output |
|
||||||
|
| `Queriable` | ABC | Has `in_object(point)`; can be hit-tested |
|
||||||
|
| `Layoutable` | ABC | Has `layout()`; arranges its own contents |
|
||||||
|
| `Interactable` | ABC | Holds a callback invoked on interaction |
|
||||||
|
| `Geometric` | mixin | `origin` and `size` as numpy arrays |
|
||||||
|
| `Hierarchical` | mixin | `parent` back-reference |
|
||||||
|
| `Styleable` | mixin | Carries a style object |
|
||||||
|
| `FontRegistry` | mixin | Deduplicates `Font` instances across a document |
|
||||||
|
| `MetadataContainer` | mixin | Key/value metadata with typed accessors |
|
||||||
|
| `BlockContainer` | mixin | Holds child `Block`s |
|
||||||
|
| `ContainerAware` | mixin | Knows the container it was added to |
|
||||||
|
|
||||||
1. **Content-focused**: Store text, structure, and semantic meaning
|
The useful shorthand: **`Renderable` and `Geometric` are the concrete markers.** An
|
||||||
2. **Layout-agnostic**: No knowledge of fonts, pixels, or rendering
|
abstract class that acquires either has crossed the line.
|
||||||
3. **Reusable**: Same content can be rendered in different formats/sizes
|
|
||||||
4. **Serializable**: Can be saved/loaded without rendering context
|
|
||||||
|
|
||||||
### Example: Abstract Word
|
**`core/query.py`** — `QueryResult` and `SelectionRange`: the result types for mapping a
|
||||||
|
pixel back to content (what was clicked, what text is selected, where it is in the
|
||||||
|
document).
|
||||||
|
|
||||||
|
**`core/highlight.py`** — `Highlight`, `HighlightColor`, `HighlightManager`. Highlights
|
||||||
|
store both pixel bounds (for drawing) and semantic bounds (word indices, for surviving
|
||||||
|
a font change).
|
||||||
|
|
||||||
|
**`core/cache.py`** — `UsageCache` / `SizedUsageCache`, bounded caches keyed by
|
||||||
|
`(font, string)` for text measurement and glyph rasterisation. Eviction is by usage
|
||||||
|
count with periodic aging, not LRU; the module docstring explains why at length. This
|
||||||
|
exists because a single page issues thousands of measurements over fewer than a
|
||||||
|
thousand distinct pairs, on hardware where an unbounded cache is not affordable.
|
||||||
|
|
||||||
|
**`core/callback_registry.py`** — `CallbackRegistry`, which owns the interactive
|
||||||
|
elements registered on a page and dispatches to them.
|
||||||
|
|
||||||
|
### `style/` — semantic styling and its resolution
|
||||||
|
|
||||||
|
This package is itself an instance of the abstract/concrete split, applied to styling:
|
||||||
|
|
||||||
|
- **`abstract_style.py`** — `AbstractStyle` (frozen dataclass, hashable) captures
|
||||||
|
*intent*: `FontFamily.SERIF`, `FontSize.LARGE`, `color="black"`. `AbstractStyleRegistry`
|
||||||
|
interns them so a document holds one instance per distinct style.
|
||||||
|
- **`concrete_style.py`** — `RenderingContext` (user preferences, DPI, accessibility
|
||||||
|
flags, available space) plus `StyleResolver`, which maps `AbstractStyle` +
|
||||||
|
`RenderingContext` → `ConcreteStyle` (a resolved font path, a pixel size, an RGB
|
||||||
|
tuple). `ConcreteStyleRegistry` caches the results.
|
||||||
|
- **`fonts.py`** — `Font`, the loaded PIL font plus its rendering attributes, and the
|
||||||
|
bundled DejaVu families (`BundledFont`).
|
||||||
|
- **`page_style.py`** — `PageStyle`: borders, padding, background, default alignment.
|
||||||
|
- **`alignment.py`** — the `Alignment` enum.
|
||||||
|
|
||||||
|
`AbstractStyle` → `StyleResolver` → `ConcreteStyle` is the mechanism by which the same
|
||||||
|
document renders differently for different readers.
|
||||||
|
|
||||||
|
### `abstract/` — document content and structure
|
||||||
|
|
||||||
|
Layout-agnostic representations of what the document contains.
|
||||||
|
|
||||||
|
**`abstract/block.py`** — block-level content, all deriving from `Block`:
|
||||||
|
|
||||||
|
`Paragraph`, `Heading` (with `HeadingLevel`), `Quote`, `CodeBlock`, `HList` (with
|
||||||
|
`ListStyle`) and `ListItem`, `Table` / `TableRow` / `TableCell`, `Image` and
|
||||||
|
`LinkedImage`, `HorizontalRule`, `PageBreak`.
|
||||||
|
|
||||||
|
**`abstract/inline.py`** — `Word`, `FormattedSpan`, `LinkedWord`, `LineBreak`. `Word`
|
||||||
|
holds text, a style, and previous/next links into the document's word sequence. It can
|
||||||
|
report whether it is a hyphenation candidate (`possible_hyphenation(language)`), but it
|
||||||
|
does not perform the split — that is a measurement decision and belongs to `Line`.
|
||||||
|
|
||||||
|
**`abstract/document.py`** — `Document`, `Chapter`, `Book`, `MetadataType`. `Book` is
|
||||||
|
the EPUB-shaped `Document` with chapters and a table of contents.
|
||||||
|
|
||||||
|
**`abstract/functional.py`** — `Link`, `Button`, `Form`, `FormField`, `LinkType`,
|
||||||
|
`FormFieldType`.
|
||||||
|
|
||||||
|
**`abstract/interactive_image.py`** — `InteractiveImage`, an `Image` that is also
|
||||||
|
`Interactable` and `Queriable`.
|
||||||
|
|
||||||
|
### `concrete/` — spatial realisation
|
||||||
|
|
||||||
|
Objects that know their position and size and can draw themselves onto a canvas.
|
||||||
|
|
||||||
|
**`concrete/text.py`** — the heart of text layout:
|
||||||
|
- `Text` — one renderable fragment (a whole word, or a hyphenated part). Requires an
|
||||||
|
`ImageDraw.Draw` at construction so it can measure itself immediately.
|
||||||
|
- `Line` — a sequence of `Text` objects with resolved spacing and a baseline.
|
||||||
|
`Line.add_word()` is where a `Word` becomes one or two `Text` objects: it measures,
|
||||||
|
and if the word overflows it tries dictionary hyphenation, then brute-force splitting,
|
||||||
|
before rejecting the word.
|
||||||
|
- `AlignmentHandler` and its subclasses (`LeftAlignmentHandler`,
|
||||||
|
`CenterRightAlignmentHandler`, `JustifyAlignmentHandler`) — the strategy objects that
|
||||||
|
turn a set of measured fragments plus an available width into concrete spacing and a
|
||||||
|
start position.
|
||||||
|
- Cache management: `configure_text_caches`, `clear_text_caches`, `text_cache_stats`,
|
||||||
|
`prewarm_text_caches`.
|
||||||
|
|
||||||
|
**`concrete/page.py`** — `Page`: a fixed-size canvas holding `Renderable` children, with
|
||||||
|
a content rectangle derived from its `PageStyle`, `can_fit_line()` for the layouter to
|
||||||
|
test against, `render()` returning a `PIL.Image`, and `query_point()` / `query_range()`
|
||||||
|
for hit-testing.
|
||||||
|
|
||||||
|
**`concrete/box.py`** — `Box`, the `Geometric` + `Renderable` + `Queriable` base for
|
||||||
|
positioned drawable objects.
|
||||||
|
|
||||||
|
**`concrete/dynamic_page.py`** — `DynamicPage` (a `Page` subclass) and `SizeConstraints`.
|
||||||
|
Adds a two-phase measure-then-layout protocol, so a container such as a table can learn
|
||||||
|
its content's intrinsic size before committing to a size allocation.
|
||||||
|
|
||||||
|
**`concrete/image.py`** — `RenderableImage`.
|
||||||
|
|
||||||
|
**`concrete/table.py`** — `TableRenderer`, `TableRowRenderer`, `TableCellRenderer`,
|
||||||
|
`TableStyle`. Cells host their own nested `Page`, which is why `Page` accepts a
|
||||||
|
non-zero `origin`.
|
||||||
|
|
||||||
|
**`concrete/functional.py`** — `LinkText`, `ButtonText`, `FormFieldText`: `Text`
|
||||||
|
subclasses that are also `Interactable`.
|
||||||
|
|
||||||
|
**`concrete/interaction_handler.py`** — `InteractionHandler`, `InteractionStateManager`:
|
||||||
|
routing taps and presses to registered elements and tracking pressed/released state.
|
||||||
|
|
||||||
|
### `layout/` — the abstract → concrete transformation
|
||||||
|
|
||||||
|
This is the package the pipeline diagram calls the layout engine.
|
||||||
|
|
||||||
|
**`layout/document_layouter.py`** — a set of layouter functions, one per content kind,
|
||||||
|
all sharing a signature shape of *(abstract element, target `Page`) → did it fit*:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# An Abstract Word knows its text content and semantic properties
|
paragraph_layouter(paragraph, page, start_word=0, pretext=None, alignment_override=None)
|
||||||
word = Word("supercalifragilisticexpialidocious", font_style)
|
-> (complete: bool, failed_word_index: int | None, remaining_pretext: Text | None)
|
||||||
word.hyphenate() # Logical operation - finds break points
|
|
||||||
parts = word.get_hyphenated_parts() # Returns ["super-", "cali-", "fragi-", ...]
|
image_layouter(image, page, max_width=None, max_height=None) -> bool
|
||||||
|
table_layouter(table, page, style=None) -> bool
|
||||||
|
pagebreak_layouter(page_break, page) -> bool
|
||||||
|
button_layouter(button, page, font=None, padding=...) -> (bool, str)
|
||||||
|
form_field_layouter(field, page, font=None, ...) -> ...
|
||||||
|
form_layouter(form, page, font=None, field_spacing=10) -> (bool, list[str])
|
||||||
```
|
```
|
||||||
|
|
||||||
## Concrete Layer (`pyWebLayout/concrete/`)
|
They **append to a page**, they do not return a list of lines. The three-part return of
|
||||||
|
`paragraph_layouter` is what makes pagination resumable: when a paragraph runs off the
|
||||||
|
bottom of a page, the caller learns which word failed and whether a hyphenated fragment
|
||||||
|
is pending, and can continue on the next page from exactly there.
|
||||||
|
|
||||||
The Concrete layer handles the **spatial representation** and actual rendering of content.
|
`DocumentLayouter` wraps a `Page` and dispatches over a list of abstract elements by
|
||||||
|
type, holding the `ConcreteStyleRegistry` for the run.
|
||||||
|
|
||||||
### Key Components
|
**`layout/ereader_layout.py`** — the paginated reading model:
|
||||||
|
- `RenderingPosition` — a serialisable cursor expressed in *abstract* coordinates
|
||||||
|
(chapter, block, word, table cell, list item, pending pretext). Because it names
|
||||||
|
document structure rather than pixels, it survives font-size and page-size changes.
|
||||||
|
- `BidirectionalLayouter` — renders a page forward or backward from a position,
|
||||||
|
returning `(Page, next_position)`.
|
||||||
|
- `ChapterNavigator` / `ChapterInfo` — a table of contents built from heading structure.
|
||||||
|
- `FontScaler`, `FontFamilyOverride` — apply scale and family changes to blocks at
|
||||||
|
layout time without mutating the abstract document.
|
||||||
|
|
||||||
#### `concrete/text.py`
|
**`layout/page_buffer.py`** — `PageBuffer` (LRU caches of rendered pages in both
|
||||||
- `Text`: Renders a specific text fragment with precise positioning
|
directions, plus position maps) and `BufferedPageRenderer` (background rendering).
|
||||||
- `Line`: Manages a line of `Text` objects with spacing and alignment
|
|
||||||
- Handles actual pixel measurements, font rendering, and positioning
|
|
||||||
|
|
||||||
#### `concrete/page.py`
|
**`layout/ereader_manager.py`** — `EreaderLayoutManager`, the top-level application
|
||||||
- `Page`: Top-level container for rendered content
|
interface (page turns, font changes, chapter jumps, progress), and `BookmarkManager`
|
||||||
- `Container`: Layout manager for organizing renderable objects
|
for persisting bookmarks and the last reading position.
|
||||||
- Handles spatial layout, pagination, and visual composition
|
|
||||||
|
|
||||||
#### `concrete/box.py`
|
**`layout/table_optimizer.py`** — column width allocation for tables.
|
||||||
- `Box`: Base class for all spatially-aware renderable objects
|
|
||||||
- Provides positioning, sizing, and rendering capabilities
|
|
||||||
|
|
||||||
### Characteristics of Concrete Classes
|
### `io/readers/` — parsing
|
||||||
|
|
||||||
1. **Rendering-focused**: Handle pixels, fonts, images, and visual output
|
**`html_extraction.py`** — `parse_html_string(html, base_font=None, document=None,
|
||||||
2. **Spatially-aware**: Know exact positions, sizes, and layout constraints
|
base_path=None) -> List[Block]`. Built from a `StyleContext` (a `NamedTuple` threaded
|
||||||
3. **Implementation-specific**: Tied to specific rendering technologies (PIL, etc.)
|
down the tree, carrying the inherited font and styling) and a table of per-tag handlers
|
||||||
4. **Non-portable**: Rendering results are tied to specific display contexts
|
(`paragraph_handler`, `heading_handler`, `table_handler`, …). This is the only place
|
||||||
|
that knows about HTML.
|
||||||
|
|
||||||
### Example: Concrete Text
|
**`epub_reader.py`** — `EPUBReader` and `read_epub(path) -> Book`: container/OPF
|
||||||
|
parsing, spine and manifest, table of contents, cover handling, and image processing
|
||||||
|
(with an e-ink processor available by default).
|
||||||
|
|
||||||
|
## Dependency rules
|
||||||
|
|
||||||
|
The direction of dependency is what keeps the split honest:
|
||||||
|
|
||||||
|
```
|
||||||
|
io/readers → abstract → core, style
|
||||||
|
layout → abstract, concrete, core, style
|
||||||
|
concrete → abstract, core, style
|
||||||
|
abstract → core, style ← must NOT import concrete
|
||||||
|
```
|
||||||
|
|
||||||
|
`abstract/` importing from `concrete/` is the violation to watch for. `concrete/`
|
||||||
|
importing from `abstract/` is expected and correct: a `Text` may point back at the
|
||||||
|
`Word` it came from, and a table cell renderer reads its abstract `TableCell`.
|
||||||
|
|
||||||
|
**Where the abstract layer touches rendering today, and why:**
|
||||||
|
|
||||||
|
- Abstract classes are constructed with `Font` objects, which carry a pixel size and a
|
||||||
|
loaded font file. This is a deliberate compromise for parsing performance (HTML
|
||||||
|
styling resolves to a `Font` once, at parse time) but it does mean the abstract layer
|
||||||
|
is not fully rendering-independent. `AbstractStyle` is the intended replacement, and
|
||||||
|
`Word` already accepts either.
|
||||||
|
- `Word.concrete` / `Word.add_concete()` ([inline.py:42](pyWebLayout/abstract/inline.py#L42),
|
||||||
|
[:134](pyWebLayout/abstract/inline.py#L134)) is a back-reference to the `Text` objects
|
||||||
|
a word became. **It is currently written and never read**, and it cannot correctly
|
||||||
|
model the relationship anyway: one `Word` becomes many `Text`s across re-layouts, and
|
||||||
|
a single slot only remembers the most recent. Treat it as vestigial. When a
|
||||||
|
word→pixels mapping is genuinely needed, build it on `Text`'s `source` back-reference
|
||||||
|
(concrete pointing at abstract), which is the safe direction — note it is currently
|
||||||
|
stored as `_source` with no public accessor, and is itself unread today.
|
||||||
|
|
||||||
|
## Worked example
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# A Concrete Text object handles actual rendering
|
from pyWebLayout.io.readers.html_extraction import parse_html_string
|
||||||
text = Text("super-", font) # Specific text fragment
|
from pyWebLayout.concrete.page import Page
|
||||||
text._calculate_dimensions() # Computes exact pixel size
|
from pyWebLayout.layout.document_layouter import DocumentLayouter
|
||||||
image = text.render() # Produces actual visual output
|
|
||||||
|
# 1. Parse: markup -> abstract blocks
|
||||||
|
blocks = parse_html_string("<h1>Chapter One</h1><p>It was a dark and stormy night.</p>")
|
||||||
|
# [Heading, Paragraph]
|
||||||
|
|
||||||
|
# 2. Lay out: abstract blocks -> concrete children appended to a Page
|
||||||
|
page = Page(size=(400, 600))
|
||||||
|
DocumentLayouter(page).layout_document(blocks)
|
||||||
|
|
||||||
|
# 3. Render: Page -> PIL.Image
|
||||||
|
image = page.render()
|
||||||
|
|
||||||
|
# 4. Query: pixel -> content
|
||||||
|
result = page.query_point((50, 40))
|
||||||
|
print(result.object_type, result.text) # e.g. "text" "Chapter"
|
||||||
```
|
```
|
||||||
|
|
||||||
## The Transformation Process
|
Inspecting the intermediate concrete objects:
|
||||||
|
|
||||||
The architecture involves a clear transformation from Abstract to Concrete:
|
|
||||||
|
|
||||||
```
|
|
||||||
Abstract Document
|
|
||||||
↓
|
|
||||||
[Parser Layer]
|
|
||||||
↓
|
|
||||||
Abstract Blocks (Paragraph, Heading, etc.)
|
|
||||||
↓
|
|
||||||
[Layout Engine]
|
|
||||||
↓
|
|
||||||
Concrete Objects (Text, Line, Page)
|
|
||||||
↓
|
|
||||||
[Rendering Engine]
|
|
||||||
↓
|
|
||||||
Visual Output (Images, PDF, etc.)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Example Transformation
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# 1. Abstract content
|
from pyWebLayout.concrete.text import Line
|
||||||
paragraph = Paragraph()
|
|
||||||
paragraph.add_word(Word("This", font))
|
|
||||||
paragraph.add_word(Word("is", font))
|
|
||||||
paragraph.add_word(Word("a", font))
|
|
||||||
paragraph.add_word(Word("test", font))
|
|
||||||
|
|
||||||
# 2. Layout transformation
|
for line in (c for c in page.children if isinstance(c, Line)):
|
||||||
layout = ParagraphLayout(line_width=200, line_height=20)
|
print([t.text for t in line.text_objects])
|
||||||
lines = layout.layout_paragraph(paragraph) # Returns List[Line]
|
# ['Chapter', 'One']
|
||||||
|
# ['It', 'was', 'a', 'dark', 'and', 'stormy', 'night.']
|
||||||
# 3. Each Line contains concrete Text objects
|
|
||||||
for line in lines:
|
|
||||||
for text_obj in line.text_objects: # List[Text]
|
|
||||||
print(f"Text: '{text_obj.text}' at position {text_obj._origin}")
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Key Architectural Principles
|
For paginated reading, drive `EreaderLayoutManager` instead of building pages by hand:
|
||||||
|
|
||||||
### 1. **Single Responsibility**
|
|
||||||
- Abstract classes: Handle content and structure
|
|
||||||
- Concrete classes: Handle rendering and layout
|
|
||||||
|
|
||||||
### 2. **Separation of Concerns**
|
|
||||||
- Text parsing/processing ≠ Text rendering
|
|
||||||
- Document structure ≠ Page layout
|
|
||||||
- Content semantics ≠ Visual presentation
|
|
||||||
|
|
||||||
### 3. **Immutable Abstract Content**
|
|
||||||
- Abstract content remains unchanged during rendering
|
|
||||||
- Multiple concrete representations can be generated from same abstract content
|
|
||||||
- Enables pagination, different formats, responsive layouts
|
|
||||||
|
|
||||||
### 4. **One-to-Many Relationships**
|
|
||||||
- One Abstract Word → Multiple Concrete Text objects (hyphenation)
|
|
||||||
- One Abstract Paragraph → Multiple Concrete Lines
|
|
||||||
- One Abstract Document → Multiple Concrete Pages
|
|
||||||
|
|
||||||
## Common Anti-Patterns to Avoid
|
|
||||||
|
|
||||||
### ❌ **Mixing Concerns**
|
|
||||||
```python
|
```python
|
||||||
# WRONG: Abstract class knowing about pixels
|
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager
|
||||||
|
|
||||||
|
manager = EreaderLayoutManager(blocks, page_size=(800, 600), document_id="my-book")
|
||||||
|
page = manager.get_current_page()
|
||||||
|
page = manager.next_page()
|
||||||
|
manager.set_font_scale(1.25) # re-lays out from the same RenderingPosition
|
||||||
|
```
|
||||||
|
|
||||||
|
## Design principles
|
||||||
|
|
||||||
|
**1. Abstract content is not mutated by layout.** Font scaling and family overrides
|
||||||
|
produce scaled copies at layout time rather than editing the document. This is what
|
||||||
|
allows the same `blocks` list to back a buffer of pages at several font sizes.
|
||||||
|
|
||||||
|
**2. Positions are expressed in abstract coordinates.** `RenderingPosition` names a
|
||||||
|
chapter, block, and word — never a pixel. A reader who changes font size stays on the
|
||||||
|
same sentence.
|
||||||
|
|
||||||
|
**3. Layouters report partial success.** Every layouter returns whether it fit, and the
|
||||||
|
text layouter also returns where it stopped. Pagination is built from this rather than
|
||||||
|
from a separate page-breaking pass.
|
||||||
|
|
||||||
|
**4. One abstract object may become many concrete ones.** One `Word` → one or two `Text`
|
||||||
|
fragments; one `Paragraph` → many `Line`s across many `Page`s; one `Document` → an
|
||||||
|
unbounded sequence of `Page`s. Any API that assumes a one-to-one mapping will be wrong
|
||||||
|
at a hyphen or a page boundary.
|
||||||
|
|
||||||
|
**5. Measurement is cached, not avoided.** Text width and glyph rasterisation are the
|
||||||
|
hot path. `core/cache.py` bounds their cost; `prewarm_text_caches` front-loads it for a
|
||||||
|
known document.
|
||||||
|
|
||||||
|
## Anti-patterns
|
||||||
|
|
||||||
|
**Concrete state stored on abstract objects.** An abstract object that remembers its
|
||||||
|
rendered width, position, or `Text` objects is wrong at the second rendering. If a
|
||||||
|
back-reference is needed, point from concrete to abstract.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# WRONG
|
||||||
class Word:
|
class Word:
|
||||||
def __init__(self, text):
|
def __init__(self, text):
|
||||||
self.text = text
|
self.rendered_width = None # invalidated by any font change
|
||||||
self.rendered_width = None # ❌ Concrete concern in abstract class
|
|
||||||
|
# RIGHT
|
||||||
|
text = Text(word.text, font, draw, source=word) # concrete knows its origin
|
||||||
```
|
```
|
||||||
|
|
||||||
### ❌ **renderable_words Concept**
|
**Treating `Word` as renderable.** Words are abstract; only `Text` draws. There is no
|
||||||
```python
|
`renderable_words` anywhere in the codebase, and there should not be.
|
||||||
# WRONG: Confusing abstract and concrete
|
|
||||||
line.renderable_words # ❌ This suggests Words are renderable
|
|
||||||
# Words are abstract - only Text objects render
|
|
||||||
```
|
|
||||||
|
|
||||||
### ✅ **Correct Separation**
|
**Assuming a layouter returns lines.** `paragraph_layouter` appends to a page and
|
||||||
```python
|
reports what did not fit. Code that expects `List[Line]` back is working from an
|
||||||
# CORRECT: Clear separation
|
outdated model.
|
||||||
abstract_word = Word("test") # Abstract content
|
|
||||||
concrete_text = Text("test", font) # Concrete rendering
|
|
||||||
line.text_objects.append(concrete_text) # Concrete objects in concrete container
|
|
||||||
```
|
|
||||||
|
|
||||||
## Benefits of This Architecture
|
## Summary
|
||||||
|
|
||||||
### 1. **Flexibility**
|
- **`core/`** — contracts and shared machinery
|
||||||
- Same content can be rendered at different sizes
|
- **`style/`** — semantic style, and its resolution to concrete rendering parameters
|
||||||
- Multiple output formats from single source
|
- **`abstract/`** — what the document says
|
||||||
- Easy to implement responsive design
|
- **`concrete/`** — where the pixels go
|
||||||
|
- **`layout/`** — the transformation between them, and pagination on top of it
|
||||||
### 2. **Testability**
|
- **`io/readers/`** — markup in
|
||||||
- Abstract logic can be tested without rendering
|
|
||||||
- Layout algorithms can be tested independently
|
|
||||||
- Visual rendering can be mocked
|
|
||||||
|
|
||||||
### 3. **Performance**
|
|
||||||
- Abstract content can be cached and reused
|
|
||||||
- Layout can be computed once for multiple renderings
|
|
||||||
- Incremental updates possible
|
|
||||||
|
|
||||||
### 4. **Maintainability**
|
|
||||||
- Clear boundaries between text processing and rendering
|
|
||||||
- Changes to rendering don't affect content parsing
|
|
||||||
- Easy to swap rendering backends
|
|
||||||
|
|
||||||
## File Organization
|
|
||||||
|
|
||||||
```
|
|
||||||
pyWebLayout/
|
|
||||||
├── abstract/ # Content and structure
|
|
||||||
│ ├── block.py # Document blocks (Paragraph, Heading, etc.)
|
|
||||||
│ ├── inline.py # Inline content (Word, etc.)
|
|
||||||
│ ├── document.py # Document structure
|
|
||||||
│ └── functional.py # Links, buttons, etc.
|
|
||||||
│
|
|
||||||
├── concrete/ # Rendering and layout
|
|
||||||
│ ├── text.py # Text and Line rendering
|
|
||||||
│ ├── page.py # Page layout and containers
|
|
||||||
│ ├── box.py # Base rendering classes
|
|
||||||
│ ├── image.py # Image rendering
|
|
||||||
│ └── functional.py # Interactive elements
|
|
||||||
│
|
|
||||||
├── typesetting/ # Layout algorithms
|
|
||||||
│ ├── paragraph_layout.py # Abstract → Concrete transformation
|
|
||||||
│ ├── flow.py # Text flow management
|
|
||||||
│ └── pagination.py # Page breaking logic
|
|
||||||
│
|
|
||||||
└── style/ # Styling and formatting
|
|
||||||
├── fonts.py # Font management
|
|
||||||
├── layout.py # Layout constants
|
|
||||||
└── alignment.py # Alignment enums
|
|
||||||
```
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
The Abstract/Concrete separation is fundamental to pyWebLayout's design. It ensures clean separation between content processing and visual rendering, enabling flexible, maintainable, and testable document processing pipelines.
|
|
||||||
|
|
||||||
**Remember**:
|
|
||||||
- **Abstract** = What to display (content, structure, semantics)
|
|
||||||
- **Concrete** = How to display it (pixels, fonts, positioning, rendering)
|
|
||||||
|
|
||||||
This architecture enables the library to handle complex document layouts while maintaining clear, understandable code organization.
|
|
||||||
|
|||||||
@@ -0,0 +1,527 @@
|
|||||||
|
# Architecture Review
|
||||||
|
|
||||||
|
Independent review of the codebase at `c5c61a3` (2026-08-06), answering one
|
||||||
|
question: **is this a well-architected library or an over-complex mess?**
|
||||||
|
|
||||||
|
It is a well-architected library with one rotten subsystem inside it. The core
|
||||||
|
design holds up; roughly a fifth of the code is speculative or non-functional,
|
||||||
|
and it is concentrated in the ereader pagination/buffering layer.
|
||||||
|
|
||||||
|
This document records the verdict, the evidence, and the findings **not already
|
||||||
|
covered** by [LAYOUT_REMEDIATION_SPEC.md](LAYOUT_REMEDIATION_SPEC.md). Where a
|
||||||
|
finding is already specced, it is cross-referenced rather than restated.
|
||||||
|
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
| ID | Finding | Severity | Status |
|
||||||
|
|----|---------|----------|--------|
|
||||||
|
| [R1](#r1--the-process-pool-crashes-on-python-314) | The process pool crashes on Python 3.14 | Critical | New; raises priority of S12 |
|
||||||
|
| [R2](#r2--the-test-suite-hangs-at-interpreter-exit) | Test suite hangs at interpreter exit | High | New; same root cause as R1 |
|
||||||
|
| [R3](#r3--font-scaling-destroys-hyperlinks) | Font scaling destroys hyperlinks | High | New |
|
||||||
|
| [R4](#r4--three-packaging-configs-that-disagree) | Three packaging configs that disagree | Medium | New (corrected) |
|
||||||
|
| [R5](#r5--monkey-patched-page-methods-with-a-conflicting-signature) | Monkey-patched `Page` methods with a conflicting signature | Medium | New |
|
||||||
|
| [R6](#r6--dead-duck-typing-cluster-in-pagepy) | Dead duck-typing cluster in `page.py` | Low | New; extends S10.3 |
|
||||||
|
| [R7](#r7--two-orphaned-subsystems) | Two orphaned subsystems | Low | New |
|
||||||
|
| [R8](#r8--backward-pagination-is-guesswork) | Backward pagination is guesswork | Medium | Partially noted in S11 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
|
||||||
|
**Well architected.** The concerns that usually decide this question are all on
|
||||||
|
the right side of the line:
|
||||||
|
|
||||||
|
- **The abstract/concrete split is real, not aspirational.** Verified
|
||||||
|
empirically: `abstract/` never imports `concrete/`; `core/` imports nothing but
|
||||||
|
itself. The single crossing ([document.py:7](../pyWebLayout/abstract/document.py#L7))
|
||||||
|
is into `style/`, which the dependency rules permit. Most codebases claiming
|
||||||
|
this layering have leaked it within a year.
|
||||||
|
- **The layouter contract is the right abstraction.** `paragraph_layouter`
|
||||||
|
returning `(fit, failed_word_index, remaining_pretext)` is what makes
|
||||||
|
pagination resumable, and the shape is consistent across content types. Layout
|
||||||
|
engines that return `List[Line]` cannot paginate without a second pass.
|
||||||
|
- **[core/cache.py](../pyWebLayout/core/cache.py) is exemplary.** Usage-ranked
|
||||||
|
eviction with periodic aging, sampled eviction instead of a maintained heap,
|
||||||
|
O(1) hit path with no reordering — every choice justified by a measurement in
|
||||||
|
the docstring.
|
||||||
|
- **[concrete/text.py](../pyWebLayout/concrete/text.py) is the strongest file.**
|
||||||
|
The glyph cache ([:581-647](../pyWebLayout/concrete/text.py#L581-L647))
|
||||||
|
reimplements PIL's internals to skip per-call setup, with a permanent graceful
|
||||||
|
fallback when the private API is absent. Alignment is a clean strategy pattern,
|
||||||
|
and `render_alignment_handler` handles last-line-of-paragraph correctly.
|
||||||
|
- **[html_extraction.py](../pyWebLayout/io/readers/html_extraction.py) is
|
||||||
|
textbook.** An immutable `StyleContext` threaded down the tree plus a handler
|
||||||
|
dispatch table — no giant if/elif, no mutable parser state.
|
||||||
|
|
||||||
|
**The complexity that is not earned** is concentrated in three places, all in the
|
||||||
|
same band of the code:
|
||||||
|
|
||||||
|
1. `layout/page_buffer.py` — 520 lines of multiprocess prefetch that has never
|
||||||
|
worked (S12, plus R1/R2 below).
|
||||||
|
2. `BidirectionalLayouter.render_page_backward` — ~100 lines of convergence
|
||||||
|
heuristics standing in for an anchor list that already exists (R8).
|
||||||
|
3. The second block dispatcher in `ereader_layout.py`, which silently drops
|
||||||
|
tables and lists (S4, S8).
|
||||||
|
|
||||||
|
The pattern is visible in the git history: work from `e000068` onward (caching,
|
||||||
|
alignment, page geometry) is markedly better than the ereader scaffolding it sits
|
||||||
|
on. This is not a mess. It is a solid library with an early prototype still
|
||||||
|
embedded in it.
|
||||||
|
|
||||||
|
**Sizing the cleanup:** R1–R7 plus S12 and S10.3 remove roughly 800–900 lines and
|
||||||
|
fix four user-visible defects. None of it requires redesigning anything.
|
||||||
|
|
||||||
|
## Test baseline
|
||||||
|
|
||||||
|
At `c5c61a3`, in a clean venv on Python 3.14.6:
|
||||||
|
|
||||||
|
```
|
||||||
|
833 passed, 2 skipped, 24 subtests passed in 11.51s
|
||||||
|
```
|
||||||
|
|
||||||
|
(The 2 skips were environmental — the review venv lacked `requests`, so the URL
|
||||||
|
image tests skipped. With the `test` extra from R4 installed the suite reports
|
||||||
|
`853 passed, 24 subtests passed in 13.53s`.)
|
||||||
|
|
||||||
|
The suite then **hangs indefinitely** rather than exiting. See R2.
|
||||||
|
|
||||||
|
## Status of the existing remediation spec
|
||||||
|
|
||||||
|
| Spec | Subject | State |
|
||||||
|
|------|---------|-------|
|
||||||
|
| S1 | Inline content in non-paragraph containers | Done (`284d521`) |
|
||||||
|
| S2 | Page geometry: origin and content rect | Done (`f18cec2`) |
|
||||||
|
| S3 | Draw/canvas lifecycle | Done (`202dacf`) |
|
||||||
|
| S11 | Partial-block progress discarded | Done (`a57da80`) |
|
||||||
|
| S13 | Word spacing and alignment | Done (`1262be6`) |
|
||||||
|
| S14 | Vertical centring in buttons and fields | Done (`c5c61a3`) |
|
||||||
|
| S4–S10, S12 | Dispatch, cells, table grid, pagination, hygiene, background rendering | Outstanding |
|
||||||
|
|
||||||
|
The spec's analysis is sound and in places sharper than this review — S12 caught
|
||||||
|
that `_render_page_worker` omits `page_size` entirely, which this review missed.
|
||||||
|
Nothing below supersedes it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## R1 — The process pool crashes on Python 3.14
|
||||||
|
|
||||||
|
**Severity: critical. Raises S12 from "useless" to "fatal".**
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
|
||||||
|
`PageBuffer` submits to a `ProcessPoolExecutor` from inside
|
||||||
|
`BufferedPageRenderer.render_page`
|
||||||
|
([page_buffer.py:431](../pyWebLayout/layout/page_buffer.py#L431)). Python 3.14
|
||||||
|
changed the default multiprocessing start method on Linux from `fork` to
|
||||||
|
`forkserver`. Under a non-`fork` start method, `submit()` reaches
|
||||||
|
`_check_not_importing_main()`, which raises unless the caller sits inside an
|
||||||
|
`if __name__ == "__main__":` guard — and the child re-imports the caller's main
|
||||||
|
module, re-executing it.
|
||||||
|
|
||||||
|
S12 documents this subsystem as delivering no benefit. On 3.14 it is worse than
|
||||||
|
that: `EreaderLayoutManager.get_current_page()` **raises** when called from
|
||||||
|
module-level script code.
|
||||||
|
|
||||||
|
### Evidence
|
||||||
|
|
||||||
|
A plain script calling `manager.get_current_page()` at module level, Python
|
||||||
|
3.14.6:
|
||||||
|
|
||||||
|
```
|
||||||
|
RuntimeError:
|
||||||
|
An attempt has been made to start a new process before the
|
||||||
|
current process has finished its bootstrapping phase.
|
||||||
|
...
|
||||||
|
File "pyWebLayout/layout/page_buffer.py", line 221, in _queue_forward_renders
|
||||||
|
future = self.executor.submit(_render_page_worker, args)
|
||||||
|
ConnectionResetError: [Errno 104] Connection reset by peer
|
||||||
|
```
|
||||||
|
|
||||||
|
With a `__main__` guard added it does not raise, and instead confirms S12's
|
||||||
|
finding on every job:
|
||||||
|
|
||||||
|
```
|
||||||
|
Background render failed for position RenderingPosition(...): cannot pickle 'ImagingCore' object
|
||||||
|
Background render failed for position RenderingPosition(...): cannot pickle 'ImagingCore' object
|
||||||
|
```
|
||||||
|
|
||||||
|
### Action
|
||||||
|
|
||||||
|
Fold into **S12**, and treat S12 as unblocked and urgent rather than phase 4. The
|
||||||
|
recommended resolution there — delete the pool, keep the LRU buffers and position
|
||||||
|
maps, replace prefetch with synchronous readahead — resolves R1 and R2 as a side
|
||||||
|
effect. S12's measurement gate still applies to the *readahead* decision; it does
|
||||||
|
not need to gate deletion of the pool, because the pool's contribution is
|
||||||
|
provably zero.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
`pyWebLayout/layout/page_buffer.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## R2 — The test suite hangs at interpreter exit
|
||||||
|
|
||||||
|
**Severity: high.** Same root cause as R1.
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
|
||||||
|
`PageBuffer.__del__` calls `shutdown()`, which calls
|
||||||
|
`executor.shutdown(wait=True)` ([page_buffer.py:342](../pyWebLayout/layout/page_buffer.py#L342)).
|
||||||
|
`EreaderLayoutManager.__del__` does the same via `renderer.shutdown()`. Running
|
||||||
|
`__del__` at interpreter shutdown and blocking on a process pool inside it
|
||||||
|
deadlocks.
|
||||||
|
|
||||||
|
### Evidence
|
||||||
|
|
||||||
|
```
|
||||||
|
833 passed, 2 skipped, 24 subtests passed in 11.51s
|
||||||
|
```
|
||||||
|
|
||||||
|
...then the process sat at ~0% CPU with idle forkserver children for 13 minutes
|
||||||
|
before being killed. Reproduced twice; both runs completed the tests in under
|
||||||
|
12s and neither exited.
|
||||||
|
|
||||||
|
This is why CI wall-clock does not resemble the 11.5s the tests actually take.
|
||||||
|
|
||||||
|
### Action
|
||||||
|
|
||||||
|
Resolved by S12's deletion of the executor. If for any reason the pool is
|
||||||
|
retained, `__del__` must not block: register an `atexit` handler or require
|
||||||
|
explicit `shutdown()`, and never `wait=True` from a finaliser.
|
||||||
|
|
||||||
|
### Acceptance criteria
|
||||||
|
|
||||||
|
- `pytest` returns to the shell within a second of printing its summary line.
|
||||||
|
- No `multiprocessing` child processes outlive the test session.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
`pyWebLayout/layout/page_buffer.py`, `pyWebLayout/layout/ereader_manager.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## R3 — Font scaling destroys hyperlinks
|
||||||
|
|
||||||
|
**Severity: high. User-visible, silent, and trivially reproducible.**
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
|
||||||
|
`BidirectionalLayouter._scale_block_fonts`
|
||||||
|
([ereader_layout.py:474-498](../pyWebLayout/layout/ereader_layout.py#L474-L498))
|
||||||
|
rebuilds a scaled block by constructing plain `Word(word.text, scaled_style)` for
|
||||||
|
every word. `LinkedWord` is a `Word` subclass
|
||||||
|
([inline.py:288](../pyWebLayout/abstract/inline.py#L288)), so the reconstruction
|
||||||
|
downgrades it and the link target is discarded.
|
||||||
|
|
||||||
|
The function returns the block unchanged when `font_scale == 1.0` and no family
|
||||||
|
override is set, which is why no test has caught this: the defect only appears
|
||||||
|
once the reader changes font size.
|
||||||
|
|
||||||
|
Two further gaps in the same function:
|
||||||
|
|
||||||
|
1. It handles only `Paragraph` and `Heading`. Every other block type is returned
|
||||||
|
unscaled, so a font-size change leaves images, tables and lists at their
|
||||||
|
original size while the text around them reflows.
|
||||||
|
2. It allocates a new `Paragraph` and a new `Word` per word on **every page
|
||||||
|
render** at any scale ≠ 1.0 — directly against the caching work in
|
||||||
|
`concrete/text.py`, and on the hot path.
|
||||||
|
|
||||||
|
### Evidence
|
||||||
|
|
||||||
|
At `c5c61a3`, parsing `<p>Go to <a href="http://x">this link</a> now.</p>`:
|
||||||
|
|
||||||
|
```
|
||||||
|
scale=1.0: LinkedWords = 2
|
||||||
|
scale=1.5: LinkedWords = 0
|
||||||
|
```
|
||||||
|
|
||||||
|
### Design
|
||||||
|
|
||||||
|
Stop reconstructing abstract blocks at layout time. Font scale and family are
|
||||||
|
*rendering context*, not document content — carrying them in a copied document
|
||||||
|
violates the "abstract content is not mutated by layout" principle in
|
||||||
|
[ARCHITECTURE.md](../ARCHITECTURE.md) in spirit, even though it copies rather
|
||||||
|
than mutates.
|
||||||
|
|
||||||
|
Preferred: thread the scale/override into the layouter and resolve fonts at
|
||||||
|
`Text` construction, where `Font` objects are already deduplicated by
|
||||||
|
`FontRegistry`. `paragraph_layouter` already accepts an `alignment_override`;
|
||||||
|
`font_scale` and `font_family_override` belong in the same place.
|
||||||
|
|
||||||
|
Minimum viable fix if the larger change is deferred: reconstruct via
|
||||||
|
`type(word)` and copy subclass state, and extend coverage to every block type.
|
||||||
|
This is strictly a stopgap — it keeps the per-page allocation cost.
|
||||||
|
|
||||||
|
### Acceptance criteria
|
||||||
|
|
||||||
|
- A document containing `<a href>` retains every `LinkedWord` after
|
||||||
|
`set_font_scale(1.5)`, and `query_point` over the rendered page still returns
|
||||||
|
`object_type="link"` with the correct target.
|
||||||
|
- An image block's rendered size is unaffected by `set_font_scale`, or scales
|
||||||
|
deliberately — not left inconsistent with the text around it.
|
||||||
|
- No new `Word`/`Paragraph` objects are allocated per page render at scale ≠ 1.0.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
`pyWebLayout/layout/ereader_layout.py`, `pyWebLayout/layout/document_layouter.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## R4 — Three packaging configs that disagree
|
||||||
|
|
||||||
|
**Severity: medium.** *Corrected: the original review claimed a clean install
|
||||||
|
fails on first import. It does not — see below.*
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
|
||||||
|
The project carries **three** sets of packaging metadata:
|
||||||
|
|
||||||
|
| File | Declares |
|
||||||
|
|------|----------|
|
||||||
|
| `pyproject.toml` `[project]` | Pillow, numpy, pyphen, beautifulsoup4, flask, ebooklib, requests |
|
||||||
|
| `setup.cfg` `[options]` | Pillow, numpy |
|
||||||
|
| `setup.py` `setup(...)` kwargs | Pillow, numpy |
|
||||||
|
|
||||||
|
`pyproject.toml`'s `[project]` table wins under any modern build backend, so the
|
||||||
|
shipped wheel is correct and `pip install pyWebLayout` works. The `setup.cfg` and
|
||||||
|
`setup.py` copies are dead, contradictory, and actively misleading — reading
|
||||||
|
either one gives the wrong answer about what the library needs.
|
||||||
|
|
||||||
|
The authoritative list is itself wrong in the other direction:
|
||||||
|
|
||||||
|
- **`flask` is a runtime dependency.** It is imported only by
|
||||||
|
`tests/abstract/test_abstract_blocks.py`, as a fixture HTTP server. Every user
|
||||||
|
installs Flask, Jinja2, Werkzeug, click, itsdangerous and blinker for nothing.
|
||||||
|
- **`ebooklib` is a runtime dependency and is never imported by the library.**
|
||||||
|
`epub_reader.py` uses `zipfile` + `xml.etree` directly. Only the *tests* use
|
||||||
|
ebooklib, to build EPUB fixtures.
|
||||||
|
- **`requests` is declared required but is optional.** `concrete/image.py:100-111`
|
||||||
|
imports it lazily and degrades to an error message on the image when absent.
|
||||||
|
- **`requires-python = ">=3.6"` is false.** The package uses dataclasses (3.7+)
|
||||||
|
and `from __future__ import annotations` (3.7+); CI tests 3.10, 3.12 and 3.13.
|
||||||
|
|
||||||
|
Net effect: a runtime install pulls 7 direct dependencies where 4 are needed.
|
||||||
|
|
||||||
|
### Action
|
||||||
|
|
||||||
|
- Consolidate on `pyproject.toml`. Reduce `setup.cfg` to its `[flake8]` section
|
||||||
|
and `setup.py` to a `setup()` shim, each with a comment saying where metadata
|
||||||
|
lives.
|
||||||
|
- Runtime deps: Pillow, numpy, pyphen, beautifulsoup4. Move flask, werkzeug,
|
||||||
|
ebooklib and requests into a `test` extra; add a `remote-images` extra for
|
||||||
|
requests; add a `dev` extra composing them.
|
||||||
|
- Set `requires-python = ">=3.10"` to match the CI matrix, and add version
|
||||||
|
classifiers.
|
||||||
|
- Add a CI step that installs the package into an empty venv with **only**
|
||||||
|
declared runtime deps and imports every top-level subpackage. This class of
|
||||||
|
defect is only caught by installing what you ship — and it is what would have
|
||||||
|
caught the original misreading.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
`pyproject.toml`, `setup.cfg`, `setup.py`, `.gitea/workflows/ci.yml`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## R5 — Monkey-patched `Page` methods with a conflicting signature
|
||||||
|
|
||||||
|
**Severity: medium.** Currently inert; a landmine if `Page` is ever refactored.
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
|
||||||
|
[ereader_layout.py:741-761](../pyWebLayout/layout/ereader_layout.py#L741-L761)
|
||||||
|
defines `_add_page_methods()` and calls it at import time. It attaches
|
||||||
|
`can_fit_line` and `available_width` to the `Page` class if they are absent.
|
||||||
|
|
||||||
|
`Page` defines both ([page.py:59](../pyWebLayout/concrete/page.py#L59),
|
||||||
|
[page.py:147](../pyWebLayout/concrete/page.py#L147)), so the patch never fires.
|
||||||
|
But the two definitions of `can_fit_line` **do not agree**:
|
||||||
|
|
||||||
|
| Source | Signature |
|
||||||
|
|--------|-----------|
|
||||||
|
| `Page` | `can_fit_line(baseline_spacing, ascent=0, descent=0)` |
|
||||||
|
| monkey patch | `can_fit_line(line_height)` |
|
||||||
|
|
||||||
|
The patched version also ignores descenders entirely — the exact bug S2 fixed. If
|
||||||
|
`Page.can_fit_line` were ever renamed or moved, this would silently reinstate
|
||||||
|
pre-S2 clipping behaviour, from an import side effect in a different package.
|
||||||
|
|
||||||
|
### Action
|
||||||
|
|
||||||
|
Delete `_add_page_methods` and its call site. Import-time monkey-patching of a
|
||||||
|
class in another module has no place here; if `Page` is missing something the
|
||||||
|
layout engine needs, it belongs on `Page`.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
`pyWebLayout/layout/ereader_layout.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## R6 — Dead duck-typing cluster in `page.py`
|
||||||
|
|
||||||
|
**Severity: low.** Extends S10.3.
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
|
||||||
|
[page.py:261-491](../pyWebLayout/concrete/page.py#L261-L491) contains a closed
|
||||||
|
cluster with no external callers:
|
||||||
|
|
||||||
|
- `_get_child_property` (:261) — called only by the four below
|
||||||
|
- `_get_child_height` (:301) — called by nothing
|
||||||
|
- `_get_child_position` (:382) — called only by `_point_in_child`
|
||||||
|
- `_point_in_child` (:435) — called by nothing
|
||||||
|
- `_get_child_size` (:466) — called only by `_point_in_child`
|
||||||
|
|
||||||
|
Verified by grep across `pyWebLayout/`, `tests/`, `examples/` and `scripts/`:
|
||||||
|
zero references outside the cluster. About 90 lines.
|
||||||
|
|
||||||
|
It exists because `Renderable` declares neither `size` nor `origin`, so the code
|
||||||
|
probes `_size`, `size`, `_height`, `height`, `_origin` and `position` in turn with
|
||||||
|
`hasattr`. `query_point` (:399) already does the right thing instead — it relies
|
||||||
|
on the `Queriable` interface.
|
||||||
|
|
||||||
|
### Action
|
||||||
|
|
||||||
|
- Delete all five methods.
|
||||||
|
- Add `origin` and `size` to the `Renderable`/`Geometric` contract in
|
||||||
|
`core/base.py` so the duck-typing cannot grow back. This is the same concern as
|
||||||
|
S10.1's render contract and can ship with it.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
`pyWebLayout/concrete/page.py`, `pyWebLayout/core/base.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## R7 — Two orphaned subsystems
|
||||||
|
|
||||||
|
**Severity: low**, but they are a large share of the "is this over-complex?"
|
||||||
|
impression: 559 lines that nothing in the library reaches.
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
|
||||||
|
**`concrete/interaction_handler.py` (310 lines).** `InteractionHandler` and
|
||||||
|
`InteractionStateManager` are referenced only by
|
||||||
|
`examples/07_pressed_state_demo.py`. No library code, no ereader path, no tests.
|
||||||
|
|
||||||
|
**`core/highlight.py` (249 lines).** `Highlight`, `HighlightColor` and
|
||||||
|
`HighlightManager` have tests (`tests/core/test_highlight.py`) but are not wired
|
||||||
|
into `EreaderLayoutManager` at all. Highlighting is not reachable through the
|
||||||
|
library's own top-level interface.
|
||||||
|
|
||||||
|
`HighlightManager` also duplicates `BookmarkManager`'s JSON persistence
|
||||||
|
(directory, `_save`, `_load`, per-document file naming) with no shared base.
|
||||||
|
|
||||||
|
### Action
|
||||||
|
|
||||||
|
Decide per subsystem, and record the decision:
|
||||||
|
|
||||||
|
- **Wire it up** — `EreaderLayoutManager` grows `add_highlight` / `highlights_for_page`
|
||||||
|
and the persistence merges with `BookmarkManager` into one document-state store.
|
||||||
|
- **Or move it out** — relocate to `examples/` or delete, and drop the tests with it.
|
||||||
|
|
||||||
|
Either is fine. Leaving a tested, documented, unreachable subsystem in `core/` is
|
||||||
|
what makes the library look larger and less coherent than it is.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
`pyWebLayout/concrete/interaction_handler.py`, `pyWebLayout/core/highlight.py`,
|
||||||
|
`pyWebLayout/layout/ereader_manager.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## R8 — Backward pagination is guesswork
|
||||||
|
|
||||||
|
**Severity: medium.** S11's closing note already flags this for audit; this
|
||||||
|
records what the audit found.
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
|
||||||
|
`render_page_backward`
|
||||||
|
([ereader_layout.py:372-472](../pyWebLayout/layout/ereader_layout.py#L372-L472))
|
||||||
|
finds the previous page by estimating a start position, rendering forward,
|
||||||
|
comparing the end against the target, and adjusting — **up to 10 times**. It then
|
||||||
|
has a fallback that jumps back up to 5 blocks and renders again, and a fallback
|
||||||
|
for *that* which renders from the start of the document.
|
||||||
|
|
||||||
|
Worst case: one "previous page" tap costs up to 12 full page layouts.
|
||||||
|
|
||||||
|
The estimator it converges from is `max(1, int(10 / font_scale))` blocks
|
||||||
|
([:684](../pyWebLayout/layout/ereader_layout.py#L684)) — a constant with no
|
||||||
|
relationship to page size, block length or font metrics.
|
||||||
|
|
||||||
|
The correct answer is usually already known.
|
||||||
|
`EreaderLayoutManager._page_history` ([ereader_manager.py:210](../pyWebLayout/layout/ereader_manager.py#L210))
|
||||||
|
records real page-start positions and serves them instantly; the refinement loop
|
||||||
|
only runs when history misses — after a jump, a bookmark, a font change, or
|
||||||
|
beyond 50 entries.
|
||||||
|
|
||||||
|
S11's note asks whether these fallbacks were compensating for the
|
||||||
|
discarded-progress bug it fixed. They were, in part: the "failed to move
|
||||||
|
backward" branch at [:446](../pyWebLayout/layout/ereader_layout.py#L446) is
|
||||||
|
reachable precisely when forward rendering fails to advance, which S11 addressed.
|
||||||
|
|
||||||
|
### Design
|
||||||
|
|
||||||
|
Replace convergence with anchors. Maintain a sorted list of known page-start
|
||||||
|
positions — chapter starts from `ChapterNavigator` (free, already built) plus
|
||||||
|
every position visited. To go back from position P: binary-search the largest
|
||||||
|
anchor A < P, render forward from A collecting page starts until reaching P, and
|
||||||
|
return the last one. Cost is bounded by the anchor spacing, and every page start
|
||||||
|
discovered on the way is itself a new anchor, so the second traversal of any
|
||||||
|
region is free.
|
||||||
|
|
||||||
|
This subsumes `_page_history`, so the two mechanisms become one.
|
||||||
|
|
||||||
|
**Sequencing:** do this after S12, and after S8 — table and list pagination
|
||||||
|
changes what a page start can be, and re-deriving anchors is cheap only once
|
||||||
|
positions round-trip through tables correctly (S8 already notes this dependency).
|
||||||
|
|
||||||
|
### Acceptance criteria
|
||||||
|
|
||||||
|
- `previous_page()` from any position issues at most *k* page layouts, where *k*
|
||||||
|
is the anchor spacing, with no iteration count and no fallback ladder.
|
||||||
|
- Forward-then-backward round-trips exactly, from a cold cache, after a chapter
|
||||||
|
jump, and after a bookmark restore.
|
||||||
|
- `_estimate_page_start`, `_adjust_start_estimate` and the three-tier fallback
|
||||||
|
are deleted, not retained alongside.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
`pyWebLayout/layout/ereader_layout.py`, `pyWebLayout/layout/ereader_manager.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recommended order
|
||||||
|
|
||||||
|
```
|
||||||
|
R4 ── packaging; independent, minutes, unblocks clean CI [done]
|
||||||
|
S12 ── delete the process pool; resolves R1 and R2 with it
|
||||||
|
R3 ── font scaling loses links; independent, user-visible
|
||||||
|
R5 ── delete the monkey patch; minutes
|
||||||
|
R6 ── delete the dead cluster (with S10.1's render contract)
|
||||||
|
R7 ── decide the two orphans; no code risk either way
|
||||||
|
S4 → S5 → S6 → S7 → S8 → S9 (existing spec, unchanged)
|
||||||
|
R8 ── after S8
|
||||||
|
```
|
||||||
|
|
||||||
|
R4, R5 and R6 are an afternoon and carry no design risk. S12 is the largest
|
||||||
|
single removal and fixes two defects at once. R3 is the one users would notice
|
||||||
|
today. Everything after that is the existing spec, which needs no revision.
|
||||||
|
|
||||||
|
## Reproducing the findings
|
||||||
|
|
||||||
|
Reviewed at `c5c61a3` on Python 3.14.6, in a venv containing
|
||||||
|
`pytest pyphen Pillow numpy beautifulsoup4 lxml ebooklib`.
|
||||||
|
|
||||||
|
- **R1**: call `EreaderLayoutManager(...).get_current_page()` from module-level
|
||||||
|
script code (no `__main__` guard).
|
||||||
|
- **R2**: `python -m pytest -q`; observe the summary line, then the hang.
|
||||||
|
- **R3**: parse HTML containing `<a href>`, call
|
||||||
|
`BidirectionalLayouter._scale_block_fonts(block, 1.5)`, count `LinkedWord`
|
||||||
|
instances in the result.
|
||||||
|
- **R5**: compare `inspect.signature(Page.can_fit_line)` against the patch body.
|
||||||
|
- **R6**: grep the five method names across `pyWebLayout/ tests/ examples/ scripts/`.
|
||||||
|
- **R8**: read the loop; no execution needed.
|
||||||
@@ -30,6 +30,7 @@ It is independent of every other spec here.
|
|||||||
| [S13](#s13--word-spacing-and-alignment) | Word spacing and alignment | 0 |
|
| [S13](#s13--word-spacing-and-alignment) | Word spacing and alignment | 0 |
|
||||||
| [S14](#s14--vertical-centring-in-buttons-and-fields) | Vertical centring in buttons and fields | 0 |
|
| [S14](#s14--vertical-centring-in-buttons-and-fields) | Vertical centring in buttons and fields | 0 |
|
||||||
| [S15](#s15--form-field-label-geometry) | Form field label geometry | 0 |
|
| [S15](#s15--form-field-label-geometry) | Form field label geometry | 0 |
|
||||||
|
| [S16](#s16--backward-page-navigation) | Backward page navigation | 0 |
|
||||||
|
|
||||||
## Design invariants
|
## Design invariants
|
||||||
|
|
||||||
@@ -1265,6 +1266,103 @@ between label and box smaller than the intended 5px.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## S16 — Backward page navigation
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
|
||||||
|
`render_page_backward` *searched* for the previous page's start: estimate a block
|
||||||
|
index, lay out forward, compare the end against the target, bisect on the block
|
||||||
|
difference, repeat up to ten times. Both the estimator and the adjuster pinned
|
||||||
|
`word_index` to 0 and moved only `block_index`.
|
||||||
|
|
||||||
|
Pages routinely start mid-block. Any such start was therefore **not in the search
|
||||||
|
space**, the loop could never match, and it fell through to a fallback that
|
||||||
|
jumped several blocks back or to the document start.
|
||||||
|
|
||||||
|
### Evidence
|
||||||
|
|
||||||
|
A document of short paragraphs around one 1200-word paragraph. Forward pagination
|
||||||
|
gives page starts at `(0,0), (2,208), (2,494), (2,780), (2,1057)`. Asking for the
|
||||||
|
page that ends where each of those begins:
|
||||||
|
|
||||||
|
```
|
||||||
|
from page 1 -> got (0,0) expected (0,0) ok (1 forward layout)
|
||||||
|
from page 2 -> got (0,0) expected (2,208) WRONG (10 forward layouts)
|
||||||
|
from page 3 -> got (0,0) expected (2,494) WRONG (10 forward layouts)
|
||||||
|
from page 4 -> got (0,0) expected (2,780) WRONG (10 forward layouts)
|
||||||
|
```
|
||||||
|
|
||||||
|
Every mid-paragraph case threw the reader to the start of the document after ten
|
||||||
|
full page layouts. The bisection was also unsound within its own space: a
|
||||||
|
document of 40 small paragraphs, where every page *does* start on a block
|
||||||
|
boundary, failed too.
|
||||||
|
|
||||||
|
This is complementary to S11 rather than caused by it. Before S11 forward
|
||||||
|
pagination dead-ended at the first page-spanning block, so mid-block starts were
|
||||||
|
never produced and the block-granular search looked adequate.
|
||||||
|
|
||||||
|
### Design
|
||||||
|
|
||||||
|
Pagination is a pure function: laying out from `q` yields a page and the position
|
||||||
|
it stopped at, `next(q)`. The page before `P` is the `q` with `next(q) == P`.
|
||||||
|
That is found by **replaying the chain forward from an anchor**, not by guessing
|
||||||
|
`q`. Three sources, in order:
|
||||||
|
|
||||||
|
1. **The recorded chain.** `render_page_forward` now records
|
||||||
|
`(font_scale, next(q)) -> q`. Stepping back to anywhere the reader has been is
|
||||||
|
exact and costs one layout. Keyed by font scale, since changing it
|
||||||
|
repaginates.
|
||||||
|
2. **Replay from an anchor.** Anchors are block starts, nearest first: the block
|
||||||
|
containing `P`, then up to `MAX_BACKWARD_ANCHORS` earlier ones, then the
|
||||||
|
document start. Lay out forward from the anchor until a page ends exactly on
|
||||||
|
`P`; that page's start is the answer. `MAX_REPLAY_PAGES` caps the walk so one
|
||||||
|
page turn cannot traverse a whole chapter.
|
||||||
|
3. **Nearest start before `P`.** If no chain passes exactly through `P` — which
|
||||||
|
happens when `P` was reached by a jump or a restored bookmark rather than by
|
||||||
|
reading forward, so it lies on no natural chain — return the last page start
|
||||||
|
before it. That overlaps `P`'s page slightly rather than skipping content,
|
||||||
|
which is the safe direction to be wrong in.
|
||||||
|
|
||||||
|
The estimator and the bisecting adjuster are deleted.
|
||||||
|
|
||||||
|
**What "correct" means here.** Each backward step returns a page ending exactly
|
||||||
|
where the reader currently is, so paging back never skips or repeats content.
|
||||||
|
That chain can differ from the one you would have seen reading forward from page
|
||||||
|
one, if you entered the document by a jump — pagination from a different starting
|
||||||
|
point is genuinely a different chain, and no algorithm can recover the original
|
||||||
|
without replaying from the start.
|
||||||
|
|
||||||
|
### Measurements
|
||||||
|
|
||||||
|
Same document, after the change:
|
||||||
|
|
||||||
|
```
|
||||||
|
warm (chain recorded by the forward pass): 4/4 exact, 1 layout each
|
||||||
|
cold, fresh layouter per call: 12/13 exact, worst 17 layouts
|
||||||
|
cold, one layouter, repeated back presses: 4 layouts per turn typical
|
||||||
|
```
|
||||||
|
|
||||||
|
The single inexact case is a target that lies on the canonical chain but not on
|
||||||
|
any chain reachable from a nearby anchor; it returns a start 15 words early,
|
||||||
|
i.e. a slightly overlapping page.
|
||||||
|
|
||||||
|
### Acceptance criteria
|
||||||
|
|
||||||
|
- For every page of a document, `render_page_backward(start[i])` returns
|
||||||
|
`start[i-1]` — verified for both a mid-paragraph-paginating document and one
|
||||||
|
where every page starts on a block boundary.
|
||||||
|
- Laying out forward from the returned position ends exactly on the requested
|
||||||
|
position.
|
||||||
|
- Forward-then-back returns to the original position.
|
||||||
|
- At the document start, backward stays there; an empty document is safe.
|
||||||
|
- Cost stays within a small bounded number of forward layouts.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
`pyWebLayout/layout/ereader_layout.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Test plan
|
## Test plan
|
||||||
|
|
||||||
Findings were reproduced with four probe scripts; each becomes a regression test
|
Findings were reproduced with four probe scripts; each becomes a regression test
|
||||||
|
|||||||
@@ -313,6 +313,13 @@ class BidirectionalLayouter:
|
|||||||
self.alignment_override = alignment_override
|
self.alignment_override = alignment_override
|
||||||
self.font_family_override = font_family_override
|
self.font_family_override = font_family_override
|
||||||
|
|
||||||
|
# Maps (font_scale, end position) -> the position the page started at.
|
||||||
|
# Filled in as pages are laid out forward, which makes "previous page"
|
||||||
|
# exact and free for anywhere the reader has already been. Keyed by font
|
||||||
|
# scale because changing it repaginates the document.
|
||||||
|
self._page_chain: Dict[Tuple[float, Tuple[int, int, int]],
|
||||||
|
RenderingPosition] = {}
|
||||||
|
|
||||||
def render_page_forward(self, position: RenderingPosition,
|
def render_page_forward(self, position: RenderingPosition,
|
||||||
font_scale: float = 1.0) -> Tuple[Page, RenderingPosition]:
|
font_scale: float = 1.0) -> Tuple[Page, RenderingPosition]:
|
||||||
"""
|
"""
|
||||||
@@ -367,109 +374,156 @@ class BidirectionalLayouter:
|
|||||||
|
|
||||||
current_pos = new_pos
|
current_pos = new_pos
|
||||||
|
|
||||||
|
# Remember this link in the chain so stepping back to it later is exact.
|
||||||
|
if self._position_compare(current_pos, position) > 0:
|
||||||
|
self._page_chain[(font_scale, self._position_key(current_pos))] = \
|
||||||
|
position.copy()
|
||||||
|
|
||||||
return page, current_pos
|
return page, current_pos
|
||||||
|
|
||||||
|
# How many block starts before the target to try as replay anchors before
|
||||||
|
# settling for the best inexact answer.
|
||||||
|
MAX_BACKWARD_ANCHORS = 4
|
||||||
|
|
||||||
|
# Ceiling on pages replayed from a single anchor, so a pathologically long
|
||||||
|
# block cannot make one page turn walk an entire chapter.
|
||||||
|
MAX_REPLAY_PAGES = 8
|
||||||
|
|
||||||
def render_page_backward(self,
|
def render_page_backward(self,
|
||||||
end_position: RenderingPosition,
|
end_position: RenderingPosition,
|
||||||
font_scale: float = 1.0) -> Tuple[Page,
|
font_scale: float = 1.0) -> Tuple[Page,
|
||||||
RenderingPosition]:
|
RenderingPosition]:
|
||||||
"""
|
"""
|
||||||
Render a page that ends at the given position, filling backward.
|
Render the page that ends at the given position - "previous page".
|
||||||
Critical for "previous page" navigation.
|
|
||||||
|
|
||||||
Uses iterative refinement to find the correct start position that
|
Pagination is a pure function: laying out from a position q yields a page
|
||||||
results in a page ending at (or very close to) the target position.
|
and the position where it stopped, next(q). The page before P is therefore
|
||||||
|
the q for which next(q) == P, and it is found by *replaying* the chain
|
||||||
|
forward from an anchor, not by guessing q.
|
||||||
|
|
||||||
|
The previous implementation searched instead: it estimated a block index
|
||||||
|
and bisected on it, pinning word_index to 0. Pages routinely start
|
||||||
|
mid-block, so the answer was frequently not in the search space at all -
|
||||||
|
the search then exhausted its iterations and fell back to a position that
|
||||||
|
was not the previous page, usually the start of the document.
|
||||||
|
|
||||||
|
Three sources are tried in order:
|
||||||
|
|
||||||
|
1. The recorded chain, from pages already laid out going forward. Exact,
|
||||||
|
and the common case when the reader is paging back and forth.
|
||||||
|
2. Replay from the start of the block containing P, then from
|
||||||
|
progressively earlier blocks. Exact when P lies on the resulting chain.
|
||||||
|
3. Failing an exact hit - which happens when P was reached by a jump or a
|
||||||
|
restored bookmark rather than by reading forward, so it is on no
|
||||||
|
natural chain - the latest page start before P. That overlaps P's page
|
||||||
|
slightly rather than skipping content, which is the safe direction to
|
||||||
|
be wrong in.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
end_position: Position where page should end
|
end_position: Position where the page should end
|
||||||
font_scale: Font scaling factor
|
font_scale: Font scaling factor
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Tuple of (rendered_page, start_position)
|
Tuple of (rendered_page, start_position)
|
||||||
"""
|
"""
|
||||||
# Handle edge case: already at beginning
|
document_start = RenderingPosition()
|
||||||
if end_position.block_index == 0 and end_position.word_index == 0:
|
|
||||||
return self.render_page_forward(end_position, font_scale)
|
|
||||||
|
|
||||||
# Start with initial estimate
|
# Nothing precedes the start of the document.
|
||||||
estimated_start = self._estimate_page_start(end_position, font_scale)
|
if self._position_compare(end_position, document_start) <= 0:
|
||||||
|
page, _ = self.render_page_forward(document_start, font_scale)
|
||||||
|
return page, document_start
|
||||||
|
|
||||||
# Iterative refinement: keep adjusting until we converge or hit max iterations
|
# 1. The chain we have already walked.
|
||||||
max_iterations = 10
|
remembered = self._page_chain.get((font_scale, self._position_key(end_position)))
|
||||||
best_page = None
|
if remembered is not None:
|
||||||
best_start = estimated_start
|
page, actual_end = self.render_page_forward(remembered, font_scale)
|
||||||
best_distance = float('inf')
|
if self._position_compare(actual_end, end_position) == 0:
|
||||||
|
return page, remembered
|
||||||
|
|
||||||
for iteration in range(max_iterations):
|
# 2/3. Replay from anchors, keeping the best inexact result as a fallback.
|
||||||
# Render forward from current estimate
|
fallback = None
|
||||||
page, actual_end = self.render_page_forward(estimated_start, font_scale)
|
for anchor in self._backward_anchors(end_position):
|
||||||
|
page, start, exact = self._replay_to(anchor, end_position, font_scale)
|
||||||
|
if page is None:
|
||||||
|
continue
|
||||||
|
if exact:
|
||||||
|
return page, start
|
||||||
|
if fallback is None:
|
||||||
|
fallback = (page, start)
|
||||||
|
|
||||||
# Calculate how far we are from target
|
if fallback is not None:
|
||||||
comparison = self._position_compare(actual_end, end_position)
|
return fallback
|
||||||
|
|
||||||
# Perfect match or close enough (within same block)
|
page, _ = self.render_page_forward(document_start, font_scale)
|
||||||
# BUT: ensure we actually moved backward (estimated_start < end_position)
|
return page, document_start
|
||||||
if comparison == 0:
|
|
||||||
# Check if we actually found a valid previous page
|
|
||||||
if self._position_compare(estimated_start, end_position) < 0:
|
|
||||||
return page, estimated_start
|
|
||||||
# If estimated_start >= end_position, we haven't moved backward
|
|
||||||
# Continue iterating to find a better position
|
|
||||||
elif iteration == 0:
|
|
||||||
# On first iteration, if we can't find a previous position,
|
|
||||||
# we're likely at or near the beginning
|
|
||||||
break
|
|
||||||
|
|
||||||
# Track best result so far
|
def _backward_anchors(self, target: RenderingPosition):
|
||||||
distance = abs(actual_end.block_index - end_position.block_index)
|
"""
|
||||||
if distance < best_distance:
|
Yield positions to replay from, nearest first.
|
||||||
best_distance = distance
|
|
||||||
best_page = page
|
|
||||||
best_start = estimated_start.copy()
|
|
||||||
|
|
||||||
# Adjust estimate for next iteration
|
Block starts are used as anchors because they are the coarsest positions
|
||||||
estimated_start = self._adjust_start_estimate(
|
that are certainly valid to lay out from. The block containing the target
|
||||||
estimated_start, end_position, actual_end)
|
comes first: when the target is mid-block, the page before it usually
|
||||||
|
starts in that same block or the one before.
|
||||||
|
"""
|
||||||
|
first_block = target.block_index if target.word_index > 0 \
|
||||||
|
else target.block_index - 1
|
||||||
|
|
||||||
# Safety: don't go before document start
|
for offset in range(self.MAX_BACKWARD_ANCHORS):
|
||||||
if estimated_start.block_index < 0:
|
block_index = first_block - offset
|
||||||
estimated_start.block_index = 0
|
if block_index < 0:
|
||||||
estimated_start.word_index = 0
|
break
|
||||||
|
yield RenderingPosition(
|
||||||
# If we exhausted iterations, return best result found
|
chapter_index=target.chapter_index,
|
||||||
# BUT: ensure we didn't return the same position (no backward progress)
|
block_index=block_index,
|
||||||
final_page = best_page if best_page else page
|
word_index=0,
|
||||||
final_start = best_start
|
|
||||||
|
|
||||||
# Safety check: if final_start >= end_position, we failed to move backward
|
|
||||||
# This can happen at the beginning of the document or when estimation failed
|
|
||||||
if self._position_compare(final_start, end_position) >= 0:
|
|
||||||
# Can't go further back - check if we're at the absolute beginning
|
|
||||||
if end_position.block_index == 0 and end_position.word_index == 0:
|
|
||||||
# Already at beginning, return as-is
|
|
||||||
return final_page, final_start
|
|
||||||
|
|
||||||
# Fallback strategy: try a more aggressive backward jump
|
|
||||||
# Start from several blocks before the current position
|
|
||||||
blocks_to_jump = max(1, min(5, end_position.block_index))
|
|
||||||
fallback_pos = RenderingPosition(
|
|
||||||
chapter_index=end_position.chapter_index,
|
|
||||||
block_index=max(0, end_position.block_index - blocks_to_jump),
|
|
||||||
word_index=0
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# Render forward from the fallback position
|
if first_block - self.MAX_BACKWARD_ANCHORS >= 0:
|
||||||
fallback_page, fallback_end = self.render_page_forward(fallback_pos, font_scale)
|
yield RenderingPosition()
|
||||||
|
|
||||||
# Verify the fallback actually moved us backward
|
def _replay_to(self,
|
||||||
if self._position_compare(fallback_pos, end_position) < 0:
|
anchor: RenderingPosition,
|
||||||
return fallback_page, fallback_pos
|
target: RenderingPosition,
|
||||||
|
font_scale: float):
|
||||||
|
"""
|
||||||
|
Lay out pages forward from `anchor`, looking for the one ending at `target`.
|
||||||
|
|
||||||
# If even the fallback didn't work, we're likely at the beginning
|
Returns:
|
||||||
# Return a page starting from the beginning
|
(page, start, exact). `exact` is True when a page ended precisely on
|
||||||
return self.render_page_forward(RenderingPosition(), font_scale)
|
the target. When the chain steps over the target instead, the last
|
||||||
|
page starting before it is returned with exact=False. (None, None,
|
||||||
|
False) means the anchor yielded nothing usable.
|
||||||
|
"""
|
||||||
|
position = anchor
|
||||||
|
last = (None, None)
|
||||||
|
|
||||||
return final_page, final_start
|
for _ in range(self.MAX_REPLAY_PAGES):
|
||||||
|
if self._position_compare(position, target) >= 0:
|
||||||
|
break
|
||||||
|
|
||||||
|
page, next_position = self.render_page_forward(position, font_scale)
|
||||||
|
comparison = self._position_compare(next_position, target)
|
||||||
|
|
||||||
|
if comparison == 0:
|
||||||
|
return page, position, True
|
||||||
|
|
||||||
|
if comparison > 0:
|
||||||
|
# Stepped over the target: this chain does not pass through it.
|
||||||
|
return last[0], last[1], False
|
||||||
|
|
||||||
|
if self._position_compare(next_position, position) <= 0:
|
||||||
|
break # no progress; give up on this anchor
|
||||||
|
|
||||||
|
last = (page, position)
|
||||||
|
position = next_position
|
||||||
|
|
||||||
|
return last[0], last[1], False
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _position_key(position: RenderingPosition) -> Tuple[int, int, int]:
|
||||||
|
"""Hashable identity of a position, for the page chain map."""
|
||||||
|
return (position.chapter_index, position.block_index, position.word_index)
|
||||||
|
|
||||||
def _scale_block_fonts(self, block: Block, font_scale: float) -> Block:
|
def _scale_block_fonts(self, block: Block, font_scale: float) -> Block:
|
||||||
"""Apply font scaling and font family override to all fonts in a block"""
|
"""Apply font scaling and font family override to all fonts in a block"""
|
||||||
|
|||||||
@@ -935,16 +935,31 @@ class EreaderLayoutManager:
|
|||||||
"""
|
"""
|
||||||
Shutdown the ereader manager and clean up resources.
|
Shutdown the ereader manager and clean up resources.
|
||||||
Call this when the application is closing.
|
Call this when the application is closing.
|
||||||
|
|
||||||
|
Idempotent: calling it twice saves the position once.
|
||||||
"""
|
"""
|
||||||
|
if getattr(self, '_shutdown_done', False):
|
||||||
|
return
|
||||||
|
self._shutdown_done = True
|
||||||
|
|
||||||
# Save current position
|
# Save current position
|
||||||
self.bookmark_manager.save_reading_position(self.current_position)
|
self.bookmark_manager.save_reading_position(self.current_position)
|
||||||
|
|
||||||
# Shutdown renderer and buffer
|
# Release cached pages
|
||||||
self.renderer.shutdown()
|
self.renderer.shutdown()
|
||||||
|
|
||||||
def __del__(self):
|
def __del__(self):
|
||||||
"""Cleanup on destruction"""
|
"""
|
||||||
self.shutdown()
|
Best-effort cleanup for callers that never called shutdown().
|
||||||
|
|
||||||
|
Finalisers run during interpreter teardown, when modules and globals
|
||||||
|
may already be torn down, so this must never raise and must never
|
||||||
|
block. Applications should call shutdown() explicitly.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
self.shutdown()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
# Convenience function for quick setup
|
# Convenience function for quick setup
|
||||||
|
|||||||
@@ -1,16 +1,35 @@
|
|||||||
"""
|
"""
|
||||||
Multi-process page buffering system for high-performance ereader navigation.
|
Page caching for ereader navigation.
|
||||||
|
|
||||||
This module provides intelligent page caching with background rendering using
|
`PageBuffer` is an LRU cache of rendered pages plus the position links between
|
||||||
multiprocessing to achieve sub-second page navigation performance.
|
them; `BufferedPageRenderer` wraps it around a `BidirectionalLayouter`.
|
||||||
|
|
||||||
|
This module used to render pages ahead of time in a `ProcessPoolExecutor`. That
|
||||||
|
never worked and has been removed — see S12 in docs/LAYOUT_REMEDIATION_SPEC.md
|
||||||
|
and R1/R2 in docs/ARCHITECTURE_REVIEW.md. In short: the worker returned
|
||||||
|
`pickle.dumps(page)`, and a Page holds a live PIL canvas, which is not
|
||||||
|
picklable, so every job failed and the result was discarded. The cost — four
|
||||||
|
interpreter copies and the whole block list shipped per job — was paid in full
|
||||||
|
for no benefit. On Python 3.14, where the default start method became
|
||||||
|
`forkserver`, submitting from module-level code raised outright.
|
||||||
|
|
||||||
|
Prefetch is not needed. Measured on the tests/data Wikipedia fixture (411
|
||||||
|
blocks) with the text caches warm, one page render costs:
|
||||||
|
|
||||||
|
800x600 p50 8.8 ms p95 15.4 ms
|
||||||
|
1072x1448 p50 13.8 ms p95 56.1 ms
|
||||||
|
|
||||||
|
A page turn is cheaper than the IPC that was meant to hide it. If a slower
|
||||||
|
target device ever changes that, the fallback is a synchronous `readahead()`
|
||||||
|
method on this class, or a single worker *thread* — layout is PIL-bound and PIL
|
||||||
|
releases the GIL — not a process pool. Making the concrete tree picklable
|
||||||
|
(Page -> Line -> Text -> Font -> FreeTypeFont) is a large surface area to
|
||||||
|
maintain for a cache.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
from typing import Dict, Optional, List, Tuple, Any
|
from typing import Dict, Optional, List, Tuple, Any
|
||||||
from collections import OrderedDict
|
from collections import OrderedDict
|
||||||
from concurrent.futures import ProcessPoolExecutor, Future
|
|
||||||
import threading
|
|
||||||
import pickle
|
|
||||||
|
|
||||||
from .ereader_layout import RenderingPosition, BidirectionalLayouter, FontFamilyOverride
|
from .ereader_layout import RenderingPosition, BidirectionalLayouter, FontFamilyOverride
|
||||||
from pyWebLayout.concrete.page import Page
|
from pyWebLayout.concrete.page import Page
|
||||||
@@ -19,57 +38,20 @@ from pyWebLayout.style.page_style import PageStyle
|
|||||||
from pyWebLayout.style.fonts import BundledFont
|
from pyWebLayout.style.fonts import BundledFont
|
||||||
|
|
||||||
|
|
||||||
def _render_page_worker(args: Tuple[List[Block],
|
|
||||||
PageStyle,
|
|
||||||
RenderingPosition,
|
|
||||||
float,
|
|
||||||
bool,
|
|
||||||
Optional[BundledFont]]) -> Tuple[RenderingPosition,
|
|
||||||
bytes,
|
|
||||||
RenderingPosition]:
|
|
||||||
"""
|
|
||||||
Worker function for multiprocess page rendering.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
args: Tuple of (blocks, page_style, position, font_scale, is_backward, font_family)
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Tuple of (original_position, pickled_page, next_position)
|
|
||||||
"""
|
|
||||||
blocks, page_style, position, font_scale, is_backward, font_family = args
|
|
||||||
|
|
||||||
# Create font family override if specified
|
|
||||||
font_family_override = FontFamilyOverride(font_family) if font_family else None
|
|
||||||
|
|
||||||
layouter = BidirectionalLayouter(blocks, page_style, font_family_override=font_family_override)
|
|
||||||
|
|
||||||
if is_backward:
|
|
||||||
page, next_pos = layouter.render_page_backward(position, font_scale)
|
|
||||||
else:
|
|
||||||
page, next_pos = layouter.render_page_forward(position, font_scale)
|
|
||||||
|
|
||||||
# Serialize the page for inter-process communication
|
|
||||||
pickled_page = pickle.dumps(page)
|
|
||||||
|
|
||||||
return position, pickled_page, next_pos
|
|
||||||
|
|
||||||
|
|
||||||
class PageBuffer:
|
class PageBuffer:
|
||||||
"""
|
"""
|
||||||
Intelligent page caching system with LRU eviction and background rendering.
|
LRU cache of rendered pages, with separate forward and backward buffers and
|
||||||
Maintains separate forward and backward buffers for optimal navigation performance.
|
the position links between adjacent pages.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, buffer_size: int = 5, max_workers: int = 4):
|
def __init__(self, buffer_size: int = 5):
|
||||||
"""
|
"""
|
||||||
Initialize the page buffer.
|
Initialize the page buffer.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
buffer_size: Number of pages to cache in each direction
|
buffer_size: Number of pages to cache in each direction
|
||||||
max_workers: Maximum number of worker processes for background rendering
|
|
||||||
"""
|
"""
|
||||||
self.buffer_size = buffer_size
|
self.buffer_size = buffer_size
|
||||||
self.max_workers = max_workers
|
|
||||||
|
|
||||||
# LRU caches for forward and backward pages
|
# LRU caches for forward and backward pages
|
||||||
self.forward_buffer: OrderedDict[RenderingPosition, Page] = OrderedDict()
|
self.forward_buffer: OrderedDict[RenderingPosition, Page] = OrderedDict()
|
||||||
@@ -81,11 +63,6 @@ class PageBuffer:
|
|||||||
self.reverse_position_map: Dict[RenderingPosition,
|
self.reverse_position_map: Dict[RenderingPosition,
|
||||||
RenderingPosition] = {} # current -> previous
|
RenderingPosition] = {} # current -> previous
|
||||||
|
|
||||||
# Background rendering
|
|
||||||
self.executor: Optional[ProcessPoolExecutor] = None
|
|
||||||
self.pending_renders: Dict[RenderingPosition, Future] = {}
|
|
||||||
self.render_lock = threading.Lock()
|
|
||||||
|
|
||||||
# Document state
|
# Document state
|
||||||
self.blocks: Optional[List[Block]] = None
|
self.blocks: Optional[List[Block]] = None
|
||||||
self.page_style: Optional[PageStyle] = None
|
self.page_style: Optional[PageStyle] = None
|
||||||
@@ -112,10 +89,6 @@ class PageBuffer:
|
|||||||
self.current_font_scale = font_scale
|
self.current_font_scale = font_scale
|
||||||
self.current_font_family = font_family
|
self.current_font_family = font_family
|
||||||
|
|
||||||
# Start the process pool
|
|
||||||
if self.executor is None:
|
|
||||||
self.executor = ProcessPoolExecutor(max_workers=self.max_workers)
|
|
||||||
|
|
||||||
def get_page(self, position: RenderingPosition) -> Optional[Page]:
|
def get_page(self, position: RenderingPosition) -> Optional[Page]:
|
||||||
"""
|
"""
|
||||||
Get a cached page if available.
|
Get a cached page if available.
|
||||||
@@ -176,125 +149,12 @@ class PageBuffer:
|
|||||||
self.position_map.pop(oldest_pos, None)
|
self.position_map.pop(oldest_pos, None)
|
||||||
self.reverse_position_map.pop(oldest_pos, None)
|
self.reverse_position_map.pop(oldest_pos, None)
|
||||||
|
|
||||||
def start_background_rendering(
|
|
||||||
self,
|
|
||||||
current_position: RenderingPosition,
|
|
||||||
direction: str = 'forward'):
|
|
||||||
"""
|
|
||||||
Start background rendering of upcoming pages.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
current_position: Current reading position
|
|
||||||
direction: 'forward', 'backward', or 'both'
|
|
||||||
"""
|
|
||||||
if not self.blocks or not self.page_style or not self.executor:
|
|
||||||
return
|
|
||||||
|
|
||||||
with self.render_lock:
|
|
||||||
if direction in ['forward', 'both']:
|
|
||||||
self._queue_forward_renders(current_position)
|
|
||||||
|
|
||||||
if direction in ['backward', 'both']:
|
|
||||||
self._queue_backward_renders(current_position)
|
|
||||||
|
|
||||||
def _queue_forward_renders(self, start_position: RenderingPosition):
|
|
||||||
"""Queue forward page renders starting from the given position"""
|
|
||||||
current_pos = start_position
|
|
||||||
|
|
||||||
for i in range(self.buffer_size):
|
|
||||||
# Skip if already cached or being rendered
|
|
||||||
if current_pos in self.forward_buffer or current_pos in self.pending_renders:
|
|
||||||
# Try to get next position from cache
|
|
||||||
current_pos = self.position_map.get(current_pos)
|
|
||||||
if not current_pos:
|
|
||||||
break
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Queue render job
|
|
||||||
args = (
|
|
||||||
self.blocks,
|
|
||||||
self.page_style,
|
|
||||||
current_pos,
|
|
||||||
self.current_font_scale,
|
|
||||||
False,
|
|
||||||
self.current_font_family)
|
|
||||||
future = self.executor.submit(_render_page_worker, args)
|
|
||||||
self.pending_renders[current_pos] = future
|
|
||||||
|
|
||||||
# We don't know the next position yet, so we'll update it when the render
|
|
||||||
# completes
|
|
||||||
break
|
|
||||||
|
|
||||||
def _queue_backward_renders(self, start_position: RenderingPosition):
|
|
||||||
"""Queue backward page renders ending at the given position"""
|
|
||||||
current_pos = start_position
|
|
||||||
|
|
||||||
for i in range(self.buffer_size):
|
|
||||||
# Skip if already cached or being rendered
|
|
||||||
if current_pos in self.backward_buffer or current_pos in self.pending_renders:
|
|
||||||
# Try to get previous position from cache
|
|
||||||
current_pos = self.reverse_position_map.get(current_pos)
|
|
||||||
if not current_pos:
|
|
||||||
break
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Queue render job
|
|
||||||
args = (
|
|
||||||
self.blocks,
|
|
||||||
self.page_style,
|
|
||||||
current_pos,
|
|
||||||
self.current_font_scale,
|
|
||||||
True,
|
|
||||||
self.current_font_family)
|
|
||||||
future = self.executor.submit(_render_page_worker, args)
|
|
||||||
self.pending_renders[current_pos] = future
|
|
||||||
|
|
||||||
# We don't know the previous position yet, so we'll update it when the
|
|
||||||
# render completes
|
|
||||||
break
|
|
||||||
|
|
||||||
def check_completed_renders(self):
|
|
||||||
"""Check for completed background renders and cache the results"""
|
|
||||||
if not self.pending_renders:
|
|
||||||
return
|
|
||||||
|
|
||||||
completed = []
|
|
||||||
|
|
||||||
with self.render_lock:
|
|
||||||
for position, future in self.pending_renders.items():
|
|
||||||
if future.done():
|
|
||||||
try:
|
|
||||||
original_pos, pickled_page, next_pos = future.result()
|
|
||||||
|
|
||||||
# Deserialize the page
|
|
||||||
page = pickle.loads(pickled_page)
|
|
||||||
|
|
||||||
# Cache the page
|
|
||||||
self.cache_page(original_pos, page, next_pos, is_backward=False)
|
|
||||||
|
|
||||||
completed.append(position)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"Background render failed for position {position}: {e}")
|
|
||||||
completed.append(position)
|
|
||||||
|
|
||||||
# Remove completed renders
|
|
||||||
for pos in completed:
|
|
||||||
self.pending_renders.pop(pos, None)
|
|
||||||
|
|
||||||
def invalidate_all(self):
|
def invalidate_all(self):
|
||||||
"""Clear all cached pages and cancel pending renders"""
|
"""Clear all cached pages"""
|
||||||
with self.render_lock:
|
self.forward_buffer.clear()
|
||||||
# Cancel pending renders
|
self.backward_buffer.clear()
|
||||||
for future in self.pending_renders.values():
|
self.position_map.clear()
|
||||||
future.cancel()
|
self.reverse_position_map.clear()
|
||||||
self.pending_renders.clear()
|
|
||||||
|
|
||||||
# Clear caches
|
|
||||||
self.forward_buffer.clear()
|
|
||||||
self.backward_buffer.clear()
|
|
||||||
self.position_map.clear()
|
|
||||||
self.reverse_position_map.clear()
|
|
||||||
|
|
||||||
def set_font_scale(self, font_scale: float):
|
def set_font_scale(self, font_scale: float):
|
||||||
"""
|
"""
|
||||||
@@ -323,7 +183,6 @@ class PageBuffer:
|
|||||||
return {
|
return {
|
||||||
'forward_buffer_size': len(self.forward_buffer),
|
'forward_buffer_size': len(self.forward_buffer),
|
||||||
'backward_buffer_size': len(self.backward_buffer),
|
'backward_buffer_size': len(self.backward_buffer),
|
||||||
'pending_renders': len(self.pending_renders),
|
|
||||||
'position_mappings': len(self.position_map),
|
'position_mappings': len(self.position_map),
|
||||||
'reverse_position_mappings': len(self.reverse_position_map),
|
'reverse_position_mappings': len(self.reverse_position_map),
|
||||||
'current_font_scale': self.current_font_scale,
|
'current_font_scale': self.current_font_scale,
|
||||||
@@ -331,28 +190,20 @@ class PageBuffer:
|
|||||||
}
|
}
|
||||||
|
|
||||||
def shutdown(self):
|
def shutdown(self):
|
||||||
"""Shutdown the page buffer and clean up resources"""
|
"""
|
||||||
if self.executor:
|
Release cached pages.
|
||||||
# Cancel pending renders
|
|
||||||
with self.render_lock:
|
|
||||||
for future in self.pending_renders.values():
|
|
||||||
future.cancel()
|
|
||||||
|
|
||||||
# Shutdown executor
|
Cheap and idempotent. There is deliberately no __del__ calling this:
|
||||||
self.executor.shutdown(wait=True)
|
blocking work in a finaliser is what deadlocked the interpreter at exit
|
||||||
self.executor = None
|
while the process pool existed.
|
||||||
|
"""
|
||||||
# Clear all caches
|
|
||||||
self.invalidate_all()
|
self.invalidate_all()
|
||||||
|
|
||||||
def __del__(self):
|
|
||||||
"""Cleanup on destruction"""
|
|
||||||
self.shutdown()
|
|
||||||
|
|
||||||
|
|
||||||
class BufferedPageRenderer:
|
class BufferedPageRenderer:
|
||||||
"""
|
"""
|
||||||
High-level interface for buffered page rendering with automatic background caching.
|
High-level interface for page rendering with an LRU cache in front of the
|
||||||
|
layouter.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self,
|
def __init__(self,
|
||||||
@@ -390,7 +241,7 @@ class BufferedPageRenderer:
|
|||||||
def render_page(self, position: RenderingPosition,
|
def render_page(self, position: RenderingPosition,
|
||||||
font_scale: float = 1.0) -> Tuple[Page, RenderingPosition]:
|
font_scale: float = 1.0) -> Tuple[Page, RenderingPosition]:
|
||||||
"""
|
"""
|
||||||
Render a page with intelligent caching.
|
Render a page, serving it from cache when possible.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
position: Position to render from
|
position: Position to render from
|
||||||
@@ -407,32 +258,18 @@ class BufferedPageRenderer:
|
|||||||
# Check cache first
|
# Check cache first
|
||||||
cached_page = self.buffer.get_page(position)
|
cached_page = self.buffer.get_page(position)
|
||||||
if cached_page:
|
if cached_page:
|
||||||
# Get next position from position map
|
# Only use the cache if we also know where the next page starts;
|
||||||
|
# otherwise fall through and compute it.
|
||||||
next_pos = self.buffer.position_map.get(position)
|
next_pos = self.buffer.position_map.get(position)
|
||||||
|
|
||||||
# Only use cache if we have the forward position mapping
|
|
||||||
# Otherwise, we need to compute it
|
|
||||||
if next_pos is not None:
|
if next_pos is not None:
|
||||||
# Start background rendering for upcoming pages
|
|
||||||
self.buffer.start_background_rendering(position, 'forward')
|
|
||||||
|
|
||||||
return cached_page, next_pos
|
return cached_page, next_pos
|
||||||
|
|
||||||
# Cache hit for the page, but we don't have the forward position
|
|
||||||
# Fall through to compute it below
|
|
||||||
|
|
||||||
# Render the page directly
|
# Render the page directly
|
||||||
page, next_pos = self.layouter.render_page_forward(position, font_scale)
|
page, next_pos = self.layouter.render_page_forward(position, font_scale)
|
||||||
|
|
||||||
# Cache the result
|
# Cache the result
|
||||||
self.buffer.cache_page(position, page, next_pos)
|
self.buffer.cache_page(position, page, next_pos)
|
||||||
|
|
||||||
# Start background rendering
|
|
||||||
self.buffer.start_background_rendering(position, 'both')
|
|
||||||
|
|
||||||
# Check for completed background renders
|
|
||||||
self.buffer.check_completed_renders()
|
|
||||||
|
|
||||||
return page, next_pos
|
return page, next_pos
|
||||||
|
|
||||||
def render_page_backward(self,
|
def render_page_backward(self,
|
||||||
@@ -440,7 +277,8 @@ class BufferedPageRenderer:
|
|||||||
font_scale: float = 1.0) -> Tuple[Page,
|
font_scale: float = 1.0) -> Tuple[Page,
|
||||||
RenderingPosition]:
|
RenderingPosition]:
|
||||||
"""
|
"""
|
||||||
Render a page ending at the given position with intelligent caching.
|
Render a page ending at the given position, serving it from cache when
|
||||||
|
possible.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
end_position: Position where page should end
|
end_position: Position where page should end
|
||||||
@@ -457,32 +295,18 @@ class BufferedPageRenderer:
|
|||||||
# Check cache first
|
# Check cache first
|
||||||
cached_page = self.buffer.get_page(end_position)
|
cached_page = self.buffer.get_page(end_position)
|
||||||
if cached_page:
|
if cached_page:
|
||||||
# Get previous position from reverse position map
|
# Only use the cache if we also know where the previous page
|
||||||
|
# starts; otherwise fall through and compute it.
|
||||||
prev_pos = self.buffer.reverse_position_map.get(end_position)
|
prev_pos = self.buffer.reverse_position_map.get(end_position)
|
||||||
|
|
||||||
# Only use cache if we have the reverse position mapping
|
|
||||||
# Otherwise, we need to compute it
|
|
||||||
if prev_pos is not None:
|
if prev_pos is not None:
|
||||||
# Start background rendering for previous pages
|
|
||||||
self.buffer.start_background_rendering(end_position, 'backward')
|
|
||||||
|
|
||||||
return cached_page, prev_pos
|
return cached_page, prev_pos
|
||||||
|
|
||||||
# Cache hit for the page, but we don't have the reverse position
|
|
||||||
# Fall through to compute it below
|
|
||||||
|
|
||||||
# Render the page directly
|
# Render the page directly
|
||||||
page, start_pos = self.layouter.render_page_backward(end_position, font_scale)
|
page, start_pos = self.layouter.render_page_backward(end_position, font_scale)
|
||||||
|
|
||||||
# Cache the result
|
# Cache the result
|
||||||
self.buffer.cache_page(start_pos, page, end_position, is_backward=True)
|
self.buffer.cache_page(start_pos, page, end_position, is_backward=True)
|
||||||
|
|
||||||
# Start background rendering
|
|
||||||
self.buffer.start_background_rendering(end_position, 'both')
|
|
||||||
|
|
||||||
# Check for completed background renders
|
|
||||||
self.buffer.check_completed_renders()
|
|
||||||
|
|
||||||
return page, start_pos
|
return page, start_pos
|
||||||
|
|
||||||
def set_font_family(self, font_family: Optional[BundledFont]):
|
def set_font_family(self, font_family: Optional[BundledFont]):
|
||||||
@@ -516,5 +340,5 @@ class BufferedPageRenderer:
|
|||||||
return self.buffer.get_cache_stats()
|
return self.buffer.get_cache_stats()
|
||||||
|
|
||||||
def shutdown(self):
|
def shutdown(self):
|
||||||
"""Shutdown the renderer and clean up resources"""
|
"""Release cached pages"""
|
||||||
self.buffer.shutdown()
|
self.buffer.shutdown()
|
||||||
|
|||||||
+37
-5
@@ -4,24 +4,56 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "pyWebLayout"
|
name = "pyWebLayout"
|
||||||
|
version = "0.1.1"
|
||||||
description = "A Python library for HTML-like layout and rendering"
|
description = "A Python library for HTML-like layout and rendering"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.6"
|
requires-python = ">=3.10"
|
||||||
license = {file = "LICENSE"}
|
license = {file = "LICENSE"}
|
||||||
authors = [
|
authors = [
|
||||||
{name = "Duncan Tourolle", email = "duncan@tourolle.paris"}
|
{name = "Duncan Tourolle", email = "duncan@tourolle.paris"}
|
||||||
]
|
]
|
||||||
dynamic = ["version"]
|
classifiers = [
|
||||||
|
"Programming Language :: Python :: 3",
|
||||||
|
"Programming Language :: Python :: 3.10",
|
||||||
|
"Programming Language :: Python :: 3.11",
|
||||||
|
"Programming Language :: Python :: 3.12",
|
||||||
|
"Programming Language :: Python :: 3.13",
|
||||||
|
"License :: OSI Approved :: MIT License",
|
||||||
|
"Operating System :: OS Independent",
|
||||||
|
]
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"Pillow",
|
"Pillow",
|
||||||
"numpy",
|
"numpy",
|
||||||
"pyphen",
|
"pyphen",
|
||||||
"beautifulsoup4",
|
"beautifulsoup4",
|
||||||
"flask",
|
|
||||||
"ebooklib",
|
|
||||||
"requests"
|
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[project.urls]
|
||||||
|
Homepage = "https://gitea.tourolle.paris/pyWebLayout"
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
# Loading images from http(s) URLs. concrete.image imports requests lazily and
|
||||||
|
# degrades to an error message on the image when it is absent, so it is not a
|
||||||
|
# hard requirement.
|
||||||
|
remote-images = ["requests"]
|
||||||
|
test = [
|
||||||
|
"pytest>=6.0",
|
||||||
|
"pytest-cov",
|
||||||
|
"flask", # fixture HTTP server in tests/abstract/test_abstract_blocks.py
|
||||||
|
"werkzeug", # make_server, same fixture
|
||||||
|
"ebooklib", # builds EPUB fixtures; the reader itself uses zipfile + ElementTree
|
||||||
|
"requests", # exercises the remote-images path
|
||||||
|
]
|
||||||
|
dev = [
|
||||||
|
"pyWebLayout[test,remote-images]",
|
||||||
|
"flake8",
|
||||||
|
"coverage-badge",
|
||||||
|
"interrogate",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.setuptools.packages.find]
|
||||||
|
include = ["pyWebLayout*"]
|
||||||
|
|
||||||
[tool.coverage.run]
|
[tool.coverage.run]
|
||||||
source = ["pyWebLayout"]
|
source = ["pyWebLayout"]
|
||||||
branch = true
|
branch = true
|
||||||
|
|||||||
@@ -1,26 +1,6 @@
|
|||||||
[metadata]
|
# Packaging metadata lives in pyproject.toml ([project]), which takes
|
||||||
name = pyWebLayout
|
# precedence over anything declared here. This file keeps only tool config
|
||||||
version = 0.1.1
|
# that has nowhere better to live.
|
||||||
author = Duncan Tourolle
|
|
||||||
author_email = duncan@tourolle.paris
|
|
||||||
description = A Python library for HTML-like layout and rendering
|
|
||||||
long_description = file: README.md
|
|
||||||
long_description_content_type = text/markdown
|
|
||||||
url = https://gitea.tourolle.paris/pyWebLayout
|
|
||||||
classifiers =
|
|
||||||
Programming Language :: Python :: 3
|
|
||||||
License :: OSI Approved :: MIT License
|
|
||||||
Operating System :: OS Independent
|
|
||||||
|
|
||||||
[options]
|
|
||||||
packages = find:
|
|
||||||
python_requires = >=3.6
|
|
||||||
install_requires =
|
|
||||||
Pillow
|
|
||||||
numpy
|
|
||||||
|
|
||||||
[options.packages.find]
|
|
||||||
include = pyWebLayout*
|
|
||||||
|
|
||||||
[flake8]
|
[flake8]
|
||||||
exclude =
|
exclude =
|
||||||
|
|||||||
@@ -1,32 +1,10 @@
|
|||||||
from setuptools import setup, find_packages
|
"""Shim for legacy `python setup.py` invocations.
|
||||||
|
|
||||||
setup(
|
All packaging metadata lives in setup.cfg. Keeping a second copy here was an
|
||||||
name="pyWebLayout",
|
active hazard: keyword arguments passed to setup() override setup.cfg, so the
|
||||||
version="0.1.1",
|
two could disagree silently and the setup.py copy would win.
|
||||||
packages=find_packages(),
|
"""
|
||||||
install_requires=[
|
|
||||||
"Pillow",
|
from setuptools import setup
|
||||||
"numpy",
|
|
||||||
],
|
setup()
|
||||||
extras_require={
|
|
||||||
"test": [
|
|
||||||
"coverage>=5.0",
|
|
||||||
],
|
|
||||||
"dev": [
|
|
||||||
"coverage>=5.0",
|
|
||||||
"pytest>=6.0",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
author="Duncan Tourolle",
|
|
||||||
author_email="duncan@tourolle.paris",
|
|
||||||
description="A Python library for HTML-like layout and rendering",
|
|
||||||
long_description=open("README.md").read(),
|
|
||||||
long_description_content_type="text/markdown",
|
|
||||||
url="https://gitea.tourolle.paris/pyWebLayout",
|
|
||||||
classifiers=[
|
|
||||||
"Programming Language :: Python :: 3",
|
|
||||||
"License :: OSI Approved :: MIT License",
|
|
||||||
"Operating System :: OS Independent",
|
|
||||||
],
|
|
||||||
python_requires=">=3.6",
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -307,6 +307,8 @@ class TestImagePIL(unittest.TestCase):
|
|||||||
|
|
||||||
if FLASK_AVAILABLE and hasattr(cls, 'flask_thread'):
|
if FLASK_AVAILABLE and hasattr(cls, 'flask_thread'):
|
||||||
cls.flask_server_running = False
|
cls.flask_server_running = False
|
||||||
|
cls.flask_server.shutdown()
|
||||||
|
cls.flask_server.server_close()
|
||||||
cls.flask_thread.join(timeout=2)
|
cls.flask_thread.join(timeout=2)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
@@ -350,10 +352,9 @@ class TestImagePIL(unittest.TestCase):
|
|||||||
"""Start a Flask server for URL testing."""
|
"""Start a Flask server for URL testing."""
|
||||||
import urllib.request
|
import urllib.request
|
||||||
import urllib.error
|
import urllib.error
|
||||||
|
from werkzeug.serving import make_server
|
||||||
|
|
||||||
cls.flask_app = Flask(__name__)
|
cls.flask_app = Flask(__name__)
|
||||||
cls.flask_port = 5555 # Use a specific port for testing
|
|
||||||
cls.flask_server_running = True
|
|
||||||
|
|
||||||
@cls.flask_app.route('/test.jpg')
|
@cls.flask_app.route('/test.jpg')
|
||||||
def serve_test_image():
|
def serve_test_image():
|
||||||
@@ -363,11 +364,12 @@ class TestImagePIL(unittest.TestCase):
|
|||||||
def health_check():
|
def health_check():
|
||||||
return 'OK', 200
|
return 'OK', 200
|
||||||
|
|
||||||
def run_flask():
|
# Bind to an ephemeral port so concurrent/leftover test runs can't clash
|
||||||
cls.flask_app.run(host='127.0.0.1', port=cls.flask_port, debug=False,
|
cls.flask_server = make_server('127.0.0.1', 0, cls.flask_app, threaded=True)
|
||||||
use_reloader=False, threaded=True)
|
cls.flask_port = cls.flask_server.server_port
|
||||||
|
cls.flask_server_running = True
|
||||||
|
|
||||||
cls.flask_thread = threading.Thread(target=run_flask, daemon=True)
|
cls.flask_thread = threading.Thread(target=cls.flask_server.serve_forever, daemon=True)
|
||||||
cls.flask_thread.start()
|
cls.flask_thread.start()
|
||||||
|
|
||||||
# Wait for server to be ready with health check
|
# Wait for server to be ready with health check
|
||||||
@@ -379,12 +381,15 @@ class TestImagePIL(unittest.TestCase):
|
|||||||
try:
|
try:
|
||||||
with urllib.request.urlopen(f'http://127.0.0.1:{cls.flask_port}/health', timeout=1) as response:
|
with urllib.request.urlopen(f'http://127.0.0.1:{cls.flask_port}/health', timeout=1) as response:
|
||||||
if response.status == 200:
|
if response.status == 200:
|
||||||
break
|
return
|
||||||
except (urllib.error.URLError, ConnectionRefusedError, OSError):
|
except (urllib.error.URLError, ConnectionRefusedError, OSError):
|
||||||
pass
|
pass
|
||||||
time.sleep(wait_interval)
|
time.sleep(wait_interval)
|
||||||
elapsed += wait_interval
|
elapsed += wait_interval
|
||||||
|
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Test Flask server did not become ready on port {cls.flask_port} within {max_wait}s")
|
||||||
|
|
||||||
def test_image_url_detection(self):
|
def test_image_url_detection(self):
|
||||||
"""Test URL detection functionality."""
|
"""Test URL detection functionality."""
|
||||||
img = Image()
|
img = Image()
|
||||||
|
|||||||
@@ -0,0 +1,171 @@
|
|||||||
|
"""
|
||||||
|
Regression tests for backward page navigation (spec S16).
|
||||||
|
|
||||||
|
The previous page of P is the position q for which laying out forward from q ends
|
||||||
|
exactly at P. The old implementation searched for q by guessing a block index and
|
||||||
|
bisecting, with word_index pinned to 0 - so a page starting mid-paragraph was not
|
||||||
|
in the search space at all. It exhausted its ten iterations and fell back to a
|
||||||
|
position that was not the previous page, typically the start of the document.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pyWebLayout.abstract.block import Paragraph
|
||||||
|
from pyWebLayout.abstract.inline import Word
|
||||||
|
from pyWebLayout.layout.ereader_layout import BidirectionalLayouter, RenderingPosition
|
||||||
|
from pyWebLayout.style import Font
|
||||||
|
from pyWebLayout.style.page_style import PageStyle
|
||||||
|
|
||||||
|
|
||||||
|
PAGE_SIZE = (800, 600)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def font():
|
||||||
|
return Font(font_size=16)
|
||||||
|
|
||||||
|
|
||||||
|
def paragraph(font, count, tag):
|
||||||
|
block = Paragraph(font)
|
||||||
|
for i in range(count):
|
||||||
|
block.add_word(Word(f"{tag}{i}", font))
|
||||||
|
return block
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def long_document(font):
|
||||||
|
"""Short paragraphs around one that spans several pages."""
|
||||||
|
return [
|
||||||
|
paragraph(font, 60, "a"),
|
||||||
|
paragraph(font, 80, "b"),
|
||||||
|
paragraph(font, 1200, "long"),
|
||||||
|
paragraph(font, 70, "c"),
|
||||||
|
paragraph(font, 90, "d"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def block_document(font):
|
||||||
|
"""Many small blocks, so every page starts on a block boundary."""
|
||||||
|
return [paragraph(font, 40, f"p{i}") for i in range(40)]
|
||||||
|
|
||||||
|
|
||||||
|
def forward_chain(layouter, limit=30):
|
||||||
|
"""The page start positions a reader would visit going forward."""
|
||||||
|
starts = []
|
||||||
|
pos = RenderingPosition()
|
||||||
|
for _ in range(limit):
|
||||||
|
starts.append(pos)
|
||||||
|
_, nxt = layouter.render_page_forward(pos, 1.0)
|
||||||
|
if nxt.block_index >= len(layouter.blocks):
|
||||||
|
break
|
||||||
|
if (nxt.block_index, nxt.word_index) == (pos.block_index, pos.word_index):
|
||||||
|
pytest.fail("forward pagination made no progress")
|
||||||
|
pos = nxt
|
||||||
|
return starts
|
||||||
|
|
||||||
|
|
||||||
|
def key(position):
|
||||||
|
return (position.chapter_index, position.block_index, position.word_index)
|
||||||
|
|
||||||
|
|
||||||
|
class TestBackwardMatchesForward:
|
||||||
|
"""The defining invariant: forward from the answer lands exactly on P."""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("document", ["long_document", "block_document"])
|
||||||
|
def test_previous_page_is_the_forward_predecessor(self, document, request):
|
||||||
|
blocks = request.getfixturevalue(document)
|
||||||
|
layouter = BidirectionalLayouter(blocks, PageStyle(), PAGE_SIZE)
|
||||||
|
starts = forward_chain(layouter)
|
||||||
|
assert len(starts) > 2, "need a few pages to test against"
|
||||||
|
|
||||||
|
for i in range(1, len(starts)):
|
||||||
|
_, got = layouter.render_page_backward(starts[i], 1.0)
|
||||||
|
assert key(got) == key(starts[i - 1]), (
|
||||||
|
f"page {i}: expected to land on page {i - 1} "
|
||||||
|
f"{key(starts[i - 1])}, got {key(got)}")
|
||||||
|
|
||||||
|
def test_result_lays_out_to_the_target(self, long_document):
|
||||||
|
"""Independent of the recorded chain: replaying the answer must reach P."""
|
||||||
|
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||||
|
starts = forward_chain(layouter)
|
||||||
|
|
||||||
|
for target in starts[1:]:
|
||||||
|
_, start = layouter.render_page_backward(target, 1.0)
|
||||||
|
_, end = layouter.render_page_forward(start, 1.0)
|
||||||
|
assert key(end) == key(target), (
|
||||||
|
f"a page starting at {key(start)} ends at {key(end)}, "
|
||||||
|
f"not at the requested {key(target)}")
|
||||||
|
|
||||||
|
def test_mid_paragraph_targets_are_reachable(self, long_document):
|
||||||
|
"""The specific regression: starts inside a block, not on its boundary."""
|
||||||
|
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||||
|
starts = forward_chain(layouter)
|
||||||
|
mid = [s for s in starts if s.word_index > 0]
|
||||||
|
assert mid, "this document should paginate mid-paragraph"
|
||||||
|
|
||||||
|
for target in mid:
|
||||||
|
_, got = layouter.render_page_backward(target, 1.0)
|
||||||
|
assert key(got) != (0, 0, 0) or key(target) == key(starts[1]), \
|
||||||
|
"backward navigation fell back to the document start"
|
||||||
|
|
||||||
|
|
||||||
|
class TestRoundTrip:
|
||||||
|
|
||||||
|
def test_forward_then_back_returns_to_the_same_place(self, long_document):
|
||||||
|
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||||
|
pos = RenderingPosition()
|
||||||
|
|
||||||
|
for _ in range(4):
|
||||||
|
_, nxt = layouter.render_page_forward(pos, 1.0)
|
||||||
|
_, back = layouter.render_page_backward(nxt, 1.0)
|
||||||
|
assert key(back) == key(pos), \
|
||||||
|
f"round trip drifted: {key(pos)} -> {key(nxt)} -> {key(back)}"
|
||||||
|
pos = nxt
|
||||||
|
|
||||||
|
|
||||||
|
class TestEdges:
|
||||||
|
|
||||||
|
def test_at_document_start_stays_there(self, long_document):
|
||||||
|
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||||
|
_, got = layouter.render_page_backward(RenderingPosition(), 1.0)
|
||||||
|
assert key(got) == (0, 0, 0)
|
||||||
|
|
||||||
|
def test_second_page_goes_back_to_the_first(self, long_document):
|
||||||
|
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||||
|
_, second = layouter.render_page_forward(RenderingPosition(), 1.0)
|
||||||
|
_, got = layouter.render_page_backward(second, 1.0)
|
||||||
|
assert key(got) == (0, 0, 0)
|
||||||
|
|
||||||
|
def test_empty_document_is_safe(self):
|
||||||
|
layouter = BidirectionalLayouter([], PageStyle(), PAGE_SIZE)
|
||||||
|
page, got = layouter.render_page_backward(RenderingPosition(), 1.0)
|
||||||
|
assert page is not None
|
||||||
|
assert key(got) == (0, 0, 0)
|
||||||
|
|
||||||
|
|
||||||
|
class TestCost:
|
||||||
|
|
||||||
|
def test_backward_is_not_wildly_more_expensive_than_forward(self, long_document):
|
||||||
|
"""The old path burned ten full layouts per call and still got it wrong."""
|
||||||
|
layouter = BidirectionalLayouter(long_document, PageStyle(), PAGE_SIZE)
|
||||||
|
starts = forward_chain(layouter)
|
||||||
|
|
||||||
|
calls = {"n": 0}
|
||||||
|
original = BidirectionalLayouter.render_page_forward
|
||||||
|
|
||||||
|
def counting(self, position, font_scale=1.0):
|
||||||
|
calls["n"] += 1
|
||||||
|
return original(self, position, font_scale)
|
||||||
|
|
||||||
|
BidirectionalLayouter.render_page_forward = counting
|
||||||
|
try:
|
||||||
|
worst = 0
|
||||||
|
for target in starts[1:]:
|
||||||
|
calls["n"] = 0
|
||||||
|
layouter.render_page_backward(target, 1.0)
|
||||||
|
worst = max(worst, calls["n"])
|
||||||
|
finally:
|
||||||
|
BidirectionalLayouter.render_page_forward = original
|
||||||
|
|
||||||
|
assert worst <= 10, f"backward navigation cost {worst} forward layouts"
|
||||||
@@ -0,0 +1,237 @@
|
|||||||
|
"""
|
||||||
|
Tests for the page caching layer.
|
||||||
|
|
||||||
|
Covers PageBuffer's LRU behaviour and BufferedPageRenderer's cache hits, plus
|
||||||
|
regressions for S12/R1/R2: the module must not start worker processes and must
|
||||||
|
not do blocking work in a finaliser.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import textwrap
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pyWebLayout.layout.page_buffer import PageBuffer, BufferedPageRenderer
|
||||||
|
from pyWebLayout.layout.ereader_layout import RenderingPosition
|
||||||
|
from pyWebLayout.abstract.block import Paragraph
|
||||||
|
from pyWebLayout.abstract.inline import Word
|
||||||
|
from pyWebLayout.style import Font
|
||||||
|
from pyWebLayout.style.page_style import PageStyle
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Fixtures
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_blocks():
|
||||||
|
"""A document long enough to paginate over several pages."""
|
||||||
|
font = Font()
|
||||||
|
blocks = []
|
||||||
|
for p in range(6):
|
||||||
|
para = Paragraph(style=font)
|
||||||
|
for w in range(120):
|
||||||
|
para.add_word(Word(f"p{p}w{w}", font))
|
||||||
|
blocks.append(para)
|
||||||
|
return blocks
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def renderer(sample_blocks):
|
||||||
|
return BufferedPageRenderer(sample_blocks, PageStyle(), buffer_size=3, page_size=(800, 600))
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# PageBuffer
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestPageBuffer:
|
||||||
|
def test_get_page_misses_when_empty(self):
|
||||||
|
buf = PageBuffer(buffer_size=3)
|
||||||
|
assert buf.get_page(RenderingPosition()) is None
|
||||||
|
|
||||||
|
def test_cache_page_round_trips(self):
|
||||||
|
buf = PageBuffer(buffer_size=3)
|
||||||
|
pos, nxt = RenderingPosition(block_index=0), RenderingPosition(block_index=1)
|
||||||
|
sentinel = object()
|
||||||
|
|
||||||
|
buf.cache_page(pos, sentinel, nxt)
|
||||||
|
|
||||||
|
assert buf.get_page(pos) is sentinel
|
||||||
|
assert buf.position_map[pos] == nxt
|
||||||
|
|
||||||
|
def test_lru_evicts_oldest_and_cleans_position_map(self):
|
||||||
|
buf = PageBuffer(buffer_size=2)
|
||||||
|
positions = [RenderingPosition(block_index=i) for i in range(4)]
|
||||||
|
for i, pos in enumerate(positions):
|
||||||
|
buf.cache_page(pos, object(), RenderingPosition(block_index=i + 1))
|
||||||
|
|
||||||
|
assert buf.get_page(positions[0]) is None, "oldest should have been evicted"
|
||||||
|
assert positions[0] not in buf.position_map, "position map must not leak evicted entries"
|
||||||
|
assert buf.get_page(positions[-1]) is not None
|
||||||
|
|
||||||
|
def test_get_page_refreshes_lru_order(self):
|
||||||
|
buf = PageBuffer(buffer_size=2)
|
||||||
|
a, b, c = (RenderingPosition(block_index=i) for i in range(3))
|
||||||
|
buf.cache_page(a, object())
|
||||||
|
buf.cache_page(b, object())
|
||||||
|
|
||||||
|
buf.get_page(a) # a becomes most recently used
|
||||||
|
buf.cache_page(c, object())
|
||||||
|
|
||||||
|
assert buf.get_page(a) is not None, "recently used entry should survive"
|
||||||
|
assert buf.get_page(b) is None, "least recently used entry should be evicted"
|
||||||
|
|
||||||
|
def test_backward_pages_land_in_the_backward_buffer(self):
|
||||||
|
buf = PageBuffer(buffer_size=3)
|
||||||
|
start, end = RenderingPosition(block_index=1), RenderingPosition(block_index=2)
|
||||||
|
|
||||||
|
buf.cache_page(start, object(), end, is_backward=True)
|
||||||
|
|
||||||
|
assert start in buf.backward_buffer
|
||||||
|
assert start not in buf.forward_buffer
|
||||||
|
assert buf.reverse_position_map[end] == start
|
||||||
|
|
||||||
|
def test_font_scale_change_invalidates(self):
|
||||||
|
buf = PageBuffer(buffer_size=3)
|
||||||
|
pos = RenderingPosition()
|
||||||
|
buf.cache_page(pos, object(), RenderingPosition(block_index=1))
|
||||||
|
|
||||||
|
buf.set_font_scale(1.5)
|
||||||
|
|
||||||
|
assert buf.get_page(pos) is None
|
||||||
|
|
||||||
|
def test_same_font_scale_keeps_cache(self):
|
||||||
|
buf = PageBuffer(buffer_size=3)
|
||||||
|
pos = RenderingPosition()
|
||||||
|
buf.cache_page(pos, object(), RenderingPosition(block_index=1))
|
||||||
|
|
||||||
|
buf.set_font_scale(1.0)
|
||||||
|
|
||||||
|
assert buf.get_page(pos) is not None
|
||||||
|
|
||||||
|
def test_shutdown_is_idempotent(self):
|
||||||
|
buf = PageBuffer(buffer_size=3)
|
||||||
|
buf.cache_page(RenderingPosition(), object())
|
||||||
|
|
||||||
|
buf.shutdown()
|
||||||
|
buf.shutdown()
|
||||||
|
|
||||||
|
assert buf.get_cache_stats()['forward_buffer_size'] == 0
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# BufferedPageRenderer
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestBufferedPageRenderer:
|
||||||
|
def test_render_page_returns_a_page_and_advances(self, renderer):
|
||||||
|
page, next_pos = renderer.render_page(RenderingPosition(), 1.0)
|
||||||
|
|
||||||
|
assert page is not None
|
||||||
|
assert next_pos != RenderingPosition()
|
||||||
|
|
||||||
|
def test_second_render_of_same_position_is_served_from_cache(self, renderer):
|
||||||
|
pos = RenderingPosition()
|
||||||
|
first, first_next = renderer.render_page(pos, 1.0)
|
||||||
|
second, second_next = renderer.render_page(pos, 1.0)
|
||||||
|
|
||||||
|
assert second is first, "identical page object means it came from the cache"
|
||||||
|
assert second_next == first_next
|
||||||
|
|
||||||
|
def test_font_scale_change_forces_a_re_render(self, renderer):
|
||||||
|
pos = RenderingPosition()
|
||||||
|
first, _ = renderer.render_page(pos, 1.0)
|
||||||
|
scaled, _ = renderer.render_page(pos, 1.5)
|
||||||
|
|
||||||
|
assert scaled is not first
|
||||||
|
|
||||||
|
def test_backward_render_round_trips_to_the_original_position(self, renderer):
|
||||||
|
start = RenderingPosition()
|
||||||
|
_, second_page_pos = renderer.render_page(start, 1.0)
|
||||||
|
|
||||||
|
_, back_to = renderer.render_page_backward(second_page_pos, 1.0)
|
||||||
|
|
||||||
|
assert back_to == start
|
||||||
|
|
||||||
|
def test_shutdown_clears_the_cache(self, renderer):
|
||||||
|
renderer.render_page(RenderingPosition(), 1.0)
|
||||||
|
renderer.shutdown()
|
||||||
|
|
||||||
|
assert renderer.get_cache_stats()['forward_buffer_size'] == 0
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# S12 / R1 / R2 regressions
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
class TestNoBackgroundProcesses:
|
||||||
|
"""
|
||||||
|
The process pool that used to live here never produced a usable page (a Page
|
||||||
|
holds a live PIL canvas and cannot be pickled), and on Python 3.14's
|
||||||
|
forkserver default it raised when driven from module-level code.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_module_declares_no_process_pool(self):
|
||||||
|
import pyWebLayout.layout.page_buffer as page_buffer
|
||||||
|
|
||||||
|
source = page_buffer.__file__
|
||||||
|
assert not hasattr(page_buffer, '_render_page_worker')
|
||||||
|
assert not hasattr(PageBuffer(), 'executor')
|
||||||
|
with open(source, encoding='utf-8') as fh:
|
||||||
|
body = fh.read().split('"""', 2)[-1] # skip the module docstring
|
||||||
|
assert 'ProcessPoolExecutor' not in body
|
||||||
|
assert 'pickle' not in body
|
||||||
|
|
||||||
|
def test_page_buffer_has_no_finaliser(self):
|
||||||
|
"""
|
||||||
|
PageBuffer.__del__ called executor.shutdown(wait=True), which deadlocked
|
||||||
|
the interpreter at exit. Cleanup must be explicit.
|
||||||
|
"""
|
||||||
|
assert '__del__' not in vars(PageBuffer)
|
||||||
|
|
||||||
|
def test_navigation_works_without_a_main_guard(self, tmp_path):
|
||||||
|
"""
|
||||||
|
R1: EreaderLayoutManager raised RuntimeError when used from module-level
|
||||||
|
script code, because submitting to a ProcessPoolExecutor under a
|
||||||
|
non-fork start method requires an `if __name__ == "__main__"` guard.
|
||||||
|
"""
|
||||||
|
script = textwrap.dedent(f"""
|
||||||
|
from pyWebLayout.io.readers.html_extraction import parse_html_string
|
||||||
|
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager
|
||||||
|
|
||||||
|
blocks = parse_html_string("<p>" + " ".join(f"w{{i}}" for i in range(2000)) + "</p>")
|
||||||
|
m = EreaderLayoutManager(blocks, page_size=(800, 600),
|
||||||
|
bookmarks_dir={str(tmp_path)!r})
|
||||||
|
m.get_current_page()
|
||||||
|
m.next_page()
|
||||||
|
m.previous_page()
|
||||||
|
m.shutdown()
|
||||||
|
print("OK")
|
||||||
|
""")
|
||||||
|
result = subprocess.run([sys.executable, "-c", script],
|
||||||
|
capture_output=True, text=True, timeout=120)
|
||||||
|
|
||||||
|
assert result.returncode == 0, result.stderr
|
||||||
|
assert "OK" in result.stdout
|
||||||
|
|
||||||
|
def test_interpreter_exits_without_explicit_shutdown(self, tmp_path):
|
||||||
|
"""
|
||||||
|
R2: a manager left to be finalised at exit must not hang. The timeout is
|
||||||
|
the assertion.
|
||||||
|
"""
|
||||||
|
script = textwrap.dedent(f"""
|
||||||
|
from pyWebLayout.io.readers.html_extraction import parse_html_string
|
||||||
|
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager
|
||||||
|
|
||||||
|
blocks = parse_html_string("<p>" + " ".join(f"w{{i}}" for i in range(500)) + "</p>")
|
||||||
|
m = EreaderLayoutManager(blocks, page_size=(800, 600),
|
||||||
|
bookmarks_dir={str(tmp_path)!r})
|
||||||
|
m.get_current_page()
|
||||||
|
# deliberately no shutdown() - rely on interpreter teardown
|
||||||
|
""")
|
||||||
|
result = subprocess.run([sys.executable, "-c", script],
|
||||||
|
capture_output=True, text=True, timeout=60)
|
||||||
|
|
||||||
|
assert result.returncode == 0, result.stderr
|
||||||
Reference in New Issue
Block a user