pyWebLayout/ARCHITECTURE.md

# pyWebLayout Architecture

This document describes how pyWebLayout is organised: the layers, what lives in each,
and the rules that govern how they may depend on one another.

## Overview

The library turns markup (HTML/EPUB) into rendered images. That pipeline is split into
layers with a strict dependency direction:

```
    io/readers/          parse markup into document structure
         ↓
    abstract/            what the document says     ─┐
         ↓                                           ├─ built on core/ + style/
    layout/              decide where things go      │
         ↓                                           │
    concrete/            what the pixels are        ─┘
         ↓
    PIL.Image
```

The central distinction is **abstract vs concrete**:

- **Abstract** — the logical content and structure of a document. A `Paragraph` knows
  it contains words in an order; it does not know how wide they are.
- **Concrete** — the spatial realisation of that content. A `Line` knows exactly which
  glyphs sit at which pixel offsets on a specific canvas.

One abstract document produces many concrete renderings: different page sizes, font
scales, and font families all re-run the concrete layer over unchanged abstract content.
That is what makes ereader features like live font scaling and reflow possible.

## Layers

### `core/` — shared foundations

Everything else is built on these. `core/` depends on nothing but `style/`.

**`core/base.py`** defines the contracts that make a class abstract or concrete:

| Contract | Kind | Meaning |
|---|---|---|
| `Renderable` | ABC | Has `render()`; produces or draws visual output |
| `Queriable` | ABC | Has `in_object(point)`; can be hit-tested |
| `Layoutable` | ABC | Has `layout()`; arranges its own contents |
| `Interactable` | ABC | Holds a callback invoked on interaction |
| `Geometric` | mixin | `origin` and `size` as numpy arrays |
| `Hierarchical` | mixin | `parent` back-reference |
| `Styleable` | mixin | Carries a style object |
| `FontRegistry` | mixin | Deduplicates `Font` instances across a document |
| `MetadataContainer` | mixin | Key/value metadata with typed accessors |
| `BlockContainer` | mixin | Holds child `Block`s |
| `ContainerAware` | mixin | Knows the container it was added to |

The useful shorthand: **`Renderable` and `Geometric` are the concrete markers.** An
abstract class that acquires either has crossed the line.

**`core/query.py`** — `QueryResult` and `SelectionRange`: the result types for mapping a
pixel back to content (what was clicked, what text is selected, where it is in the
document).

**`core/highlight.py`** — `Highlight`, `HighlightColor`, `HighlightManager`. Highlights
store both pixel bounds (for drawing) and semantic bounds (word indices, for surviving
a font change).

**`core/cache.py`** — `UsageCache` / `SizedUsageCache`, bounded caches keyed by
`(font, string)` for text measurement and glyph rasterisation. Eviction is by usage
count with periodic aging, not LRU; the module docstring explains why at length. This
exists because a single page issues thousands of measurements over fewer than a
thousand distinct pairs, on hardware where an unbounded cache is not affordable.

**`core/callback_registry.py`** — `CallbackRegistry`, which owns the interactive
elements registered on a page and dispatches to them.

### `style/` — semantic styling and its resolution

This package is itself an instance of the abstract/concrete split, applied to styling:

- **`abstract_style.py`** — `AbstractStyle` (frozen dataclass, hashable) captures
  *intent*: `FontFamily.SERIF`, `FontSize.LARGE`, `color="black"`. `AbstractStyleRegistry`
  interns them so a document holds one instance per distinct style.
- **`concrete_style.py`** — `RenderingContext` (user preferences, DPI, accessibility
  flags, available space) plus `StyleResolver`, which maps `AbstractStyle` +
  `RenderingContext` → `ConcreteStyle` (a resolved font path, a pixel size, an RGB
  tuple). `ConcreteStyleRegistry` caches the results.
- **`fonts.py`** — `Font`, the loaded PIL font plus its rendering attributes, and the
  bundled DejaVu families (`BundledFont`).
- **`page_style.py`** — `PageStyle`: borders, padding, background, default alignment.
- **`alignment.py`** — the `Alignment` enum.

`AbstractStyle` → `StyleResolver` → `ConcreteStyle` is the mechanism by which the same
document renders differently for different readers.

### `abstract/` — document content and structure

Layout-agnostic representations of what the document contains.

**`abstract/block.py`** — block-level content, all deriving from `Block`:

`Paragraph`, `Heading` (with `HeadingLevel`), `Quote`, `CodeBlock`, `HList` (with
`ListStyle`) and `ListItem`, `Table` / `TableRow` / `TableCell`, `Image` and
`LinkedImage`, `HorizontalRule`, `PageBreak`.

**`abstract/inline.py`** — `Word`, `FormattedSpan`, `LinkedWord`, `LineBreak`. `Word`
holds text, a style, and previous/next links into the document's word sequence. It can
report whether it is a hyphenation candidate (`possible_hyphenation(language)`), but it
does not perform the split — that is a measurement decision and belongs to `Line`.

**`abstract/document.py`** — `Document`, `Chapter`, `Book`, `MetadataType`. `Book` is
the EPUB-shaped `Document` with chapters and a table of contents.

**`abstract/functional.py`** — `Link`, `Button`, `Form`, `FormField`, `LinkType`,
`FormFieldType`.

**`abstract/interactive_image.py`** — `InteractiveImage`, an `Image` that is also
`Interactable` and `Queriable`.

### `concrete/` — spatial realisation

Objects that know their position and size and can draw themselves onto a canvas.

**`concrete/text.py`** — the heart of text layout:
- `Text` — one renderable fragment (a whole word, or a hyphenated part). Requires an
  `ImageDraw.Draw` at construction so it can measure itself immediately.
- `Line` — a sequence of `Text` objects with resolved spacing and a baseline.
  `Line.add_word()` is where a `Word` becomes one or two `Text` objects: it measures,
  and if the word overflows it tries dictionary hyphenation, then brute-force splitting,
  before rejecting the word.
- `AlignmentHandler` and its subclasses (`LeftAlignmentHandler`,
  `CenterRightAlignmentHandler`, `JustifyAlignmentHandler`) — the strategy objects that
  turn a set of measured fragments plus an available width into concrete spacing and a
  start position.
- Cache management: `configure_text_caches`, `clear_text_caches`, `text_cache_stats`,
  `prewarm_text_caches`.

**`concrete/page.py`** — `Page`: a fixed-size canvas holding `Renderable` children, with
a content rectangle derived from its `PageStyle`, `can_fit_line()` for the layouter to
test against, `render()` returning a `PIL.Image`, and `query_point()` / `query_range()`
for hit-testing.

**`concrete/box.py`** — `Box`, the `Geometric` + `Renderable` + `Queriable` base for
positioned drawable objects.

**`concrete/dynamic_page.py`** — `DynamicPage` (a `Page` subclass) and `SizeConstraints`.
Adds a two-phase measure-then-layout protocol, so a container such as a table can learn
its content's intrinsic size before committing to a size allocation.

**`concrete/image.py`** — `RenderableImage`.

**`concrete/table.py`** — `TableRenderer`, `TableRowRenderer`, `TableCellRenderer`,
`TableStyle`. Cells host their own nested `Page`, which is why `Page` accepts a
non-zero `origin`.

**`concrete/functional.py`** — `LinkText`, `ButtonText`, `FormFieldText`: `Text`
subclasses that are also `Interactable`.

**`concrete/interaction_handler.py`** — `InteractionHandler`, `InteractionStateManager`:
routing taps and presses to registered elements and tracking pressed/released state.

### `layout/` — the abstract → concrete transformation

This is the package the pipeline diagram calls the layout engine.

**`layout/document_layouter.py`** — a set of layouter functions, one per content kind,
all sharing a signature shape of *(abstract element, target `Page`) → did it fit*:

```python
paragraph_layouter(paragraph, page, start_word=0, pretext=None, alignment_override=None)
    -> (complete: bool, failed_word_index: int | None, remaining_pretext: Text | None)

image_layouter(image, page, max_width=None, max_height=None)   -> bool
table_layouter(table, page, style=None)                        -> bool
pagebreak_layouter(page_break, page)                           -> bool
button_layouter(button, page, font=None, padding=...)          -> (bool, str)
form_field_layouter(field, page, font=None, ...)               -> ...
form_layouter(form, page, font=None, field_spacing=10)         -> (bool, list[str])
```

They **append to a page**, they do not return a list of lines. The three-part return of
`paragraph_layouter` is what makes pagination resumable: when a paragraph runs off the
bottom of a page, the caller learns which word failed and whether a hyphenated fragment
is pending, and can continue on the next page from exactly there.

`DocumentLayouter` wraps a `Page` and dispatches over a list of abstract elements by
type, holding the `ConcreteStyleRegistry` for the run.

**`layout/ereader_layout.py`** — the paginated reading model:
- `RenderingPosition` — a serialisable cursor expressed in *abstract* coordinates
  (chapter, block, word, table cell, list item, pending pretext). Because it names
  document structure rather than pixels, it survives font-size and page-size changes.
- `BidirectionalLayouter` — renders a page forward or backward from a position,
  returning `(Page, next_position)`.
- `ChapterNavigator` / `ChapterInfo` — a table of contents built from heading structure.
- `FontScaler`, `FontFamilyOverride` — apply scale and family changes to blocks at
  layout time without mutating the abstract document.

**`layout/page_buffer.py`** — `PageBuffer` (LRU caches of rendered pages in both
directions, plus position maps) and `BufferedPageRenderer` (background rendering).

**`layout/ereader_manager.py`** — `EreaderLayoutManager`, the top-level application
interface (page turns, font changes, chapter jumps, progress), and `BookmarkManager`
for persisting bookmarks and the last reading position.

**`layout/table_optimizer.py`** — column width allocation for tables.

### `io/readers/` — parsing

**`html_extraction.py`** — `parse_html_string(html, base_font=None, document=None,
base_path=None) -> List[Block]`. Built from a `StyleContext` (a `NamedTuple` threaded
down the tree, carrying the inherited font and styling) and a table of per-tag handlers
(`paragraph_handler`, `heading_handler`, `table_handler`, …). This is the only place
that knows about HTML.

**`epub_reader.py`** — `EPUBReader` and `read_epub(path) -> Book`: container/OPF
parsing, spine and manifest, table of contents, cover handling, and image processing
(with an e-ink processor available by default).

## Dependency rules

The direction of dependency is what keeps the split honest:

```
io/readers  →  abstract  →  core, style
layout      →  abstract, concrete, core, style
concrete    →  abstract, core, style
abstract    →  core, style          ← must NOT import concrete
```

`abstract/` importing from `concrete/` is the violation to watch for. `concrete/`
importing from `abstract/` is expected and correct: a `Text` may point back at the
`Word` it came from, and a table cell renderer reads its abstract `TableCell`.

**Where the abstract layer touches rendering today, and why:**

- Abstract classes are constructed with `Font` objects, which carry a pixel size and a
  loaded font file. This is a deliberate compromise for parsing performance (HTML
  styling resolves to a `Font` once, at parse time) but it does mean the abstract layer
  is not fully rendering-independent. `AbstractStyle` is the intended replacement, and
  `Word` already accepts either.
- `Word.concrete` / `Word.add_concete()` ([inline.py:42](pyWebLayout/abstract/inline.py#L42),
  [:134](pyWebLayout/abstract/inline.py#L134)) is a back-reference to the `Text` objects
  a word became. **It is currently written and never read**, and it cannot correctly
  model the relationship anyway: one `Word` becomes many `Text`s across re-layouts, and
  a single slot only remembers the most recent. Treat it as vestigial. When a
  word→pixels mapping is genuinely needed, build it on `Text`'s `source` back-reference
  (concrete pointing at abstract), which is the safe direction — note it is currently
  stored as `_source` with no public accessor, and is itself unread today.

## Worked example

```python
from pyWebLayout.io.readers.html_extraction import parse_html_string
from pyWebLayout.concrete.page import Page
from pyWebLayout.layout.document_layouter import DocumentLayouter

# 1. Parse: markup -> abstract blocks
blocks = parse_html_string("<h1>Chapter One</h1><p>It was a dark and stormy night.</p>")
# [Heading, Paragraph]

# 2. Lay out: abstract blocks -> concrete children appended to a Page
page = Page(size=(400, 600))
DocumentLayouter(page).layout_document(blocks)

# 3. Render: Page -> PIL.Image
image = page.render()

# 4. Query: pixel -> content
result = page.query_point((50, 40))
print(result.object_type, result.text)   # e.g. "text" "Chapter"
```

Inspecting the intermediate concrete objects:

```python
from pyWebLayout.concrete.text import Line

for line in (c for c in page.children if isinstance(c, Line)):
    print([t.text for t in line.text_objects])
# ['Chapter', 'One']
# ['It', 'was', 'a', 'dark', 'and', 'stormy', 'night.']
```

For paginated reading, drive `EreaderLayoutManager` instead of building pages by hand:

```python
from pyWebLayout.layout.ereader_manager import EreaderLayoutManager

manager = EreaderLayoutManager(blocks, page_size=(800, 600), document_id="my-book")
page = manager.get_current_page()
page = manager.next_page()
manager.set_font_scale(1.25)   # re-lays out from the same RenderingPosition
```

## Design principles

**1. Abstract content is not mutated by layout.** Font scaling and family overrides
produce scaled copies at layout time rather than editing the document. This is what
allows the same `blocks` list to back a buffer of pages at several font sizes.

**2. Positions are expressed in abstract coordinates.** `RenderingPosition` names a
chapter, block, and word — never a pixel. A reader who changes font size stays on the
same sentence.

**3. Layouters report partial success.** Every layouter returns whether it fit, and the
text layouter also returns where it stopped. Pagination is built from this rather than
from a separate page-breaking pass.

**4. One abstract object may become many concrete ones.** One `Word` → one or two `Text`
fragments; one `Paragraph` → many `Line`s across many `Page`s; one `Document` → an
unbounded sequence of `Page`s. Any API that assumes a one-to-one mapping will be wrong
at a hyphen or a page boundary.

**5. Measurement is cached, not avoided.** Text width and glyph rasterisation are the
hot path. `core/cache.py` bounds their cost; `prewarm_text_caches` front-loads it for a
known document.

## Anti-patterns

**Concrete state stored on abstract objects.** An abstract object that remembers its
rendered width, position, or `Text` objects is wrong at the second rendering. If a
back-reference is needed, point from concrete to abstract.

```python
# WRONG
class Word:
    def __init__(self, text):
        self.rendered_width = None    # invalidated by any font change

# RIGHT
text = Text(word.text, font, draw, source=word)   # concrete knows its origin
```

**Treating `Word` as renderable.** Words are abstract; only `Text` draws. There is no
`renderable_words` anywhere in the codebase, and there should not be.

**Assuming a layouter returns lines.** `paragraph_layouter` appends to a page and
reports what did not fit. Code that expects `List[Line]` back is working from an
outdated model.

## Summary

- **`core/`** — contracts and shared machinery
- **`style/`** — semantic style, and its resolution to concrete rendering parameters
- **`abstract/`** — what the document says
- **`concrete/`** — where the pixels go
- **`layout/`** — the transformation between them, and pagination on top of it
- **`io/readers/`** — markup in