---
title: "architecture"
description: "The tokenizer, reclassifier pipeline, renderer and package structure."
source: https://twinkleplop.pngwn.at/docs/architecture
---
# architecture

The tokenizer, reclassifier pipeline, renderer and package structure.

> This page is a work in progress.

Twinkleplop uses a shared tokenizer with a separate grammar for each language. Grammars define the rules the tokenizer uses to recognise tokens.

┌─────────────────────────────────────────────────────┐ │ language package grammar + reclassifiers │ ├─────────────────────────────────────────────────────┤ │ @twinkleplop/core compiler · tokenizer · │ │ reclassifier · generator │ └─────────────────────────────────────────────────────┘ language-specific data language-agnostic runtime

## design principles

▲ Performance first

The tokenizer scans characters and stores tokens in typed arrays to reduce processing time and memory use.

◆ Separation of concerns

The runtime, language grammars and reclassifier pipeline are separate. Embedded languages are handled after tokenization.

● Declarative grammars

Grammars use a small set of helper functions. The compiler converts them into lookup tables for the tokenizer.

[read →](https://twinkleplop.pngwn.at/docs/grammar)

◐ User-controlled fidelity

The fidelity option controls which passes run to identify functions, classes and other specific token types.

[read →](https://twinkleplop.pngwn.at/docs/fidelity)

## state machine

The tokenizer uses a state machine with a stack. The stack tracks nested constructs, such as a template literal inside another template literal's interpolation.

Grammars describe states and rules. The compiler turns them into flat typed arrays: a transition table indexed by computed integers, dense character maps for ASCII classification, range lists for everything above 127, and a keyword set checked after scanning an identifier span. The tokenizer uses numeric indexes to look up transitions.

> Grammars are JavaScript or TypeScript modules. Their rules can contain symbols and tagged objects, so they cannot be serialised as JSON.

## reclassification

Reclassifiers handle two tasks after tokenization:

- **Multi-token lookahead.** Recognising that `foo` in `const foo = () => {}` is a function would need chained probe states and balanced-paren matching at every `=` in the file.
- **Cross-language embedding.** Handing `<script>` content to a JavaScript tokenizer would otherwise mean duplicating an entire sub-language into the host grammar.

These operations run in a pipeline between the tokenizer and renderer. You can use the tokenizer directly if you only need the grammar's tokens.

The fidelity option selects which reclassifier passes run. Most passes propose token types, and the pipeline uses precedence to resolve competing proposals. See [reclassifiers](https://twinkleplop.pngwn.at/docs/reclassifier).

## HTML rendering

The renderer builds an HTML string in a single pass. This avoids the cost of creating an intermediate tree.

Use overlays and line or token hooks to add classes and attributes to the generated elements. These options cannot change the element structure. See [render options](https://twinkleplop.pngwn.at/docs/render_options).

## package layout

A pnpm workspace of `lib/*` and `languages/*`.

```text title="workspace"
lib/core          runtime, compiler, DSL, reclassifier, generator
lib/annotation    marker plugins, shiki-notation compatibility
lib/markdown-*    rehype / remark / markdown-it over one core
lib/theme-*       generated stylesheets
lib/twoslash*     twoslash integrations
languages/*       one package per grammar
```

Every language is its own package so consumers install only what they use, and grammars can be versioned independently of the runtime. Themes are generated CSS stylesheets.