# Phase 1 · Module 7 — Tokenizer, rendering & parsing > Estimated time: 1 day · Pinned to commit `1aedee4` · Some exercises need a > build and the model; each is tagged. ## Where this fits F0.1 said the model only sees integer token IDs. Module 7 is **how the bytes become those integers and back** — and how DS4 builds the rich, structured prompts the model expects (chat turns, tool calls, thinking markers) out of that same token alphabet. The prompt is the model's only interface; this module is everything that happens at that interface. It is also one of the most security-relevant modules: parsing is where *structure* meets *untrusted data*, and the engineering here is the same engineering that prevents SQL injection and XSS. ## Read first - [`ds4.c` §"Tokenizer and Chat Prompt Encoding"](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L14250) — the tokenizer and the prompt builder. - The "Chat Template And Encoding" section of [`MODEL_CARD.md`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/MODEL_CARD.md) — the spec for DS4's chat format and the special tokens. - [`ds4.c` → `bpe_tokenize_text()`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L14708) — the actual BPE encoder. - [`ds4.c` → `encode_chat_prompt()`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L14840) — chat-prompt builder. ## The DS4 detail ### Three small problems, one section of code A real LLM engine has to solve three problems that look related but aren't: 1. **Tokenize plain text** — turn bytes into a sequence of integer token IDs. 2. **Render structured prompts** — build the integer stream a chat conversation requires, with special tokens marking roles, reasoning, and tool calls. 3. **Parse model output** — recognize structured output (tool calls, `` regions) coming back from the model as a stream of tokens. DS4 does all three within `ds4.c`'s "Tokenizer and Chat Prompt Encoding" section, plus a DSML parser (Module 9 / Module 10 walk through that one in context). Together they are *the* model interface. ### Tokenization: byte-level BPE in two stages The tokenizer is **GPT-2-style byte-level BPE** — same family as most modern LLMs. "Byte-level" means it operates on raw byte sequences (a roundtrip from unicode bytes through a printable alias), not on Unicode characters. This is what lets the tokenizer represent *any* input — code, emoji, non-Latin scripts, binary garbage — without an out-of-vocabulary failure. Encoding happens in two stages ([`bpe_tokenize_text`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L14708)): **Stage 1 — Pre-tokenization.** The text is first *split* into pieces by a regex-style pattern. DeepSeek's tokenizer declares `tokenizer.ggml.pre = "joyai-llm"`; DS4 reimplements that exact pre-split: - runs of digits up to 3 long (so `12345` becomes `123` + `45`); - runs of CJK characters; - runs of letters (possibly with leading whitespace); - runs of punctuation, whitespace, etc. Why pre-split at all? Because BPE merges are *learned* on already-split pieces, and the model's training pipeline used this exact split. If DS4 pre-split differently — even by one character class — the merges would no longer apply consistently, and tokenization would diverge from the model's training distribution. The pre-tokenizer is a strict spec, not an implementation detail. **Stage 2 — BPE merging.** Each pre-split piece goes to [`bpe_emit_piece`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L14533): 1. **Byte-encode**: map each raw byte to a printable Unicode codepoint via a fixed table (`gpt2_byte_to_codepoint`). This is the "byte-level" trick: instead of dealing with raw bytes (which include unprintable ones), every byte gets aliased to a printable character. The model and vocabulary are trained on those aliased forms. 2. **Greedy merge**: repeatedly find the *lowest-rank* adjacent pair (per the GGUF's `tokenizer.ggml.merges` table) and merge them. This is BPE's actual algorithm — keep merging adjacent pairs in priority order until nothing else merges. 3. **Look up** each remaining symbol in the vocabulary's `token → id` hash to emit token IDs. The model only ever sees those token IDs. Module 1 / F0.5's `DS4_N_VOCAB = 129280` is the size of that ID space. ### Special tokens — same alphabet, structural meaning Mixed in with the 129,280 ordinary tokens are a handful of **special tokens** that don't represent text — they represent *structure*. From F0.1, you met: | Token | Role | |---|---| | `<｜begin▁of▁sentence｜>` | start of every conversation | | `<｜end▁of▁sentence｜>` | end of an assistant turn — the EOS the sampler stops on | | `<｜User｜>` | starts a user turn | | `<｜Assistant｜>` | starts an assistant turn | | ``, `` | bracket the model's hidden reasoning | | `｜DSML｜` | marker inside DeepSeek-Style Markup Language (tool calls) | | `<｜latest_reminder｜>` | prefix for a reminder injected before the latest turn | These are *single tokens* — they each occupy one entry in the vocabulary, with one token ID. They are not parsed character by character. The tokenizer must recognize them as literals (when present in text it's tokenizing) and pass their IDs through verbatim, not BPE-merge their constituent characters. DS4's `tokenize_rendered_chat_vocab` ([`ds4.c`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L14901)) does exactly this: it scans for known special-token literal substrings, emits their token IDs, and BPE-encodes only the spans between them. The decoder ([`ds4_token_text`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L15046)) is the inverse: known special-token IDs are output as their literal text; ordinary IDs are reversed through the byte-encoding to recover raw bytes. ### Building a chat prompt A chat prompt is not a string the user types — it is a *constructed* sequence of special tokens and content. DS4's [`encode_chat_prompt`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L14840) builds it in a fixed shape that matches DeepSeek's reference renderer (the official `encoding_dsv4.py` script, per Module 1's Provenance section): ``` <｜begin▁of▁sentence｜> [optional Think Max prefix] <- only in Think Max mode [optional system text] <｜User｜> the user message <｜Assistant｜> <- non-thinking mode opens with instead ``` For a multi-turn conversation, each old turn is rendered fully (including its `...` reasoning and final `<｜end▁of▁sentence｜>`), and the new turn is appended in the same shape. Two things about this are load-bearing: - **Non-thinking mode** opens an assistant turn with an immediate `` rather than ``. That single token tells the model "skip hidden reasoning and produce the visible answer." It's how thinking mode is controlled: the prompt itself, by which token it puts right after `<｜Assistant｜>`, instructs the model what to do. - **Schema and system text positioning** matters for the KV cache (Module 6). DS4's server deliberately renders tool schemas *before* the system text and user content, so the *dynamic* part (the current user message) is at the *tail*. Cache trimming and prefix matching then work cleanly: the stable prefix (system + schema) stays intact, and only the tail varies between requests. ### DSML — DeepSeek-Style Markup Language for tool calls When the model wants to call a tool, it doesn't output JSON. It outputs **DSML** — a small markup language DeepSeek defined for this purpose. A tool call looks roughly like: ``` ｜DSML｜tool_calls invoke name="read" parameter name="path" string="true">README.md /invoke /tool_calls｜DSML｜ ``` (Stylized; the actual delimiters use the `｜DSML｜` special-token marker around the structural words.) This is a *generated* format — the model emits DSML when it decides to call a tool, character by character (well, token by token). A parser at the consumer side (`ds4-server` or `ds4-agent`) recognizes the DSML block as it streams, extracts the tool name and parameters, and dispatches the call. Module 9 covers the server's parser in context; Module 10 covers the agent's. The shape is the same: a small recursive-descent parser walking the token stream, extracting attributes, decoding `string="true"` parameter bodies (raw strings, unescaped) vs `string="false"` parameter bodies (JSON values). When tool results come *back* into the model, the chat renderer wraps them in a structural marker: `...result body...`. This brings the next subsection. ### Escaping ``: parsing meets untrusted data A tool result body is, in general, arbitrary bytes — file contents, shell output, search results, whatever the tool produced. The renderer's job is to put those bytes between `` and `` so the model can read them. But what if the bytes themselves contain `` somewhere in the middle? Without escaping, the model would see the wrapper close prematurely, and whatever followed would be interpreted as part of the chat flow rather than as tool output. DS4 handles this by escaping the *exact closing sentinel* `` in the body — replacing it with a form the model recognizes as data, not structure. Literal `<`, `>`, and `&` characters from the body are preserved unchanged; only the precise closing wrapper bytes are altered. This is the minimal-intrusion form of escaping: change as little as possible while making the boundary unambiguous. This is the *same problem* as SQL injection, XSS, command injection, and every related class of bug in the security canon. The shape is universal: **structure meets data, and unless the boundary is encoded so data cannot masquerade as structure, the data can break out and become structure.** A SQL parameter that contains `'; DROP TABLE users; --` is the same hazard as a tool result containing ``; the fix is the same in both domains — escape the structural delimiters. It is worth pausing on: every byte that flows from a tool into the model is *untrusted*. A file the model asked to read might have been written by an adversary. A web page a search tool returned might be hostile. The escaping is not paranoia — it is the only thing standing between "the model read a file" and "the model executed text that pretended to be its own next turn." ## Zoom out — transformer The prompt is the model's **entire interface**. Everything the model can be told — task, context, tools available, role, thinking mode, prior turns — arrives as one stream of token IDs. Module 7 is the engineering layer that *builds* that stream from human-friendly inputs (chat messages, JSON tool schemas, file contents) and *parses* the stream coming back (text, thinking regions, tool calls). You will see the same boundary in any LLM serving system: a tokenizer + template + parser triple, sometimes scattered across multiple files, sometimes (as in DS4) concentrated in one section. The deep idea transformer-side: **format-shape your prompts to match what the model was trained to expect**. DeepSeek V4 was trained with this exact special-token layout; deviating even slightly — a missing `` token, a mis-rendered `<｜User｜>` — produces measurably worse outputs. Hence Module 1's Provenance discipline: DS4's prompt format was matched *byte-for-byte* to the official `encoding_dsv4.py` script. ## Zoom out — CS **Parsing and formal languages.** This whole module is *parsing*, in two flavors: - *Generative* (rendering): given structured input (chat messages, tool results), produce a string. That is a *code generator* — the same general idea as a templating engine, a serializer, or a code-gen tool. The trick is making the output *unambiguous to parse* on the other end. (Reference: any compilers text on pretty-printing; "Crafting Interpreters" Part I.) - *Consumptive* (parsing): given a stream of tokens, recognize structure. DS4's DSML parser is a small **recursive-descent parser** — a parser written by hand as a set of mutually recursive functions. (Reference: "Crafting Interpreters" Parts I–II for the canonical introduction; any compilers text — Aho/Sethi/Ullman, Appel — for the theory.) BPE itself is *also* a parser, of a peculiar kind. It is a **greedy priority-merge algorithm**: at every step pick the highest-priority adjacent pair from a finite rule table and merge. It is not regular, not context-free, not in any classical grammar class; it is just a fixed iterative procedure trained on data. The output happens to be deterministic given the rule table, and *that* determinism is what every other part of the system depends on. (For learners interested in the BPE math: the original Sennrich, Haddow, Birch paper "Neural Machine Translation of Rare Words with Subword Units" is short and readable.) **Escaping and injection safety.** The `` problem is one instance of a class: | Domain | Structural delimiter | Injection if not escaped | |---|---|---| | SQL | `'`, `;`, `--` | SQL injection | | HTML | `<`, `>`, `&` | XSS | | Shell | `;`, `|`, backticks | command injection | | URL | `&`, `=`, `?`, `#` | parameter smuggling | | LLM tool result | `` | prompt injection via structure break | The pattern is universal: **wherever structure meets data, escape the structural delimiters or expect data to become structure.** DS4's `` escaping is in the same family of defenses as parameterized SQL queries and React's automatic HTML escaping. The lesson transfers directly to anything you build that mixes structure and content. (To study this properly: the OWASP injection cheat sheets; any web-security text on input handling.) **A second-order observation:** the *parser's job* is to refuse to be tricked. DSML parsers (server, agent) only recognize tool calls that start at a structural position, never inside string parameters; the renderer escapes the one closing sentinel that could otherwise reach a structural position from inside a string. The two halves cooperate: the renderer produces output the parser can never confuse, and the parser refuses to find structure where the renderer would never have put it. Neither half is sufficient on its own. ## Exercise **Part A — paper, no build needed.** 1. **Pre-tokenization matters.** Suppose two implementations of a BPE tokenizer use *different* pre-split rules (one splits on whitespace only; the other splits on digit/letter/punctuation transitions). Even with the exact same merge table, they will produce different token sequences for most text. Why? And which is more dangerous — using a different pre-split than the model was trained with, or using a different merge table? 2. **The `` problem, generalized.** Imagine a search tool returns a web page that contains the literal text `` inside a code block on the page. Without escaping, what does the model see? With DS4's escaping, what does the model see instead? **Part B — [needs build + model].** 3. Run `./ds4 --dump-tokens -p "Hello, world!"`. Read the output — each line shows one token's ID and its bytes. Identify the special tokens, if any. 4. Run `./ds4 --dump-tokens -p "<｜User｜>hi"` and confirm the literal `<｜User｜>` becomes one token, not many. ## Deep check DS4's tokenizer is byte-level BPE with a `joyai-llm` pre-tokenizer. The pre-tokenizer is reimplemented in C (not loaded from the GGUF metadata). Why must the C reimplementation match DeepSeek's reference behaviour **exactly** — not "mostly" — and what would happen if a single character class were misclassified (say, a CJK character treated as ordinary letter)?

Answer

Because BPE merges are *learned* on the assumption of a specific pre-split. A merge like "th + e → the" only exists in the merge table because the training pipeline pre-split text into pieces where `th` and `e` could become neighbors. If DS4 pre-splits differently — sticking a character into a different piece, or splitting where the training pipeline didn't — then the adjacency the merge depends on no longer occurs, and the merge never fires. Concretely with a CJK misclassification: a CJK run that DeepSeek's pre-tokenizer treats as one piece (so merges apply within it) might, under a buggy reimplementation, get split into single characters or get glued to adjacent Latin letters. The resulting token IDs would differ from what the training pipeline produced. The model has *never seen those token IDs* in that distribution; outputs would be measurably worse, and in pathological cases the model could fail catastrophically — gibberish or silent mistranslations on inputs containing the affected character class. This is also why Module 1's Provenance discipline matters so much: "validated against official logits" means *the same input bytes produce the same token IDs produce the same logits* — token-for-token. A pre-tokenizer bug breaks the very first link in that chain, and every downstream check fails on the same input. The pre-tokenizer is the contract between DS4 and DeepSeek's training pipeline; "mostly right" is just "subtly wrong on inputs nobody tested."

## Transferable takeaway The prompt is the model's entire interface. Two engineering lessons follow: **(1)** *Match the training pipeline byte-for-byte.* Tokenizers, special tokens, and chat formats are not approximations — they are contracts with what the model was trained to expect, and small deviations have measurable costs. **(2)** *Treat structure-meets-data as a security boundary.* Every place a parser meets untrusted bytes, there is an injection class waiting; the cure is unambiguous escaping of the structural delimiters, not vigilance or pattern-matching. DS4's `` defense is the same shape as parameterized SQL queries — recognize that family and reach for it whenever you put untrusted text into a structured stream. → Next: Module 8 — The CLI *(pending)*.

13 Tokenizer And Parsing