# Phase 0 · F0.1 — Tokens & Language Models > Foundations · Estimated time: ½ day · Needs no build or model. Phase 0 builds the vocabulary the rest of the course assumes. Read these five files before Module 1. They teach general ideas about language models — but every example is grounded in DS4's real numbers, so the foundation and the codebase connect from the start. ## What you'll understand After this file you will know: - what a language model fundamentally *does*; - what a **token** is, and why it is neither a word nor a letter; - what a **tokenizer** and a **vocabulary** are; - the single most important fact for reading any LLM code: **the model only ever sees integers, never text**; - vocabulary you will own: *token, token ID, tokenizer, vocabulary, BPE, special token*. ## A language model is a next-token predictor Strip away the mystique. At its core, a language model does exactly one thing: > **Given some text, predict what comes next.** That is the whole job. Chat, code, reasoning, translation — all of it is this one operation, repeated. Give the model: ``` The capital of France is ``` and it predicts that what most plausibly comes next is ` Paris`. Append that, feed the whole thing back in, and it predicts the next piece again. A long answer is just "predict what comes next," run hundreds or thousands of times in a loop. Hold onto this. Everything else in the course is *machinery in service of doing that one prediction well and fast.* ## The unit of prediction is a token "Predict what comes next" — next *what*? You might guess "next letter" or "next word." Both are wrong, and the reasons matter. - **Not letters.** Too fine-grained. Predicting one letter at a time carries almost no meaning per step, and it makes every sequence punishingly long. - **Not words.** Too coarse. There is no finite list of "all words" — names, typos, code identifiers, numbers, other languages, emoji. A model needs a *fixed, finite* set of things it can predict. So language models use a unit in between: the **token**. A token is a *chunk* of text — usually a common word or a word-piece. Roughly: - common words become a single token: `the`, ` France`, ` is`; - rarer or longer words split into several: `tokenizer` → `token` + `izer`; - a leading space is normally part of the token (` France`, not `France`) — that is how the model knows where words begin. So the text `The capital of France is` is, to the model, a sequence of **5 tokens**, not 24 letters and not 5 words by coincidence — 5 *chunks the tokenizer chose*. ## The tokenizer: text becomes integers The **tokenizer** is the component that converts between human text and the model's units. It works in both directions: - **encode** — text → tokens → token IDs; - **decode** — token IDs → tokens → text. A **token ID** is just an integer: each token has a number. And here is the fact to burn in: > **The model never sees text. It sees a list of integers.** Input is integers. Output is an integer (the predicted next token). Text exists only at the very edges — encoded on the way in, decoded on the way out: ``` "The capital of France is" <-- text (what humans see) | v TOKENIZER (encode) ["The"][" capital"][" of"][" France"][" is"] <-- tokens (chunks) | v look up each token's ID in the vocabulary [ 504, 6864, 295, 6181, 349 ] <-- token IDs (integers) | v [ M O D E L ] works ENTIRELY in token IDs | v [ 12001 ] <-- predicted next token ID | v TOKENIZER (decode) " Paris" ``` *(The ID numbers above are illustrative — the real IDs come from DS4's vocabulary file. The shape is what matters: integers in, an integer out.)* This is why, when you debug an LLM, you constantly convert between text and token IDs — they are two views of the same thing, and bugs love to hide in the gap between them. ## The vocabulary: a fixed dictionary of every token Every token the model can possibly read or write lives in one fixed list: the **vocabulary**. A token's ID is simply its position in that list. The vocabulary is decided once, before training, and never changes. **In DS4, the vocabulary has exactly 129,280 tokens.** That is the constant `DS4_N_VOCAB` in the engine. The model's *entire universe* of expressible text — every prompt, every answer, in every language — is sequences drawn from those 129,280 pieces. The very next file, F0.2, shows the model producing exactly that: one score for each of these 129,280 tokens, every time it predicts what comes next. ## How the tokens are chosen: BPE The tokens are not hand-picked by a human. They are *learned* from a large pile of text by an algorithm called **BPE — Byte-Pair Encoding**. The idea, simplified: 1. Start with the most basic units (individual bytes). 2. Find the most *frequent* adjacent pair and merge it into a new, single token. 3. Repeat thousands of times. Frequent sequences (` the`, ` of`, common word stems) get merged early and end up as whole tokens; rare sequences never get merged and stay fragmented. That is exactly why `cat` is one token but `antidisestablishmentarianism` is several — the common word was frequent enough to merge into one unit; the rare word's pieces never were. DS4 uses **byte-level BPE**; you do not need its mechanics yet (Module 7 covers the tokenizer in depth) — just the intuition that the vocabulary is *learned from frequency*. ## Special tokens A few entries in the vocabulary are not ordinary text at all — they are **special tokens**: structural markers. They have IDs like any token, but they *mean* something rather than spelling something. DS4's special tokens include: - `<｜User｜>` and `<｜Assistant｜>` — mark whose turn it is in a chat; - `` and `` — mark the model's hidden reasoning; - an end-of-sequence token — marks "I am done." This is how a plain next-token predictor can hold a *conversation*: the chat structure (who said what) is encoded as special tokens mixed into the same integer stream. Module 7 shows exactly how DS4 builds that stream. ## Grounded in DS4 Where these ideas live in this repository: - **`DS4_N_VOCAB = 129280`** — the vocabulary size, a fixed constant in [`ds4.c`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L89). It sits in an `enum` of fixed model constants that Module 1 examines closely. - DS4's tokenizer is a **byte-level BPE** tokenizer. It is built at load time from token and merge tables stored inside the GGUF model file. - The command **`./ds4 --dump-tokens -p "your text here"`** runs *only* the tokenizer: it converts your text to tokens, prints them, and exits before any inference. It is the direct way to watch encoding happen — you will use it in Module 7. (It needs a build and the model file, since the tokenizer's tables live inside the GGUF, so it is a **[needs model]** exercise.) - Special tokens such as `<｜User｜>` are recognized by the tokenizer as single tokens, not spelled out letter by letter. ## Check yourself 1. The word `cat` is almost certainly **one** token. The word `antidisestablishmentarianism` is almost certainly **several**. Why? 2. The model is given the text `The capital of France is`. In what *form* does the model actually receive it — and how many things is it, roughly? 3. DS4's vocabulary is 129,280 tokens. When the model "predicts the next token," how many candidates is it choosing between?

Answers

1. BPE merges *frequent* sequences into single tokens. `cat` is a common word — frequent enough that its letters were merged into one token early. `antidisestablishmentarianism` is rare — its byte sequences were never frequent enough to merge, so it stays as several word-piece tokens. 2. The model receives it as a **list of integer token IDs** — never as text. Roughly **5 things** (5 tokens: `The`, ` capital`, ` of`, ` France`, ` is`), not 24 letters. The exact count is whatever DS4's tokenizer chooses. 3. **129,280** — one candidate for every entry in the vocabulary. Every "next token" prediction is a choice across all 129,280 possible tokens. (*How* it chooses among them is F0.2.)

## What this unlocks You now understand what the **tokenize** and **detokenize** steps of an inference do, and you own the words *token, token ID, tokenizer, vocabulary, BPE, special token*. The open thread: the model does not just blurt out one next token — it produces a *score for every one of the 129,280 tokens*, and something has to turn those scores into a single choice. That is **F0.2 — Logits & Sampling**. → Next: F0.2 — Logits & Sampling *(pending)*.

01 (Foundation) Tokens