# Phase 1 · Module 1 — The model & the cost of attention > Estimated time: 1–1.5 days · Pinned to commit `1aedee4` · Needs no build or > model. ## Where this fits Module 0 mapped the *program*. This module maps the *model* the program exists to serve. Almost everything DS4 does — the compressed KV cache, the asymmetric quantization, the disk-checkpoint design — is a consequence of three or four facts about DeepSeek V4 Flash. Learn those facts here, anchored to the exact constants and the exact mechanism in the code, and the rest of the course becomes "oh, *that* is why." **Prerequisites: all of Phase 0.** This module uses *attention*, *heads*, *query / key / value*, *KV cache*, *sliding-window attention*, *MoE*, *RoPE*, and *big-O* as known vocabulary. ## Read first - [`MODEL_CARD.md`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/MODEL_CARD.md) — the whole file. DS4's synopsis of the official DeepSeek V4 Flash model card; the teaching source for the architecture. - [`ds4.c` → `enum { DS4_N_LAYER … }`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L86), §"Fixed DeepSeek V4 Flash Shape" — the model's shape as hard-coded constants. - [`ds4.c` → `ds4_layer_compress_ratio()`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L411) — the five-line function that defines the per-layer attention pattern. - [`ds4.c` → `indexer_allowed_decode_one()`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L6943) — the CPU reference for the indexer: how DS4 decides which compressed rows attention may see. - [`ds4.c` → `validate_compress_ratio_metadata()`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L2474) — the validator that enforces the layout against the GGUF file. ## The DS4 detail ### The model is a compile-time constant Most inference engines discover a model's shape at load time from metadata. DS4 does the opposite. Open [`ds4.c` §"Fixed DeepSeek V4 Flash Shape"](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L86): the entire architecture is an `enum` of fixed integers. ```c enum { DS4_N_LAYER = 43, DS4_N_EMBD = 4096, DS4_N_VOCAB = 129280, DS4_N_HEAD = 64, DS4_N_HEAD_KV = 1, DS4_N_HEAD_DIM = 512, /* … */ DS4_N_ROT = 64, /* … */ DS4_N_EXPERT = 256, DS4_N_EXPERT_USED = 6, DS4_N_EXPERT_SHARED = 1, DS4_N_FF_EXP = 2048, /* … */ DS4_N_SWA = 128, /* … */ DS4_N_INDEXER_HEAD = 64, DS4_N_INDEXER_HEAD_DIM = 128, DS4_N_INDEXER_TOP_K = 512, DS4_N_HC = 4, /* … */ }; ``` *(Excerpt — the real enum has 22 members; ones not used in this module are elided at each `/* … */`. Open [`ds4.c#L86`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L86) for the full list.)* The comment above it is the thesis of the whole engine: *"These constants define the single model family this program accepts."* This is what "not a generic GGUF runner" means *mechanically* — the shape is not data, it is **source code**. Every later optimization is allowed to assume these exact numbers and write fixed-size, branch-free paths. You already met most of these constants in Phase 0: 64 query heads but 1 key/value head (F0.3); 64 of 512 head dimensions rotated by RoPE (F0.5); 256 experts, 6 used, 1 shared (F0.5); a 128-token raw window (F0.4). The ones this module is *about* are the last new group — `DS4_N_INDEXER_*` — and the per-layer pattern they serve. ### The bet: what one-model specialization costs and buys Hard-coding the architecture is a deliberate, double-edged bet. It is worth naming both edges plainly. **What it buys:** - *Speed and simplicity.* Fixed shapes mean fixed-size, branch-free code paths — no dynamic shape handling, no generic dispatch. The kernels can be tuned for exactly these dimensions. - *Co-design.* Because the engine knows the model exactly, the GGUF file and the engine can be built for each other (you will see this in Module 2's quantization). - *Validatability.* One known model can be checked, token by token, against the official implementation's outputs. **What it costs:** - *Zero generality.* DS4 runs this model and nothing else. - *Per-model maintenance.* A new model — even DeepSeek's own *next* version of Flash — needs source edits and a recompile, not a config change. - *Brittleness.* Any mismatch between the GGUF and the hard-coded numbers is fatal (see "enforced, not trusted" below). This is the classic specialization trade: maximum performance and simplicity for exactly **one** target, in exchange for **no** flexibility. The project takes that bet on purpose — its own README calls it a "deliberately narrow bet." Most of the engineering in this course only makes sense once you accept that DS4 is optimizing the daylights out of a single, fixed target. ### The shape is enforced, not trusted A constant in `ds4.c` is only a *claim* about the model. The other half is [`ds4.c` → `validate_compress_ratio_metadata()`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L2474). At load time DS4 reads the GGUF metadata key `deepseek4.attention.compress_ratios` and, for every layer, compares it to `ds4_layer_compress_ratio(il)`: ```c const uint32_t expected = ds4_layer_compress_ratio(il); if (got != expected) { fprintf(stderr, "ds4: unexpected DeepSeek4 compression ratio at layer %u: got %u, expected %u\n", il, got, expected); exit(1); } ``` On any mismatch the process **exits**. This is the engine's whole philosophy in one function: it would rather refuse to run than run a model it was not built for. "Specialized engine" is not marketing — it is `exit(1)`. ### How the architecture was known A fair question: how did the author know all of this precisely enough to implement it? Not by reverse-engineering. There are three external sources, and the repo points at all of them: - The **official DeepSeek V4 model card** — `MODEL_CARD.md` is the author's distilled synopsis of it. It is the architecture spec. - DeepSeek's official **`encoding_dsv4.py`** — the reference prompt-rendering script, described in [`MODEL_CARD.md`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/MODEL_CARD.md). DS4's prompt format was matched to it exactly. - **Official output vectors** — DS4 is validated against logits "obtained with the official implementation" (`tests/test-vectors`). Even where the spec is ambiguous, the author could compare DS4's numbers to the real model's, token by token. The workflow: read the spec, copy the exact prompt format, then *prove correctness* against official outputs. That last step — validation against official vectors — is Module 11's subject. ## DeepSeek V4's attention, in depth This is the heart of the model, and the reason it needs a dedicated engine. You have the cost problem from F0.4: plain attention is **O(n²)**, impossible at a million tokens. DeepSeek V4's answer is not one trick but a *structured combination* of three. This section builds it. ### Three ways to see the past At any layer, a token's "past" is split into the **recent** past and the **deep** past, and the model treats them very differently. The **recent past** is handled the same way in *every* layer: a **raw sliding window** of the last 128 tokens (F0.4), full-resolution, uncompressed. That is the high-fidelity local context. The **deep past** — everything older than 128 tokens — is where the layers diverge. There are two regimes, and a layer uses one or the other: | Regime | Resolution of the deep past | Coverage | What bounds its cost | |---|---|---|---| | **Coarse-complete** (ratio 128) | very low — one compressed row per 128 tokens | *all* of history | only `n/128` rows exist | | **Fine-selective** (ratio 4 + indexer) | high — one compressed row per 4 tokens | only the *retrieved-relevant* parts | an indexer caps attention at 512 rows | Read those as two different answers to the same question — "how do you make the deep past affordable?": - **Coarse-complete** keeps the cost down by making every row *cheap and few*: pool 128 tokens into one row, so a million-token history is only ~8,000 rows. The model sees *all* of history, but only as a blurry summary. - **Fine-selective** keeps the *rows* high-resolution (one per 4 tokens — so a million tokens is ~250,000 rows, far too many to attend to) and instead bounds the cost by *not looking at most of them*: an **indexer** picks the 512 most relevant rows and attention ignores the rest. The model sees the deep past in sharp detail, but only the parts it decided are relevant *right now*. One gives you *complete but blurry*; the other gives you *sharp but selective*. Neither alone is enough — and that is why the model alternates them. ### The per-layer pattern — and why it alternates The whole pattern is one function, [`ds4.c` → `ds4_layer_compress_ratio()`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L411): ```c static uint32_t ds4_layer_compress_ratio(uint32_t il) { if (il >= DS4_N_LAYER) ds4_die("…layer index is outside the fixed model layout"); if (il < 2) return 0; return (il & 1u) == 0 ? 4u : 128u; } ``` - **Layers 0 and 1** → ratio `0`: the raw 128-token window only, no deep past at all. The input-most layers work on full-resolution *local* signal before any compression enters the model. - **Even layers from 2 on** → ratio `4`: raw window + the **fine-selective** regime (compressed + indexer). - **Odd layers from 3 on** → ratio `128`: raw window + the **coarse-complete** regime. The `(il & 1u)` is just an even/odd test. So after layer 1 the model **alternates**: fine-selective, coarse-complete, fine-selective, coarse-complete, all the way up. *Why alternate, instead of picking one?* Because the two regimes are complementary, and the model benefits from interleaving them through its depth: - The **coarse-complete** layers guarantee that *nothing is ever fully out of view* — there is always a low-resolution signal from the entire history, so the model keeps a continuous sense of the global gist. - The **fine-selective** layers provide *sharp recall* — when something specific from far back matters (a name, a number, a definition), the indexer can retrieve those exact rows at full resolution. A token, passing up through 43 layers, is therefore repeatedly handed *both* a blurry view of everything and a sharp view of the specifically-relevant — at alternating depths. That interleaving is the architecture. It is also why this is one function and not a hyperparameter table: the pattern is fixed, total, and load-bearing. ### The indexer: a learned retrieval mechanism The fine-selective regime hinges entirely on one component — the **indexer** — and it is worth understanding properly, because "an indexer picks the relevant rows" hides a real mechanism. The CPU reference is [`ds4.c` → `indexer_allowed_decode_one()`](https://github.com/antirez/ds4/blob/1aedee468d782f4159a0b3d36816c4a3787bd580/ds4.c#L6943). The indexer is **a small, separate, learned attention-like scorer.** Its only job: for the current token, score *every* compressed row for relevance, so the real attention can skip the low-scoring ones. Concretely, for a ratio-4 layer: 1. **It has its own compressed stream.** A ratio-4 layer keeps *two* compressed caches: one the real attention reads from, and a *second* one built only for the indexer to score against. (Two learned compressors: one output is for *attending*, the other for *deciding what to attend to*.) 2. **It builds a query.** From the same query representation the main attention uses, a projection produces an indexer query — **64 indexer heads**, each **128**-dimensional (`DS4_N_INDEXER_HEAD`, `DS4_N_INDEXER_HEAD_DIM`). 3. **It scores every compressed row.** For each compressed row, it takes a dot product with each of the 64 indexer-query heads, clamps negatives to zero (a ReLU), and combines the 64 results with learned per-head weights into a single **relevance score** for that row. Scores, not probabilities — for selection you only need an *ordering*. 4. **Top-k selects.** The 512 highest-scoring rows are marked visible (`DS4_N_INDEXER_TOP_K`); the rest are masked out. The layer's attention then runs over the raw 128-token window **plus only those 512 selected rows**. ``` A ratio-4 layer, deep in a long context. Suppose the compressed history has grown to 9,000 rows. current token ──► INDEXER — scores every compressed row for relevance │ (a cheap multi-head dot-product per row) ▼ compressed rows: [ r0 r1 r2 r3 ... r8999 ] relevance score: [ .1 .9 .2 .7 ... .3 ] │ ▼ TOP-K: keep the 512 highest-scoring rows [ r1 r3 … 512 rows total … ] │ ▼ the layer's ATTENTION attends over: the raw 128-token window + only those 512 selected rows (the other ~8,500 compressed rows are masked out — never looked at) ``` One important nuance from the code: selection only kicks in **once there are more than 512 compressed rows**. Below that, `top_k == n_comp` and *every* compressed row is allowed — the indexer prunes nothing. So the indexer is dormant at short context and only starts mattering once the deep past has truly grown large. It is a long-context mechanism. The deep idea: the indexer makes attention **content-addressed**. A plain transformer attends to history *by position* (everything in the window). DS4's fine-selective layers attend to history *by relevance* — the model learns to *retrieve*. That is what lets a ratio-4 layer keep one row per 4 tokens (sharp) without paying for all 250,000 of them at a million-token context. ## Zoom out — transformer Step back from DS4 to the general picture. **Attention is a design space, not a fixed recipe.** Phase 0 taught plain attention (every token attends to every earlier one) and sliding-window attention (only the last *w*). DeepSeek V4 Flash is one carefully chosen point in that space: raw window **+** compressed history **+** learned retrieval, alternating per layer. The model card's names for the combination are "Compressed Sparse Attention" and "Heavily Compressed Attention"; you have now seen what those names actually denote. One more architectural term you will keep meeting: **mHC — Manifold-Constrained Hyper-Connections** (`DS4_N_HC = 4`). F0.3 mentioned it: where a standard transformer carries one residual stream between layers, this model carries **four**, mixed by a learned, normalized combination. You do not need its math here — just register that "HC split" and "HC expand" in later modules are the model reducing four streams to one before a sublayer and expanding back to four after. Module 4 covers the mechanics. ## Zoom out — CS **Architecture is driven by asymptotic complexity.** This is the deep idea of the module, and it is pure computer science. You have the wall from F0.4: plain attention is **O(n²)** in context length, and at *n* = 1,000,000 that is a *trillion* operations per layer. No faster kernel rescues the wrong complexity *class*. DeepSeek V4's three-part design is, read through this lens, three different ways of escaping O(n²): - **The raw window** caps the recent past at a constant `w = 128` → that term is **O(1)** in n. - **Coarse-complete (ratio 128)** reduces the deep past to `n/128` rows. Still technically O(n), but with a 128× smaller constant — cheap enough to keep at every coarse layer. - **Fine-selective (ratio 4 + indexer)** is the most interesting. The compressed rows are O(n/4) — *worse* — but the indexer's **top-k selection** caps what attention actually consumes at a constant 512 rows → **O(1)** again. That last move deserves its own CS name. "Find the *k* largest of *m* values" is the classic **selection problem**. The subtle part is *where the cost goes*. The indexer must still *score* all *m* compressed rows — scoring is unavoidably **O(m)**. But scoring is *cheap per row* (one multi-head dot product). The expensive operation — full attention with value-blending — is then run on only *k* = 512 rows, **O(1)** in n. So top-k selection does not eliminate the linear term; it **moves the expensive work off the unbounded set onto a bounded one**. Cheap-and-linear to decide; expensive-but-constant to execute. The lesson generalizes far past LLMs. When a system must scale, you do not chase constant factors — you find the term whose *growth class* is wrong and change it: by **bounding** it (a fixed window, a fixed top-k) or by **selecting** cheaply instead of processing everything expensively. DeepSeek V4 does both, three times over. To study the idea properly: any algorithms text on *asymptotic notation* and *the selection problem* (e.g. CLRS). ## Exercise *(No build or model needed — paper and reading.)* 1. **Trace the layer pattern.** By hand, evaluate `ds4_layer_compress_ratio(il)` for `il = 0 … 11`. For each, also write which regime it is: *raw-only*, *fine-selective*, or *coarse-complete*. 2. **When does the indexer actually do anything?** A ratio-4 layer has a compressed history of (a) 300 rows, (b) 9,000 rows. In which case does the indexer prune, and in which does it pass everything through? (Hint: re-read `indexer_allowed_decode_one` and `DS4_N_INDEXER_TOP_K`.) 3. **Quantify "284B total, 13B active."** Using the `enum` constants, estimate the routed-expert parameters. One routed expert is three matrices: gate (`N_EMBD → N_FF_EXP`), up (`N_EMBD → N_FF_EXP`), down (`N_FF_EXP → N_EMBD`). Compute (a) one expert, (b) all 256 experts × 43 layers, (c) only the 6 experts active per token × 43 layers. Compare (b) to 284B and (c) to 13B.

What you should have found

**1.** Layers 0,1 → `0` *raw-only*. Then 2→`4` *fine-selective*, 3→`128` *coarse-complete*, 4→`4`, 5→`128`, 6→`4`, 7→`128`, 8→`4`, 9→`128`, 10→`4`, 11→`128`. After the two raw layers, the model alternates fine-selective (even) and coarse-complete (odd). **2.** (a) 300 rows: `DS4_N_INDEXER_TOP_K` is 512, and 300 < 512, so `top_k == n_comp` and **every row is allowed — the indexer prunes nothing.** (b) 9,000 rows: 9,000 > 512, so the indexer scores all 9,000 and **selects only the top 512.** The indexer is a long-context mechanism — dormant until the compressed past exceeds 512 rows. **3.** Approximate, weights only: (a) one expert ≈ `3 × 4096 × 2048 ≈ 25.2M`. (b) `25.2M × 256 × 43 ≈ 277B` — almost the entire 284B model; the routed experts *are* the model by mass (which is why Module 2 quantizes them and little else). (c) `25.2M × 6 × 43 ≈ 6.5B` active in the routed experts; the rest of the ~13B active budget is the always-on dense parts (shared expert, attention, embeddings, output). Total vs active diverge by the routing ratio 6/256.

## Deep check The model has `DS4_N_HEAD = 64` attention heads but `DS4_N_HEAD_KV = 1`. What does that single `1` tell you about the size of the KV cache, and how does it combine with the per-layer compression to make the project's headline claim — "the KV cache is a first-class disk citizen" — physically possible?

Answer

`DS4_N_HEAD_KV = 1` means all 64 query heads share **one** key/value head (F0.3's grouped-query attention, taken to its limit). A vanilla 64-head design stores 64 separate K and V per token per layer; this model stores **one** — a ~64× reduction *before any compression runs*. Then the per-layer compression stacks on top: the raw window is bounded to 128 tokens; the coarse-complete layers keep one row per 128 tokens; the fine-selective layers keep one row per 4 tokens but the indexer means attention only ever touches 512 of them. So the deep-history KV state is shrunk first by the single shared KV head, then again by 4×/128× pooling. The result is a KV cache small enough that *persisting* it is cheap — small enough to write to an SSD and reload (Module 6), instead of treating it as something that must live in RAM and be recomputed. The headline feature is downstream of a handful of numbers in an `enum`.

## Transferable takeaway A model's architecture is a set of cost decisions, and a good engine encodes those decisions as enforced constants rather than discovered data. You learned to read DeepSeek V4 Flash as: 64 query heads but 1 KV head; a raw window plus two complementary views of the deep past — coarse-complete and fine-selective — alternating per layer; and a learned **indexer** that turns attention from *position-addressed* into *content-addressed*. And you learned the CS reason it is shaped that way: a 1M-token context is impossible in O(n²), and the escape is always the same move — bound the term whose growth class is wrong, or select cheaply instead of processing everything. **When you next meet a system that scales surprisingly well, look for the place its designers changed a complexity class — it is never an accident.** → Next: Module 2 — Loading & quantization *(pending)*.

07 The Model