Large Language Models

How large language models work under the hood, and how they are made faster and cheaper to run. Read them in order; each note builds on the one before it.

• Attention

  How a token decides which other tokens matter, from Q, K, V to multi-head attention.

• KV Cache

  How KV Cache speeds up LLM inference by reusing Keys and Values, the GPU memory it costs, and how PagedAttention, GQA and MQA tame it.

• Speculative Decoding

  How a small draft model proposes tokens and a large target model verifies them together, cutting the sequential steps of LLM generation.

• Quantization

  How storing model weights in fewer bits cuts memory, bandwidth and cost, and what it trades away in numerical precision.

• Fine Tuning

  Updating a pretrained model's weights on a task-specific dataset to change its behavior, and when that beats prompting or RAG.

• Mixture of Experts

  How MoE models use a router to activate only a subset of experts per token, trading engineering complexity for more capacity at a lower compute cost.