The Pipeline Is the Model
A language model is a pipeline of choices that turns text, compute, and feedback into behavior, and each stage prices the one after it.
Building language models from first principles.
One overview plus 17 focused notes on the full path: data, tokenization, transformers, training, systems, evaluation, inference, and alignment. The goal is a working mental model.
Notes
Sorted in sequence orderTokenization Is Where Text Becomes Compute
Efficiency decides how much scale you can afford, and the tokenizer is where that budget is won or spent. Byte Pair Encoding is how you win it.
Training Is Math Plus Bookkeeping
A training step costs about 6 times tokens times parameters, and AdamW needs about 16 bytes for each parameter. Those two numbers price a run before you launch it.
Transformers Are Controlled Editing Machines
Open models converge on one decoder block because those choices train stably and serve cheaply. Pre-norm, RMSNorm, SwiGLU, RoPE: guess a new model's architecture and you will be right.
Mixture of Experts Routes Capacity
Mixture of experts buys more parameters at the same FLOPs for each token, then charges the gain back in routing discipline, load balance, and all-to-all traffic.
GPUs Reward Memory Discipline
Compute grew faster than memory bandwidth, so the bytes you move set the ceiling. FlashAttention keeps attention exact and never writes the n by n score matrix to global memory.
Kernels Save Time by Moving Less Data
A fused GeLU cut 8.1 ms to 1.1 ms, and the hand-written kernel lost to the tuned library. Measure first, then fuse the memory-bound chains the profiler names.
Parallelism Starts With the Bottleneck
Compute, memory, communication, and batch size compete for one run. Name the one that runs out first, because that is the only wall a split can answer.
Distributed Training Is a Scheduling Problem
Collectives set the step time. A measured all_reduce reached 277 GB/s against an NVLink peak near 900, so measure the communication your parallel plan assumed.
Scaling Laws Make Expensive Choices Less Blind
Loss falls on a log-log straight line, and that line picks model size and token count before you spend the budget. Chinchilla lands near twenty tokens for each parameter.
Inference Is Cache, Latency, and Scheduling
Generation is memory-bound because every sequence drags its own KV cache. The wins come from shrinking that cache, sharing it, or spending compute to skip decode steps.
Scaling Laws Are Planning Tools
Twenty tokens for each parameter is a floor. The teams that published their own measurements landed near 39, near 96, and near 192.
Evaluation Is a Contract
A benchmark number carries no meaning apart from the choices that produced it. Contamination and length bias decide leaderboards without ever appearing on them.
Data Is the Model Diet
Two 1B models on the same 100B tokens land several points apart on data alone. Extraction, filtering, deduplication, and mixture set the ceiling.
Data Quality Beats Token Count
Every filter that reads the whole web must cost orders of magnitude less than the training it protects. One cheap pattern does language ID, quality, toxicity, and deduplication.
Post-Training Teaches the System What to Reward
Post-training decides how capability shows up, and every shortcut in the feedback becomes behavior. A rater with one minute rewards confident structure, so the model writes it.
Preference Optimization Is Proxy Design
Push a proxy reward hard enough and the true win rate turns down. The reasoning systems that work moved to rewards a program can check.
Reward Optimization Needs a Baseline
A 0/1 reward leaves most gradients at zero. The baseline that fixes it is the mean reward across several samples of the same prompt.
The one hour path.
Building a language model, data to deployment If you only have an hour, read kernels, scaling laws, inference, and alignment. That path shows the full loop: make training efficient, predict scale, serve the model, then shape behavior.