The Pipeline Is the Model

A language model is a pipeline of choices that turns text, compute, and feedback into behavior. Every stage sets the terms for the stage after it. Most LLM tutorials teach you to call an API. This sequence builds the system behind it.

Collect the data, tokenize it, build a transformer, train it, evaluate it, and serve it. Every hard part gets an answer you can check. By the end you can name what every number in a result cost to produce.

Each stage decides what the next stage can do

The chain runs from data collection through tokenization, architecture, optimization, hardware, scaling, evaluation, inference, and post-training. Each stage carries one assignment.

• Data decides what signal exists.
• Tokens decide how text becomes compute.
• Architecture decides what can be represented.
• Training decides what the weights record.
• Evaluation and serving decide whether anyone can use the system.

Tokenization is the first stage, and the token count it produces is what a training run pays for. Resource accounting belongs to training, where it answers whether a planned run fits the hardware. The training loop decides whether that run stays stable. Evaluation runs last, beside serving, and it gates whether the work leaves your machine.

A rough tokenizer makes training harder. Careless data makes the loss dishonest, and a loose evaluation makes a weak model look ready. Each failure lands one stage after the choice that caused it.

Small models teach the full process

You will not train GPT-4 here. The models are small, and the process is the same. The choices that are important at 7B parameters are the choices that are important at 70B.

The numbers shrink. The work stays the same: measure, debug, account for what you spent, and know what changed. A small model becomes a toy when you skip a stage.

A small run gives you the whole chain at a price you can pay many times. Run it end to end, break it, and read what the break cost.

Going straight to architecture is the expensive mistake

Architecture is important, and it is still one link in the chain. A beautiful model trained on careless data is a careless system. No block design recovers signal the data never carried.

When a result disappoints, walk the chain backwards from the number: evaluation, then training, then tokens, then data. Stop at the first stage you cannot explain, and fix that one. Rebuilding the later link is the expensive way to learn that the earlier link was the problem.

See the whole chain once, then go back to the hard parts

Read the chain once at speed. A pass that stops to master every step never finishes. The first pass buys the shape and nothing else, and after it you know which parts are hard.

Then return to the parts that were difficult. If a part stays difficult, slow down and stay on it until you can explain it in your own words. A part you can only repeat in borrowed words is a part you do not have yet. The goal is to build it again without the notes.

The Builder Test

Describe the path from raw text to one generated token. Do not omit a step. Mark every step that sounds like magic, and study those first. The description is finished when every step names what it decides.

What Carries

The pipeline is the object you are building. When something breaks, find the break before you reach for a bigger model. The bigger model hides that break and charges you on every run after. The chain starts at tokenization, and that is a budget decision before it is a text decision.