Tokenization Is Where Text Becomes Compute

Reports put GPT-4 near 1.8 trillion parameters and its training cost near 100 million dollars. At that price, efficiency is the difference between a run you can afford and one you cannot. The tokenizer is the first stage where it is won or spent, and it is settled before training starts.

Scale matters, and efficiency decides how much scale you can afford

Training clusters reach about 200,000 H100 GPUs. At those prices you cannot waste compute. Frontier labs are usually more careful about it than academic runs are.

Most people quote the bitter lesson as proof that only scale matters. I read it as a function with two terms.

You buy accuracy two ways. Spend more resources on the same algorithms, or improve efficiency and get more from the resources you hold. ImageNet training between 2012 and 2019 gained about 44x in algorithmic efficiency at the same accuracy. Without those gains, that accuracy costs about 44 times more compute.

Most of the components predate GPT-3. Shannon used language models to estimate the entropy of English. Classical NLP put them inside machine translation and speech recognition. Attention, Adam, and the transformer arrived through the 2010s.

ELMo and BERT showed that one base model fine-tunes to many tasks. Researchers ran early 100B-parameter experiments with mixture of experts and multi-GPU parallelism. Frontier labs supplied the engineering and the decision to spend large amounts of money. Scaling laws became the design principle over the top. The ideas sat on the shelf. Efficiency and money moved them.

The field already ran the resources-only experiment. By 2007 Google trained 5-gram models on about two trillion tokens, more text than GPT-3 saw. Those models stayed shallow, and none of the behavior people now expect from a language model appeared. Resources bought better word statistics and stopped there.

Efficiency is the theme under every choice here, because compute is scarce today and text is abundant. Filter the corpus so that compute does not go to junk tokens. Shorten sequences so that the same budget covers more text. Train one epoch over a huge dataset, since more unique data once beats the same data twice.

The same pressure sets the defaults further down the chain. Architecture choices survive because they train and serve efficiently. A well-tuned configuration beats a naive one by an order of magnitude on the same hardware. Fine-tuning lets a smaller base model handle a narrow task, which cuts the serving bill.

That regime can flip. When compute grows faster than high-quality data, one pass over the full dataset stops being optimal. Repeated passes or careful sampling matter more. Architectures shaped by compute efficiency change when data becomes the bottleneck. Name the factor that limits you, then let the choices follow from it.

Small experiments hide two things that only appear in large runs

Frontier labs keep the training data, the architecture, and the procedure private. They name competition and safety as the reasons. So the money and the information sit inside a few buildings, and the rest of us learn on small models. Small models hide two effects that decide what happens in a large run.

The cost profile moves first. In small transformers, attention layers and MLP layers carry similar FLOPs. Near 175B parameters, MLP FLOPs are the largest part of the bill. Optimize attention on a small model and you can miss the cost that dominates at the top.

Threshold behavior hides the second one. Some tasks show no gain as training FLOPs rise, then improve quickly past a point. In-context learning is one of them. A small run reports that the approach fails, and the report is true only at the size you ran. So write the scale beside every negative result you carry out of a small experiment.

A tokenizer is a budget decision before it is a text decision

Efficiency is a chain of decisions, and the pipeline note put tokenization at the head of it. Every token the tokenizer emits is compute that somebody pays for later. The stages after it inherit whatever it decides.

A tokenizer maps raw strings to sequences of integer tokens and back. Reversibility is the hard requirement, because decoding has to return the string you started with. What it decides after that is budget: sequence length, vocabulary size, training cost, and which scripts get represented cheaply.

The model reads integer IDs. The tokenizer decides which chunks get an ID of their own. Everything else gets spelled out piece by piece. That vocabulary is the alphabet the model gets, and it never sees the text any other way.

Compression ratio is the number that turns a text decision into a compute bill. Divide bytes by tokens. GPT-2 on sample text runs about 1.6 bytes per token. Change the tokenizer and that ratio moves, and the length of every sequence you ever train on moves with it.

The same ratio prices serving. Prefill reads the whole prompt at once and stays compute bound. Decode emits one token at a time for every user and stays latency sensitive. The total serving cost across all users can pass the training cost. The tokenizer sets the token count on both bills.

Look at what a GPT-style tokenizer produces. Tokens are word pieces rather than whole words. Spaces ride inside tokens, so " hello" and "hello" are two different tokens. Numbers break into pieces that do not match the digit groups people use.

Character, byte, and word tokenizers each break in a different place

Each naive scheme fails somewhere different, and the failure tells you what a learned tokenizer has to fix.

Character tokenizers treat each Unicode character as one token. The scheme is simple and reversible. The vocabulary runs very large and fills with rare characters. Compression stays mediocre, and some characters take more than one byte anyway.

Byte tokenizers convert to UTF-8 and treat each byte as a token. The vocabulary holds 256 entries, and nothing falls outside it. Compression is one byte per token, so sequences run long and the attention cost climbs quickly. Byte-only models stay off the frontier for that reason, elegant and too expensive to run.

Word tokenizers split with a regex into words and non-word spans. Frequent words become single tokens and compression is strong. The vocabulary grows without limit, many words appear exactly once, and any new word arrives out of vocabulary. Training and evaluation both get harder.

Every naive scheme lands in one of three states: too big, too slow, or brittle. A tokenizer learned from data is the way out of all three.

Byte Pair Encoding learns its merges from the data

BPE started in 1994 as a compression method, and tokenization runs the same loop. Start from bytes rather than words. Merge frequent adjacent pairs into new tokens, and learn which pairs those are from the corpus you will encode.

Start with a byte sequence, values 0 to 255. Keep an empty merge list and a vocabulary that maps token IDs to byte strings. Then repeat for a fixed number of merges.

• Count how often every adjacent token pair appears in the data.
• Find the most frequent pair.
• Create a new token ID for that pair.
• Record the merge rule from the left and right tokens to the new ID.
• Replace every occurrence of the pair with the new token.

Each merge adds one vocabulary entry and shortens the sequence wherever that pattern is common.

Frequent patterns collapse into single tokens, and sequences get shorter. Rare patterns stay composed of smaller tokens, so nothing arrives unknown. You keep the compression of a word tokenizer and the coverage of a byte tokenizer.

Encoding converts the string to bytes, then applies the merge rules in the learned order. Decoding maps the IDs back to byte strings, joins them, and decodes to Unicode. More merges buy more compression, because more common sequences collapse to a single token.

The merge list is what you ship. You learn it once and apply it in that exact order at encode time. Apply the rules out of order and the same string tokenizes differently. It stays fixed for the life of the model. The text in the merge corpus is the text it compresses well later.

Real implementations pre-tokenize first. Split the text with a word-like regex, then run the merges inside each segment. Skip that step and merges run across whitespace, which gives tokens that straddle word boundaries. Pre-tokenization strongly controls how the finished tokens behave, and it is where spaces and punctuation get their rules.

A naive implementation is easy to read and slow, because it rescans whole sequences across every merge. Production tokenizers look only at the merges the current sequence allows. Their data structures avoid the full rescan. They also carry special tokens for start, end, and padding.

Vocabulary size buys sequence length and charges you elsewhere

Every merge you add buys shorter sequences and charges you somewhere else. A larger vocabulary packs more text into fewer tokens. It also grows the embedding table and leaves each rare token with fewer examples to learn from.

The charge lands unevenly across text. A common English word costs one token. A rare name, a typo, a code string, or a language the merges never saw costs many tokens. Each of those tokens spends context and compute that the common word did not.

Script coverage is decided in the same place. A vocabulary learned mostly on English gives other scripts few merges, so those strings fall back toward bytes. The same sentence then costs more tokens on every request.

The Builder Test

Calling an API teaches you the surface. Running text through your own tokenizer teaches you the bill. Take these strings and read the token counts.

• A few names, including some outside the Latin alphabet.
• A long URL from a page you use.
• A block of code from the repository you work in.
• A line of math with symbols and digits mixed together.
• A paragraph in a language other than English.

Compare each count against the bytes per token you measured on ordinary prose. Whatever comes back long is what the model finds expensive. Fix it in the tokenizer, or pay for it in every run after.

Then run the same strings through a second tokenizer and read the two counts side by side. The gap between them is what the choice is worth. Keep that file, and every tokenizer you consider later gets measured against it.

What Carries

Tokens are the interface between language and computation. A rough interface shows up as wasted context and as text the model cannot copy back cleanly. Neither symptom looks like a tokenizer problem when you meet it.

Byte-level models can remove tokenization one day, and none of them leads at the frontier now. So the tokenizer stays a decision you own and defend with a number.

The tokenizer fixes how much compute each string costs. The next thing to price is one training step, before you run it.