Modeling Is Where Text Becomes Geometry

Tokenization is the first modeling choice, and it lands before a single weight moves. Text becomes tokens. Tokens become vectors. Vectors move through the model, and somewhere in that path strings become geometry.

When output looks strange on code, math, rare names, or non-English text, check the tokenizer. Do that before you invent a psychological story about what the model believes. Each of those four cases turns on how the text was cut.

Text Becomes Tokens Before It Becomes Anything Else

The tokenizer cuts every string into pieces before the model sees anything. Split too finely and the model spends its context on the spelling of common words. Split too coarsely and rare forms, morphology, code, and multilingual text get harder. BPE and unigram tokenizers are compromises between those two costs.

Nothing downstream recovers a boundary the tokenizer erased. A rare name that arrives as fragments stays fragments for every layer above it. The pieces are the only view of the string the model gets.

Text the vocabulary does not cover gets cut into more and smaller pieces. The same sentence then spends more of the context window and arrives with less structure. That cost falls unevenly across languages. The first harms note measured it from the other side. It called the gap a quality-of-service harm and named the group the average was carrying.

BPE Merges Frequent Pairs, Unigram Picks Likely Segmentations

BPE starts from small symbols and repeatedly merges the most frequent adjacent pair into a larger token. The merge table records what was common in the corpus it was fit on. Frequent strings become single pieces. Everything else gets assembled out of leftovers.

Unigram tokenization, the SentencePiece style, learns a vocabulary and then picks the segmentation it scores as most likely. BPE builds pieces up from frequency. Unigram chooses among segmentations a learned vocabulary already scores. Both are fit on a corpus, and both freeze one view of what counts as normal text.

Both choices are locked once training starts. The embedding table is indexed by token ID. A new vocabulary sends every ID somewhere else, and the trained table is dead weight. Changing the tokenizer means training again, so price that decision before the run.

Embeddings Make The Workspace, Attention Moves The Context

Embeddings map token IDs into vectors, which gives the model a space to compute in. Similarity, context, and prediction become vector operations. That space is the working memory, and the tokenizer already decided what occupies it.

Recurrent models walk a sequence one step at a time. Transformers process many positions in parallel, and attention carries information between them. Self-attention builds a contextual representation for each token by mixing in the positions it reads. Positional information keeps the order in place, because attention alone does not tell first from last.

Four choices decide what information can move between positions. Attention decides which positions read each other, and depth decides how many times the model edits what they read. Width gives each position more room. Context length sets the reach.

Modeling Choices Become Infrastructure Costs

Attention is why a long context is expensive. Every position that can read every other position needs memory to hold what it reads. That memory grows with the length you allow. The architecture decision arrives later as a training constraint and as a serving bill.

The parallelism note names the walls this creates. Context length and width set the memory wall before the run starts. Once the model no longer fits on one accelerator, the split has to send activations and gradients across the interconnect. Those messages compete with the work itself.

That makes the tokenizer a serving decision as well as a modeling one. Measure pieces per request on your own traffic before you argue about context length.

Check The Representation Before You Blame The Reasoning

The overview note set the rule for this unit. When a model surprises you, name the layer that did the work. The representation is the layer to read first, because everything a reasoning story explains sits on top of it.

Weird spelling, broken code indentation, rare names, and math notation are all shapes a tokenizer problem takes. From the outside each one looks like a failure of understanding. The mechanism is a boundary in the wrong place.

The risk is treating the representation as invisible plumbing. Bad token boundaries, weak embeddings, a short context, and a poor attention pattern all cost the model capacity. They take it before training gets a fair chance. Fix the representation before you spend a run compensating for it.

The Builder Test

Take one sentence your system handles badly. Trace it through the whole path: tokens, IDs, embeddings, attention context, next-token scores. Write down what each step produced, in that order.

Start at the token boundaries, because they are visible and cost nothing to print. Stop at the first step that sounds like magic. That is the step to slow down on.

What Carries

The model edits vectors until the next-token distribution becomes useful. The representation decides what it can edit. Tokenizer, context length, and attention pattern set that limit. All three are chosen before any data moves.

Architecture is potential. The objective decides which potential the model practices.