Read The Stack Sideways
A language-model claim stays unfinished until it names the dataset, the interface, the measurement, the user, the failure mode, and the cost of an error.
Understanding and Developing Large Language Models.
How language models are built, measured and misused
Notes
Sorted in orderLanguage Models Are Probability Machines
A language model assigns probabilities to strings, and temperature plus top-k decide whether those probabilities read as deterministic autocomplete or as a stochastic writer.
Capabilities Are Interfaces
A capability belongs to the model and the interface together. The claim is incomplete until it names the prompt, context, decoding rule, benchmark, and failure examples.
Harms Start With Measurement
Representational, allocative, and quality-of-service harm each need different evidence, and one aggregate score hides the subgroup it fails.
Every Moderation Threshold Picks An Error
Every moderation threshold picks which error you prefer to make. A stricter filter blocks the wrong people, teaches attackers the boundary, and does not make a product safer by itself.
Data Is the Model Before the Model
The dataset is the first behavior specification a model receives. WebText used outbound Reddit links as its proxy for quality, and every corpus picks a proxy like it.
Security Begins With Memorization
Extraction attacks turn generation into a search for memorized text. Once a secret is trained in, you cannot cleanly delete it.
Law Is Part of the Model Boundary
Public access, permission to copy, permission to train, and permission to deploy are four different questions. A rights register answers all four before a takedown forces the answer.
Modeling Is Where Text Becomes Geometry
Tokenization is the first modeling choice. When output looks strange on code, math, rare names, or non-English text, check the tokenizer before you invent a psychological story.
Training Turns Architecture Into Behavior
A model learns whatever game the training objective writes down, and a lower loss can still hide worse behavior on the task users need.
Parallelism Is the Hidden Curriculum
If communication dominates compute, an expensive cluster waits instead of learning. The scarce resource picks the parallel strategy, so name the wall before you name the tool.
Scaling Laws Make Compute Legible
Small probe runs estimate the slope before a frontier run spends real money, and the curve they return speaks for the setup that produced it.
Selective Architectures Move The Hard Problem Into Routing
Mixture-of-experts wakes a few experts per token and retrieval moves knowledge into an index. Selection becomes the new failure point, and it needs its own measurement.
Adaptation Is How General Models Become Useful
Take the least invasive adaptation that clears the reliability bar. A model can learn the format of helpfulness without learning the work.
Environmental Impact Is a Design Constraint
You pay for a training run once. You pay for inference on every call, and past billions of tokens the recurring bill outgrows the run.