CS336 Notes: Lecture 4 - Mixture of Experts
Mixture of Experts (MoE): adding capacity without proportional compute, routing, load balancing, and what makes MoE stable.
Read
Blog
Transformers
Filter
CS336 Notes: Lecture 3 - Architectures and Hyperparameters
What modern LLMs converge on: pre-norm, RMSNorm, SwiGLU, RoPE, and stability tricks.
Read
CS336 Notes: Lecture 1 - Overview and Tokenization
A tour of modern LLMs, the "bitter lesson" through the lens of efficiency, and why BPE tokenization matters.
Read
CS336 Notes: Course Overview
My notes on Stanford CS336: Language Modeling From Scratch: how to build a large language model end to end, from data to deployment.
Read