CS336 Notes: Lecture 5 - GPUs
GPU fundamentals for LLM training: memory hierarchy, arithmetic intensity, kernel optimization, FlashAttention, and bandwidth limits.
Read
Blog
Deep-learning
Filter
CS336 Notes: Lecture 4 - Mixture of Experts
Mixture of Experts (MoE): adding capacity without proportional compute, routing, load balancing, and what makes MoE stable.
Read
CS336 Notes: Lecture 3 - Architectures and Hyperparameters
What modern LLMs converge on: pre-norm, RMSNorm, SwiGLU, RoPE, and stability tricks.
Read
CS336 Notes: Lecture 2 - PyTorch and Resource Accounting
Resource accounting for LLM training: compute estimates, memory budgets, dtypes, tensors, and mixed precision.
Read
CS336 Notes: Course Overview
My notes on Stanford CS336: Language Modeling From Scratch: how to build a large language model end to end, from data to deployment.
Read