CS336 Notes: Lecture 2 - PyTorch and Resource Accounting
Resource accounting for LLM training: compute estimates, memory budgets, dtypes, tensors, and mixed precision.
Read
Blog
Machine-learning
Filter
CS336 Notes: Lecture 1 - Overview and Tokenization
A tour of modern LLMs, the "bitter lesson" through the lens of efficiency, and why BPE tokenization matters.
Read
CS336 Notes: Course Overview
My notes on Stanford CS336: Language Modeling From Scratch: how to build a large language model end to end, from data to deployment.
Read