CS336 Notes: Lecture 6 - Kernels and Triton
Writing efficient GPU kernels with Triton: profiling, benchmarking, kernel fusion, and when to hand-optimize versus using torch.compile.
Read
Blog
Gpu
Filter
CS336 Notes: Lecture 5 - GPUs
GPU fundamentals for LLM training: memory hierarchy, arithmetic intensity, kernel optimization, FlashAttention, and bandwidth limits.
Read
CS336 Notes: Lecture 2 - PyTorch and Resource Accounting
Resource accounting for LLM training: compute estimates, memory budgets, dtypes, tensors, and mixed precision.
Read