CS336 Notes: Lecture 6 - Kernels and Triton
Writing efficient GPU kernels with Triton: profiling, benchmarking, kernel fusion, and when to hand-optimize versus using torch.compile.
Read
Blog
Filter
Writing efficient GPU kernels with Triton: profiling, benchmarking, kernel fusion, and when to hand-optimize versus using torch.compile.
GPU fundamentals for LLM training: memory hierarchy, arithmetic intensity, kernel optimization, FlashAttention, and bandwidth limits.
Mixture of Experts (MoE): adding capacity without proportional compute, routing, load balancing, and what makes MoE stable.
What modern LLMs converge on: pre-norm, RMSNorm, SwiGLU, RoPE, and stability tricks.
Resource accounting for LLM training: compute estimates, memory budgets, dtypes, tensors, and mixed precision.