Parallelism Starts With the Bottleneck

Compute, memory, communication bandwidth, and batch size compete inside one training run. The split you choose is correct only when it answers whichever one runs out first.

One GPU cannot train a frontier language model. The model does not fit, the data does not fit, and the compute takes too long.

Four resources compete, and batch size is the one people forget

Model needs grow faster than GPUs do. A large model strains a single GPU in two places. Billions of weights do not fit in its memory. Long sequences and deep networks build intermediate tensors that do not fit either.

Batch size is the fourth resource, and some splits spend it. You fix that supply when you plan the run, so what one split takes another cannot have.

The unit of compute is now the data center. Near-linear scaling is the goal. Double the GPUs and you want about double the capacity and throughput.

Where a split runs decides what it costs. A cluster has three rungs.

• A node holds several GPUs, often eight, and very fast NVLink or NVSwitch links connect them.
• Across nodes, a fabric such as InfiniBand carries the traffic, slower than NVLink and higher in latency.
• Past a few hundred GPUs, the traffic crosses switch tiers that are slower again.

An algorithm that talks constantly belongs inside a node, where the bandwidth is high. Across nodes, the same algorithm has to justify every byte it sends.

Allreduce equals reducescatter plus allgather

Five collectives show up everywhere in distributed training, and each moves a known amount of traffic.

• Allreduce reduces data across all ranks, for example with a sum, and gives every rank the same result. The bandwidth cost is about twice the data size.
• Broadcast copies the input from one rank to all ranks, at about one times the data size.
• Reduce takes data from many ranks and leaves the reduced result on one.
• Allgather starts with one chunk on each rank and ends with the full concatenation on every rank.
• Reducescatter reduces the chunks across ranks, splits the result, and leaves each rank holding its own shard.

Run a reducescatter and then an allgather and you have done an allreduce. In a bandwidth-limited regime the two paths cost the same. The same bytes move either way, so sharding moves that traffic in two pieces. ZeRO and FSDP stand on that equality.

Data parallelism is simple and pays for it in memory

Naive data parallel SGD copies the model and splits the batch. The global batch is B and the cluster has M GPUs, so each GPU processes B/M examples. Each GPU runs forward and backward, computes its gradients, and allreduces them. Every GPU then holds the full gradient and applies the same update locally.

Compute scales well. Communication for each step is about twice the number of parameters, and a large enough batch hides it behind compute.

Every GPU also stores a full copy of the parameters, the gradients, and the optimizer state. Adam-style training keeps five items for each parameter.

The bf16 parameter and the bf16 gradient take 2 bytes each. The fp32 master weight, the first moment, and the second moment take 4 bytes each. The bill is about sixteen bytes for each parameter.

The accounting note already charged that bill. Sharding starts here because this is where the waste sits. A 7.5B model on 64 accelerators can need about 120 GB of parameter-related memory. In naive data parallel most of that holds copies of the same numbers.

Every GPU computes the same update from the same state. Does every GPU need to hold that state?

ZeRO removes the copies without adding bandwidth

ZeRO means Zero Redundancy Optimizer, and each stage deletes one kind of copy.

Stage 1 shards the optimizer state and leaves the parameters and gradients replicated. Each GPU computes gradients on its mini-batch. A reducescatter sends each summed gradient shard to its owner. The owner updates its own shard, because it holds the optimizer state for that shard. An allgather then rebuilds the full updated parameters on every GPU.

Reducescatter plus allgather costs the same bandwidth as the one allreduce it replaces. Stage 1 adds almost nothing to the wire and divides the optimizer state memory by the number of GPUs.

Stage 2 keeps that and shards the gradients too. Each layer produces its gradients during the backward pass, and a reducescatter sends them to their owners at once. The GPU then frees the gradient buffers it does not own.

After the backward pass each GPU holds one gradient shard and one optimizer-state shard, with the parameters still replicated. The GPUs update their shards, and an allgather restores the full parameters. Traffic stays near twice the parameters for each step, in smaller and more frequent messages. Gradient memory now falls by that same factor.

Stage 3, also called FSDP, shards the parameters as well, and no GPU holds the whole model at one time.

• Forward: allgather the parameter shards for one layer, run that layer on local data, then free the full parameters.
• Backward: allgather the parameters again if needed, compute the gradients, and reducescatter them so each owner keeps its shard.
• Then the owners update their shards and free the full parameters and the temporary gradients.

Communication for each step rises to about three times the parameters, against two for simpler data parallelism. Overlap is what makes that acceptable. While the GPU computes layer k, the network reads the parameters for layer k+1.

Memory for each GPU lands near the minimum. You pay in traffic and you get a model that fits.

Data parallelism runs out when the batch cannot grow

Replicas need batch to feed them. You cannot run more replicas than the global batch size allows. Past the critical batch size, a bigger batch returns smaller and smaller gains.

Sharding does nothing for activation memory at Stages 1 and 2. Stage 3 relieves the parameters and the optimizer state, and activations still dominate at long sequences.

Activation memory climbs through the forward pass and falls through the backward pass. The peak usually sits in the middle of the backward pass.

Two terms stay large in a rough per-layer view. One grows with sequence times batch times hidden size, for the MLP and the pointwise operations. The other grows with sequence squared times batch, for attention.

If the batch cannot grow and the activations are large, data parallelism is finished. The next split must cut the model itself.

Pipeline splits depth and tensor parallel splits width

Pipeline parallel cuts the network by depth. GPU 0 takes the early layers, GPU 1 the middle layers, GPU 2 the later ones. Activations flow forward from stage to stage, and gradients flow back from the last stage to the first.

With one microbatch most GPUs sit idle, and utilization lands near one over the number of stages. Microbatching fills the pipe. GPU 0 sends microbatch 0 forward and starts microbatch 1 at once.

The bubble fraction scales as (S - 1) / M, where S counts stages and M counts microbatches. M must be much larger than S to keep the bubble small.

Pipeline parallel buys memory. Each GPU stores only its own layers, and it ships point-to-point activations that survive slower links. It spends batch size through microbatches, and efficient zero-bubble schedules are hard to build and to maintain.

Tensor parallel cuts by width. Most transformer compute sits in matmuls, so this split shards the large weight matrices. Each GPU computes a partial result, and collectives assemble the correct activations.

That puts a communication point in every layer. The split needs very high bandwidth, so it stays inside a fast node. Eight GPUs on NVLink or NVSwitch is the usual case.

Placement is priced. From 1 to 8 inside a node, tensor parallel costs 10 to 12 percent of throughput. At 16 or 32 across nodes it can cost 40 to 65 percent, because the links are slower.

Tensor parallel does not spend batch size, and that is why it goes first inside the node. It matches matmul-heavy work and it is sensitive to the hardware topology.

Fit the model first, then buy throughput

A plan that does not fit has no throughput to measure. Set tensor parallel to the number of GPUs in one node, for example TP = 8. If the model still does not fit, add FSDP or Stage 3 across nodes. You can add pipeline parallel as well, or both.

Once the memory fits, add data-parallel replicas to raise the total FLOPs. Data parallelism tolerates slower links, and it is the easiest split to reason about.

Batch size is a hard budget, and both pipeline parallel and data parallelism draw on it. If the batch is small and you want fewer synchronizations, use gradient accumulation.

Activations are the last term to fall. Naive tensor parallel leaves pointwise operations such as LayerNorm and Dropout unsharded. Sequence parallelism shards that work along the sequence dimension, so each GPU handles about sequence over TP tokens. When you need a full-sequence view, an allgather or a reducescatter provides it.

Recomputation is the trade the GPU note already priced, applied here to activations. Flash-style attention never stores the full attention matrices. Tensor parallel, sequence parallelism, and recomputation together drop the activation memory on each device.

Production runs mix all three splits. Megatron-style training spans about 1B to 1T parameters. Tensor parallel stops near 8, pipeline parallel grows with depth, and data parallelism balances batch against compute. That mix reaches about 40 to 50 percent of theoretical GPU FLOPs.

The Builder Test

Name the wall in one sentence before you choose a split. Then say which resource the split spends to buy relief.

• Memory is the wall: shard the optimizer state first, and the traffic bill does not change.
• One matrix is too large: split the operation inside the node, and pay in intra-node bandwidth.
• The batch divides cleanly: add data-parallel replicas, and pay out of the batch budget.

Write that sentence down before you write the config.

What Carries

Parallelism is a response to the thing that does not fit. A split that answers no wall still costs bandwidth.

The plan so far lives on paper. The next step is to measure what the collectives cost on your links.