Tensor, Pipeline, Expert, Data and Sequence Parallelism

By the end of this lesson you will be able to define each of the five parallelism strategies in one sentence, name the collective operation it performs and how often per token it performs it, write down how many bytes cross a link for a given split, and answer the question that decides every home-cluster design: which of these can survive a home network, and which cannot.

The five names get used loosely, including by people selling hardware. They are not interchangeable and they are not alternatives to each other: real systems compose several at once. The Megatron scaling paper is explicit about that, saying it shows “how different types of parallelism methods (tensor, pipeline, and data parallelism) can be composed to scale to thousands of GPUs and models with trillions of parameters”.

The thing being split

A transformer is a stack of identical layers. Each layer holds two blocks of weights: the attention projections and the feed-forward network. A token’s journey through one layer takes a vector of hidden_size numbers in and produces a vector of hidden_size numbers out. That vector is the only thing that has to travel between layers.

Two quantities set the entire communication budget for the rest of this lesson, so name them now.

Pseudocode — not a real command

activation_bytes = batch_size x tokens_in_flight x hidden_size x bytes_per_element
layer_count      = the number of transformer layers in the model

During decode, tokens_in_flight is one per sequence, so activation_bytes is small: for a model with a hidden size of 4,096 at two bytes per element, one token’s activation vector is eight kilobytes. Hold that number. Everything below is about how many times per token something of that size crosses a wire.

Tensor parallelism

Definition. Every weight matrix in every layer is cut into pieces and each device holds one piece. All devices work on the same token at the same time, each computing a partial result, and they combine those partial results before the layer can continue.

This is the approach the original Megatron-LM paper introduced, describing it as “a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters”. Its appeal was that it needed no new tooling: the paper notes the approach “does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch”. The authors trained an 8.3 billion parameter model on 512 GPUs and report sustaining “15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline”.

The collective. An all-reduce: every device contributes its partial sum and every device receives the total. It happens twice per layer in the forward pass, once after the attention block and once after the feed-forward block.

The cost. For a two-device split, each all-reduce moves roughly one activation tensor’s worth of bytes per device. Two per layer, across every layer:

Pseudocode — not a real command

tensor_parallel_bytes_per_token  ~  2 x layer_count x activation_bytes

For the eight-kilobyte activation above and a 64-layer model, the arithmetic gives about one megabyte crossing the link for every single token, and it has to happen before the next layer can start. That is the number that decides whether tensor parallelism is possible on your network, and the next lesson turns it into a link speed.

What it buys. Both devices read their half of the weights at the same time, so the bandwidth ceiling from the previous lesson roughly doubles. This is the only split that makes a single conversation faster.

Pipeline parallelism

Definition. The layers are dealt out in contiguous blocks. Device one holds layers 1 to 32, device two holds layers 33 to 64. A token passes through the first device, then crosses to the second.

The collective. None. It is a plain point-to-point send of the activation vector, once at each stage boundary. With two devices there is one boundary.

Pseudocode — not a real command

pipeline_parallel_bytes_per_token  ~  (stages - 1) x activation_bytes

Eight kilobytes per token across one boundary, against a megabyte for the tensor split of the same model. The arithmetic gives a ratio of roughly 128 to 1 for that example, and it scales with the layer count: the deeper the model, the more lopsided the comparison.

The cost. Idle time, usually called the bubble. While device one works on a token, device two has nothing to do, and afterwards the roles reverse. Serving engines hide this by keeping several requests in flight so that each stage always has something to work on, which is why pipeline parallelism helps throughput much more than it helps a single user. The Megatron scaling paper addresses the same problem at training scale, proposing “a novel interleaved pipeline parallelism schedule that can improve throughput by 10+% with memory footprint comparable to existing approaches”.

What it buys. Capacity, cheaply, over a slow link. This is why it is the cluster shape that works at home.

!Important

llama.cpp’s RPC backend is a layer split by another name. Its README says that “by default, llama.cpp distributes model weights and the KV cache across all available devices - both local and remote - in proportion to each device’s available memory”, with --tensor-split to override the proportions. Despite the flag’s name, what crosses the network per token is the small between-layers transfer, which is exactly why it tolerates ordinary Ethernet. Part 19 builds it.

Expert parallelism

Definition. In a mixture-of-experts model, each layer’s feed-forward block is replaced by many parallel experts and a router that sends each token to a small number of them. Expert parallelism puts different experts on different devices.

GShard scaled this arrangement to “beyond 600 billion parameters using automatic sharding”, training a multilingual translation model “on 2048 TPU v3 accelerators in 4 days”. The Switch Transformer work then simplified the routing: its abstract says the authors “simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs”, and reports “up to 7x increases in pre-training speed with the same computational resources”.

The collective. Every device may hold experts that tokens on every other device need, so the tokens are exchanged in a pattern where each device potentially sends to and receives from each other device. The collective that does this is called an all-to-all, and there are two per mixture-of-experts layer: one to dispatch the tokens to their experts and one to bring the results back. Neither abstract cited above names the collective; they describe the architecture and the sharding, and the all-to-all is the standard way it is implemented.

The cost. All-to-all is the least forgiving pattern on a home network, because its volume depends on the routing, which changes with the input. A batch whose tokens all route to one device’s experts creates a hotspot that no amount of average bandwidth fixes.

What it buys. It is the only way to hold a very large sparse model when no single device can hold all the experts of one layer. For a home cluster it is usually the wrong tool: a mixture-of-experts model split by layers, so that whole layers with all their experts live together, keeps the routing local and only pays the small between-layers transfer.

▤Memory budget

Mixture-of-experts models are the reason home clusters are interesting at all. gpt-oss-120b activates about 5B of its 117B parameters per token and Qwen3-235B-A22B about 22B of 235B, both recorded in the model reference. Capacity is what you need a second machine for; the per-token read stays small, so the bandwidth ceiling stays reachable. All three named here are Apache-2.0.

Data parallelism

Definition. Each device holds a complete copy of the model and works on different inputs.

The collective. For inference, nothing at all: two replicas never speak. For training, an all-reduce of the gradients once per optimisation step, which is a large transfer at a low frequency rather than a small one at a high frequency.

The cost. Memory. Every replica needs the whole model, so this does nothing for capacity.

What it buys. Throughput that scales almost linearly with machines, and the simplest failure model in this part: if one replica dies, the other still answers. The router in front of them is the gateway from Part 9. When people describe a home cluster that “just works”, this is usually what they have.

Sequence and context parallelism

Definition. The other four split the model. This one splits the input: a long sequence is cut into blocks and each device holds one block’s share of the keys and values.

Ring Attention is the clearest formulation. Its abstract describes an approach “which leverages blockwise computation of self-attention and feedforward to distribute long sequences across multiple devices while fully overlapping the communication of key-value blocks with the computation of blockwise attention”, and claims it “enables training and inference of sequences that are up to device count times longer than those achievable by prior memory-efficient Transformers, without resorting to approximations or incurring additional communication and computation overheads”.

The collective. Key-value blocks passed around a ring of devices, overlapped with the arithmetic so that the transfer is hidden behind work that was happening anyway.

The cost. It needs a link fast enough that the overlap actually holds. If moving a block takes longer than computing on one, the ring stalls and the claimed absence of overhead does not survive.

What it buys. Context length that scales with the number of devices. At home this matters when the key-value cache, not the weights, is what will not fit, which happens with very long documents and is the case Part 22 returns to.

Why tensor parallelism stays in the box

Put the five in one table of communication behaviour and the design rule writes itself.

Strategy	What is split	Collective	How often	Volume per event
Tensor	every weight matrix	all-reduce	twice per layer, per token	one activation tensor
Pipeline	contiguous blocks of layers	point-to-point send	once per stage boundary, per token	one activation tensor
Expert	the experts in each MoE layer	all-to-all	twice per MoE layer, per token	the routed tokens
Data	nothing; replicas	none for inference	never	nothing
Sequence	the sequence and its KV cache	ring exchange	per block, per step	one key-value block

Tensor and pipeline move the same amount per event. The difference is entirely the frequency: 2 x layer_count events per token against stages - 1. A memory bus can absorb that frequency because it is measured in tens of nanoseconds per access. A network link cannot, because even a perfect one has a fixed cost per exchange measured in microseconds, and there are hundreds of exchanges per token.

vLLM states the resulting rule directly. Its parallelism guidance says that “if the model is too large for a single GPU but fits on a single node with multiple GPUs, use tensor parallelism”, and that “if the model is too large for a single node, combine tensor parallelism with pipeline parallelism”, setting “tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes”. Tensor parallelism inside a chassis, pipeline parallelism between chassis. Part 20 configures exactly this on a pair of DGX Sparks.

The standard composition on two machines with two accelerators each

• workerMachine A, device 0layers 1-32, first half of every matrix
• workerMachine A, device 1layers 1-32, second half of every matrix
• workerMachine B, device 0layers 33-64, first half of every matrix
• workerMachine B, device 1layers 33-64, second half of every matrix

• Machine A, device 0 ⟷ connected to Machine A, device 1tensor parallel: all-reduce twice per layer, over the internal bus
• Machine B, device 0 ⟷ connected to Machine B, device 1tensor parallel: all-reduce twice per layer, over the internal bus
• Machine A, device 1 ⟷ connected to Machine B, device 0pipeline parallel: one send per token, over the network

The frequent, latency-sensitive traffic stays inside a chassis; the network carries one small transfer per token. Reverse this arrangement and the same hardware performs far worse.

The interconnect ladder

Everything above turns into one question: how fast is the path between the two pieces of the split? The ladder from fastest to slowest is worth keeping in your head, because a design decision is really a decision about which rung a given collective is allowed to land on.

Where a split can land, fastest first

1. On-package memoryUnified memory or a GPU's own memory. Hundreds of gigabytes per second on every track in this course. This is where "not split at all" lives.
2. NVLink between GPUs in one chassisA dedicated GPU-to-GPU link. Not present on the GeForce 40 and 50 series this course uses on Track N, as the hardware reference records, so a multi-card desktop here is a PCIe machine.
3. PCIe between cards in one desktopAn order of magnitude below on-package memory, and the realistic home ceiling for tensor parallelism inside a box.
4. A direct high-speed adapter link between two machinesThe two QSFP ports on a DGX Spark, or Thunderbolt 5 between two Macs. Fast enough to consider more than a layer split, and the subject of the next lesson.
5. Ordinary wired EthernetFrom 2.5 gigabit upwards. Comfortable for a layer split, hostile to an all-reduce at every layer.
6. Wi-FiShared, variable and reordering. Adequate for a router talking to a client; never the path a model is split across.

A split is a decision about which rung its collective lands on. Frequent collectives belong on the top rungs; a once-per-token send survives near the bottom.

Choosing, in order

Picking a strategy for machines you already own

1. 1Does the model fit on one machine?If yes, and you need more requests served, use data parallelism: a second replica behind the gateway.
2. 2Does it fit across the machines you have?Then pipeline parallelism, or llama.cpp RPC's proportional layer split, which is the same idea. One small transfer per token.
3. 3Do your machines each hold several accelerators?Tensor parallelism inside each machine, pipeline parallelism between them. This is vLLM's stated rule and Part 20's configuration.
4. 4Is the KV cache, not the weights, what does not fit?That is a sequence-parallel or a disaggregation problem. Part 22 separates prefill from decode for exactly this case.
5. 5Is it a mixture-of-experts model?Prefer a layer split that keeps each layer's experts together, so the routing stays local and the network never carries an all-to-all.

Trace one request through each split

Tensor parallelism divides operations within layers and introduces communication at those layer boundaries. Pipeline parallelism assigns groups of layers to stages; one request moves between stages, while multiple microbatches can fill the pipeline. Expert parallelism routes work to the devices holding selected experts. Data parallel serving uses replicas to handle different requests, while data parallel training also synchronises updates.

For a paper exercise, draw two devices and label what each stores, computes and transmits for one token. Then add several independent requests. A pipeline that is underfilled for one request can improve utilisation with more work in flight, but the latency of each request still includes traversal and communication.

Choose the split according to the limiting resource and engine support. Adding devices cannot divide every allocation or remove every replicated tensor. Record effective memory per rank and the topology seen by the runtime. The term “parallel” is incomplete unless you name what is partitioned, what remains replicated and how often devices must synchronise.

Recap

Five strategies, distinguished by what they split and what they must therefore communicate. Tensor splits every matrix and pays an all-reduce twice per layer per token: the only split that speeds up one conversation, and the one that needs the fastest link. Pipeline splits the layers into blocks and pays one small send per stage boundary per token: the split that buys capacity over an ordinary network, at the cost of idle stages that batching hides. Expert splits a mixture-of-experts layer’s experts and pays an all-to-all whose volume depends on the routing, which is the least predictable pattern to put on a home link. Data splits nothing, replicates everything, communicates nothing during inference, and is the easiest throughput win available. Sequence splits the input rather than the model and passes key-value blocks around a ring, which is how context grows with device count.

The design rule falls out of frequency rather than volume: tensor and pipeline move the same bytes per event, but tensor does it hundreds of times per token and pipeline once. Keep the frequent collectives on the fastest rung of the interconnect ladder you own, and let the network carry only the once-per-token transfer.

The next lesson takes 2 x layer_count x activation_bytes and turns it into a link speed you can buy.

Check your understanding

Question 1. A 64-layer model is split across two machines. Roughly how many activation-sized transfers cross the link per token under tensor parallelism, and how many under pipeline parallelism?

• 64 and 64
• 128 and 1
• 1 and 128
• 2 and 2

Show the answer and why

Answer: 128 and 1

Tensor parallelism runs an all-reduce after the attention block and after the feed-forward block of every layer: two per layer, so 128 for a 64-layer model. Pipeline parallelism with two stages has one boundary, so one transfer. Same size per transfer, wildly different frequency, and frequency is what a network cannot absorb.

Question 2. Your two machines each hold two GPUs and the model needs all four. Which arrangement does vLLM's guidance point at?

• Tensor parallel across all four GPUs
• Pipeline parallel across all four GPUs
• Tensor parallel within each machine, pipeline parallel between the two machines
• Data parallel with two replicas

Show the answer and why

Answer: Tensor parallel within each machine, pipeline parallel between the two machines

The documented rule is to set tensor parallel size to the number of GPUs per node and pipeline parallel size to the number of nodes. The frequent all-reduce then travels over the internal bus, and the network carries one transfer per token. A data-parallel arrangement would need each machine to hold the whole model, which is the case being ruled out.

Question 3. Which strategy makes a single user's conversation decode faster, rather than only making a larger model possible or serving more users?

• Pipeline parallelism
• Data parallelism
• Tensor parallelism
• Expert parallelism

Show the answer and why

Answer: Tensor parallelism

Only tensor parallelism has all devices reading their share of the weights at the same time, which is what lifts the bandwidth ceiling for one sequence. Pipeline reads sequentially, data parallelism does not change one replica, and expert parallelism addresses capacity for sparse models. The catch is the all-reduce at every layer, which is why this split needs a link far faster than a house has.

Question 4. Why is expert parallelism a poor fit for a home network even when the model is sparse?

• Mixture-of-experts models cannot be split across machines at all
• Its all-to-all exchange has a volume that depends on the routing, so a batch can create a hotspot that average bandwidth does not fix
• It requires more memory than tensor parallelism
• The experts must all be identical

Show the answer and why

Answer: Its all-to-all exchange has a volume that depends on the routing, so a batch can create a hotspot that average bandwidth does not fix

Routing is data-dependent, so the traffic pattern changes with the input and the worst case is much worse than the average. A layer split that keeps each layer's experts on the same machine avoids the problem entirely: the routing stays inside one box and only the small between-layers activation crosses the wire.

Question 5. True or false: sequence parallelism helps when the weights do not fit in memory.

• True
• False

Show the answer and why

Answer: False

Sequence parallelism splits the input and its key-value cache, not the weights. Every device still needs the model. It is the answer when a very long context is what overflows, which is a different problem from the weights not fitting, and it is why Part 22 treats long-prompt work as its own architecture.

Check my answersClear