Combining Parallelism Strategies

When deploying massive architectures like meta-llama/Meta-Llama-3.1-70B-Instruct or 100B+ parameter models, a single GPU or even a single high-density node often lacks the memory capacity and computational throughput to meet production SLAs. While Tensor Parallelism efficiently splits individual matrix multiplications, it quickly bottlenecks on the communication bandwidth of the physical interconnect. Pushing Tensor Parallelism across slow network boundaries destroys generation speed. To scale horizontally across multiple physical machines without incurring massive latency penalties, infrastructure operators must combine intra-node Tensor Parallelism with inter-node Pipeline Parallelism, creating a multi-dimensional distributed topology.

The Physics of Interconnects and Topology Limits

The decision to combine parallelism strategies is strictly dictated by hardware interconnect speeds. Tensor Parallelism requires an all-reduce communication step at nearly every layer of the transformer. This requires moving massive activation tensors across GPUs continuously. On an enterprise chassis like an HGX A100 or H100, GPUs are connected via NVLink, providing 600 GB/s to 900 GB/s of bidirectional bandwidth. This makes Tensor Parallelism highly efficient within a single physical node.

However, when you exceed the 8 GPUs typically available in a single node, you must cross the network boundary using PCIe Gen4/Gen5, InfiniBand, or RoCE Ethernet. These inter-node connections peak between 50 GB/s and 100 GB/s. Running Tensor Parallelism over these slower connections will cause your hardware to spend more time waiting for network transfers than performing actual matrix multiplications.

Pipeline Parallelism solves this by slicing the model horizontally. Instead of splitting individual layers, it assigns sequential blocks of layers to different nodes. Node 1 processes layers 1 through 40, then sends a single hidden state tensor across the network to Node 2, which processes layers 41 through 80. By combining both methods, you confine the bandwidth-heavy Tensor Parallelism to the high-speed NVLink within each node and use Pipeline Parallelism to jump across the slower Ethernet connections between nodes.

Logical routing of a 2D Parallel topology combining Tensor Parallelism (TP=2) inside the node and Pipeline Parallelism (PP=2) across nodes.

Sizing Memory Footprints in Multi-Dimensional Topologies

Calculating the exact memory requirement per GPU in a combined topology requires dividing the model footprint across both dimensions. Tensor Parallelism splits the number of attention heads and hidden dimensions, while Pipeline Parallelism splits the total number of layers. Consequently, the memory scaling equation for both model weights and the KV cache is inversely proportional to the product of both parallel sizes.

$Memory_{per\_gpu} = \frac{Parameters \times bytes\_per\_param}{TP \times PP} + \frac{KV\_Cache_{total}}{TP \times PP} + Memory_{framework}$

For example, deploying meta-llama/Meta-Llama-3.1-70B-Instruct in an FP16 precision format (2 bytes per parameter) on two local nodes. Each node contains 4x RTX 4090 24GB GPUs. We have 8 total GPUs, so we set a Tensor Parallel size (TP) of 4 and a Pipeline Parallel size (PP) of 2.

The base weight memory footprint of the 70B parameter model is 140 GB. Weight Memory per GPU = 140 GB / (4 * 2) = 17.5 GB.

By utilizing this 2D combined topology, the static weights occupy exactly 17.5 GB of the available 24 GB VRAM on each RTX 4090. This leaves a healthy 6.5 GB margin per GPU for the dynamic KV cache and CUDA context overhead. If you attempted to run this entirely with TP=8 across nodes without Pipeline Parallelism, the memory footprint would be identical, but the decoding speed would plummet due to network bottlenecking on the all-reduce operations.

Implementing 2D Parallelism with Open Source Frameworks

Modern inference engines like vLLM abstract the complexity of cross-node scheduling and micro-batching. Under the hood, distributed deployments rely on Ray to manage the cluster topology and discover hardware nodes. To serve the 70B model across the multi-node setup calculated above, you first initialize a Ray cluster on your primary machine and attach the secondary machine to it.

# On Node 1 (IP: 192.168.1.100): Start the Ray head node
ray start --head --port=6379

# On Node 2: Connect to the primary node's Ray cluster
ray start --address=192.168.1.100:6379

# On Node 1: Launch the vLLM server with combined parallelism flags
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 2 \
  --dtype float16 \
  --max-model-len 8192 \
  --enforce-eager

In this configuration, vLLM automatically detects all 8 GPUs across the Ray cluster. It places the first half of the Llama 3 layers on Node 1 (sharded 4 ways via TP) and the second half on Node 2 (also sharded 4 ways).

Integrating Expert Parallelism for MoE Architectures

Mixture of Experts (MoE) models like mistralai/Mixtral-8x7B-Instruct-v0.1 introduce a third optimization vector known as Expert Parallelism (EP). While a model might contain 47 billion total parameters, it only activates roughly 13 billion parameters for any given token. Expert Parallelism isolates the MoE layers, placing specific expert networks on dedicated GPUs.

When combining Expert Parallelism with Tensor Parallelism, the dense parameters (like self-attention layers and embeddings) are replicated or sharded via Tensor Parallelism, while the expert feed-forward blocks are distributed completely independently based on the EP size.

$Memory_{moe\_gpu} = \left(\frac{Base\_Params}{TP} + \frac{Expert\_Params}{EP}\right) \times bytes\_per\_param$

If we load Mixtral 8x7B in FP16 on a workstation with 4x A100 80GB GPUs, setting both TP=2 and EP=2, we must separate the base parameters from the experts. The model contains roughly 3 billion base parameters and 44 billion expert parameters.

Base Memory Allocation = (3B / 2) * 2 bytes = 3 GB Expert Memory Allocation = (44B / 2) * 2 bytes = 44 GB Total Static Weight Memory = 47 GB per GPU.

This leaves 33 GB of VRAM on each A100 for continuous batching and the KV cache. Combining TP and EP ensures the high-bandwidth self-attention operations are parallelized efficiently, while routing prevents any single GPU from holding the entirety of the massive expert parameter count.

参考文献

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia, 2021 Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC21) DOI: 10.1145/3458817.3476209 - Details the implementation of combining tensor and pipeline parallelism across GPU clusters to minimize interconnect communication bottlenecks.
Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning, Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, 2022 OSDI 2022 - Formulates distributed model scaling as an optimization problem dividing operations between intra-node tensor sharding and inter-node pipeline stages.
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale, Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He, 2022 ICML 2022 - Explains expert parallelism and multi-dimensional distributed topologies for routing sparse models across multiple GPUs.
vLLM Distributed Serving Documentation, vLLM Team, 2024 vLLM Official Documentation - Provides configuration parameters and setup instructions for deploying language models across multiple hardware nodes using Ray and vLLM.

魏明 (Wei-Ming Thor)

• 创始人与工程师, ApX Machine Learning

专注于大语言模型架构分析与硬件容量规划，维护 ApX VRAM 计算器。