Interconnect Requirements for Distributed LLMs

When a large language model outgrows the memory capacity of a single GPU, operators must distribute the model weights and runtime activations across multiple accelerators. The speed at which text generates is no longer restricted strictly by the internal memory bandwidth of the GPU. Instead, the physical connections linking the GPUs become the primary bottleneck. The type of distributed topology you select dictates exactly how much data must travel across these interconnects, and mismatched topologies will stall compute operations while hardware waits for data packets to arrive.

Interconnect Hardware and Bandwidth Tiers

Before mapping distributed workloads, infrastructure operators must quantify the physical bandwidth available between accelerators. Interconnect speeds vary by orders of magnitude depending on the hardware architecture.

• PCIe Gen 4 and Gen 5: Standard consumer and workstation motherboards connect GPUs via Peripheral Component Interconnect Express. A standard PCIe Gen 4 x16 slot provides roughly 32 GB/s of unidirectional bandwidth. Bidirectional data transfer rates can theoretically reach 64 GB/s, but protocol overhead typically reduces practical throughput to around 25 GB/s. This is the operating reality for multi-GPU setups using hardware like the RTX 3090 or RTX 4090 24GB.
• NVLink and Enterprise Fabrics: Enterprise accelerators bypass the motherboard entirely. The NVIDIA A100 80GB utilizes third-generation NVLink to achieve 600 GB/s of direct GPU-to-GPU bandwidth. The H100 80GB pushes this to 900 GB/s. Older consumer cards like the RTX 3090 maintain a bridge NVLink connection at 112 GB/s, though NVIDIA removed this feature from the 40-series.
• Unified Memory Fabrics: Apple Silicon architectures, such as the M2 or M3 Ultra with 128GB to 192GB of unified RAM, utilize an internal ultra-fusion fabric connecting compute chiplets. This acts as a high-speed interconnect delivering up to 800 GB/s of memory bandwidth directly shared between the CPU and GPU cores without discrete transfer boundaries.
• Inter-Node Networking: When scaling to multiple server chassis, GPUs must communicate over traditional networks. A standard 100 Gigabit Ethernet (100GbE) or InfiniBand connection provides 12.5 GB/s of bandwidth. High-end clusters utilize 400GbE to achieve 50 GB/s between nodes.

Hardware interconnect tiers ordered by practical throughput capabilities.

Tensor Parallelism and the All-Reduce Bottleneck

Tensor Parallelism shards individual matrix multiplications across multiple GPUs. If you distribute a model across four GPUs using Tensor Parallelism, every single transformer layer must synchronize its partial results before passing the activation to the next layer. This synchronization is executed via an all-reduce communication pattern.

The total volume of data transmitted over the interconnect during a single forward pass all-reduce operation is calculated by the following equation:

$Data_{AllReduce} = 2 \times \left( 2 \times Batch \times SeqLen \times HiddenDim \right)$

The inner 2 represents the byte size of an FP16 or BF16 parameter. The outer 2 accounts for the multiplier intrinsic to the all-reduce communication pattern.

To understand the interconnect penalty, we can calculate the prefill phase overhead for loading a dense model like meta-llama/Meta-Llama-3.1-70B-Instruct across four RTX 4090 GPUs. The Llama 3.1 70B architecture has a hidden dimension of 8192 and contains 80 stacked transformer layers. If an operator submits a single prompt with a sequence length of 4096 tokens, the communication volume per layer is:

$Data_{AllReduce} = 2 \times \left( 2 \times 1 \times 4096 \times 8192 \right) = 134,217,728 \text{ bytes (134.2 MB)}$

Across all 80 layers, the GPUs must exchange 10.7 GB of data just to process the initial prompt.

If these GPUs are connected via PCIe Gen 4 x16 with a practical throughput of 25 GB/s, transmitting 10.7 GB takes approximately 428 milliseconds of pure network time. This is half a second added directly to the Time-to-First-Token (TTFT) SLA, completely ignoring the time spent doing actual matrix computation. If the same model runs on four A100 GPUs connected via 600 GB/s NVLink, the communication phase completes in a negligible 18 milliseconds.

Because of this heavy communication tax, Tensor Parallelism is strictly recommended for intra-node environments with NVLink or extremely high-bandwidth direct connections.

Pipeline Parallelism and Point-to-Point Transfers

Pipeline Parallelism places different sequential segments of a model on different hardware. If you split Llama 3.1 70B using a pipeline degree of 2, GPU 0 holds layers 1 through 40, and GPU 1 holds layers 41 through 80.

These GPUs do not need to synchronize continuously. They only communicate exactly once at the boundary between layer 40 and layer 41. The operation is a simple point-to-point (P2P) transfer of the forward hidden states. The data volume required at the pipeline boundary is defined as:

$Data_{P2P} = 2 \times Batch \times SeqLen \times HiddenDim$

Using the same 4096-token prompt for Llama 3.1 70B, the boundary transfer volume drops dramatically:

$Data_{P2P} = 2 \times 1 \times 4096 \times 8192 = 67,108,864 \text{ bytes (67.1 MB)}$

A 67.1 MB payload is trivial for interconnects. Even over a standard 100G Ethernet inter-node network yielding 12.5 GB/s, this transfer completes in about 5 milliseconds. Consequently, Pipeline Parallelism is the standard topology for splitting models across separate server nodes or across consumer GPUs that lack NVLink bridge interfaces.

Expert Parallelism and MoE Routing Overhead

Mixture of Experts models like mistralai/Mixtral-8x7B-Instruct-v0.1 introduce an additional communication burden known as Expert Parallelism. In MoE architectures, a router evaluates each token and sends it to a specific expert layer.

If the model is split across four GPUs, there is a high probability that a token processed on GPU 0 must be routed to an expert residing on GPU 3. This forces an all-to-all communication pattern where tokens are continuously scattered and gathered across the interconnect during the decode phase. Over standard PCIe links, this rapid, low-latency scatter-gather routing creates severe traffic collisions. Operators hosting MoE models on consumer hardware often observe that Expert Parallelism scales poorly compared to pure Tensor Parallelism, requiring fallback to unified memory architectures or direct NVLink connections to maintain acceptable decode speeds.

Configuring Distributed Topologies in Software

When deploying production infrastructure, orchestration frameworks require explicit topology mapping to match the physical hardware constraints. Using vLLM, a standard production serving engine, operators assign tensor and pipeline dimensions via CLI arguments.

To deploy a 70B model across an eight-GPU cluster composed of two physically separate nodes linked by 100G Ethernet, you must isolate the high-bandwidth operations within the nodes and push the low-bandwidth operations across the network.

vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --pipeline-parallel-size 2 \
    --gpu-memory-utilization 0.90

The --tensor-parallel-size 4 argument groups four GPUs together to execute the rapid all-reduce synchronization locally over the server motherboard or internal NVLink. The --pipeline-parallel-size 2 argument instructs the framework to divide the model depth in half, transmitting the 67 MB boundary states across the Ethernet link connecting the two separate machines.

Hardware mapping matching Tensor Parallelism to local, high-bandwidth interconnects and Pipeline Parallelism to slower network bounds.

Aligning the parallelism strategy with the physical interconnect specifications ensures the compute cores spend the maximum amount of time performing arithmetic, rather than idling while waiting for token data to traverse a constrained physical wire.