Mixture of Experts Parameters Base Versus Routed

Mixture of Experts (MoE) architectures change how infrastructure operators calculate hardware requirements for language models. Unlike dense models where every parameter is read from High Bandwidth Memory (HBM) and multiplied for every single token, MoE models decouple the total resident memory footprint from the active compute required per token. When sizing infrastructure for an MoE model, you must calculate two distinct numbers: the total VRAM required to hold the model weights on the GPU, and the subset of those weights that actively consume memory bandwidth during execution.

Base Parameters Versus Routed Experts

In a standard dense transformer, the architecture consists of attention layers and feed-forward (MLP) layers stacked sequentially. In an MoE architecture, the attention layers remain shared across all tokens. These are the base parameters. The standard dense feed-forward layers are replaced by a routing mechanism and a set of independent feed-forward networks called experts. These are the routed parameters.

When a token passes through an MoE layer, the router assigns it to a small subset of the available experts, usually one or two. This is often referred to as top-1 or top-2 routing. The GPU only needs to read the attention weights and the weights of the selected experts from memory for that specific token, leaving the unselected experts idle in VRAM.

Token data flowing through shared base attention parameters and being selectively routed to active experts while bypassing idle experts.

Sizing VRAM for MoE Architectures

Calculating the physical footprint for an MoE model requires adding the base parameters to the total sum of all routed experts. Marketing naming conventions like "8x7B" are often misunderstood. A model like mistralai/Mixtral-8x7B-Instruct-v0.1 does not have 56 billion parameters. It has exactly 46.7 billion total parameters.

This happens because the base parameters are not duplicated by a factor of 8. Only the feed-forward experts are duplicated. The total parameter equation is defined as:

$Total\_Parameters = Base\_Parameters + (Number\_of\_Experts \times Parameters\_per\_Expert)$

To size the hardware for FP16 precision, where each parameter occupies 2.0 bytes, apply the standard conversion:

$VRAM_{Weights} = Total\_Parameters \times 2\_bytes$

For Mixtral 8x7B in FP16, this requires $46.7 \times 2 = 93.4$ GB of VRAM just to load the weights. This model will immediately out of memory (OOM) on an 80GB A100 accelerator or a dual-RTX 4090 workstation (48GB total) without quantization. By converting the model to a 4-bit integer format like GGUF (Q4_K_M), the footprint shrinks to roughly 0.6 bytes per parameter. The VRAM requirement drops to $46.7 \times 0.6 = 28$ GB. This fits comfortably on a Mac Studio with 64GB of unified memory or a dual-GPU 48GB setup with plenty of capacity left over for the KV cache.

Active Compute and Memory Bandwidth Impact

While total VRAM dictates the physical hardware capacity required, the active parameters dictate inference speed and latency SLAs.

During the decode phase of text generation, token generation is strictly bound by memory bandwidth. You calculate the active parameter footprint to measure how much data must cross the HBM interface per token:

$Active\_Parameters = Base\_Parameters + (Active\_Experts\_per\_Token \times Parameters\_per\_Expert)$

Mixtral 8x7B uses top-2 routing. For any single token, the active parameter count is exactly 12.9 billion.

At FP16 precision, generating one token requires moving 25.8 GB of data from HBM to the streaming multiprocessors. If you run this on a workstation with ~1,000 GB/s of memory bandwidth, your theoretical maximum single-batch decode speed is $1000 / 25.8 = 38.7$ tokens per second, resulting in a Time-Per-Output-Token (TPOT) of 25.8ms. In terms of physical capacity, you are hosting a 47B model. In terms of generation speed and compute constraints per token, you are running a 13B model.

Inspecting the Architecture with Python

Inspecting an MoE model with Hugging Face Transformers exposes the shared attention layers alongside the routed expert layers:

from transformers import AutoModelForCausalLM
import torch

# Load a small MoE model to inspect the parameter routing structure
model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True
)

# Print the first transformer layer to see base vs expert separation
first_layer = model.model.layers[0]
print(first_layer)

# Expected Output breakdown:
# 1. self_attn (Base: Multi-Head Attention shared by all tokens)
# 2. mlp.gate (Base: The routing mechanism)
# 3. mlp.experts (Routed: A ModuleList containing the individual expert MLPs)

Batch Sizing and Expert Thrashing

The efficiency of an MoE model behaves differently under concurrent loads. In a single-batch scenario, only 2 experts out of 8 are loaded from memory per layer.

As concurrent requests scale and the batch size grows, you process multiple tokens simultaneously. Each token in the batch is routed independently. At a sufficiently high batch size, usually around 16 or 32 for an 8-expert model, every expert in the layer will be requested by at least one token in the batch.

When this happens, the GPU must load all 46.7B parameters from memory for the forward pass, losing the memory bandwidth amortization benefits of the sparse architecture. This effect is known as expert thrashing. It forces your throughput scaling to hit a memory bandwidth ceiling much faster than an equivalent dense 13B model. When performing capacity planning for high-concurrency production endpoints, you must benchmark MoE models assuming near-dense memory bandwidth utilization at peak loads.