All Courses

Mixture of Experts: Core Concepts and Hands-on Implementation

Chapter 1: Foundations of Mixture of Experts Models

Overview of Sparsely-Gated MoE Architecture

The Gating Network: Formulation and Function

Expert Networks: Specialization and Capacity

Mathematical Formulation of the MoE Layer

Load Balancing and Auxiliary Losses

Challenges in MoE Training: Expert Collapse

Comparison with Dense Model Scaling

Hands-on: Implementing a Basic MoE Layer

Chapter 2: Advanced Routing Mechanisms

Analysis of Top-k Gating and its Variants

Noisy Top-k Gating for Load Balancing

Hash-based Routing for Deterministic Selection

Switch Transformers: Simplified Routing

Soft MoE: Differentiable Routing

Analyzing Routing Decisions and Specialization

Hands-on: Implementing Different Routing Strategies

Chapter 3: Training and Optimization of Large-Scale MoEs

Expert Parallelism for Distributed Training

Combining Model, Data, and Expert Parallelism

Capacity Factor and its Impact on Performance

Techniques for Mitigating Router Z-Loss Instability

Precision and its Effects: BFloat16 Training

Fine-tuning Strategies for Pre-trained MoE Models

Practice: Configuring a Distributed Training Job

Chapter 4: Efficient Inference with MoE Models

Inference Challenges: Memory and Latency

Expert Offloading to CPU or NVMe

Batching Strategies for Sparse Activation

Model Distillation for MoE Compression

Quantization Techniques for MoE Layers

Speculative Decoding with MoE Models

Hands-on: Building an Optimized Inference Pipeline

Chapter 5: Integrating MoE into Modern Architectures

Replacing FFNs with MoE Layers in Transformers

Placement of MoE Layers: Frequency and Location

MoE in Vision Transformers (ViT)

MoE in Multi-modal Models

Architectural Variants and their Properties

Analyzing Parameter vs. FLOPs Trade-offs

Practice: Modifying a Transformer to use MoE

Mixture of Experts: Core Concepts and Hands-on Implementation

A technical course on building and deploying Mixture of Experts (MoE) models. It provides a detailed examination of the MoE architecture, from foundational mathematics to advanced implementation strategies. You will learn to construct, train, and optimize sparse models, with a focus on advanced routing algorithms, distributed training techniques, and efficient inference methods required for large-scale applications. The material covers the integration of MoE layers into modern Transformer architectures and the practical considerations for managing their performance.

Prerequisites Deep Learning & Transformers

Level:

Advanced

Advanced MoE Implementation
Implement various routing mechanisms for MoE layers, including noisy top-k and switch-style routing.
Large-Scale Training
Apply expert parallelism and other distributed training techniques to scale MoE models effectively.
Performance Optimization
Develop and apply load balancing loss functions to prevent expert collapse and improve training stability.
Efficient Inference
Construct optimized inference pipelines for sparse models using techniques like expert offloading and quantization.
Architectural Integration
Integrate MoE layers into existing Transformer models and analyze the performance trade-offs.

There are no prerequisite courses for this course.

There are no recommended next courses at the moment.

Login to Write a Review

Share your feedback to help other learners.

© 2026 ApX Machine LearningEngineered with