Advanced AI Infrastructure Design and Optimization
Chapter 1: Chapter 1: Architectural Patterns for AI Platforms
MLOps Principles at Scale
Compute Selection: CPU, GPU, and TPU Architectures
High-Bandwidth Interconnects for Distributed Systems
Storage Solutions for Large-Scale AI Datasets
Networking Topologies for ML Clusters
Hands-on Practical: Environment and Tooling Setup
Chapter 2: Chapter 2: Engineering Distributed Model Training
Data Parallelism with Synchronous and Asynchronous Updates
Model and Pipeline Parallelism for Large Models
Implementing Training with Horovod
Leveraging Microsoft DeepSpeed for ZeRO and Offloading
Fault Tolerance and Checkpointing in Long-Running Jobs
Hands-on Practical: Distributed Training with PyTorch FSDP
Chapter 3: Chapter 3: Advanced Resource Orchestration with Kubernetes
Managing ML Workflows with KubeFlow Pipelines
Advanced GPU Scheduling and Sharing
Cluster Autoscaling for Dynamic ML Workloads
Strategies for Using Spot and Preemptible Instances
Multi-Tenancy with Namespaces, Quotas, and Priority Classes
Practice: Configure a GPU-Aware Autoscaling Group
Chapter 4: Chapter 4: High-Performance Model Inference and Serving
Architecting Inference Services for Latency and Throughput
Model Optimization with TensorRT and ONNX Runtime
Model Quantization Techniques: INT8 and FP8
Serving Multiple Models with NVIDIA Triton Inference Server
A/B Testing and Canary Deployments for Models
Hands-on Practical: Deploying an Optimized Model on Triton
Chapter 5: Chapter 5: Scalable Data Management and Feature Engineering
Designing and Implementing a Feature Store
Real-time vs. Batch Feature Computation
Data Versioning and Lineage with DVC and Pachyderm
High-Throughput Data Processing with Spark and Ray
Managing Data Lakes and Data Warehouses for AI
Practice: Build a Basic Feature Ingestion Pipeline
Chapter 6: Chapter 6: Financial Operations and Governance for AI
Applying FinOps Principles to ML Workloads
Cost Attribution and Showback Models for ML Teams
Optimizing Cloud Storage Costs for Datasets
Right-Sizing Compute for Training and Inference
Automating Cost Anomaly Detection
Governance Policies for Resource Consumption
Practice: Analyzing a Cloud Cost and Usage Report
Data Parallelism with Synchronous and Asynchronous Updates
Was this section helpful?
- Scaling Distributed Machine Learning with the Parameter Server, Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, Bor-Yiing Su, 2014 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI '14) (USENIX Association) - This seminal paper introduces the parameter server architecture, a distributed framework fundamental for understanding asynchronous data parallelism in large-scale machine learning.
- Horovod: fast and easy distributed deep learning in TensorFlow, Alexander Sergeev, Mike Del Balso, 2018 arXiv preprint arXiv:1802.05799 DOI: 10.48550/arXiv.1802.05799 - Discusses Horovod, a popular framework that uses the All-Reduce collective operation for efficient synchronous data parallel training, offering insights into its implementation and benefits.
- DistributedDataParallel (DDP), PyTorch Team, 2024 - Official documentation for PyTorch's primary synchronous data parallelism module, providing practical guidance for its usage and configuration.
- Distributed Machine Learning Systems, Jinjun Cai, Song Guo, Min-Hua Huang, Yang Li, Xiaoyong Li, Wenlong Ma, Yong-Min Wang, Jianxiong Xiao, Xiaochao Wang, Xilong Wu, Junyuan Xie, Chunjing Xu, Shuai Zheng, Wenqiang Zhang, 2021 Synthesis Lectures on Data Mining and Knowledge Discovery (Morgan & Claypool Publishers) DOI: 10.2200/S01099ED1V01Y202105DMK019 - A comprehensive book covering various aspects of distributed machine learning systems, including different parallelism strategies, communication patterns, and system designs.