Advanced PyTorch
Chapter 1: PyTorch Internals and Autograd
Tensor Implementation Details
Understanding the Computational Graph
Autograd Engine Mechanics
Custom Autograd Functions: Forward and Backward
Higher-Order Gradient Computation
Inspecting Gradients and Graph Visualization
Memory Management Considerations
Hands-on Practical: Building Custom Autograd Functions
Chapter 2: Advanced Neural Network Architectures
Implementing Transformers from Components
Advanced Attention Mechanisms
Graph Neural Networks with PyTorch Geometric
Normalizing Flows for Generative Modeling
Neural Ordinary Differential Equations
Meta-Learning Algorithms
Practice: Implementing a Custom GNN Layer
Chapter 3: Optimization Techniques and Training Strategies
Sophisticated Optimizers Overview
Advanced Learning Rate Scheduling
Regularization Methods
Gradient Clipping and Accumulation
Mixed-Precision Training with torch.cuda.amp
Strategies for Handling Large Datasets
Automated Hyperparameter Tuning
Hands-on Practical: Implementing Mixed-Precision Training
Chapter 4: Model Deployment and Performance Optimization
TorchScript Fundamentals: Tracing vs Scripting
Model Quantization Techniques
Model Pruning Strategies
Performance Analysis with PyTorch Profiler
Optimizing Kernels with External Libraries
Exporting Models to ONNX Format
Serving Models with TorchServe
Practice: Profiling and Quantizing a Model
Chapter 5: Distributed Training and Parallelism
Fundamental Concepts of Distributed Computing
Data Parallelism with DistributedDataParallel (DDP)
Tensor Model Parallelism
Pipeline Parallelism Implementation
Fully Sharded Data Parallelism (FSDP)
Using torch.distributed Primitives
Setting up Distributed Environments
Hands-on Practical: Setting up a DDP Training Script
Chapter 6: Custom Extensions and Interoperability
Building Custom C++ Extensions
Building Custom CUDA Extensions
Working with the ATen Library
Interfacing PyTorch with NumPy
Extending torch.nn with Custom Modules
Extending torch.optim with Custom Optimizers
Foreign Function Interfaces (FFI)
Practice: Building a Simple CUDA Extension
Advanced Attention Mechanisms
Was this section helpful?
- Longformer: The Long-Document Transformer, Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020 arXiv preprint arXiv:2004.05150 DOI: 10.48550/arXiv.2004.05150 - Introduces an attention mechanism with a complexity linear with sequence length by combining local and global attention patterns.
- BigBird: A Self-Attention Mechanism for Long Context Transformers, Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed, 2020 Neural Information Processing Systems (NeurIPS) DOI: 10.48550/arXiv.2007.14062 - Presents a sparse attention mechanism that achieves linear complexity and can approximate full attention.
- Rethinking Attention with Performers, Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller, 2020 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.2009.14794 - Proposes the Performer model, which uses positive orthogonal random features to approximate softmax attention with linear complexity.
- Linformer: Self-Attention with Linear Complexity, Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao Ma, 2020 arXiv preprint arXiv:2006.04768 DOI: 10.48550/arXiv.2006.04768 - Describes an attention mechanism that achieves linear complexity by projecting the key and value matrices to a lower dimension.