Advanced JAX: Performance, Optimization, and Scale
Chapter 1: Advanced JAX Transformations and Control Flow
Review of Core Transformations: jit, grad, vmap
Mastering lax.scan for Sequential Operations
Conditional Execution with lax.cond
Looping with lax.while_loop
Combining Control Flow and Transformations
Advanced Masking Techniques
Understanding Closures and JAX Staging
Practical: Implementing Complex Recurrent Logic
Chapter 2: Optimizing JAX Code for Performance
Profiling JAX Code on CPU, GPU, and TPU
Understanding JAX Computation Graphs (jaxpr)
The Role of XLA Compilation
Memory Layout and Its Impact on Performance
Avoiding Recompilation
Fusion and Operator Optimization
Asynchronous Dispatch
Practice: Optimizing a Numerical Computation
Chapter 3: Distributed Computing with JAX
Introduction to Parallelism Concepts
Device Management in JAX
Single-Program Multiple-Data (SPMD) with pmap
Implementing Data Parallelism using pmap
Collective Communication Primitives (psum, pmean, etc.)
Handling Axis Names in pmap
Nested pmap and Advanced Partitioning
Introduction to Multi-Host Programming
Practice: Distributed Data-Parallel Training
Chapter 4: Advanced Automatic Differentiation Techniques
Review of Forward- and Reverse-Mode Autodiff
Jacobian-Vector Products (JVPs) with jax.jvp
Vector-Jacobian Products (VJPs) with jax.vjp
Higher-Order Derivatives
Computing Full Jacobians and Hessians
Custom Differentiation Rules with jax.custom_vjp
Custom Differentiation Rules with jax.custom_jvp
Differentiation through Control Flow Primitives
Handling Non-Differentiable Functions
Practice: Implementing a Custom Gradient
Chapter 5: Interoperability and Custom Operations
Integrating JAX with NumPy
Zero-Copy Data Sharing with DLPack
Calling External CPU/GPU Code with jax.experimental.host_callback
Using jax.pure_callback for Side-Effect Free Calls
Introduction to JAX Primitives
Defining Custom Primitives
Implementing Abstract Evaluation Rules
Implementing Lowering Rules for Backends (CPU/GPU/TPU)
Defining Differentiation Rules for Custom Primitives
Practice: Integrating a C++ Function
Chapter 6: Large-Scale Model Training Techniques
Overview of Challenges in Large Model Training
Introduction to JAX Ecosystem Libraries (Flax, Haiku)
Managing Model Parameters and State
Combining pmap with Training Frameworks
Gradient Accumulation
Gradient Checkpointing (Re-materialization)
Mixed Precision Training
Model Parallelism Strategies
Optimization Algorithms for Large Scale
Practice: Implementing Gradient Checkpointing
Gradient Accumulation
Was this section helpful?
- Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - Introduces the concept of mini-batch gradient descent and discusses techniques to approximate large batches.
- Language Models are Few-Shot Learners, Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020 Advances in Neural Information Processing Systems (NeurIPS) 33 DOI: 10.48550/arXiv.2005.14165 - Describes the training of GPT-3, a very large language model, where techniques like gradient accumulation are essential for memory management.
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He, 2020 International Conference for High Performance Computing, Networking, Storage and Analysis (SC) DOI: 10.48550/arXiv.1910.02054 - Presents a set of memory optimization techniques, including aspects that leverage or complement gradient accumulation for very large-scale model training.
- Large Model Training Best Practices, Flax Developers, 2024 (Flax) - Official Flax guide detailing strategies for training large models, explicitly covering gradient accumulation as a core technique.