使用 DeepSpeed ZeRO 优化

这部分内容有帮助吗？

参考文献

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Samyam Rajbhandari, Cong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He, 2020 SC '20: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (ACM) DOI: 10.1145/3418579.3441319 - 阐述了ZeRO概念及其三个阶段，解释了内存优化机制。
DeepSpeed ZeRO-powered Data Parallelism, DeepSpeed Team, 2024 - 在DeepSpeed框架中配置和使用ZeRO优化的官方资源。
DeepSpeed: Extreme-Scale Model Training for Everyone, Jeff Rasley, Samyam Rajbhandari, Kazem Cheshmi, Chris Ping, Yuxiong He, 2020 KDD '20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Association for Computing Machinery (ACM)) DOI: 10.1145/3394486.3403154 - 介绍了DeepSpeed框架，ZeRO是其核心组件之一，并讨论了其训练大规模模型的能力。