硬件加速的考量

这部分内容有帮助吗？

参考文献

Sparsely-Gated Mixture-of-Experts Layers, Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean, 2017 arXiv preprint arXiv:1701.06538 DOI: 10.48550/arXiv.1701.06538 - 介绍了稀疏门控混合专家模型，阐述了其架构以及与稀疏性和动态路由相关的挑战。
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, William Fedus, Barret Zoph, Noam Shazeer, 2022 Journal of Machine Learning Research, Vol. 23 (JMLR) DOI: 10.55986/joxg921 - 描述了一种简化的MoE架构，并讨论了扩展性的工程实践，包括分布式训练和推理中的通信和内存优化。
A Domain-Specific Architecture for Deep Neural Networks, Norman P. Jouppi, Cliff Young, David Patil, Dustin Patterson, Gaetano Agostini, Shumeet Baluja, Keren Bergman, Ry Chiang, Sheng Li, Mike Ni, Vijay Nivargi, Paul Norman, Mike Reddi, Kevin Smith, David Sprague, Greg Thorson, Rajat Wadia, Kevin Walker, David Wang, Hongbo Wei, Christof Zabriskie, 2017 ACM SIGARCH Computer Architecture News, Vol. 45 (ACM) DOI: 10.1145/3144819.3144824 - 描述了谷歌张量处理单元（TPU）的架构，解释了其脉动阵列设计和高带宽内存，这些特性加速了机器学习工作负载。
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022 NeurIPS 2022 DOI: 10.48550/arXiv.2208.07339 - 提出了一种针对大型Transformer模型执行8位矩阵乘法的方法，显著减少了推理的内存占用和计算量，对MoE模型有影响。