MoE 层中的负载均衡

这部分内容有帮助吗？

参考文献

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, William Fedus, Barret Zoph, Noam Shazeer, 2021 Journal of Machine Learning Research DOI: 10.48550/arXiv.2101.03961 - 本文介绍了Switch Transformer架构、辅助负载均衡损失公式以及用于专家混合模型的容量因子。
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean, 2017 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1701.06538 - 这篇基础性论文介绍了稀疏门控专家混合层，并讨论了为实现专家有效利用而进行负载均衡的挑战。
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen, 2020 arXiv preprint DOI: 10.48550/arXiv.2006.16668 - 本文描述了一个用于大型模型的条件计算框架，详细介绍了专家混合层的有效负载均衡策略和分布式训练。