MoE 中的路由机制

这部分内容有帮助吗？

参考文献

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean, 2017 arXiv preprint arXiv:1701.06538 DOI: 10.48550/arXiv.1701.06538 - 介绍了稀疏门控专家混合（MoE）层的基本概念，详细阐述了Top-k门控、噪声Top-k门控以及负载平衡损失的初步概念。
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen, 2020 arXiv preprint arXiv:2006.16668 DOI: 10.48550/arXiv.2006.16668 - 改进了MoE的路由机制，为超大模型引入了增强的负载平衡策略和专家容量的概念。
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, William Fedus, Barret Zoph, Noam Shazeer, 2022 Journal of Machine Learning Research, Vol. 23 DOI: 10.48550/arXiv.2101.03961 - 介绍了Switch Transformers，这是一种使用k=1路由的高效MoE架构，强调了专家容量和负载平衡等实用考虑因素，用于扩展模型。