分析参数与FLOPs的权衡

这部分内容有帮助吗？

参考文献

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean, 2017 International Conference on Learning Representations (ICLR) 2017 - 介绍了稀疏门控混合专家（MoE）层，展示了如何在不显著增加每令牌计算成本的情况下大幅提升模型容量。
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, William Fedus, Barret Zoph, and Noam Shazeer, 2022 Journal of Machine Learning Research, Vol. 23 (JMLR, Inc. and Microtome Publishing) - 介绍了Switch Transformer，将MoE层简化为每令牌一个活跃专家（k=1），展示了其将模型扩展到万亿参数的能力，并解决了通信和负载均衡方面的挑战。
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts, Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Quoc V. Le, and Zhifeng Chen, 2022 arXiv preprint arXiv:2201.05824 DOI: 10.48550/arXiv.2201.05824 - 详细介绍了GLaM，一种MoE架构，与同等质量的密集模型相比，以显著更少的训练FLOP实现了高性能，强调了效率和规模。
A Survey of Mixture of Experts, Xufeng Lin, Yiming Qian, Yuanyang Liu, Huadong Liu, Xizhen Sun, Jianyang Li, Guanyu Chen, Qingyu Jin, Meng Zhang, and Bo Xu, 2023 arXiv preprint arXiv:2308.14073 (arXiv) DOI: 10.48550/arXiv.2308.14073 - 全面概述了混合专家模型，涵盖其历史、架构变体、训练技术以及效率和负载均衡等实际考虑。