视觉Transformer (ViT) 中的MoE

这部分内容有帮助吗？

参考文献

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, 2020 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.2010.11929 - 介绍了视觉Transformer (ViT) 架构，它是将混合专家模型应用于图像数据的基础。
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean, 2017 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1701.06538 - 提出了稀疏门控混合专家层的基础概念，是现代混合专家模型架构的核心组成部分。
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, William Fedus, Barret Zoph, Noam Shazeer, 2021 arXiv preprint, Vol. 23 DOI: 10.48550/arXiv.2101.03961 - 描述了Switch Transformer架构，展示了混合专家模型如何在大规模参数模型中保持计算效率。
Vision MoE: An Empirical Study of Scaling Laws for MoE in Vision, William Fedus, Jeff Dean, Zhifeng Chen, Yuanzhong Xu, Anna Goldie, Basil Mustafa, Anushan Fernando, George Tucker, Yonghui Wu, David So, Blake Hechtman, Barret Zoph, David R. So, Aditya Sharma, Hieu Pham, Quoc V. Le, Paul Barham, Daniel N. Freeman, Albin Cassirer, Jiantao Jiao, Shibo Wang, Claire Cui, Ewa Dominowska, H. Yang, A. Mirhoseini, 2022 International Conference on Machine Learning (ICML) DOI: 10.48550/arXiv.2203.05605 - 研究了混合专家模型在视觉Transformer中的应用，详细说明了大型视觉模型的扩展性与性能。