稀疏注意力机制简介

这部分内容有帮助吗？

参考文献

Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, 2017 Advances in Neural Information Processing Systems, Vol. 30 (Neural Information Processing Systems) DOI: 10.5555/3295222.3295349 - 介绍Transformer架构和自注意力机制的基础论文，强调其相对于序列长度的二次复杂度。
Longformer: The Long-Document Transformer, Iz Beltagy, Matthew E. Peters, and Arman Cohan, 2020 arXiv preprint arXiv:2004.05150 DOI: 10.48550/arXiv.2004.05150 - 介绍Longformer模型，该模型结合滑动窗口和全局注意力模式，高效处理超长序列。
Big Bird: Transformers for Longer Sequences, Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed, 2020 Advances in Neural Information Processing Systems, Vol. 33 (Curran Associates, Inc.) DOI: 10.5555/3455793.3455928 - 提出BigBird，一种结合全局、局部和随机注意力的稀疏注意力机制，实现线性复杂度，同时在长序列任务上保持良好性能。