Transformer组件中的初始化

这部分内容有帮助吗？

参考文献

Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, 2017 Advances in Neural Information Processing Systems, Vol. 30 (NeurIPS) DOI: 10.5555/3295222.3295252 - 介绍了Transformer架构，是关于组件初始化讨论的基础。
Understanding the difficulty of training deep feedforward neural networks, Xavier Glorot, Yoshua Bengio, 2010 Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Vol. 9 (PMLR) - 提出了Xavier初始化，一种通过保持信号方差来稳定深度网络训练的方法。
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015 Proceedings of the IEEE International Conference on Computer Vision (ICCV 2015) (IEEE) DOI: 10.1109/ICCV.2015.122 - 介绍了Kaiming初始化，一种专门为ReLU及类似非线性激活函数设计的方差保持方法。
torch.nn.init, PyTorch Contributors, 2022 (PyTorch Foundation) - PyTorch初始化函数的官方文档，包括normal_、kaiming_uniform_、zeros_和ones_。