内存与计算开销

这部分内容有帮助吗？

参考文献

LoRA: Low-Rank Adaptation of Large Language Models, Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, 2021 arXiv DOI: 10.48550/arXiv.2106.09685 - 介绍了LoRA方法，通过向Transformer层注入可训练的低秩矩阵来适应预训练模型，大幅减少了可训练参数数量和内存占用。
Parameter-Efficient Transfer Learning for NLP, Neil Houlsby, Andrei Giurgiu, Stanislau Padolski, Quentin de Latour, Max Vladutu, Albert Verga, Quincy Hatcliff, Jason Riesa, Anna Schiff, Shauna Horn, Melvin Johnson, George Dahl, Orhan Firat, 2019 Proceedings of the 36th International Conference on Machine Learning (ICML), Vol. 97 - 提出了适配器模块，即插入在Transformer层之间的小型神经网络层，实现了通过少量额外参数进行高效微调。
Prefix-Tuning: Optimizing Continuous Prompts for Generation, Xiang Lisa Li, Percy Liang, 2021 Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Vol. Volume 1: Long Papers (Association for Computational Linguistics) DOI: 10.18653/v1/2021.acl-long.353 - 介绍了前缀微调，在每个Transformer层的输入中添加一小段可训练向量（前缀），冻结基础模型并减少了可训练参数。
The Power of Scale for Parameter-Efficient Prompt Tuning, Brian Lester, Rami Al-Rfou, Noah Constant, 2021 Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) (Association for Computational Linguistics) DOI: 10.18653/v1/2021.emnlp-main.243 - 讨论了提示微调，仅优化一小组可训练的token（软提示），在大型模型上以最小参数开销达到与完全微调相当的性能。