趋近智

模型大小如何衡量

这部分内容有帮助吗？

参考文献

Language Models are Few-Shot Learners, Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei, 2020 Advances in Neural Information Processing Systems (NeurIPS) DOI: 10.48550/arXiv.2005.14165 - 介绍了GPT-3，一个拥有1750亿参数的语言模型，展示了规模显著增加的模型的能力。这项工作为LLM中的“大型”设定了新基准，并显示了参数数量对少样本学习的影响。
Scaling Laws for Neural Language Models, Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020 arXiv preprint arXiv:2001.08361 DOI: 10.48550/arXiv.2001.08361 - 本文系统地研究了神经语言模型性能如何随模型大小（参数）、数据集大小和训练计算资源而扩展。它为参数数量与模型容量、性能和资源需求之间的关系提供了理论和实证见解。