选择优化器超参数 (lr, betas, eps, weight_decay)

这部分内容有帮助吗？

参考文献

Decoupled Weight Decay Regularization, Ilya Loshchilov and Frank Hutter, 2019 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1711.05101 - 介绍了AdamW，它将权重衰减与自适应梯度更新解耦，这是LLM优化中的标准做法。
Adam: A Method for Stochastic Optimization, Diederik P. Kingma and Jimmy Ba, 2015 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1412.6980 - 介绍了Adam优化器的原始论文，解释了其一阶和二阶矩估计。
Scaling Laws for Neural Language Models, Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020 arXiv preprint DOI: 10.48550/arXiv.2001.08361 - 讨论了模型性能如何随模型大小、数据集大小和计算量而扩展，为LLM的超参数选择提供信息。
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He, 2017 arXiv preprint DOI: 10.48550/arXiv.1706.02677 - 展示了学习率随大批量大小的线性缩放规则，与LLM训练相关。