奖励模型架构

这部分内容有帮助吗？

参考文献

Training language models to follow instructions with human feedback, Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray, 2022 Advances in Neural Information Processing Systems, Vol. 35 DOI: 10.48550/arXiv.2203.02155 - 一篇基础性论文，描述了作为对齐大型语言模型RLHF流程一部分的奖励模型架构和训练过程。它涵盖了使用预训练LLM骨干网络和标量回归头来学习人类偏好的方法。
Constitutional AI: Harmlessness from AI Feedback, Yuntao Bai, Saurav Kadavath, Sandeep Trehan, John Chu, Long Nguyen, Andy Jones, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Etienne Fort, Zac Hatfield-Dodds, Danny Hernandez, Andrew Jones, Nicholas Joseph, Nelson Elhage, Zac Evans, Liane Lovitt, Cameron McKnight, Da Yan, Daniela Amodei, Sam McCandlish, Dario Amodei, and Tom Brown, 2022 arXiv preprint arXiv:2212.08073 DOI: 10.48550/arXiv.2212.08073 - 描述了一种利用AI反馈对齐大型语言模型的方法，该方法高度依赖于从人类偏好训练奖励模型（称为“偏好模型”），以评估回复的有用性和无害性。
Learning to summarize with human feedback, Mark Stiennon, Long Ouyang, Jeff Wu, Daniel Ziegler, Ryan Lowe, Jeffrey Schulman, Harish Agarwal, Noah Fiedel, Basri B. Erdogdu, and Kai Guo, 2020 Advances in Neural Information Processing Systems, Vol. 33 (Curran Associates, Inc.) DOI: 10.55919/00735 - 一项早期工作，将奖励模型和RLHF应用于文本摘要。它有效展示了使用Transformer模型学习人类对文本生成质量偏好的核心原则。
A Survey of Reinforcement Learning from Human Feedback, Yuanzhi Zhao, Zili Wang, Yuxin Li, Runji Lin, Xuanfan Ni, Shangqian Leng, and Jiangjie Chen, 2023 arXiv preprint arXiv:2312.09114 DOI: 10.48550/arXiv.2312.09114 - 一项全面调查，回顾了RLHF的各个方面，包括对不同奖励模型架构、训练技术及其在整体对齐流程中作用的详细讨论。