RLAIF 系统的超参数调整

这部分内容有帮助吗？

参考文献

Proximal Policy Optimization Algorithms, John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017 arXiv preprint arXiv:1707.06347 DOI: 10.48550/arXiv.1707.06347 - 介绍PPO算法的奠基性论文，详细阐述了裁剪目标、GAE及其在稳定策略优化中的实际应用。
Training language models to follow instructions with human feedback, Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe, 2022 NeurIPS DOI: 10.48550/arXiv.2203.02155 - 一篇里程碑式的论文，展示了如何通过人类反馈强化学习（RLHF）来对齐大型语言模型，并提供了PM和PPO调优的实际考量。
Constitutional AI: Harmlessness from AI Feedback, Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan, 2022 arXiv preprint arXiv:2212.08073 DOI: 10.48550/arXiv.2212.08073 - 介绍了宪法式AI，一种使用AI反馈（RLAIF）对齐LLM的方法，与课程章节主题直接相关。
Optuna: A Next-generation Hyperparameter Optimization Framework, Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, Masanori Koyama, 2019 Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Association for Computing Machinery) DOI: 10.1145/3292500.3330701 - 描述了Optuna，一个高效的超参数优化框架，支持贝叶斯优化和其他高级调优策略。