RLHF: Reinforcement Learning from Human Feedback
Chapter 1: Foundations of RLHF for Language Model Alignment
The AI Alignment Problem in LLMs
Limitations of Supervised Fine-Tuning
Reinforcement Learning Principles Refresher
Introduction to the RLHF Process
Setting Up the Development Environment
Chapter 2: Supervised Fine-Tuning (SFT) Phase
Role of SFT in the RLHF Pipeline
Curating High-Quality SFT Datasets
SFT Implementation Details
Evaluating SFT Model Performance
Hands-on Practical: SFT Execution
Chapter 3: Reward Modeling from Human Preferences
Concept of Learning from Preferences
Human Preference Data Collection
Preference Dataset Formats and Structures
Reward Model Architectures
Training Objectives for Reward Models
Calibration of Reward Models
Potential Issues in Reward Modeling
Hands-on Practical: Training a Reward Model
Chapter 4: RL Fine-Tuning with Proximal Policy Optimization (PPO)
PPO Algorithm for RLHF Context
Policy and Value Network Implementation
The Role of the KL Divergence Penalty
Calculating Advantages and Returns
PPO Hyperparameter Tuning for LLMs
Common PPO Implementation Libraries (TRL)
Troubleshooting PPO Training Instability
Practice: Implementing the PPO Update Step
Chapter 5: Integrating the Full RLHF Pipeline
Workflow Orchestration
Model Loading and Initialization
Generating Responses with the Policy Model
Scoring Responses with the Reward Model
Synchronizing Models During Training
Code Structure for an End-to-End RLHF System
Hands-on Practical: Running a Simplified RLHF Loop
Chapter 6: Advanced RLHF Techniques and Alternatives
Direct Preference Optimization (DPO)
Reinforcement Learning from AI Feedback (RLAIF)
Improving Sample Efficiency in RLHF
Addressing Reward Hacking Explicitly
Multi-Objective Reward Models
Contextual and Conditional RLHF
Practice: Comparing PPO and DPO Concepts
Chapter 7: Evaluation, Analysis, and Deployment
Metrics for Evaluating Aligned Models
Human Evaluation Protocols
Automated Evaluation Suites
Analyzing Policy Shift During RL Tuning
Red Teaming and Safety Testing
Computational Costs and Scalability
Deployment Considerations for RLHF Models
Hands-on Practical: Analyzing RLHF Run Logs
Automated Evaluation Suites
Was this section helpful?
- TruthfulQA: Measuring How Models Mimic Human Falsehoods, Stephanie Lin, Jacob Hilton, and Owain Evans, 2021 ACL 2022 (main conference) DOI: 10.48550/arXiv.2109.07958 - Introduces a benchmark for factual accuracy in large language models (LLMs).
- HELM: Holistic Evaluation of Language Models, Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher RĂ©, Diana Acosta-Navas, Drew J. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, Yuta Koreeda, 2023 Transactions on Machine Learning Research (TMLR) DOI: 10.48550/arXiv.2211.09110 - Presents a standardized framework for comprehensive LLM evaluation across many metrics and scenarios.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica, 2023 NeurIPS 2023 Datasets and Benchmarks Track DOI: 10.48550/arXiv.2306.05685 - Examines the reliability of using LLMs as automated evaluators and introduces MT-Bench for multi-turn dialogue assessment.
- EleutherAI/lm-evaluation-harness, EleutherAI, 2024 (EleutherAI) - Official repository for the EleutherAI language model evaluation framework, providing tools for running many NLP evaluation tasks.
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan, 2022 arXiv preprint arXiv:2204.05862 DOI: 10.48550/arXiv.2204.05862 - Describes an early approach to RLHF, including evaluation strategies for model helpfulness and harmlessness during alignment.