RLHF: Reinforcement Learning from Human Feedback
Chapter 1: Foundations of RLHF for Language Model Alignment
The AI Alignment Problem in LLMs
Limitations of Supervised Fine-Tuning
Reinforcement Learning Principles Refresher
Introduction to the RLHF Process
Setting Up the Development Environment
Chapter 2: Supervised Fine-Tuning (SFT) Phase
Role of SFT in the RLHF Pipeline
Curating High-Quality SFT Datasets
SFT Implementation Details
Evaluating SFT Model Performance
Hands-on Practical: SFT Execution
Chapter 3: Reward Modeling from Human Preferences
Concept of Learning from Preferences
Human Preference Data Collection
Preference Dataset Formats and Structures
Reward Model Architectures
Training Objectives for Reward Models
Calibration of Reward Models
Potential Issues in Reward Modeling
Hands-on Practical: Training a Reward Model
Chapter 4: RL Fine-Tuning with Proximal Policy Optimization (PPO)
PPO Algorithm for RLHF Context
Policy and Value Network Implementation
The Role of the KL Divergence Penalty
Calculating Advantages and Returns
PPO Hyperparameter Tuning for LLMs
Common PPO Implementation Libraries (TRL)
Troubleshooting PPO Training Instability
Practice: Implementing the PPO Update Step
Chapter 5: Integrating the Full RLHF Pipeline
Workflow Orchestration
Model Loading and Initialization
Generating Responses with the Policy Model
Scoring Responses with the Reward Model
Synchronizing Models During Training
Code Structure for an End-to-End RLHF System
Hands-on Practical: Running a Simplified RLHF Loop
Chapter 6: Advanced RLHF Techniques and Alternatives
Direct Preference Optimization (DPO)
Reinforcement Learning from AI Feedback (RLAIF)
Improving Sample Efficiency in RLHF
Addressing Reward Hacking Explicitly
Multi-Objective Reward Models
Contextual and Conditional RLHF
Practice: Comparing PPO and DPO Concepts
Chapter 7: Evaluation, Analysis, and Deployment
Metrics for Evaluating Aligned Models
Human Evaluation Protocols
Automated Evaluation Suites
Analyzing Policy Shift During RL Tuning
Red Teaming and Safety Testing
Computational Costs and Scalability
Deployment Considerations for RLHF Models
Hands-on Practical: Analyzing RLHF Run Logs
Reward Model Architectures
Was this section helpful?
- Training language models to follow instructions with human feedback, Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray, 2022 Advances in Neural Information Processing Systems, Vol. 35 DOI: 10.48550/arXiv.2203.02155 - A foundational paper describing the Reward Model architecture and training process as part of the RLHF pipeline for aligning large language models. It covers the use of a pre-trained LLM backbone and a scalar regression head to learn human preferences.
- Constitutional AI: Harmlessness from AI Feedback, Yuntao Bai, Saurav Kadavath, Sandeep Trehan, John Chu, Long Nguyen, Andy Jones, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Etienne Fort, Zac Hatfield-Dodds, Danny Hernandez, Andrew Jones, Nicholas Joseph, Nelson Elhage, Zac Evans, Liane Lovitt, Cameron McKnight, Da Yan, Daniela Amodei, Sam McCandlish, Dario Amodei, and Tom Brown, 2022 arXiv preprint arXiv:2212.08073 DOI: 10.48550/arXiv.2212.08073 - Describes an approach to aligning LLMs using AI feedback, which heavily relies on training a Reward Model from human preferences (referred to as a "preference model") to evaluate responses for helpfulness and harmlessness.
- Learning to summarize with human feedback, Mark Stiennon, Long Ouyang, Jeff Wu, Daniel Ziegler, Ryan Lowe, Jeffrey Schulman, Harish Agarwal, Noah Fiedel, Basri B. Erdogdu, and Kai Guo, 2020 Advances in Neural Information Processing Systems, Vol. 33 (Curran Associates, Inc.) DOI: 10.55919/00735 - An early work applying Reward Modeling and RLHF to text summarization. It effectively illustrates the core principles of using transformer models to learn human preferences for text generation quality.
- A Survey of Reinforcement Learning from Human Feedback, Yuanzhi Zhao, Zili Wang, Yuxin Li, Runji Lin, Xuanfan Ni, Shangqian Leng, and Jiangjie Chen, 2023 arXiv preprint arXiv:2312.09114 DOI: 10.48550/arXiv.2312.09114 - A comprehensive survey reviewing various aspects of RLHF, including detailed discussions on different Reward Model architectures, training techniques, and their role in the overall alignment pipeline.