Aligning Large Language Models (LLMs) to follow human intentions and safety guidelines presents significant challenges, especially as models become more capable. While initial methods have shown promise, they encounter difficulties when applied at scale or to complex alignment goals. This chapter examines the limitations of existing techniques and establishes the need for more scalable approaches to AI oversight.

You will learn about:

• The specific shortcomings of Supervised Fine-Tuning (SFT) when used for detailed alignment tasks.
• The operational bottlenecks and potential biases associated with Reinforcement Learning from Human Feedback (RLHF), particularly concerning scalability.
• The concept of scalable oversight and why it's necessary for managing the behavior of advanced AI systems.
• The rationale for incorporating AI-generated feedback into the alignment process.
• An overview of the theoretical foundations supporting AI-assisted alignment methods, setting the stage for later chapters on Constitutional AI and RLAIF.

By understanding these foundational problems and concepts, you'll gain a clear perspective on why techniques like Constitutional AI and RLAIF were developed and the specific issues they aim to address.