Self-Instruct: Aligning LLMs with Your Own Instructions, Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi, 2022ACL 2023DOI: 10.48550/arXiv.2212.10560 - Introduces a method for generating synthetic instruction-following data using LLMs themselves, a key technique for fine-tuning and improving model capabilities without extensive human labeling.
The Curse of Recursion: Training on Generated Data Makes Models Forget, Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson, 2023arXiv preprint arXiv:2305.17493DOI: 10.48550/arXiv.2305.17493 - Investigates the phenomenon of 'model collapse' where training generative models repeatedly on synthetically generated data can lead to degradation in performance and loss of accuracy to original data distributions.