Comparative Analysis: Parameters vs Performance Trade-offs
Was this section helpful?
Prefix-Tuning: Optimizing Continuous Prompts for Generation, Xiang Lisa Li, Percy Liang, 2021Annual Meeting of the Association for Computational Linguistics (ACL)DOI: 10.48550/arXiv.2101.00190 - This paper introduces Prefix-Tuning, a method for fine-tuning by optimizing a continuous prefix prepended to the input, showing its effectiveness for text generation tasks.
Power of Scale for Parameter-Efficient Prompt Tuning, Brian Lester, Rami Al-Rfou, Noah Constant, 2021Conference on Empirical Methods in Natural Language Processing (EMNLP)DOI: 10.48550/arXiv.2104.08691 - This work presents Prompt Tuning, an extremely parameter-efficient method that learns soft prompt embeddings for task adaptation, achieving strong performance with larger models.
QLoRA: Efficient Finetuning of Quantized LLMs, Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer, 2023Conference on Neural Information Processing Systems (NeurIPS)DOI: 10.48550/arXiv.2305.14314 - This paper describes QLoRA, an optimization for LoRA that quantizes the base model to 4-bit, enabling fine-tuning of very large models on consumer GPUs, which is highly relevant to memory efficiency.