字节对编码 (BPE) 算法

这部分内容有帮助吗？

参考文献

A New Algorithm for Data Compression, Philip Gage, 1994 C/C++ Users Journal, Vol. 12 - 首次提出字节对编码作为通用数据压缩算法的论文。
Neural Machine Translation of Rare Words with Subword Units, Rico Sennrich, Barry Haddow, and Alexandra Birch, 2016 Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Association for Computational Linguistics) DOI: 10.18653/v1/P16-1162 - 将BPE应用于神经机器翻译的开创性工作，使其在自然语言处理领域得到广泛应用。
Language Models are Unsupervised Multitask Learners, Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, 2019 (OpenAI) - 在GPT-2模型中引入字节级BPE（BBPE）进行分词，解决了任意Unicode文本处理的挑战。
Natural Language Processing with Transformers: Building Innovative Applications with 🤗 Transformers, Lewis Tunstall, Leandro von Werra, and Thomas Wolf, 2022 (O'Reilly Media) - 在现代大语言模型和Hugging Face生态系统的背景下，提供了包括BPE在内的分词技术的实用和最新概述。