分词对困惑度的影响

这部分内容有帮助吗？

参考文献

Neural Machine Translation of Rare Words with Subword Units, Rico Sennrich, Barry Haddow, and Alexandra Birch, 2016 Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Association for Computational Linguistics) DOI: 10.18653/v1/P16-1162 - 介绍了用于子词分词的字节对编码（BPE），这是GPT-2等许多现代语言模型中使用的基础方法，与讨论的子词示例直接相关。
SentencePiece: A language independent subword tokenizer and detokenizer for Neural Text Processing, Taku Kudo and John Richardson, 2018 Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (Association for Computational Linguistics) DOI: 10.18653/v1/D18-2012 - 介绍了SentencePiece，一个实现了包括WordPiece在内的子词分词算法的工具包，对理解BERT等分词器的工作方式有帮助。
Tokenizers in the transformers library, Hugging Face team, 2024 - Hugging Face transformers库中分词器的官方文档，解释了如何加载和使用不同的分词算法，与Python代码示例直接相关。