优化分词可显著减少低资源模型的令牌数，提升性能。
When Every Token Counts: Optimal Segmentation for Low-Resource Language Models
- 通过实验发现最优BPE配置比贪心分词更高效
- 小模型上令牌数量减少，生成与分类任务性能提升
- 适合多语言及低资源场景下的模型优化
传统贪心分词在自然语言处理中至关重要，影响文本到令牌的转换并直接决定模型表现。尽管子词分词器如字节对编码（BPE）被广泛使用，其在不同模型规模和语言下的最优性仍存疑问。本文通过大量实验表明，最优BPE配置相比贪心分词显著降低令牌数量，在令牌节省率和性能方面均有提升，尤其在小型模型上效果明显。我们在多种内在与外在任务中评估了分词性能，涵盖生成与分类任务。结果表明，以压缩为目标的分词策略在多语言和低资源语言应用中具有显著优势，为未来研究和包容性NLP提供了新方向。
原文摘要 · Abstract (English)
Traditional greedy tokenization methods have been a critical step in Natural Language Processing (NLP), influencing how text is converted into tokens and directly impacting model performance. While subword tokenizers like Byte-Pair Encoding (BPE) are widely used, questions remain about their optimality across model scales and languages. In this work, we demonstrate through extensive experiments that an optimal BPE configuration significantly reduces token count compared to greedy segmentation, yielding improvements in token-saving percentages and performance benefits, particularly for smaller models. We evaluate tokenization performance across various intrinsic and extrinsic tasks, including generation and classification. Our findings suggest that compression-optimized tokenization strategies could provide substantial advantages for multilingual and low-resource language applications, highlighting a promising direction for further research and inclusive NLP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。