提出简化版Unigram分词算法，兼顾压缩效率与实现复杂度。
Which Pieces Does Unigram Tokenization Really Need?
- 改进实现方案，降低实际应用门槛
- 新算法训练损失略高但压缩效果更好
- 适合关注分词效率与部署便利性的研究者
Unigram分词算法为字节对编码提供了概率化的替代方案。尽管理论优雅，其实现复杂，限制了其应用范围，仅在SentencePiece及其衍生工具中使用。本文通过提供清晰的实现指南和参数选择建议，弥合了理论与实践之间的差距。同时，我们提出一种更简单的算法，在接受稍高的训练损失前提下，实现了更好的压缩性能。
原文摘要 · Abstract (English)
The Unigram tokenization algorithm offers a probabilistic alternative to the greedy heuristics of Byte-Pair Encoding. Despite its theoretical elegance, its implementation in practice is complex, limiting its adoption to the SentencePiece package and adapters thereof. We bridge this gap between theory and practice by providing a clear guide to implementation and parameter choices. We also identify a simpler algorithm that accepts slightly higher training loss in exchange for improved compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。