改进分词算法，让低资源语言更公平
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
- 在每步合并时优先提升最差语言的压缩效果
- 跨语言分词不平等减少89%，全球压缩率几乎不变
- 适合关注多语言公平性的NLP研究者
分词是大多数自然语言处理流程的第一步，却常被忽视。现有基于频率的分词算法倾向于主流语言，导致低资源语言分词结果更长、形态不合理，甚至充满未登录词占位符。这加剧了不同语言用户间的计算与经济不平等。为此，我们提出公平感知的字节对编码（Parity-aware BPE），在每一步合并时采用公平最大化策略，优先提升当前压缩最差语言的效率，以微小全局压缩损失换取跨语言公平性。实验表明，该方法可使各语言分词成本的吉尼系数相对经典BPE降低最高89%，对整体压缩率影响极小，且下游语言模型性能无系统性下降。
原文摘要 · Abstract (English)
Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines. Standard algorithms for learning tokenizers rely on frequency-based objectives, which favor languages dominant in the training data and consequently leave lower-resource languages with tokenizations that are disproportionately longer, morphologically implausible, or even riddled with $<UNK>$ placeholders. This phenomenon ultimately amplifies computational and financial inequalities between users from different language backgrounds. To remedy this, we introduce Parity-aware Byte Pair Encoding (BPE), a variant of the widely-used BPE algorithm. At every merge step, Parity-aware BPE applies a fair-max rule that maximizes the compression gain of the currently worst-compressed language, trading a small amount of global compression for cross-lingual parity. We find empirically that Parity-aware BPE reduces tokenization inequality -- operationalized by the Gini coefficient of per-language token costs -- by up to 89% relative to Classical BPE. This comes with negligible impact on global compression rate and no evidence of systematic degradation in downstream LM performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。