用双语数据让目标语言分词更懂源语言，提升语言建模效果
Conditional Unigram Tokenization with Parallel Data
- 根据双语数据中的源语言词，动态调整目标语言分词概率
- 语言建模困惑度持续降低，但机器翻译质量未提升
- 适合关注跨语言分词优化的研究者或多语言模型开发者
我们提出条件无监督分词（conditional unigram tokenization），通过利用平行语料中的源语言词，对目标语言分词概率进行条件化建模。在固定源语言分词器的前提下，该方法学习能最大化跨语言语义对齐的目标分词器。我们在四种不同语系、资源水平的语言对上评估该分词器，考察其内在属性及在机器翻译与语言建模任务上的表现。结果表明，该分词器保持与标准无监督分词器相近的统计特性，但在机器翻译上未见性能提升；而在语言建模中则呈现一致的困惑度下降。我们推测，条件概率估计随词表大小呈二次增长，造成数据效率瓶颈。研究提示，实现实用的跨语言分词可能需要替代参数化方式。
原文摘要 · Abstract (English)
We introduce conditional unigram tokenization, a novel approach that extends unigram tokenization by conditioning target token probabilities on source-language tokens from parallel data. Given a fixed source tokenizer, our method learns a target tokenizer that maximizes cross-lingual semantic alignment. We evaluate our tokenizer on four language pairs across different families and resource levels, examining intrinsic properties and downstream performance on machine translation and language modeling. While our conditional tokenizer maintains comparable statistical properties to standard unigram tokenizers, results are mixed: we observe no improvements in machine translation quality, but find consistent perplexity reductions in language modeling. We hypothesize that quadratic scaling of conditional probability estimation with respect to the vocabulary size creates a data efficiency bottleneck. Our findings suggest that alternative parameterizations may be necessary for practical cross-lingual tokenization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。