arXiv:2608.08847cs.CL2026-08

用显式边界标记解决子词分词中同词多形问题,提升语言建模效果。

Explicit Boundary Markers for Subword Vocabularies

  • 引入显式词边界标记替代空格,避免同一词因空格/大小写分裂
  • 下游任务中所有标记方案均降低每字节比特数,平均提升0.5~1.2%
  • 适合需高精度语言建模的NLP任务,尤其关注词形一致性的场景

子词分词器在使用空格的书写系统中,会将常见词以带前导空格和不带前导空格两种形式表示,导致同一词被拆分为多个独立嵌入。不同形式的词可能被不同方式切分(如' together'为单一词条,而'together'拆为'to|gether'),大小写更使一个词分裂成最多六种形式。本文提出一种显式词边界标记机制,用边界符号分隔词语,空格由一对边界标记表示。通过两种移位编码处理首字母大写和全大写,实现同一词的内部表示复用。该方案缓解了重复条目问题,但未提升分词压缩率——在六种语言上,两种词汇学习算法的最佳标记方案字符/词元比仅比基线低1%以内。然而,所有测试标记方案在下游语言建模任务中均取得更低的比特/字节值,表明重复分裂带来压缩外成本,影响模型性能。

原文摘要 · Abstract (English)

Subword tokenizers represent many common words twice in space-using writing systems, once with a leading space and once without. The two entries have separate embeddings in models, so occurrences of one word are divided across rows that are trained independently, and the two forms need not even segment the string the same way: " together" may be a single entry while the same word without a preceding space is tokenized as "to|gether". Capitalization divides a word further, into as many as six forms. We introduce an alternative to standard whitespace conventions using an explicit word boundary marker, which prevents such duplication. Words are delimited by the boundary markers, and spaces between words are represented as pairs of such markers. Two shift codes do the same for title case and upper case, allowing one internal representation of a word to be re-used across different settings. Switching to this convention mitigates the duplicate-entry issue, but does not improve tokenization compression: for both vocabulary-learning algorithms, the best marker scheme stays within one percent of the baseline in characters per token, averaged across six languages. It does result in better language modeling performance. Every marker scheme tested downstream reaches lower bits per byte than the baseline, suggesting that duplication carries a cost that compression does not capture.

分词优化语言建模子词边界标记

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。