arXiv:2605.28128cs.CL2026-05被引 1

通过字符对齐投影修复中文分词错误,提升非标准文本标注稳定性。

Chinese Word Boundary Recovery through Character Alignment Projection

论文配图:Chinese Word Boundary Recovery through Character Alignment Projection
图 1 · 摘自论文原文
  • 基于字符级对齐,将清洁文本的分词边界投影回噪声源文本。
  • 在学习者语料上,相比直接分词,错误减少37.2%。
  • 适合中文教学、低质量文本处理场景使用。

中文分词在非标准文本中尤为脆弱,语言学习者错误及其他字符级偏差会破坏下游标注与评估所依赖的词边界。本文将中文词边界恢复建模为基于对齐的投影任务:给定带噪源句和更清洁的目标句,先在字符层面进行对齐,再将目标侧的词边界投影回源端。除提出方法外,还构建了两个评估资源:基于MuCGEC的手工校验学习者中文基准,以及源自中文Penn Treebank的受控合成基准。实验表明,直接分词在学习者输入中易产生复合词碎片化问题,而提出的两步投影方法能利用修正后的目标文本纠正源端的过度切分错误。结果证明词边界恢复不同于常规分词,且对齐投影为噪声输入下的中文标注与评估提供了可解释的稳定机制。

原文摘要 · Abstract (English)

Chinese word segmentation is especially fragile in non-standard text, where language learner errors and other character-level divergences disrupt the word boundaries assumed by downstream annotation and evaluation. This paper formulates Chinese word boundary recovery as an alignment-based projection task. Given a noisy source sentence and a cleaner target counterpart, we first align the two strings at the character level and then project target-side word boundaries back onto the source. Beyond the recovery method itself, we introduce two evaluation resources: a manually checked learner Chinese benchmark based on MuCGEC and a controlled synthetic benchmark derived from the Chinese Penn Treebank. Experiments show that direct segmentation remains vulnerable to compound fragmentation in learner input, whereas the proposed two step projection method corrects many over-segmentation errors by using the corrected target to recover source-side word spans. The results show that word boundary recovery is distinct from ordinary segmentation and that alignment projection provides a principled mechanism for stabilizing Chinese annotation and evaluation under noisy input.

中文分词噪声鲁棒对齐投影

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。