arXiv:2603.27653cs.CL2026-03中稿 · CAWL 2026

量化文字系统对变音符号的依赖程度,揭示其如何影响自动修复效果。

The Degree of Language Diacriticity and Its Effect on Tasks

  • 用信息论方法衡量变音符号的频率、歧义和结构多样性。
  • 多变音符号语言中结构复杂度与修复准确率相关性更强。
  • 为跨语言文本处理提供可量化的书写系统复杂度基准。

变音符号用于澄清发音、区分近义词或改变语义,在多种文字系统中起关键作用,但其对语言技术的影响尚未在跨语言层面得到系统量化。现有研究多聚焦单一语言,缺乏数据驱动的跨语言评估框架。本文提出一种基于语料库的信息论框架,通过频率、模糊性和结构多样性等指标量化变音符号复杂度,覆盖15种语言的24个语料库,涵盖单变音与多变音文字系统。进一步分析该复杂度与变音符号恢复任务性能的关系,对比BERT与RNN模型表现。结果表明,整体上变音复杂度越高,恢复准确率越低;在单变音系统中,频率与结构指标一致性高;而在多变音系统中,结构复杂度对性能的预测力超过频率指标。研究证明,书写系统的可度量特征直接影响模型表现,凸显正字法复杂性的功能性价值。

原文摘要 · Abstract (English)

Diacritics are orthographic marks that clarify pronunciation, distinguish similar words, or alter meaning. They play a central role in many writing systems, yet their impact on language technology has not been systematically quantified across scripts. While prior work has examined diacritics in individual languages, there's no cross-linguistic, data-driven framework for measuring the degree to which writing systems rely on them and how this affects downstream tasks. We propose a data-driven framework for quantifying diacritic complexity using corpus-level, information-theoretic metrics that capture the frequency, ambiguity, and structural diversity of character-diacritic combinations. We compute these metrics over 24 corpora in 15 languages, spanning both single- and multi-diacritic scripts. We then examine how diacritic complexity correlates with performance on the task of diacritics restoration, evaluating BERT- and RNN-based models. We find that across languages, higher diacritic complexity is strongly associated with lower restoration accuracy. In single-diacritic scripts, where character-diacritic combinations are more predictable, frequency-based and structural measures largely align. In multi-diacritic scripts, however, structural complexity exhibits the strongest association with performance, surpassing frequency-based measures. These findings show that measurable properties of diacritic usage influence the performance of diacritic restoration models, demonstrating that orthographic complexity is not only descriptive but functionally relevant for modeling.

变音符号正字法语言建模跨语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。