arXiv:2603.17912cs.CLstat.ML2026-03

用预训练模型的注意力机制量化语言距离,揭示语言亲缘关系。

Pretrained Multilingual Transformers Reveal Quantitative Distance Between Human Languages

  • 利用多语言模型的注意力矩阵,通过最优传输计算语言间表征距离。
  • 在多种语言上验证了语言距离与地理、历史关系高度一致。
  • 该方法可提升低资源翻译性能,适合语言学与AI交叉研究者。

理解人类语言间的距离是语言学、人类学及追溯人类演化史的核心问题。尽管语言学长期提供丰富的定性描述,但统一且可扩展的定量测量方法仍缺乏。本文提出一种新方法,利用预训练多语言模型作为系统性语言测量工具。具体而言,我们发现这些模型自发产生的注意力机制可提供一种鲁棒且分词无关的语言距离度量——注意力传输距离(ATD)。通过将注意力矩阵视为概率分布,并使用最优传输衡量其几何差异,我们量化了翻译过程中不同语言间的表征距离。将ATD应用于大规模多样化的语言集,结果表明其能高保真恢复已知语言分类,并揭示与地理和接触相关的关系模式。此外,将ATD作为正则化项可提升低资源机器翻译的迁移性能。本研究为使用人工神经网络检验语言学假说提供了严谨基础,使多语言模型成为定量语言学探索的强大工具,推动更公平的多语言人工智能发展。

原文摘要 · Abstract (English)

Understanding the distance between human languages is central to linguistics, anthropology, and tracing human evolutionary history. Yet, while linguistics has long provided rich qualitative accounts of cross-linguistic variation, a unified and scalable quantitative approach to measuring language distance remains lacking. In this paper, we introduce a method that leverages pretrained multilingual language models as systematic instruments for linguistic measurement. Specifically, we show that the spontaneously emerged attention mechanisms of these models provide a robust, tokenization-agnostic measure of cross-linguistic distance, termed Attention Transport Distance (ATD). By treating attention matrices as probability distributions and measuring their geometric divergence via optimal transport, we quantify the representational distance between languages during translation. Applying ATD to a large and diverse set of languages, we demonstrate that the resulting distances recover established linguistic groupings with high fidelity and reveal patterns aligned with geographic and contact-induced relationships. Furthermore, incorporating ATD as a regularizer improves transfer performance in low-resource machine translation. Our results establish a principled foundation for testing linguistic hypotheses using artificial neural networks. This framework transforms multilingual models into powerful tools for quantitative linguistic discovery, facilitating more equitable multilingual AI.

语言距离注意力机制多语言模型量化语言学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。