发现多语言模型在跨语言转换时知识错乱，提出轻量方法提升一致性。
Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models
- 通过可解释性分析发现模型前几层用通用概念空间，最后几层才转为语言特定表示。
- 最终层语言转换失败会导致目标语言回答错误，即使其他语言正确。
- 提出线性捷径方法跳过最后一层计算，显著提升准确率和跨语言一致性。
多语言语言模型（MLMs）虽在多语言间存储事实知识，但常对语义等价的跨语言提示给出不一致回答。现有研究指出该问题，但成因未明。本文采用机制可解释性方法，探究MLMs中的跨语言不一致现象。结果表明，模型在多数层中以语言无关的概念空间编码知识，仅在最后几层转向语言特定空间。最终层的语言转换失败常导致目标语言预测错误，即便其他语言答案正确。为此，我们提出线性捷径方法，绕过最后几层计算，有效提升预测准确率与跨语言一致性。研究揭示了MLMs内部机制，并提供一种轻量高效的解决方案。
原文摘要 · Abstract (English)
Multilingual language models (MLMs) store factual knowledge across languages but often struggle to provide consistent responses to semantically equivalent prompts in different languages. While previous studies point out this cross-lingual inconsistency issue, the underlying causes remain unexplored. In this work, we use mechanistic interpretability methods to investigate cross-lingual inconsistencies in MLMs. We find that MLMs encode knowledge in a language-independent concept space through most layers, and only transition to language-specific spaces in the final layers. Failures during the language transition often result in incorrect predictions in the target language, even when the answers are correct in other languages. To mitigate this inconsistency issue, we propose a linear shortcut method that bypasses computations in the final layers, enhancing both prediction accuracy and cross-lingual consistency. Our findings shed light on the internal mechanisms of MLMs and provide a lightweight, effective strategy for producing more consistent factual outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。