大模型跨语言推理不一致，因各语言表征差异大
Language-Specific Latent Process Hinders Cross-Lingual Performance
- 通过表征相似性分析发现模型在不同语言间使用不共享的表征
- 大模型虽更擅长跨语言知识获取但表征分离更严重
- 引导小模型向共享语义空间对齐可提升跨语言一致性
大型语言模型（LLMs）具备跨语言迁移能力，但在用不同语言提问时会产生不一致输出。为理解模型如何实现跨语言知识泛化，我们测量了不同语言间的表征相似性，并运用logit lens解析模型解决多语言多项选择推理问题时的隐含步骤。分析显示，模型预测不一致且准确率较低，是因为其依赖于跨语言差异较大的表征，而非共享语义空间。尽管更大模型更具多语言能力，但其隐藏状态更易脱离共享表征，但仍更善于从不同语言中提取知识。最后，我们证明通过引导小模型的潜在处理过程向共享语义空间对齐，可增强其跨语言推理性能，实现更优的知识迁移和与英语输出的一致性。
原文摘要 · Abstract (English)
Large language models (LLMs) are demonstrably capable of cross-lingual transfer, but can produce inconsistent output when prompted with the same queries written in different languages. To understand how language models are able to generalize knowledge from one language to the others, we measure representation similarity between languages, and apply the logit lens to interpret the implicit steps taken by LLMs to solve multilingual multi-choice reasoning questions. Our analyses reveal LLMs predict inconsistently and are less accurate because they rely on representations that are dissimilar across languages, rather than working in a shared semantic space. While larger models are more multilingual, we show their hidden states are more likely to dissociate from the shared representation compared to smaller models, but are nevertheless more capable of retrieving knowledge embedded across different languages. Finally, we demonstrate that knowledge sharing in small models can be facilitated by steering their latent processing towards the shared semantic space. This improves the models' multilingual reasoning performance, as a result of more knowledge transfer from, and better output consistency with English.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。