从结构视角揭示大模型多语言能力差异与训练影响
Multilinguality of Large Language Models From a Structural Perspective
- 通过表征结构分析语言模型的多语言特性
- 低资源语言与英语结构差异更大，高/中资源语言更接近
- 语言特异性微调改变结构但保持跨语言关系
大型语言模型（LLMs）虽以英语为主进行预训练和后训练，仍能有效处理多种语言。以往研究聚焦于词元表示，揭示了模型对非英语文本的处理机制，但未能捕捉语言固有的结构性特征。本研究通过表征结构分析，揭示低资源语言在结构上比高、中资源语言更远离英语，且语言特定的后训练会改变其结构，同时维持不同语言间的关联性。
原文摘要 · Abstract (English)
Large language models (LLMs) have excelled in processing multiple languages through pre- and post-training on multilingual data, even though English dominates the training data. Prior work focusing on token representations has revealed how those LLMs process non-English text. Although these analyses have provided insightful findings, they fail to capture a structural view, which is an inherent property of language. In this study, we explore the multilinguality of LLMs through representational structural analysis. Our findings reveal that low-resource languages are structurally more different from English than high- and mid-resource languages, and that language-specific post-training alters their structures while preserving inter-language relationships.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。