发现跨语言生成关键的检索过渡头，提升多语言推理能力
Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads
- 提出检索过渡头（RTH），连接上下文检索与目标语言生成
- 在4个基准上，遮蔽RTH导致性能下降比遮蔽检索头更严重
- 适合研究多语言大模型机制或改进跨语言推理的学者
近期研究发现Transformer中部分注意力头为检索头，负责从上下文中提取信息。本文首次研究多语言场景下的检索头，发现其常在多种语言间共享。进一步拓展至跨语言设置，识别出检索-过渡头（RTH），它控制向特定目标语言输出的转换。实验表明，RTH与普通检索头不同，对多语言大模型的思维链推理更为关键。在四个多语言基准（MMLU-ProX、MGSM、MLQA、XQuaD）和两个模型家族（Qwen-2.5、Llama-3.1）上，遮蔽RTH导致的性能下降显著大于遮蔽检索头。本工作通过分离出负责映射目标语言的注意力头，深化了对多语言大模型机制的理解。
原文摘要 · Abstract (English)
Recent work has identified a subset of attention heads in Transformer as retrieval heads, which are responsible for retrieving information from the context. In this work, we first investigate retrieval heads in multilingual contexts. In multilingual language models, we find that retrieval heads are often shared across multiple languages. Expanding the study to cross-lingual setting, we identify Retrieval-Transition heads(RTH), which govern the transition to specific target-language output. Our experiments reveal that RTHs are distinct from retrieval heads and more vital for Chain-of-Thought reasoning in multilingual LLMs. Across four multilingual benchmarks (MMLU-ProX, MGSM, MLQA, and XQuaD) and two model families (Qwen-2.5 and Llama-3.1), we demonstrate that masking RTH induces bigger performance drop than masking Retrieval Heads (RH). Our work advances understanding of multilingual LMs by isolating the attention heads responsible for mapping to target languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。