发现并利用大模型中负责多语言处理的注意力头，提升翻译准确性。
Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models
- 提出快速评估注意力头重要性的算法LAHIS，仅需一次前后传播。
- 发现特定语言与通用注意力头共存，前者可减少跨语言干扰。
- 轻量级适配只需20个参数，显著提升XQuAD测试准确率。
大型语言模型（LLMs）在多语言理解和生成方面日益重要，但其内部机制仍不清晰。本文研究多头自注意力（MHA）在多语言能力中的作用，提出语言注意力头重要性评分（LAHIS），通过一次前向和反向传播即可高效识别关键注意力头。在Aya-23-8B、Llama-3.2-3B和Mistral-7B-v0.1上应用该方法，揭示了语言特异性和语言通用性注意力头的存在。前者支持跨语言注意力传递，引导模型聚焦目标语言，缓解非目标语言生成问题。此外，我们设计一种轻量级适配策略，学习软注意力掩码以调控语言头输出，仅需20个可调参数即可提升XQuAD准确率。本工作从MHA视角增强了多语言LLMs的可解释性与性能。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly support multilingual understanding and generation. Meanwhile, efforts to interpret their internal mechanisms have emerged, offering insights to enhance multilingual performance. While multi-head self-attention (MHA) has proven critical in many areas, its role in multilingual capabilities remains underexplored. In this work, we study the contribution of MHA in supporting multilingual processing in LLMs. We propose Language Attention Head Importance Scores (LAHIS), an effective and efficient method that identifies attention head importance for multilingual capabilities via a single forward and backward pass through the LLM. Applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we reveal the existence of both language-specific and language-general heads. Language-specific heads enable cross-lingual attention transfer to guide the model toward target language contexts and mitigate off-target language generation issue, contributing to addressing challenges in multilingual LLMs. We also introduce a lightweight adaptation that learns a soft head mask to modulate attention outputs over language heads, requiring only 20 tunable parameters to improve XQuAD accuracy. Overall, our work enhances both the interpretability and multilingual capabilities of LLMs from the perspective of MHA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。