发现大模型翻译中语义与语言分离的注意力机制，仅改1%头就能实现指令自由翻译。
Translation Heads: Disentangling meaning from language in LLM-based machine translation
- 通过分析注意力头，将翻译拆解为语言生成与语义保持两个任务。
- 在20个翻译方向上，各任务由稀疏特定注意力头负责，改动1%头即可媲美指令提示。
- 适合对大模型可解释性、低资源翻译优化感兴趣的开发者与研究者。
机制可解释性旨在揭示神经网络如何实现其能力，但大语言模型的规模限制了以往机器翻译领域的可解释性研究仅能停留在词级分析。本文从机制视角研究句级机器翻译，通过分析注意力头理解大模型如何内部编码和分配翻译功能。我们将翻译分解为两个子任务：生成目标语言文本（即目标语言识别）和保持输入句子的语义等价性。在三个开源模型家族及20个翻译方向上，我们发现不同的稀疏注意力头集合分别专用于每个子任务。基于此发现，我们构建了子任务特异性控制向量，仅修改1%相关注意力头即可实现无需指令提示的翻译性能，接近指令提示效果；而选择性消融这些头会分别破坏对应翻译功能。
原文摘要 · Abstract (English)
Mechanistic Interpretability (MI) seeks to explain how neural networks implement their capabilities, but the scale of Large Language Models (LLMs) has limited prior MI work in Machine Translation (MT) to word-level analyses. We study sentence-level MT from a mechanistic perspective by analyzing attention heads to understand how LLMs internally encode and distribute translation functions. We decompose MT into two subtasks: producing text in the target language (i.e. target language identification) and preserving the input sentence's meaning (i.e. sentence equivalence). Across three families of open-source models and 20 translation directions, we find that distinct, sparse sets of attention heads specialize in each subtask. Based on this insight, we construct subtask-specific steering vectors and show that modifying just 1% of the relevant heads enables instruction-free MT performance comparable to instruction-based prompting, while ablating these heads selectively disrupts their corresponding translation functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。