揭秘大模型翻译的底层机制,发现关键组件仅占5%参数却决定翻译质量。
Exploring the Translation Mechanism of Large Language Models
- 通过细粒度路径修补技术定位翻译核心组件。
- 仅调整少于5%的参数就能显著提升翻译性能。
- 成果适合对模型可解释性与高效微调感兴趣的读者。
尽管大语言模型在多语言翻译中表现卓越,其内部核心翻译机制,甚至在基础词汇层面仍不清晰。本文提出系统性框架,从计算组件视角解析大模型翻译机制。首次引入子空间干预的路径修补方法,实现精准的因果分析,识别出对翻译任务至关重要的组件,并以人类可理解的方式描述其行为模式。实验表明,翻译主要由少数关键组件驱动:特定注意力头负责提取源语言、翻译指示符和位置特征,随后由特定多层感知机(MLPs)整合并处理为以英语为中心的中间表示,最终生成翻译结果。实证显示,仅对少于5%的参数进行针对性微调,即可提升翻译性能,同时保持通用能力。该结果表明这些关键组件能有效泛化至句子级翻译,并有助于理解更复杂的翻译任务。
原文摘要 · Abstract (English)
While large language models (LLMs) demonstrate remarkable success in multilingual translation, their internal core translation mechanisms, even at the fundamental word level, remain insufficiently understood. To address this critical gap, this work introduces a systematic framework for interpreting the mechanism behind LLM translation from the perspective of computational components. This paper first proposes subspace-intervened path patching for precise, fine-grained causal analysis, enabling the detection of components crucial to translation tasks and subsequently characterizing their behavioral patterns in human-interpretable terms. Comprehensive experiments reveal that translation is predominantly driven by a sparse subset of components: specialized attention heads serve critical roles in extracting source language, translation indicators, and positional features, which are then integrated and processed by specific multi-layer perceptrons (MLPs) into intermediary English-centric latent representations before ultimately yielding the final translation. The significance of these findings is underscored by the empirical demonstration that targeted fine-tuning a minimal parameter subset ($<5\%$) enhances translation performance while preserving general capabilities. This result further indicates that these crucial components generalize effectively to sentence-level translation and are instrumental in elucidating more intricate translation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。