发现大模型翻译的内在启动机制,提升训练效率与准确性。
Finding the Translation Switch: Discovering and Exploiting the Task-Initiation Features in LLMs
- 用稀疏自编码器识别翻译任务的共激活特征。
- 激活这些特征可准确翻译,删除则引发幻觉。
- 基于机制选择难样本,提升微调效率与鲁棒性。
大型语言模型(LLMs)在未经过特定任务微调的情况下仍具备强大的翻译能力,但其内部机制仍不清晰。本文利用稀疏自编码器(SAEs),提出一种新框架以识别任务特异性特征。方法首先找出在翻译输入中频繁共激活的特征,再通过基于PCA的一致性度量筛选出功能一致的特征,成功分离出一组关键的‘翻译启动’特征。因果干预实验表明,增强这些特征能引导模型正确翻译,而移除它们则导致幻觉和偏离任务的输出,证实其为模型内生翻译能力的核心组件。进一步地,基于这一机制洞察,我们提出一种高效微调的数据选择策略:优先训练那些无法自然激活该启动特征的‘机制困难’样本。实验显示,该方法显著提升数据效率并抑制幻觉。此外,这些机制在同系列更大模型中具有可迁移性。本工作不仅解码了LLM翻译机制的核心,还为利用内部机制构建更鲁棒、高效的模型提供了范式。代码已公开于 https://github.com/flamewei123/AAAI26-translation-Initiation-Features。
原文摘要 · Abstract (English)
Large Language Models (LLMs) frequently exhibit strong translation abilities, even without task-specific fine-tuning. However, the internal mechanisms governing this innate capability remain largely opaque. To demystify this process, we leverage Sparse Autoencoders (SAEs) and introduce a novel framework for identifying task-specific features. Our method first recalls features that are frequently co-activated on translation inputs and then filters them for functional coherence using a PCA-based consistency metric. This framework successfully isolates a small set of **translation initiation** features. Causal interventions demonstrate that amplifying these features steers the model towards correct translation, while ablating them induces hallucinations and off-task outputs, confirming they represent a core component of the model's innate translation competency. Moving from analysis to application, we leverage this mechanistic insight to propose a new data selection strategy for efficient fine-tuning. Specifically, we prioritize training on **mechanistically hard** samples-those that fail to naturally activate the translation initiation features. Experiments show this approach significantly improves data efficiency and suppresses hallucinations. Furthermore, we find these mechanisms are transferable to larger models of the same family. Our work not only decodes a core component of the translation mechanism in LLMs but also provides a blueprint for using internal model mechanism to create more robust and efficient models. The codes are available at https://github.com/flamewei123/AAAI26-translation-Initiation-Features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。