通过识别语言敏感神经元，实现大模型翻译微调的精准参数更新。
LANDeRMT: Detecting and Routing Language-Aware Neurons for Selectively Finetuning LLMs to Machine Translation
- 基于任务感知能力区分通用与特定语言神经元，实现选择性微调。
- 在多个语种对上超越强基线，翻译质量显著提升。
- 适合需要多语言翻译且避免灾难性遗忘的研究与应用
近期大语言模型在少样本双语监督下已展现出多语言翻译的潜力。但微调时面临灾难性遗忘和参数干扰问题。为此，我们提出LANDeRMT——一种语言感知神经元检测与路由框架，可针对机器翻译任务选择性微调大模型。该方法评估神经元对翻译任务的感知能力，并将其分为通用型与语言特异型神经元。基于此分类，在微调中实施选择性参数更新，缓解参数干扰与遗忘问题。进一步设计了基于条件感知的动态路由机制，根据翻译信号调节模型中通用与特定语言能力的容量分配。实验表明，LANDeRMT在多种语言对上显著优于多个强基线，有效提升了翻译质量。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have shown promising results in multilingual translation even with limited bilingual supervision. The major challenges are catastrophic forgetting and parameter interference for finetuning LLMs when provided parallel training data. To address these challenges, we propose LANDeRMT, a \textbf{L}anguage-\textbf{A}ware \textbf{N}euron \textbf{De}tecting and \textbf{R}outing framework that selectively finetunes LLMs to \textbf{M}achine \textbf{T}ranslation with diverse translation training data. In LANDeRMT, we evaluate the awareness of neurons to MT tasks and categorize them into language-general and language-specific neurons. This categorization enables selective parameter updates during finetuning, mitigating parameter interference and catastrophic forgetting issues. For the detected neurons, we further propose a conditional awareness-based routing mechanism to dynamically adjust language-general and language-specific capacity within LLMs, guided by translation signals. Experimental results demonstrate that the proposed LANDeRMT is very effective in learning translation knowledge, significantly improving translation quality over various strong baselines for multiple language pairs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。