arXiv:2604.02881cs.CLcs.AI2026-04被引 2

多语言翻译模型合并效果差,因微调后语言特征分布不兼容。

One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging

  • 在权重空间合并多语言微调模型,但性能下降明显。
  • 不同目标语言间合并时,生成层表示差异显著增大。
  • 适合研究多语言模型融合机制与失败原因的团队。

权重空间模型合并可在不访问原始训练数据的情况下整合独立微调的模型,是联合训练的实用替代方案。尽管在多任务场景中表现良好,其在多语言环境下的行为仍不清晰。本文通过在大规模双语语料上全量微调语言模型,并评估标准合并策略,系统研究了多语言机器翻译中的权重空间合并。实验表明,合并会降低性能,尤其当目标语言不同时。通过分段条件神经元选择性和逐层中心核对齐分析内部表示,发现语言特异性神经元集中于嵌入层和上层Transformer块,而中间层则跨语言共享。关键发现:微调会重新分配而非强化语言特异性——受监督和相关语言的神经元排他性下降,未监督语言的神经元更孤立。这种重分配导致高层表示分化加剧,进而影响生成。结果表明,多语言微调可能改变模型几何结构,削弱标准权重空间合并假设的适用性。本工作为多语言翻译中合并失败提供了解释。

原文摘要 · Abstract (English)

Weight-space model merging combines independently fine-tuned models without accessing original training data, offering a practical alternative to joint training. While merging succeeds in multitask settings, its behavior in multilingual contexts remains poorly understood. We systematically study weight-space merging for multilingual machine translation by fully fine-tuning language model on large-scale bilingual corpora and evaluating standard merging strategies. Our experiments reveal that merging degrades performance, especially when target languages differ. To explain this failure, we analyze internal representations using span-conditioned neuron selectivity and layer-wise centered kernel alignment. We find that language-specific neurons concentrate in embedding layers and upper transformer blocks, while intermediate layers remain largely shared across languages. Critically, fine-tuning redistributes rather than sharpens language selectivity: neurons for supervised and related languages become less exclusive, while those for unsupervised languages grow more isolated. This redistribution increases representational divergence in higher layers that govern generation. These findings suggest that multilingual fine-tuning may reshape geometry in ways that reduce compatibility with standard weight-space merging assumptions. Our work thus provides an explanation for why merging fails in multilingual translation scenarios.

多语言模型合并翻译表示分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。