arXiv:2506.00875cs.CL2025-06ACL被引 6

通过显式连接跨语言表征,提升大模型多语言能力。

CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning

  • 在隐层层面融合中英文激活值,建立跨语言连接。
  • 6个基准测试22种语言上超越普通微调方法。
  • 适合需要均衡多语言性能的模型优化场景。

当前大型语言模型因训练语料以英语为主,常表现出多语言能力不平衡。现有数据级微调方法(如数据增强或知识蒸馏)虽引入隐含的跨语言对齐,却忽视了更深层次的潜在层交互。本文提出CC-Tuning,一种新型多语言微调范式,在隐层层面显式建立跨语言连接机制。训练时,通过融合英语与非英语输入的前馈激活值,使模型同时受益于双语资源;该过程由可学习的决策模块控制,以识别有益激活。推理阶段,利用可学习的变换矩阵在单语环境下模拟跨语言连接。在涵盖22种语言的6个基准测试中,CC-Tuning表现优于标准微调,且为数据级增强提供了强有力的隐层替代方案。进一步分析验证了其实用性及潜在层跨语言交互在提升大模型多语言性能中的价值。

原文摘要 · Abstract (English)

Current large language models (LLMs) often exhibit imbalanced multilingual capabilities due to their English-centric training corpora. To address this, existing fine-tuning approaches operating at the data-level (e.g., through data augmentation or distillation) typically introduce implicit cross-lingual alignment, overlooking the potential for more profound, latent-level cross-lingual interactions. In this work, we propose CC-Tuning, a novel multilingual fine-tuning paradigm that explicitly establishes a cross-lingual connection mechanism at the latent level. During training, CC-Tuning fuses the feed forward activations from both English and non-English inputs, enabling the model to benefit from both linguistic resources. This process is facilitated with a trainable Decision Maker that identifies beneficial activations. Furthermore, during inference, a Transform Matrix is utilized to simulate the cross-lingual connection under monolingual setting through representation transformation. Our experiments on six benchmarks covering 22 languages show that CC-Tuning outperforms vanilla SFT and offers a strong latent-level alternative to data-level augmentation methods. Further analysis also highlights the practicality of CC-Tuning and the potential of latent-level cross-lingual interactions in advancing the multilingual performance of LLMs.

多语言微调隐层连接

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。