arXiv:2604.20720cs.LGcs.AI2026-04

通过智能选数据提升多语言模型性能,避免不同语言间干扰。

COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Sampling

论文配图:COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Sampling
图 1 · 摘自论文原文
  • 根据语义分布选择训练数据,优先补充薄弱语义类别。
  • 在多个模型和任务上显著优于传统方法,最高提升6.3%准确率。
  • 适合需要持续更新的多语言应用,如跨语言客服系统。

大型语言模型在不同语言间常存在性能差异,直接多语言微调易因跨语言干扰导致性能下降。为此,我们提出COMPASS(COntinual Multilingual PEFT with Adaptive Semantic Sampling),一种以数据为中心的轻量级多语言适配框架。该方法基于参数高效微调(PEFT),在精心筛选的辅助多语言数据子集上训练语言特异性适配器。核心是分布感知采样策略,利用多语言嵌入与聚类识别现有数据与目标使用分布间的语义缺口,优先选取代表性不足的语义簇数据,最大化正向跨语言迁移并最小化干扰。进一步扩展为持续学习框架COMPASS-ECDA,实时监测生产环境中的数据分布变化,动态更新适配器以防止模型退化,同时兼顾新知识学习与旧知识保留。在三种模型架构(Phi-4-Mini、Llama-3.1-8B、Qwen2.5-7B)及多个挑战性多语言基准(Global-MMLU、MMLU-ProX)上,包括未见过的长上下文任务(OneRuler),COMPASS始终优于基于语言相似性的基线方法,提供了一种高效、可持续的多语言模型开发与维护方案。

原文摘要 · Abstract (English)

Large language models (LLMs) often exhibit performance disparities across languages, with naive multilingual fine-tuning frequently degrading performance due to negative cross-lingual interference. To address this, we introduce COMPASS (COntinual Multilingual PEFT with Adaptive Semantic Sampling), a novel data-centric framework for adapting LLMs to target languages. COMPASS leverages parameter-efficient fine-tuning (PEFT) by training lightweight, language-specific adapters on a judiciously selected subset of auxiliary multilingual data. The core of our method is a distribution-aware sampling strategy that uses multilingual embeddings and clustering to identify semantic gaps between existing training data and a target usage distribution. By prioritizing auxiliary data from under-represented semantic clusters, COMPASS maximizes positive cross-lingual transfer while minimizing interference. We extend this into a continual learning framework, COMPASS-ECDA, which monitors for data distribution shifts in production and dynamically updates adapters to prevent model staleness, balancing adaptation to new data with the preservation of existing knowledge. Across three different model architectures (Phi-4-Mini, Llama-3.1-8B, and Qwen2.5-7B) and multiple challenging multilingual benchmarks (Global-MMLU, MMLU-ProX), including unseen long-context tasks (OneRuler), we demonstrate that COMPASS consistently outperforms baseline methods guided by linguistic similarity, providing an effective, efficient, and sustainable solution for developing and maintaining high-performing multilingual models in dynamic environments.

多语言参数高效持续学习语义采样

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。