arXiv:2603.18863cs.CL2026-03

对齐越强,跨语言迁移反而越差?原因在此。

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

  • 分析四种模型在不同语种对上的对齐效果
  • 发现对齐与任务梯度几乎正交,优化一个难助另一个
  • 提示应谨慎选择损失函数,避免盲目对齐

更好的跨语言对齐常被假设能提升跨语言迁移效果。然而,尽管显式对齐技术提升了嵌入相似性,却往往无法改善词级别下游任务表现。本文研究发现,对齐目标与下游任务目标基本正交,且对齐带来的收益在不同语言和任务类型间差异显著。通过分析四种XLM-R编码器在不同语言对上对齐后,再进行词性标注或句子分类微调的表现,结合嵌入距离、梯度相似性和梯度大小等表征分析,我们发现:(1) 嵌入距离无法可靠预测任务性能的提升或下降;(2) 对齐梯度与任务梯度常接近正交,说明优化其中一个对另一个帮助甚微。综合来看,这解释了为何‘更好’的对齐并未带来‘更好’的跨语言迁移。基于此,本文提出实际建议:在跨语言对齐与任务微调结合时,需注意损失函数的选择。

原文摘要 · Abstract (English)

Better cross-lingual alignment is often assumed to yield better cross-lingual transfer. However, explicit alignment techniques -- despite increasing embedding similarity -- frequently fail to improve token-level downstream performance. In this work, we show that this mismatch arises because alignment and downstream task objectives are largely orthogonal, and because the downstream benefits from alignment vary substantially across languages and task types. We analyze four XLM-R encoder models aligned on different language pairs and fine-tuned for either POS Tagging or Sentence Classification. Using representational analyses, including embedding distances, gradient similarities, and gradient magnitudes for both task and alignment losses, we find that: (1) embedding distances alone are unreliable predictors of improvements (or degradations) in task performance and (2) alignment and task gradients are often close to orthogonal, indicating that optimizing one objective may contribute little to optimizing the other. Taken together, our findings explain why ``better'' alignment often fails to translate into ``better'' cross-lingual transfer. Based on these insights, we provide practical guidelines for combining cross-lingual alignment with task-specific fine-tuning, highlighting the importance of careful loss selection.

跨语言对齐微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。