通过对齐中间层表示，提升微调大模型的跨语言迁移能力。
Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
- 聚焦模型中间层，设计跨语言表示对齐目标。
- 在低资源语言上实现显著性能提升，尤其在槽位填充等任务中。
- 无需重新训练，可融合已有模块，适合多语言应用部署。
尽管大语言模型通过微调在特定任务上表现出色，但跨语言应用仍面临性能差异和数据稀缺的挑战。通过对1000多个语言对的模型内部表示分析，我们发现中间层具有最强的跨语言对齐潜力。基于此，我们在任务特定训练中引入中间层对齐目标。在槽位填充、机器翻译和结构化文本生成任务上的实验表明，该方法显著提升了跨语言迁移效果，尤其在低资源语言上表现突出。该方法对对齐语言的选择不敏感，且能泛化到对齐阶段未见的语言。此外，我们证明了独立训练的对齐模块可与现有任务模块合并，无需全量重训即可增强跨语言能力。代码已公开（https://github.com/dannigt/mid-align）。
原文摘要 · Abstract (English)
While large language models demonstrate remarkable capabilities at task-specific applications through fine-tuning, extending these benefits across diverse languages is essential for broad accessibility. However, effective cross-lingual transfer is hindered by LLM performance gaps across languages and the scarcity of fine-tuning data in many languages. Through analysis of LLM internal representations from over 1,000+ language pairs, we discover that middle layers exhibit the strongest potential for cross-lingual alignment. Building on this finding, we propose a middle-layer alignment objective integrated into task-specific training. Our experiments on slot filling, machine translation, and structured text generation show consistent improvements in cross-lingual transfer, especially to lower-resource languages. The method is robust to the choice of alignment languages and generalizes to languages unseen during alignment. Furthermore, we show that separately trained alignment modules can be merged with existing task-specific modules, improving cross-lingual capabilities without full re-training. Our code is publicly available (https://github.com/dannigt/mid-align).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。