通过微调模型内部表示，用少量资源提升多语言大模型性能。
Improving Multilingual Language Models by Aligning Representations through Steering
- 在单层残差流中添加学习向量，实现对多语言表征的精准调整。
- 在7个基线方法上均表现更优，接近商用翻译系统水平。
- 与监督微调互补，适合资源有限但需快速优化多语言能力的场景。
本文研究大型语言模型如何表征非英语词元，这一问题虽重要却未受充分关注。我们提出一种轻量级干预方法——表示引导（representation steering），即在单一模型层的残差流中添加一个学习向量，以增强多语言表现。通过在七个先进基线方法（包括提示优化、监督微调、上下文学习、跨语言迁移及基于翻译的方法）上的广泛实验，验证该方法显著优于多数现有方案。尤其在保持极低资源消耗的前提下，性能达到与生产级翻译系统相当的水平。进一步分析表明，该方法与监督微调具有互补性，能直接高效地重校准模型内部表示。这些发现凸显了激活层面干预在提升大模型多语言能力方面的巨大潜力。
原文摘要 · Abstract (English)
This paper investigates how Large Language Models (LLMs) represent non-English tokens -- a question that remains underexplored despite recent progress. We propose a lightweight intervention method using representation steering, where a learned vector is added to the residual stream at a single model layer to enhance multilingual performance. Through extensive experiments across seven competitive baselines -- including prompt optimization, supervised fine-tuning (SFT), in-context learning, cross-lingual transfer, and translation-based methods-we show that our approach consistently outperforms most alternatives. In particular, it achieves performance on par with production-grade translation systems while requiring far fewer resources. We further explore the complementarity between our method and SFT, demonstrating that steering offers a direct, efficient way to realign internal representations. These findings underscore the potential of activation-level interventions as a powerful tool for improving the multilingual capabilities of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。