不训练不翻译,用向量工程提升低资源语言推理能力
Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering
- 推理时注入预计算向量,让非英语推理更接近英文表现
- 在泰语和斯瓦希里语上平均提升5.48%,最高达7.54%
- 适合追求多语言公平性的模型部署与应用
大型语言模型(LLMs)和大型视觉-语言模型(LVLMs)具备强大推理能力,但在英语之外的低资源语言中表现显著落后,引发多语言应用中的公平性问题。现有方法或依赖昂贵的多语言训练,或使用外部翻译工具进行提示,均存在资源消耗大且受翻译质量影响的问题。为此,我们提出一种无需训练的推理时方法——表示工程增强多语言推理(MRRE),不依赖额外训练数据或工具。MRRE在推理过程中于特定层依次注入两个预计算向量:跨语言推理增强向量,将非英语推理表示引导至英语空间以激活多语言推理能力;目标语言输出锚定向量,恢复目标语言输出分布以保持输入输出语言一致性。在四个推理基准上对六种先进LLMs和LVLMs的全面实验表明,MRRE在非英语语言中平均提升5.48%,低资源语言(泰语、斯瓦希里语)最高提升7.54%,同时输入输出语言一致性提升3.78%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) demonstrate strong reasoning capabilities, yet their performance in English significantly outperforms that in low-resource languages, raising fairness concerns in multilingual applications. Existing approaches either rely on costly multilingual training or employ prompting with external translation tools, both of which are resource-intensive and sensitive to translation quality. To address these limitations, we propose a training-free inference-time method to enhance Multilingual Reasoning capabilities via Representation Engineering (MRRE) without using any additional training data or tools. MRRE sequentially injects two precomputed vectors at specific layers during inference processing: cross-lingual reasoning enhancement vectors, which steer non-English reasoning representations toward English space to unlock multilingual reasoning, and target-language output anchoring vectors, which restore the distribution of the target language to preserve input-output language consistency. Comprehensive experiments across six advanced LLMs and LVLMs on four reasoning benchmarks demonstrate that MRRE consistently enhances non-English reasoning by an average gain of 5.48% and up to 7.54% in low-resource languages (Thai and Swahili), while improving input-output language consistency by 3.78%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。