arXiv:2606.06835cs.CL2026-06

用强化学习自动决定何时翻译,让大模型更省成本地跨语言工作。

Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learning

论文配图:Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learning
图 1 · 摘自论文原文
  • 通过奖励驱动学习翻译决策策略,避免盲目自信
  • 低资源语言提升23.5分,零样本扩展至9种新语言
  • 适合需要低成本多语言部署的AI系统开发者

大语言模型在不同语言间的性能差距明显,原生训练需大量语料,多数语言难以实现。翻译可作为替代方案:将输入转为模型主导语言,瞬间激活其全部能力。但对已能处理的语言频繁翻译会浪费资源,而让模型自主判断又因过度自信导致漏翻。现有方法依赖人工规则或外部路由,需手动设计。本文提出一种仅基于奖励学习的统一策略,实现语言与领域自适应的自我判断,仅在无法原生处理时才调用翻译。基于我们构建的保答案翻译流水线数据,在22种语言、3个资源层级(高、低、极低)和5个领域上对Qwen3-4B进行强化学习微调,并引入置信度门控的GSPO算法以实现成本敏感的工具使用。该策略在高资源语言上提升+4.6分,低资源语言+23.5分,极低资源语言+17.5分;在零样本下拓展至9种未见语言。相比始终翻译的上限策略,仅用66%成本即保持全收益,在低资源层优于其他惩罚机制最高达+24.3,且在高资源层以极低工具调用量匹配全收益。对于两种零曝光合成语言,策略正确学会始终翻译,较过度自信基线提升+18.7。

原文摘要 · Abstract (English)

The performance gap across languages in LLMs is well documented, and closing it natively requires pretraining or fine-tuning on corpora that, for most languages, are quite limited. Translation offers an alternative: converting an input into the model's dominant language unlocks its full capabilities at once. But translating every input is wasteful for languages the model already handles, while leaving the choice to the model fails too, as LLMs are overconfident and skip the tool even when they cannot understand the input. Prior work resolves this with language-specific rules, heuristics, or external routers, each requiring manual engineering. We instead learn a single policy that decides when to translate from reward alone, developing language- and domain-adaptive introspection that invokes translation only when it cannot solve a task natively. Using data from our answer-preserving translation pipeline, we continue RL on the post-trained Qwen3-4B across 22 languages in 3 resource tiers (High, Low, XLow) and 5 domains, and introduce confidence-gated GSPO for cost-sensitive tool use. The gated policy lifts reward over the untrained baseline by +4.6 on High, +23.5 on Low, and +17.5 on XLow, and extends zero-shot to 9 held-out languages. Against an unconstrained policy that almost always translates (the reward upper bound), it preserves full reward at 66% of the cost, outperforming other cost penalties by up to +24.3 on the low-resource tiers while matching the free policy's reward on high-resource tiers at a fraction of the tool use. On 2 synthetic languages with zero prior exposure it correctly learns to always translate, improving +18.7 over the overconfident baseline.

强化学习多语言工具使用成本优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。