arXiv:2608.20362cs.CLcs.LG2026-08

多语言数学推理中,验证器会因格式差异产生语言偏见,影响训练效果。

Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck

  • 设计可复用的多语言奖励审计流程,识别语言特异性错误
  • 日文正确答案被误拒率高达64.2%,远超英文和中文
  • 跨语言选择瓶颈显著,需跨语言支持才能有效修复

强化学习结合可验证奖励(RLVR)是训练大模型数学推理的标准方法,其中答案验证器被视为语言无关的奖励函数。我们发现,在多语言场景下该假设失效:精确匹配验证器将格式与书写体差异转化为语言相关的假负样本噪声。本文提出一套可复用的多语言RLVR奖励审计协议,包括验证器鲁棒性测试套件、滚动诊断流程及针对日、英、中文答案的语言条件奖励误差指标。在MGSM k=8的滚动结果中,不同语言下可信正确答案被拒绝率差异显著:Qwen3-8B在日语上假负率为0.642,英文为0.122,中文为0.073。纯数值探针定位问题根源在于最终答案接口:接口模型使奖励误差VLB归零,但准确率差距仍存。进一步揭示跨语言选择瓶颈:在MGSM250的滚动中,仅使用目标本地聚合规则且无可信标签时,可弥补55%-78%的平均选择差距,且超过95%的修复依赖真正的跨语言支持。该瓶颈在483题MATH-500数据集上重现。受控训练审计显示,规则-GRPO提升可信准确率,但奖励误差VLB仍高。核心启示是:多语言RLVR奖励必须按语言和答案接口进行审计后再优化。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) is a standard recipe for training large language models on mathematical reasoning, where an answer verifier serves as a language-neutral reward function. We show that this assumption fails in multilingual settings: an exact-match verifier turns format and script variation into language-dependent false-negative reward noise. We introduce a reusable protocol for auditing multilingual RLVR rewards: a verifier-robustness suite, a rollout-diagnosis procedure, and language-conditioned reward-error metrics for Japanese, English, and Chinese answers. On MGSM rollouts with k=8, the exact-match proxy rejects trusted-correct answers at sharply different rates by language across Qwen3-4B, Qwen3-8B, and Llama-3.1-8B-Instruct; for Qwen3-8B, the false-negative rate reaches 0.642 on JP against 0.122 on EN and 0.073 on CN. A plain-numeric probe localizes the mechanism to the final-answer interface: an interface model drives reward-error VLB to zero while the residual accuracy gap is unchanged. We then expose a cross-lingual selection bottleneck: on MGSM250 rollouts, a target-local aggregation rule using no trusted labels closes 55-78% of the average selection gap, and over 95% of repairs require genuine cross-lingual support. The bottleneck replicates on a 483-problem MATH-500 set. A controlled training audit shows that rule-GRPO raises trusted accuracy while the reward-error VLB stays high. The unifying message is operational: multilingual RLVR rewards should be audited by language and by answer interface before they are optimized.

多语言强化学习数学推理验证器偏差

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。