让大模型在小语种中推理,监督微调提升流畅度,强化学习修复格式错误。
Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

- 用监督微调让模型98%时间按问题语言推理,减少3倍令牌消耗。
- 模型在小语种推理时准确率波动大,真实改进在无法被准确率捕捉的层面。
- 提出六维可测量行为指标,避免长度相关误导,适合评估低资源语言模型。
选取三个前沿混合专家模型(阿里巴巴、OpenAI、NVIDIA;每模型活跃参数3.6-4.0B),微调其在低资源语言中的推理能力。在准确率基准上几乎无变化:仅随机种子变动就造成7.7分波动,超过所有数据与训练配方的影响。真正改变存在于准确率无法观测之处:基线模型从不以希腊语进行推理(1000次测试中0次),即使答案正确,其思考过程用户无法阅读、审计或修正。经监督微调(SFT)后,各发布检查点在约98%样本中按问题语言推理,其中一模型仅需3倍少的令牌,语法质量全面提升,通用能力保持在基线附近,未丢失原有能力。我们提出六个行为维度,均排除输出长度相关干扰,自检发现六个指标曾失效,均被控制项捕获。SFT自身缺陷仍存在:四分之一回答跳过指定格式,信息泄露至推理通道,明确指令‘用英语思考’仅半数响应。采用预注册的可验证奖励信号进行强化学习,彻底修复前两项缺陷(回退率从24%降至2.5%,泄露率从3.5%降至0.0%),并提升第三项+9.1个百分点,而希腊语推理习惯不受准确率梯度影响。我们发布五个检查点,方法论、控制机制和预注册流程可迁移至任意低资源语言,希腊语为实证案例。
原文摘要 · Abstract (English)
Take three frontier mixture-of-experts models (Alibaba, OpenAI, NVIDIA; 3.6-4.0B active parameters each) and fine-tune them to reason in a low-resource language. On accuracy benchmarks almost nothing happens, and the benchmark itself is noise at this scale: changing only the random seed moves the score by 7.7 points, more than every data and recipe effect we measured. That null is our first result. The real changes live where accuracy cannot see. Base models never think in Greek: 0 of 1,000 reasoning traces, even when the question is Greek, so the model answers correctly while reasoning in a form its user cannot read, audit, or correct. After supervised fine-tuning (SFT), every released checkpoint reasons in the language of the question on ~98% of items, one family at 3x fewer tokens, with judged grammaticality improving on all four models and general ability within a few points of each base: nothing was forgotten, and fluency was gained. We propose six behavioural dimensions that make such changes measurable, each gated to reject any metric that correlates with output length, and we report how our own instruments lied: six failures, each caught by a control. What SFT cannot do is fix its own defects: a quarter of answers skip the requested format, answers leak into the reasoning channel, and an explicit "think in English" is obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training, fixes the first two outright (fallback 24% to 2.5%, leak 3.5% to 0.0%, both against a flat random-reward control) and moves the third (+9.1pp), while the Greek reasoning habit survives an accuracy-only gradient untouched. We release five checkpoints. The instruments, the controls and the pre-registration travel to any low-resource language; Greek is the case that let us measure them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。