小模型通过强化学习实现强生成选择，提升推理效率
Learning Generative Selection for Best-of-N
- 用大规模数学与代码数据构建筛选任务，训练小模型进行生成选择
- 1.7B参数模型在多个基准上超越提示和多数投票，接近大模型表现
- 能泛化到更强模型输出，适合资源受限场景下的高效推理
通过并行采样扩展测试时计算可显著提升大语言模型的推理能力，但常受限于 Best-of-N 选择质量。生成选择方法（如 GenSelect）缓解此瓶颈，然而强选择性能仍主要局限于大模型。本文表明，小规模推理模型可通过针对性强化学习获得强大生成选择能力。我们从大规模数学与代码指令数据集中构建筛选任务，仅保留包含正确与错误候选解的实例，并使用 DAPO 训练 1.7B 参数模型以奖励正确选择。在数学（AIME24、AIME25、HMMT25）与代码（LiveCodeBench）推理基准上，我们的模型持续优于提示与多数投票基线，常接近甚至超过更大模型。此外，这些增益可泛化至对更强模型输出的选择，尽管训练仅基于弱模型输出。总体而言，结果确立了强化学习作为小模型中实现强生成选择的可扩展路径，支持高效测试时扩展。
原文摘要 · Abstract (English)
Scaling test-time compute via parallel sampling can substantially improve LLM reasoning, but is often limited by Best-of-N selection quality. Generative selection methods, such as GenSelect, address this bottleneck, yet strong selection performance remains largely limited to large models. We show that small reasoning models can acquire strong GenSelect capabilities through targeted reinforcement learning. To this end, we synthesize selection tasks from large-scale math and code instruction datasets by filtering to instances with both correct and incorrect candidate solutions, and train 1.7B-parameter models with DAPO to reward correct selections. Across math (AIME24, AIME25, HMMT25) and code (LiveCodeBench) reasoning benchmarks, our models consistently outperform prompting and majority-voting baselines, often approaching or exceeding much larger models. Moreover, these gains generalize to selecting outputs from stronger models despite training only on outputs from weaker models. Overall, our results establish reinforcement learning as a scalable way to unlock strong generative selection in small models, enabling efficient test-time scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。