用反思生成框架让小模型达到大模型推理水平
Test-Time Scaling with Reflective Generative Model
- 共享主干网络，用任务头分别预测和打分推理轨迹
- 仅用5300万参数实现自监督过程奖励模型
- 支持测试时扩展，三档思考长度适配不同需求
我们提出首个反思生成模型MetaStone-S1，通过新的反思生成范式实现了OpenAI o3-mini的性能。该范式聚焦高质量推理轨迹选择，包含两项创新：1）政策与过程奖励模型共享主干网络，使用任务特定头分别预测和评分推理轨迹，仅引入5300万额外参数；2）无需过程级标注，提出自监督过程奖励模型，可直接从结果奖励中学习高质量推理轨迹选择。得益于反思生成范式，MetaStone-S1天然支持测试时扩展，提供低、中、高三种可控思考长度的推理模式。实验表明，仅320亿参数的MetaStone-S1即可达到OpenAI o3-mini系列的性能水平。为支持研究社区，我们已将MetaStone-S1开源至https://github.com/MetaStone-AI/MetaStone-S1。
原文摘要 · Abstract (English)
We introduce our first reflective generative model MetaStone-S1, which obtains OpenAI o3-mini's performance via the new Reflective Generative Form. The new form focuses on high-quality reasoning trajectory selection and contains two novelties: 1) A unified interface for policy and process reward model: we share the backbone network and use task-specific heads for reasoning trajectory predicting and scoring respectively, introducing only 53M extra parameters for trajectory scoring. 2) Eliminating the reliance on process-level annotation: we provide a self-supervised process reward model, which can directly learn the high-quality reasoning trajectory selection from the outcome reward. Equipped with the reflective generative form, MetaStone-S1 is naturally suitable for test-time scaling, and we provide three reasoning effort modes (low, medium, and high) based on the controllable thinking length. Experiments demonstrate that our MetaStone-S1 achieves comparable performance to OpenAI o3-mini's series with only 32B parameter size. To support the research community, we have open-sourced MetaStone-S1 at https://github.com/MetaStone-AI/MetaStone-S1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。