用简单方法提升大模型推理能力，开源模型在数学竞赛题上超o1-preview。
s1: Simple test-time scaling
- 构建1000条高质量推理题数据集s1K，筛选高难度、多样、优质题目。
- 通过强制延长或中断思考过程，动态控制测试时计算量，提升答案正确率。
- 模型可扩展至无干预推理，数学竞赛准确率提升27%，代码全开源。
测试时缩放是一种利用额外测试时计算资源提升语言模型性能的有前景方法。OpenAI的o1模型展示了这一潜力，但未公开具体方法，引发大量复现尝试。本文探索最简实现方式以获得强推理能力：首先，基于难度、多样性与质量三项标准，构建包含1000个问题及推理链的数据集s1K；其次，提出预算强制机制（budget forcing），通过强制终止或反复添加“Wait”来调控模型思考长度，促使模型自我校验，修复错误推理。在s1K上对Qwen2.5-32B-Instruct进行监督微调，并结合预算强制后，s1-32B模型在MATH和AIME24竞赛数学题上相较o1-preview最高提升27%。进一步地，通过预算强制扩展，模型可在无需测试时干预的情况下外推性能：AIME24准确率从50%提升至57%。模型、数据与代码已开源于https://github.com/simplescaling/s1。
原文摘要 · Abstract (English)
Test-time scaling is a promising new approach to language modeling that uses extra test-time compute to improve performance. Recently, OpenAI's o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. We seek the simplest approach to achieve test-time scaling and strong reasoning performance. First, we curate a small dataset s1K of 1,000 questions paired with reasoning traces relying on three criteria we validate through ablations: difficulty, diversity, and quality. Second, we develop budget forcing to control test-time compute by forcefully terminating the model's thinking process or lengthening it by appending "Wait" multiple times to the model's generation when it tries to end. This can lead the model to double-check its answer, often fixing incorrect reasoning steps. After supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and equipping it with budget forcing, our model s1-32B exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). Further, scaling s1-32B with budget forcing allows extrapolating beyond its performance without test-time intervention: from 50% to 57% on AIME24. Our model, data, and code are open-source at https://github.com/simplescaling/s1
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。