S*通过混合缩放提升代码生成准确率，3B模型可超GPT-4o-mini。
S*: Test Time Scaling for Code Generation
- 融合并行与串行测试时计算，动态生成对比输入增强判断。
- 3B模型经S*后超越GPT-4o-mini，顶尖模型达85.7%准确率。
- 适合追求高效推理的开发者及代码生成研究者。
增大大语言模型的测试时计算量在多个领域展现出潜力，但在代码生成中仍研究不足，尽管数学领域已有广泛探索。本文提出S*，首个混合测试时缩放框架，显著提升代码生成的覆盖率与选择准确性。S*在现有并行缩放基础上引入串行缩放，突破性能边界；同时设计新型选择机制，自适应生成用于成对比较的区分性输入，并结合执行反馈信息，稳健识别正确解。我们在12个大语言模型和大推理模型上评估，结果表明：(1) S*在不同模型家族与规模下均持续提升性能，使3B模型超越GPT-4o-mini；(2) 非推理模型经S*后超越推理模型——GPT-4o-mini + S*在LiveCodeBench上优于o1-preview 3.7%；(3) S*进一步提升最先进推理模型表现——DeepSeek-R1-Distill-Qwen-32B + S*在LiveCodeBench达85.7%，接近o1(high)的88.5%。代码将开源于https://github.com/NovaSky-AI/SkyThought。
原文摘要 · Abstract (English)
Increasing test-time compute for LLMs shows promise across domains but remains underexplored in code generation, despite extensive study in math. In this paper, we propose S*, the first hybrid test-time scaling framework that substantially improves the coverage and selection accuracy of generated code. S* extends the existing parallel scaling paradigm with sequential scaling to push performance boundaries. It further leverages a novel selection mechanism that adaptively generates distinguishing inputs for pairwise comparison, combined with execution-grounded information to robustly identify correct solutions. We evaluate across 12 Large Language Models and Large Reasoning Model and show: (1) S* consistently improves performance across model families and sizes, enabling a 3B model to outperform GPT-4o-mini; (2) S* enables non-reasoning models to surpass reasoning models - GPT-4o-mini with S* outperforms o1-preview by 3.7% on LiveCodeBench; (3) S* further boosts state-of-the-art reasoning models - DeepSeek-R1-Distill-Qwen-32B with S* achieves 85.7% on LiveCodeBench, approaching o1 (high) at 88.5%. Code will be available under https://github.com/NovaSky-AI/SkyThought.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。