用困惑度选最优代码,比微调更省算力且效果更好
Surprisal-Guided Selection: Compute-Optimal Test-Time Strategies for Execution-Grounded Code Generation
- 基于困惑度筛选高不确定性正确代码,提升生成质量
- 最佳采样策略达90%成功率,远超微调的30.6%
- 适合需要高效生成的代码优化等执行验证任务
测试时训练(TTT)通过梯度更新在推理阶段调整语言模型。但适应真的是正确策略吗?我们在可验证执行基础(VEG)任务中研究计算最优的测试时策略,如GPU内核优化这类由确定性评估器提供密集连续奖励信号的领域。以KernelBench为基准,使用1200亿参数模型(GPT-OSS-120B结合LoRA适配),发现搜索优于最小化适应(1-5步梯度更新):Best-of-N采样在K=64时达到90%任务成功率(20个任务中18个成功),而TTT最佳检查点仅30.6%(3次种子平均),其“等效K”低于1,甚至不如单样本推理。失败原因是过度锐化:梯度更新将多样性压缩至平庸解而非发现最优解。我们的核心贡献是困惑度引导选择:选择最高困惑度(最低置信度)的正确样本,成功率从50%提升至80%,提高30%。扩展至困惑度引导的Top3匹配了理想表现(100%)。这一零成本策略经长度控制分析验证,恢复了理想性能。对于密集奖励的VEG任务,算力应分配给样本多样性与智能选择,而非梯度适应。该原则可能推广至其他最优解位于分布尾部的执行基础领域。
原文摘要 · Abstract (English)
Test-time training (TTT) adapts language models through gradient-based updates at inference. But is adaptation the right strategy? We study compute-optimal test-time strategies for verifiable execution-grounded (VEG) tasks, domains like GPU kernel optimization where a deterministic evaluator provides dense, continuous reward signals. Using KernelBench as our testbed and a 120B-parameter model (GPT-OSS-120B with LoRA adaptation), we find that search outperforms minimal adaptation (1-5 gradient steps): Best-of-N sampling achieves 90% task success (18/20 tasks) at K=64 across the full KernelBench L1 eval set while TTT's best checkpoint reaches only 30.6% (3-seed mean), with TTT's "equivalent K" falling below 1, worse than single-sample inference. The failure mode is over-sharpening: gradient updates collapse diversity toward mediocre solutions rather than discovering optimal ones. Our main contribution is surprisal-guided selection: selecting the highest-surprisal (lowest-confidence) correct sample yields 80% success vs. 50% for most-confident selection, a 30% improvement. Extending to surprisal-guided-top3 matches oracle performance at 100%. This zero-cost strategy, validated through length-controlled analysis, recovers oracle performance. For dense-reward VEG tasks, compute should be allocated to sample diversity and intelligent selection rather than gradient adaptation. The surprisal-guided selection principle may generalize to other execution-grounded domains where optimal solutions occupy the distribution tail.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。