不训练不调优，用分布锐化提升大模型推理效率
Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
- 通过缩放低温度分布实现分布锐化，无需迭代采样
- 在数学、问答、代码任务上超越单次GRPO，延迟降低10倍以上
- 适合追求高效推理的部署场景，无需验证器或外部奖励
强化学习后训练是提升大语言模型推理能力的主要方法，但越来越多证据表明其收益主要来自分布锐化而非新能力获取。已有研究显示，使用马尔可夫链蒙特卡洛（MCMC）从大模型的幂分布中采样，可在无外部奖励情况下恢复与强化学习相当的性能；然而MCMC计算成本过高，难以广泛使用。本文提出一种理论支持的替代方案，无需迭代MCMC即可逼近全局幂分布。我们推导出新公式：全局幂分布可由逐标记缩放的低温度分布近似，缩放因子反映未来轨迹质量。基于此，我们设计了一种无需训练、无需验证器的自回归分布锐化算法。我们在四个大模型的数学、问答和代码任务上评估该方法，结果表明其性能达到或超过单次GRPO，且相比MCMC采样推理延迟降低超10倍。
原文摘要 · Abstract (English)
Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a novel formulation showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks across four LLMs, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over 10x compared to MCMC-based sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。