让大模型在推理时实时进化策略，提升复杂问题解决能力
Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution
- 通过在线优化机制，让模型根据失败尝试动态调整推理策略
- 40亿参数模型在LiveCodeBench上达49.71%准确率，超越更庞大的GPT-4o
- 适合需要高可靠推理的智能系统研发者使用
大语言模型在复杂、长周期推理任务中表现不稳定，根源在于其策略固定不变。现有测试时扩展方法仅将执行反馈当作外部信号用于轨迹筛选或重写，未能将其内化为推理策略的改进依据。受波普尔“猜想与反驳”认识论启发，我们提出政策之思（PoT）框架，将推理重构为实例内在线优化过程。PoT首先通过高效探索机制生成多样候选解，再利用组相对策略优化（GRPO）基于执行反馈更新临时的LoRA适配器。这种闭环设计实现了推理先验的动态、实例化优化。实验表明，该方法显著提升性能：40亿参数模型在LiveCodeBench上达到49.71%准确率，超越参数更大的GPT-4o与DeepSeek-V3。
原文摘要 · Abstract (English)
Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption. Current test-time scaling methods treat execution feedback merely as an external signal for filtering or rewriting trajectories, without internalizing it to improve the underlying reasoning strategy. Inspired by Popper's epistemology of "conjectures and refutations," we argue that intelligence requires real-time evolution of the model's policy through learning from failed attempts. We introduce Policy of Thoughts (PoT), a framework that recasts reasoning as a within-instance online optimization process. PoT first generates diverse candidate solutions via an efficient exploration mechanism, then uses Group Relative Policy Optimization (GRPO) to update a transient LoRA adapter based on execution feedback. This closed-loop design enables dynamic, instance-specific refinement of the model's reasoning priors. Experiments show that PoT dramatically boosts performance: a 4B model achieves 49.71% accuracy on LiveCodeBench, outperforming GPT-4o and DeepSeek-V3 despite being over 50 smaller.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。