用树搜索优化大模型自我纠错，提升代码生成准确率。
TGPR: Tree-Guided Policy Refinement for Robust Self-Debugging of LLMs
- 结合贝叶斯树搜索与策略优化，动态探索成功失败路径。
- 在MBPP上pass@1提升4.2个百分点，APPS上pass@10提升12.51个百分点。
- 适合需要高可靠性推理的自动化编程与复杂任务求解场景。
迭代精炼是使大语言模型解决复杂推理与问题求解任务的有前景范式。然而，如何有效搜索可能的精炼路径构成关键挑战。现有方法多依赖预设启发式规则，面临探索-利用困境，且无法根据过往精炼结果自适应调整。本文提出树引导策略精炼（TGPR），融合GRPO与基于Thompson采样的树搜索。TGPR主动探索失败与成功路径，获得更密集的训练轨迹和更自适应的策略。在HumanEval、MBPP和APPS基准测试中，相比一个竞争力强的GRPO基线，本方法在MBPP上实现pass@1绝对提升4.2个百分点，在APPS上实现pass@10绝对提升12.51个百分点。除代码调试外，TGPR还提供一种将学习策略与结构化搜索方法结合的系统性框架，为增强大模型的迭代精炼与状态化推理能力提供了通用解决方案。
原文摘要 · Abstract (English)
Iterative refinement has been a promising paradigm to enable large language models (LLMs) to resolve difficult reasoning and problem-solving tasks. One of the key challenges, however, is how to effectively search through the enormous search space of possible refinements. Existing methods typically fall back on predefined heuristics, which are troubled by the exploration-exploitation dilemma and cannot adapt based on past refinement outcomes. We introduce Tree-Guided Policy Refinement (TGPR), a novel framework that combines GRPO with a Thompson-Sampling-based tree search. TGPR explores both failed and successful refinement paths actively, with denser training trajectories and more adaptive policies. On HumanEval, MBPP, and APPS benchmarks, our method achieves up to +4.2 percentage points absolute improvement in pass@1 (on MBPP) and up to +12.51 percentage points absolute improvement in pass@10 (on APPS) compared to a competitive GRPO baseline. Apart from debugging code, TGPR focuses on a principled approach to combining learned policies with structured search methods, offering a general framework for enhancing iterative refinement and stateful reasoning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。