用伪代码规划提升大模型智能体推理能力
PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization
- 用伪代码替代自然语言规划，提升结构化逻辑表达
- 在多个基准测试中超越当前最优基线，减少动作错误与遗漏
- 适合需要高效、通用推理的AI系统开发者
大型语言模型（LLM）智能体在处理复杂交互问题上展现出强大能力。现有方法主要依赖自然语言计划引导推理，但存在冗长低效的问题，且任务特异性限制了跨任务泛化能力。为此，本文探索伪代码风格计划（P-code Plan），以捕捉推理的结构性逻辑。实验发现，P-code Plan显著增强了智能体的泛化能力与执行效率。基于此，提出一种基于伪代码规划引导的偏好优化方法PGPO，引入两种面向规划的奖励机制，进一步提升生成高质量P-code Plan及后续推理的能力。大量实验表明，PGPO在代表性智能体基准上表现优异，优于当前领先基线；分析显示其能有效降低推理过程中的动作错误与遗漏。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents have demonstrated impressive capabilities in handling complex interactive problems. Existing LLM agents mainly generate natural language plans to guide reasoning, which is verbose and inefficient. NL plans are also tailored to specific tasks and restrict agents' ability to generalize across similar tasks. To this end, we explore pseudocode-style plans (P-code Plan) to capture the structural logic of reasoning. We find that P-code Plan empowers LLM agents with stronger generalization ability and more efficiency. Inspired by this finding, we propose a pseudocode-style Planning Guided Preference Optimization method called PGPO for effective agent learning. With two planning-oriented rewards, PGPO further enhances LLM agents' ability to generate high-quality P-code Plans and subsequent reasoning. Experiments show that PGPO achieves superior performance on representative agent benchmarks and outperforms the current leading baselines. Analyses reveal the advantage of PGPO in reducing action errors and omissions during reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。