让大模型学会按原则行动，自动优化决策策略。
PRACT: Optimizing Principled Reasoning and Acting of LLM Agent
- 用文本梯度从执行轨迹中提炼行动原则。
- 在四个环境中提升性能，自适应调整原则。
- 支持无奖励场景，适合复杂任务规划。
我们提出一种名为原理性推理与行动（PRAct）的框架，通过轨迹数据学习并强制执行行动原则。核心是利用反思与优化引擎生成的文本梯度来推导行动原则。为适配具体任务需求，我们设计了反射性原则优化（RPO）框架，执行后通过反思器批评当前原则，并由优化器进行更新。RPO在两种场景下实现：基于环境奖励的Reward-RPO，以及无需外部奖励的Self-RPO。此外，还提出了RPO-Traj和RPO-Batch两种方法以适应不同设置。在四个环境中的实验表明，采用RPO框架的PRAct代理能有效学习并应用行动原则，显著提升性能。
原文摘要 · Abstract (English)
We introduce the Principled Reasoning and Acting (PRAct) framework, a novel method for learning and enforcing action principles from trajectory data. Central to our approach is the use of text gradients from a reflection and optimization engine to derive these action principles. To adapt action principles to specific task requirements, we propose a new optimization framework, Reflective Principle Optimization (RPO). After execution, RPO employs a reflector to critique current action principles and an optimizer to update them accordingly. We develop the RPO framework under two scenarios: Reward-RPO, which uses environmental rewards for reflection, and Self-RPO, which conducts self-reflection without external rewards. Additionally, two RPO methods, RPO-Traj and RPO-Batch, is introduced to adapt to different settings. Experimental results across four environments demonstrate that the PRAct agent, leveraging the RPO framework, effectively learns and applies action principles to enhance performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。