让大模型同时探索多个环境，提升决策多样性与任务成功率。
DPEPO: Diverse Parallel Exploration Policy Optimization for LLM-based Agents
- 让智能体并行交互多个环境，共享不同轨迹的经验。
- 在ALFWorld和ScienceWorld上达到当前最高成功率。
- 适合需要广泛探索的复杂任务场景研究者使用。
基于大语言模型的智能体遵循顺序式“推理-执行”范式，在诸多复杂任务中表现优异。然而，这类方法因每步仅与单一环境交互，导致探索能力有限、环境理解不充分。本文提出一种新范式，使智能体可同时与多个环境交互，并共享跨轨迹经验。在此基础上，我们进一步提出DPEPO强化学习算法，鼓励智能体进行多样化的并行探索。DPEPO包含两个阶段：初始监督微调（SFT）赋予基础的并行推理与动作生成能力；随后是采用分层奖励机制的强化学习阶段。设计了轨迹级成功奖励及两项步骤级奖励——多样动作奖励与多样状态转移奖励，主动惩罚行为冗余，促进广泛探索。在ALFWorld和ScienceWorld上的大量实验表明，DPEPO实现了当前最优成功率，同时效率与强序列基线相当。
原文摘要 · Abstract (English)
Large language model (LLM) agents that follow the sequential "reason-then-act" paradigm have achieved superior performance in many complex tasks.However, these methods suffer from limited exploration and incomplete environmental understanding, as they interact with only a single environment per step. In this paper, we first introduce a novel paradigm that enables an agent to interact with multiple environments simultaneously and share cross-trajectory experiences. Building upon this paradigm, we further propose DPEPO, a reinforcement learning (RL) algorithm that encourages the agent to perform diverse parallel exploration. There are two stages in DPEPO: initial supervised fine-tuning (SFT) imparts basic parallel reasoning and action generation, followed by reinforcement learning stage with a hierarchical reward scheme. We design a parallel trajectory-level success reward and two step-level rewards: Diverse Action Reward and Diverse State Transition Reward, which actively penalize behavioral redundancy and promote broad exploration. Extensive experiments on ALFWorld and ScienceWorld show that DPEPO achieves state-of-the-art (SOTA) success rates, while maintaining comparable efficiency to strong sequential baselines. (Code is available at https://github.com/LePanda026/Code-for-DPEPO)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。