让大模型智能体分阶段专注任务,避免简单任务霸占学习资源。
Phase-Aware Mixture of Experts for Agentic Reinforcement Learning
- 引入阶段感知路由,动态识别任务阶段边界并分配专家
- 专家按阶段统一调度,避免碎片化分配导致能力分散
- 适合需要多阶段决策的复杂智能体任务,如规划与推理
强化学习已赋予大语言模型智能体解决复杂任务的强大能力。然而,现有方法通常采用单一策略网络,导致‘简单性偏差’:简单任务占用大部分参数并主导梯度更新,使复杂任务缺乏足够表达能力。一种可行方案是在策略网络中引入混合专家(MoE)架构,使不同专家专注于特定任务,从而避免简单任务占据全部参数。但传统MoE存在令牌级路由缺陷:路由器为每个令牌独立分配专家,将具有时序一致性的任务模式打散,削弱专家专长。本文提出相位感知混合专家(PA-MoE)。其首先设计轻量级相位路由器,直接从强化学习目标中学习潜在阶段边界,无需预定义阶段类别;随后,该路由器实现时间上一致的专家分配,确保专家保持阶段特异性能力。实验表明,所提方法有效提升了复杂任务的学习效率与表现。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has equipped LLM agents with a strong ability to solve complex tasks. However, existing RL methods normally use a \emph{single} policy network, causing \emph{simplicity bias} where simple tasks occupy most parameters and dominate gradient updates, leaving insufficient capacity for complex tasks. A plausible remedy could be employing the Mixture-of-Experts (MoE) architecture in the policy network, as MoE allows different parameters (experts) to specialize in different tasks, preventing simple tasks from dominating all parameters. However, a key limitation of traditional MoE is its token-level routing, where the router assigns each token to specialized experts, which fragments phase-consistent patterns into scattered expert assignments and thus undermines expert specialization. In this paper, we propose \textbf{Phase-Aware Mixture of Experts (PA-MoE)}. It first features a lightweight \emph{phase router} that learns latent phase boundaries directly from the RL objective without pre-defining phase categories. Then, the phase router allocates temporally consistent assignments to the same expert, allowing experts to preserve phase-specific expertise. Experimental results demonstrate the effectiveness of our proposed PA-MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。