用高斯分布动态选择引导深度，提升长任务强化学习效率
Agent-G$^2$: Gaussian Guidance for Agentic Reinforcement Learning
- 以高斯分布随机选取引导轨迹深度，避免固定值或逐样本探测
- 在ALFWorld上比最优基线高7.4分，仅需1/3的采样成本
- 适合需要高效探索的复杂长程决策任务研究者
基于提示的强化学习通过保留专家轨迹前缀来缓解长时序任务中的奖励稀疏问题，其效果取决于引导深度——即保留多少轨迹。现有方法将深度视为确定性标量：调度策略对所有样本使用统一值，忽略任务差异；逐样本探测虽精确但需额外采样。我们发现有效引导集中在某一深度区间，其信息量呈高斯分布，而非单一最优点。为此提出Agent-G²框架：从在线收集的回滚数据中估计高斯中心与方差，动态生成每任务引导深度，无需探针采样或训练预测器。中心融合全局基线与聚类难度，方差捕捉簇内变异性。在ALFWorld和WebShop上使用Qwen2.5-1.5B / 7B-Instruct评估，Agent-G²在ALFWorld上优于最强提示法、无提示法及辅助强化学习基线2.3 / 3.9 / 7.4分，且采样成本不足逐样本探测的三分之一。
原文摘要 · Abstract (English)
Hint-based reinforcement learning addresses reward sparsity in long-horizon agentic tasks by retaining a prefix of an expert trajectory before each rollout, letting the policy explore from a state closer to success. Its effectiveness hinges on the guidance depth: how much of the trajectory to keep. Existing methods treat this depth as a deterministic scalar. Scheduled approaches share one value across samples and ignore per-task heterogeneity; per-sample probing estimates it separately at the cost of extra rollouts. We find that useful guidance occupies a band of depths whose informativeness profile is approximately Gaussian around the band center, rather than concentrating at a single optimal point. We propose Agent-G$^2$, a Gaussian guidance framework that draws the depth per task from a Gaussian whose center and spread are estimated online from rollouts already collected for policy optimization, requiring no probe rollouts or learned depth predictor. The center combines a global baseline with per-cluster difficulty, and the spread tracks within-cluster variance. We evaluate Agent-G$^2$ on ALFWorld and WebShop on Qwen2.5-1.5B / 7B-Instruct. Agent-G$^2$ outperforms the strongest hint-based, hint-free, and Aux-RL baselines on ALFWorld by 2.3 / 3.9 / 7.4 points at under one-third the rollout cost of per-sample probing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。