arXiv:2602.19225cs.AI2026-02ACL被引 26

让大模型智能体更懂任务难易,精准分配奖励。

Proximity-Based Multi-Turn Optimization: Practical Credit Assignment for LLM Agent Training

  • 根据任务难度动态调整梯度强度,避免误奖
  • 在真实场景中提升性能,计算开销几乎为零
  • 可直接接入现有训练流程,适合工业应用

多轮大模型智能体正广泛应用于客服自动化、电商辅助和交互式任务管理等生产系统,准确区分高价值信息信号与随机噪声对高效训练至关重要。现实中,简单任务失败可能源于偶然波动,而复杂任务成功才反映真正能力突破。然而,现有基于分组的策略优化方法依赖批内统计偏差,当任务难度变化时容易错判奖励归属。为此,我们提出近邻驱动的多轮优化(ProxMO),一种专为实际部署设计的实用且稳健框架。ProxMO通过两种轻量机制融合全局上下文:成功率感知调制根据每轮任务难度动态调节梯度强度;近邻软聚合则在步骤级通过连续语义加权生成基准。在ALFWorld和WebShop基准上的大量实验表明,ProxMO显著优于现有基线,且计算成本极低。消融实验进一步验证了两种机制的独立有效性与协同作用。关键的是,ProxMO可无缝兼容标准GRPO框架,便于在现有工业训练流程中快速部署。代码已开源。

原文摘要 · Abstract (English)

Multi-turn LLM agents are becoming pivotal to production systems, spanning customer service automation, e-commerce assistance, and interactive task management, where accurately distinguishing high-value informative signals from stochastic noise is critical for sample-efficient training. In real-world scenarios, a failure in a trivial task may reflect random instability, whereas success in a high-difficulty task signifies a genuine capability breakthrough. Yet, existing group-based policy optimization methods rigidly rely on statistical deviation within discrete batches, frequently misallocating credit when task difficulty fluctuates. To address this issue, we propose Proximity-based Multi-turn Optimization (ProxMO), a practical and robust framework engineered specifically for the constraints of real-world deployment. ProxMO integrates global context via two lightweight mechanisms: success-rate-aware modulation dynamically adapts gradient intensity based on episode-level difficulty, while proximity-based soft aggregation derives baselines through continuous semantic weighting at the step level. Extensive evaluations on ALFWorld and WebShop benchmarks demonstrate that ProxMO yields substantial performance gains over existing baselines with negligible computational cost. Ablation studies further validate the independent and synergistic efficacy of both mechanisms. Crucially, ProxMO offers plug-and-play compatibility with standard GRPO frameworks, facilitating immediate, low-friction adoption in existing industrial training pipelines. Our implementation is available at: \href{https://anonymous.4open.science/r/proxmo-B7E7/README.md}{https://anonymous.4open.science/r/proxmo}.

大模型智能体奖励分配强化学习工业落地

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。