提出梯度耦合机制，解决智能体强化学习泛化差问题
Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learning
- 用分类器辅助训练，让模型区分好坏动作
- 在多个任务上显著提升泛化性能
- 适合研究智能体决策与鲁棒性的人参考
强化学习是训练自主智能体的主要范式，但这些智能体常表现出较差的泛化能力，难以适应训练中未出现的场景。本文揭示了其根本原因——梯度耦合现象：在复杂任务中，不同状态高度相似，导致梯度更新产生破坏性干扰。一个状态中的最优动作更新可能意外提升另一相似状态下的次优动作概率。为此，我们提出新目标，使智能体同时具备分类能力，区分好与坏动作。这一辅助机制促使模型学习解耦的动作嵌入，缓解负向梯度干扰，从而提升泛化表现。大量实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is a dominant paradigm for training autonomous agents, yet these agents often exhibit poor generalization, failing to adapt to scenarios not seen during training. In this work, we identify a fundamental cause of this brittleness, a phenomenon which we term "gradient coupling." We hypothesize that in complex agentic tasks, the high similarity between distinct states leads to destructive interference between gradients. Specifically, a gradient update that reinforces an optimal action in one state can inadvertently increase the likelihood of a suboptimal action in a similar, yet different, state. To solve this, we propose a novel objective where the actor is trained to simultaneously function as a classifier that separates good and bad actions. This auxiliary pressure compels the model to learn disentangled embeddings for positive and negative actions, which mitigates negative gradient interference and improve the generalization performance. Extensive experiments demonstrate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。