提出新强化学习框架,让对话双方协同进化更适应对手变化。
IB-RL: Isolated Bilateral Reinforcement Learning for Strategic Dialogue Agents

- 双方独立优化,通过联合推演共同进化。
- 在车辆销售任务中达成89.6%成功率,优于基线5个百分点。
- 适合需要应对多变对手的智能对话系统研究者。
强化学习在数学推理和代码执行等奖励固定的任务中表现优异,但战略对话环境是另一个会动态适应的智能体,传统方法常以固定对手训练,导致策略依赖特定对手而泛化能力差。本文提出隔离双边强化学习(IB-RL),使两个角色通过联合推演共同演化,同时各自独立优化收益、动作掩码和更新路径。在车辆电话销售任务中,IB-RL达到89.6% Success@1,优于最佳单边基线的84.6%;在谈判任务中,与DeepSeek V4 Pro对抗时达成98.4%一致率,高于基线的86.4%。结果表明,严格隔离下的双角色联合训练能显著提升策略对未知对手的泛化能力。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has achieved strong results in improving large language models (LLMs) on tasks with stationary, verifiable rewards, such as mathematical reasoning and code execution. In these settings, the environment follows fixed rules and does not adapt strategically to the agent. Strategic dialogue differs in this respect: the environment is another agent that adapts to the policy, and success depends on the interaction between the two sides. Despite this interactive nature, current RL approaches typically train a target agent against a fixed counterpart or simulator. We find that this training paradigm encourages the policy to exploit counterpart-specific regularities rather than learn strategies that generalize across counterparts. We call this problem the static-counterpart mismatch, which we quantify directly in our experiments. To address it, we propose Isolated Bilateral Reinforcement Learning (IB-RL), in which the two roles coevolve through joint rollouts while each role optimizes its own reward through fully independent advantages, action masks, and update paths. We evaluate frozen policies against fully independent held-out counterparts in both domains. On Vehicle TeleSales, IB-RL achieves 89.6% Success@1, compared to 84.6% for the best unilateral RL baseline. On Deal-or-NoDeal, it reaches 98.4% agreement against DeepSeek V4 Pro, compared to 86.4% for the best unilateral baseline. These results indicate that jointly training both roles with strict peragent isolation produces policies that generalize more effectively to unseen counterparts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。