arXiv:2603.00656cs.AI2026-03被引 5

通过信息增益奖励优化用户交互,让智能体更精准地获取关键信息。

InfoPO: Information-Driven Policy Optimization for User-Centric Agents

  • 基于信息增益设计细粒度奖励,识别有价值的交互回合。
  • 在多任务中优于提示和强化学习基线,提升决策准确率。
  • 适合需要复杂人机协作的智能体系统,如客服、编程助手。

现实世界中用户对大模型智能体的请求往往不完整,智能体需通过多轮交互获取缺失信息并做出正确决策。然而,现有基于多轮GRPO的方法常依赖轨迹级奖励计算,导致信用分配困难且优势信号不足。为此,我们提出InfoPO(信息驱动策略优化),将多轮交互建模为不确定性主动降低过程,通过比较反馈前后动作分布变化,计算信息增益奖励,以量化各交互回合的信息价值。该信号与任务结果通过自适应方差门控融合,既识别信息重要性又保持任务目标导向。在意图澄清、协同编程和工具增强决策等多样化任务中,InfoPO consistently 超越提示方法和多轮强化学习基线。其在用户模拟器变化下表现稳健,并有效泛化至环境交互任务。整体上,InfoPO 提供了一种原则性强且可扩展的机制,用于优化复杂人机协作。代码已开源:https://github.com/kfq20/InfoPO。

原文摘要 · Abstract (English)

Real-world user requests to LLM agents are often underspecified. Agents must interact to acquire missing information and make correct downstream decisions. However, current multi-turn GRPO-based methods often rely on trajectory-level reward computation, which leads to credit assignment problems and insufficient advantage signals within rollout groups. A feasible approach is to identify valuable interaction turns at a fine granularity to drive more targeted learning. To address this, we introduce InfoPO (Information-Driven Policy Optimization), which frames multi-turn interaction as a process of active uncertainty reduction and computes an information-gain reward that credits turns whose feedback measurably changes the agent's subsequent action distribution compared to a masked-feedback counterfactual. It then combines this signal with task outcomes via an adaptive variance-gated fusion to identify information importance while maintaining task-oriented goal direction. Across diverse tasks, including intent clarification, collaborative coding, and tool-augmented decision making, InfoPO consistently outperforms prompting and multi-turn RL baselines. It also demonstrates robustness under user simulator shifts and generalizes effectively to environment-interactive tasks. Overall, InfoPO provides a principled and scalable mechanism for optimizing complex agent-user collaboration. Code is available at https://github.com/kfq20/InfoPO.

智能体强化学习人机协作信息增益

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。