让大模型主动预测用户每轮互动偏好，提升对话协作效率
Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction
- 用隐式过程奖励模型从稀疏结果反推每轮交互的细粒度奖励
- 在数学辅导等3个任务上，收敛速度优于现有方法
- 适合需要主动引导的对话系统研发者使用
多轮人机协作是自适应教学、对话推荐和专业咨询等交互服务的核心。但强化学习优化此类交互面临中间奖励信号稀疏和用户响应高随机性的挑战。为此，我们提出隐式轮次策略优化（ITPO），通过隐式过程奖励模型，从稀疏的结果信号中推导出细粒度的轮次级过程奖励。相比易波动的词级别奖励，轮次级信号更具鲁棒性，还可结合归一化机制进一步提升训练稳定性。我们在数学辅导、文档撰写和医疗推荐三个代表性多轮协作任务上评估了ITPO，结果表明：当与PPO、GRPO或RLOO结合时，ITPO始终实现更优的收敛性能。轨迹分析显示，ITPO推断出的轮次偏好在语义上与人类判断高度一致。代码已公开于https://github.com/Graph-COM/ITPO。
原文摘要 · Abstract (English)
Multi-turn human-AI collaboration is fundamental to deploying interactive services such as adaptive tutoring, conversational recommendation, and professional consultation. However, optimizing these interactions via reinforcement learning is hindered by the sparsity of verifiable intermediate rewards and the high stochasticity of user responses. To address these challenges, we introduce Implicit Turn-wise Policy Optimization (ITPO). ITPO leverages an implicit process reward model to derive fine-grained, turn-wise process rewards from sparse outcome signals. Unlike volatile token-level rewards, these turn-level signals exhibit superior robustness and may utilize a normalization mechanism to further enhance training stability. We evaluate ITPO across three representative multi-turn collaborative tasks: math tutoring, document writing, and medical recommendation. Empirical results demonstrate that ITPO, when combined with PPO, GRPO, or RLOO, consistently achieves improved convergence than existing baselines. Elaborate trajectory analysis confirms that ITPO infers turn-wise preferences that are semantically aligned with human judgment. Code is publicly available at https://github.com/Graph-COM/ITPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。