让大模型学会在对话中判断问题能否回答，避免过早作答导致的性能下降。
Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewards
- 用分阶段难度递增的训练策略，逐步提升对话复杂度。
- 对话性能衰减从62.6%提升至75.1%，拒绝回答率从33.5%升至73.4%。
- 适合需要可靠多轮交互的智能客服、医疗咨询等高可信场景。
大语言模型在单轮指令跟随中表现良好，但在多轮对话中常出现「对话迷失」（LiC）现象，即随着信息逐步揭示，性能显著下降。受可验证奖励强化学习（RLVR）启发，我们提出一种基于可验证准确率与拒答奖励的课程强化学习框架（RLAAR），引导模型不仅生成正确答案，还判断问题在当前对话阶段是否可解。该方法采用能力门控的课程机制，按指令片段难度逐步增加对话挑战，稳定训练过程并提升可靠性。通过多轮、在线策略采样与混合奖励系统，RLAAR教会模型在求解与明智拒答之间取得平衡，减少过早回答行为。在LiC基准测试中，该方法将性能衰减从62.6%降至75.1%，同时将校准后的拒答率从33.5%提升至73.4%，为构建可信多轮大模型提供了可行方案。
原文摘要 · Abstract (English)
Large Language Models demonstrate strong capabilities in single-turn instruction following but suffer from Lost-in-Conversation (LiC), a degradation in performance as information is revealed progressively in multi-turn settings. Motivated by the current progress on Reinforcement Learning with Verifiable Rewards (RLVR), we propose Curriculum Reinforcement Learning with Verifiable Accuracy and Abstention Rewards (RLAAR), a framework that encourages models not only to generate correct answers, but also to judge the solvability of questions in the multi-turn conversation setting. Our approach employs a competence-gated curriculum that incrementally increases dialogue difficulty (in terms of instruction shards), stabilizing training while promoting reliability. Using multi-turn, on-policy rollouts and a mixed-reward system, RLAAR teaches models to balance problem-solving with informed abstention, reducing premature answering behaviors that cause LiC. Evaluated on LiC benchmarks, RLAAR significantly mitigates LiC performance decay (62.6% to 75.1%) and improves calibrated abstention rates (33.5% to 73.4%). Together, these results provide a practical recipe for building multi-turn reliable and trustworthy LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。