arXiv:2603.04783cs.AIcs.CL2026-03ACL被引 5

让大模型在多轮对话中不再固执,及时纠正错误

Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction

  • 用单轮最优表现作锚点,引导模型在多轮中修正错误
  • 实验显示性能显著优于传统微调和放弃回答的方法
  • 跨领域通用性强,无需外部验证器也能生效

尽管大模型在单轮提供完整信息时表现出强大的推理能力,但在多轮交互中却存在明显脆弱性。当信息逐步揭示或需要更新时,模型常无法整合新约束,导致性能大幅下降。我们称此现象为‘上下文惯性’:模型固守先前的推理路径,即使用户后续明确纠正或提供新数据,也选择忽略。为此,我们提出强化学习结合单轮锚点(RLSTA)训练方法,利用模型在单轮任务中的优异表现作为稳定内部锚点,生成奖励信号,使多轮输出与锚点对齐。该方法有效打破上下文惯性,实现基于最新信息的自我校准。实验表明,RLSTA显著优于标准微调与放弃回答策略;在数学到编程等跨领域场景中展现强泛化能力,且无需外部验证器即可生效,具备广泛适用潜力。代码已开源。

原文摘要 · Abstract (English)

While LLMs demonstrate strong reasoning capabilities when provided with full information in a single turn, they exhibit substantial vulnerability in multi-turn interactions. Specifically, when information is revealed incrementally or requires updates, models frequently fail to integrate new constraints, leading to a collapse in performance compared to their single-turn baselines. We term the root cause as \emph{Contextual Inertia}: a phenomenon where models rigidly adhere to previous reasoning traces. Even when users explicitly provide corrections or new data in later turns, the model ignores them, preferring to maintain consistency with its previous (incorrect) reasoning path. To address this, we introduce \textbf{R}einforcement \textbf{L}earning with \textbf{S}ingle-\textbf{T}urn \textbf{A}nchors (\textbf{RLSTA}), a generalizable training approach designed to stabilize multi-turn interaction across diverse scenarios and domains. RLSTA leverages the model's superior single-turn capabilities as stable internal anchors to provide reward signals. By aligning multi-turn responses with these anchors, RLSTA empowers models to break contextual inertia and self-calibrate their reasoning based on the latest information. Experiments show that RLSTA significantly outperforms standard fine-tuning and abstention-based methods. Notably, our method exhibits strong cross-domain generalization (e.g., math to code) and proves effective even without external verifiers, highlighting its potential for general-domain applications. Code is available at https://github.com/Tencent/RLSTA.

大模型多轮对话强化学习自我修正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。