测试大模型在对话中切换话题时的准确度，发现多数模型会误保留旧上下文。
Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs
- 构建合成数据集模拟不同难度的话题切换，评估模型理解能力
- 仅部分强指令模型能准确识别话题转换，开源模型普遍表现差
- 所有模型均存在位置偏差，适合关注长对话鲁棒性的研究者阅读
用户与大语言模型进行多轮对话时常会修改请求或转向新话题，但模型常忽略这些变化，继续沿用无关上下文，导致回答不准确。本文通过压力测试多轮对话理解能力，聚焦两个子任务：(1) 判断当前轮次是否发生话题切换或请求细化；(2) 从历史对话中筛选相关上下文。为此，我们基于多个真实领域数据集构建合成基准，模拟不同难度的话题转换。评估了十种大模型（包括开源、闭源及推理型）的零样本性能，结果表明：仅少数推理型和强指令模型在话题检测上表现准确；开源模型即使有明确提示也常携带过时上下文；所有模型均存在位置偏差。基于此，我们探讨了提升大模型长期多轮交互鲁棒性的关键启示。
原文摘要 · Abstract (English)
Users interacting with Large Language Models (LLMs) in a multi-turn conversation routinely refine their requests or pivot to new topics. LLMs, however, often miss these topic shifts and carry over irrelevant context from previous turns, leading to inaccurate responses. In this paper, we stress-test the multi-turn understanding of LLMs and study the following two sub-tasks: (1) detecting whether the user pivots or refines in the current turn, and (2) shortlisting relevant context from previous turns. To this end, we construct synthetic benchmarks based on real-world datasets from varied domains, as to simulate context shifts of different levels of difficulty. We then evaluate the zero-shot performance of ten LLMs (open-weight, closed-source and reasoning), and demonstrate that only some reasoning and strongly instructed LLMs are accurate in detecting pivots; open-weight LLMs struggle with the task and frequently carry stale context even with explicit cues; and all models suffer from a position bias. Based on the results, we discuss key takeaways for improving long-term robustness in multi-turn capabilities for LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。