构建中文长对话理解六维测评基准，揭示模型真实短板。
Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding
- 设计六类控制任务，覆盖记忆、执行、指代等核心能力
- 209个任务跨度12-76轮，含干扰与口语化表达
- 主流模型平均仅41.1%正确率，无模型全胜
长时多轮对话已成常态，正确响应依赖于对早期细节的回忆、后续修订的追踪、目标对象的识别以及条件未满足时的动作抑制。现有评测多集中于短对话，难以全面评估长程交互中的上述能力，尤其在中文场景下，且缺乏对模型失败机制的深入洞察。为此，我们分析真实聊天机器人失败案例，提炼出六种常见失效机制，并据此构建了六种受控评估模式，形成面向深度多轮对话理解的中文基准Hy-MultiTurn。六个评估维度涵盖约束记忆、精确执行、约束合成、对象定位、动作抑制和指代消解。共设计209个受控任务，对话长度12至76轮不等，引入无关话题干扰与口语化表达提升难度。对22个前沿模型配置的评估显示，该基准普遍具有挑战性：即使最强模型GPT-5.5也仅在41.1%的响应中满足所有要求，且无模型在全部六项模式中表现最优。
原文摘要 · Abstract (English)
Long-running multi-turn interactions with chatbots and agents are now common, and a correct response often depends on remembering earlier details, tracking later revisions, identifying intended objects or referents, and withholding action when required conditions are unmet. Existing multi-turn benchmarks typically cover short exchanges and do not fully evaluate these capabilities in long multi-turn interactions, particularly in Chinese, while offering limited insight into how and why models fail. To address these limitations, we analyze real chatbot failures to identify six recurring mechanisms and use them to define six controlled evaluation modes in Hy-MultiTurn, a Chinese benchmark for deep multi-turn dialogue understanding. The six modes evaluate constraint memory, precise execution, constraint synthesis, object localization, action suppression, and reference resolution. Across the six modes, we construct 209 controlled tasks spanning 12-76 turns, with dialogue length, irrelevant-topic distraction, and colloquial phrasing adding further difficulty. Evaluation of 22 frontier model configurations shows that Hy-MultiTurn is broadly challenging, as even GPT-5.5, the strongest overall configuration, satisfies all requirements in only 41.1 percent of responses and no model performs best in all six modes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。